SIGGRAPH 2025 method lets editors place CG-style lights inside photographs

Simon Fraser University researchers have published a relighting technique that gives photographs the kind of explicit light control normally reserved for 3D graphics packages. The paper, "Physically Controllable Relighting of Photographs" by Chris Careaga and Yağız Aksoy, appears in the SIGGRAPH 2025 proceedings, and its project page recently drew front-page attention on Hacker News.

How it works

Most photo editors treat lighting as a set of tonal adjustments applied across pixels. This method instead reconstructs enough of the scene in 3D to place real light sources into it. According to the project abstract, the pipeline begins with mid-level computer vision: monocular estimates of geometry and intrinsic components are used to infer a colored mesh of the scene. The user then defines a lighting setup in 3D on that mesh — a spot light here, a point light there, or a full environmental illumination — and a path-tracing engine renders the scene under the new configuration.

That render is physically grounded but visually approximate, since the mesh is only an estimate of the real scene. To close the gap, the approximate output is passed through a feed-forward neural renderer that predicts the final photorealistic image. The division of labor is deliberate: the rendering engine contributes physical accuracy, while the network contributes the realism that a render of an estimated mesh cannot achieve on its own.

Self-supervised training on ordinary photos

Training a neural renderer for relighting normally requires paired examples of the same scene under different lighting, which are expensive to capture. The researchers' alternative is a differentiable rendering process that reconstructs the illumination already present in real-world photographs. That reconstruction provides the supervision needed to train the neural renderer directly on raw image collections, which the authors describe as the key to generalizing beyond controlled datasets to arbitrary photographs.

A line of research coming together

The project page situates the work on top of three earlier papers by the same group: intrinsic image decomposition via ordinal shading (ACM Transactions on Graphics, 2023), colorful diffuse intrinsic image decomposition (SIGGRAPH Asia 2024, which received a Best Paper Award Honorable Mention), and intrinsic harmonization for illumination-aware compositing (SIGGRAPH Asia 2023). Since the new pipeline depends on estimates of intrinsic components like shading and albedo, it draws directly on this accumulated work on separating surface appearance from illumination.

Reception and licensing

The work has attracted attention outside academia. SFU's Computing Science department issued a media release framing it as bringing Blender-like lighting control to any photograph, PetaPixel published a story on the 3D relighting results, and the YouTube channel Two Minute Papers produced a video about the research.

Unlike much graphics research, the method is not being given away openly for now. The project page states that the methodology is patent pending and directs licensing inquiries to SFU's Technology Licensing Office, so practical availability will likely depend on a commercial agreement.

Why it matters

Lighting is one of the hardest things to change after a photograph is taken, because doing it convincingly requires the scene's shape and material properties — information a flat image does not contain. By reconstructing that information automatically and then re-rendering through a path tracer, this work turns lighting from a pixel-level effect into an object-level scene edit: add a lamp, aim it, and the photograph responds. That has clear appeal for product photography, compositing and post-production, where reconciling lighting between elements is a routine and tedious task.

The training recipe may matter as much as the result. Using differentiable rendering to generate supervision from unmodified image collections sidesteps the data bottleneck that constrains many inverse-rendering approaches, a pattern other research groups can adopt. The pending patent, meanwhile, signals that the university sees a commercial path — so if the technique reaches users, it will likely arrive embedded in an editing product rather than as a standalone release.