Objects with Lighting: A Real-World Dataset for Evaluating Reconstruction and Rendering for Object Relighting

Reconstructing an object from photos and placing it virtually in a newenvironment goes beyond the standard novel view synthesis task as theappearance of the object has to not only adapt to the novel viewpoint but alsoto the new lighting conditions and yet evaluations of inverse rendering methodsrely on novel view synthesis data or simplistic synthetic datasets forquantitative analysis. This work presents a real-world dataset for measuringthe reconstruction and rendering of objects for relighting. To this end, wecapture the environment lighting and ground truth images of the same objects inmultiple environments allowing to reconstruct the objects from images taken inone environment and quantify the quality of the rendered views for the unseenlighting environments. Further, we introduce a simple baseline composed ofoff-the-shelf methods and test several state-of-the-art methods on therelighting task and show that novel view synthesis is not a reliable proxy tomeasure performance. Code and dataset are available athttps://github.com/isl-org/objects-with-lighting .