Material Palette: Extraction of Materials from a Single Image, CVPR'24

Authors: Ivan Lopes (Inria), Fabio Pizzati (Oxford Uni.), Raoul de Charette (Inria)

Page | arXiv | Github | Dataset (soon)

Abstract:

In this paper, we propose a method to extract Physically-Based-Rendering (PBR) materials from a single real-world image. We do so in two steps: first, we map regions of the image to material concepts using a diffusion model, which allows the sampling of texture images resembling each material in the scene. Second, we benefit from a separate network to decompose the generated textures into Spatially Varying BRDFs (SVBRDFs), providing us with materials ready to be used in rendering applications. Our approach builds on existing synthetic material libraries with SVBRDF ground truth, but also exploits a diffusion-generated RGB texture dataset to allow generalization to new samples using unsupervised domain adaptation (UDA). Our contributions are thoroughly evaluated on synthetic and real-world datasets. We further demonstrate the applicability of our method for editing 3D scenes with materials estimated from real photographs. Our code is available at https://github.com/astra-vision/MaterialPalette.

To cite our paper, please use:

@inproceedings{lopes2023material,
  author = {Lopes, Ivan and Pizzati, Fabio and de Charette, Raoul},
  title = {Material Palette: Extraction of Materials from a Single Image},
  booktitle = {arXiv},
  year = {2023},
  project = {https://astra-vision.github.io/MaterialPalette/}
}