What is it about?

We address the new problem of language-guided semantic style transfer of 3D indoor scenes. The input is a 3D indoor scene mesh and several phrases that describe the target scene. Firstly, 3D vertex coordinates are mapped to RGB residues by a multi-layer perceptron. Secondly, colored 3D meshes are differentiablly rendered into 2D images, via a viewpoint sampling strategy tailored for indoor scenes. Thirdly, rendered 2D images are compared to phrases, via pre-trained vision-language models. Lastly, errors are back-propagated to the multi-layer perceptron to update vertex colors corresponding to certain semantic categories. We did large-scale qualitative analyses and A/B user tests, with the public ScanNet and SceneNN datasets.

Featured Image

Why is it important?

We did large-scale qualitative analyses and A/B user tests, with the public ScanNet and SceneNN datasets. We demonstrate: (1) visually pleasing results that are potentially useful for multimedia applications. (2) rendering 3D indoor scenes from viewpoints consistent with human priors is important. (3) incorporating semantics significantly improve style transfer quality. (4) an HSV regularization term leads to results that are more consistent with inputs and generally rated better.

Perspectives

In this paper we propose the first language-driven semantic style transfer algorithm for 3D indoor scenes, named LASST. The inputs are a 3D indoor scene mesh and several phrases specifying target styles. The mesh is rendered differentiablly into multiple 2D images and compared with text prompts in the feature space of a pre-trained vision language model. We identify the importance of better vision language alignment through semantic masks, a viewpoint sampling strategy that incorporates human viewing priors, and an HSV regularization loss that discourages too much drift from input colors. With a large scale A/B user tests, we demonstrate that our design choices are well recognized. We also provide comprehensive qualitative results showing the pros and cons of our method.

Bu Jin

Read the Original

This page is a summary of: Language-guided Semantic Style Transfer of 3D Indoor Scenes, October 2022, ACM (Association for Computing Machinery),
DOI: 10.1145/3552482.3556555.
You can read the full text:

Read

Resources

Contributors

The following have contributed to this page