Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Field

Xiangyu Wang1*, Jingsen Zhu2*, Qi Ye1†, Yuchi Huo3,2, Yunlong Ran1, Zhihua Zhong2, Jiming Chen1
1Zhejiang University, Key Lab of CS&AUS of Zhejiang Province, 2State Key Lab of CAD&CG, Zhejiang University 3Zhejiang Lab,
ICCV 2023
Paper arXiv Code
Zotero Connector friendly
Seal-3D

Seal-3D: The first interactive pixel level NeRF editing tool. We design an interactive user editing method and system Seal-3D, which achieves instant (~1s) preview (left) by our novel pretraining strategy. High-quality editing results can be further obtained by a short period (in 1 or 2 minutes) of finetuning. The editing results of our implemented editing tools (right) are view-consistent with rich shading details (e.g. shadows) on the original surface (left).

Abstract

With the popularity of implicit neural representations, or neural radiance fields (NeRF), there is a pressing need for editing methods to interact with the implicit 3D models for tasks like post-processing reconstructed scenes and 3D content creation. While previous works have explored NeRF editing from various perspectives, they are restricted in editing flexibility, quality, and speed, failing to offer direct editing response and instant preview. The key challenge is to conceive a locally editable neural representation that can directly reflect the editing instructions and update instantly. To bridge the gap, we propose a new interactive editing method and system for implicit representations, called Seal-3D, which allows users to edit NeRF models in a pixel-level and free manner with a wide range of NeRF-like backbone and preview the editing effects instantly. To achieve the effects, the challenges are addressed by our proposed proxy function mapping the editing instructions to the original space of NeRF models and a teacher-student training strategy with local pretraining and global finetuning. A NeRF editing system is built to showcase various editing types. Our system can achieve compelling editing effects with an interactive speed of about 1 second.

Video

Results

BibTeX

@misc{wang2023seal3d,
      title={Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields}, 
      author={Xiangyu Wang and Jingsen Zhu and Qi Ye and Yuchi Huo and Yunlong Ran and Zhihua Zhong and Jiming Chen},
      year={2023},
      eprint={2307.15131},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}