Skip to content
/ NRHints Public

Official Code Release for [SIGGRAPH 2023] Relighting Neural Radiance Fields with Shadow and Highlight Hints

License

Notifications You must be signed in to change notification settings

iamNCJ/NRHints

Repository files navigation

Relighting Neural Radiance Fields with Shadow and Highlight Hints

Chong Zeng · Guojun Chen · Yue Dong · Pieter Peers · Hongzhi Wu · Xin Tong

SIGGRAPH 2023 Conference Proceedings


Project Page | Paper | arXiv | Data


Setup

Environment

The code is developed and tested on Linux servers with NVIDIA GPU(s). We support Python 3.8+ and PyTorch 1.11+. After getting a required Python environment, you can setup the rest of requirements by running:

git clone https://github.com/iamNCJ/NRHints.git
cd NRHints
pip install -r requirements.txt

Data

Our data is compatible with NeRF Blender Dataset, except that we have extra fields in each frame for point light position.

You can download our data here.

Usage

Configuration System

We use tyro for configuration management. Description to all configurations can be found by running python main.py -h.

Training

python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX

Refer to train_synthetic.sh and train_real.sh for training on synthetic and real data, respectively.

Note:

  1. Our code automatically detects the number of GPUs and uses all of them for training. If you want to use a subset of GPUs, you can set the CUDA_VISIBLE_DEVICES environment variable.
  2. For training on real captured scenes, we recommend turning on camera optimization by using config:nr-hints-cam-opt, which can significantly reduce the blurry effects. Since this is an improvement after the paper submission, details are described in the author's version.

Testing

python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX --config.evaluation-only True

Refer to eval_synthetic.sh and eval_real.sh for testing on synthetic and real data, respectively.

Our pretrained models can be downloaded here.

Data and Models

Real Captured Scenes

Object Data Pre-trained model
Cat Link Link
Cluttered Scene Link Link
Pixiu Statuette Link Link
Ornamental Fish Link Link
Cat on Decor Link Link
Cup and Fabric Link Link
Pikachu Statuette Link Link

Synthetic Rendered Scenes

Note: Our synthetic data rendering scripts are released at here.

Object Data Pre-trained model
Diffuse Link Link
Metallic Link Link
Glossy-Metal Link Link
Rough-Metal Link Link
Anisotropic-Metal Link Link
Plastic Link Link
Glossy-Plastic Link Link
Rough-Plastic Link Link
Short-Fur Link Link
Long-Fur Link Link
Translucent Link Link
Fur-Ball Link Link
Basket Link Link
Layered Woven Ball Link Link
Drums Link Link
Hotdog Link Link
Lego Link Link

You can use the script download_data.sh to download all data.

Citation

Cite as below if you find this repository is helpful to your project:

@inproceedings {zeng2023nrhints,
    title      = {Relighting Neural Radiance Fields with Shadow and Highlight Hints},
    author     = {Chong Zeng and Guojun Chen and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
    booktitle  = {ACM SIGGRAPH 2023 Conference Proceedings},
    year       = {2023}
}

Acknowledgement

Some code snippets are borrowed from NeuS and Nerfstudio. Thanks for these great projects.

About

Official Code Release for [SIGGRAPH 2023] Relighting Neural Radiance Fields with Shadow and Highlight Hints

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published