🎥 Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text

🔥 News:

• 🥰 Check out our new gradio demo by simply running python app.py.

• 🆓 Try out Director3D for free with our Google Colab Demo.

📖 Generation Results

❗ All videos are rendered with generated camera trajectories and 3D Gaussians, the only inputs are text prompts!

little_gallery.mp4

👀 See more than 200 examples in our Gallery.

🔧 Installation

• create a new conda enviroment

conda create -n director3d python=3.9
conda activate director3d

• install pytorch (or use your own if it is compatible with xformers)

conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia

• install xformers for momory-efficient attention

conda install xformers -c xformers

• install pip packages

pip install kiui scipy opencv-python-headless kornia omegaconf imageio imageio-ffmpeg  seaborn==0.12.0 plyfile ninja tqdm diffusers transformers accelerate timm einops matplotlib plotly typing argparse gradio kaleido
pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable"
pip install "git+https://github.com/ashawkey/diff-gaussian-rasterization.git"

• clone this repo:

git clone https://github.com/imlixinyang/director3d.git
cd director3d

• download the pre-trained model by:

wget https://huggingface.co/imlixinyang/director3d/resolve/main/model.ckpt?download=true -O model.ckpt

🧐 General Usage

You can generate 3D scenes with camera trajectories by running the following command:

python inference.py --export_all --text "a delicious hamburger on a wooden table."

This will take about 5 minutes per sample on a single A100 GPU (or 7 minutes per sample on a single RTX 3090 GPU). The results can be found in ./exps/tmp.

💡 Code Overview

Core code of three key components of Director3D can be found in:

• Cinematographer - Trajectory Diffusion Transformer (Traj-DiT)

system_traj_dit.py

• Decorator - Gaussian-driven Multi-view Latent Diffusion Model (GM-LDM)

system_gm_ldm.py
gm_ldm.py

• Detailer - SDS++

modules/refiners/sds_pp_refiner.py

Citation

License

Licensed under the CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)

The code is released for academic research use only.

If you have any questions, please contact me via [email protected].

👊 I'm looking for research collaboration in Real-world 3D/4D/Video Generation, please contact me if you're interested.