| # Preparation | |
| This is the official implementation of *LucidFusion: Generating 3D Gaussians with Arbitrary Unposed Images*. | |
| ## Install | |
| ``` | |
| # Xformers is required! please refer to https://github.com/facebookresearch/xformers for details. | |
| # For example, we use torch 2.3.1 + cuda 11.8 | |
| conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia | |
| # [linux only] cuda 11.8 version | |
| pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu118 | |
| # For 3D Gaussian Splatting, we use the official installation. Please refer to https://github.com/graphdeco-inria/gaussian-splatting for details | |
| git clone https://github.com/graphdeco-inria/gaussian-splatting --recursive | |
| pip install ./diff-gaussian-rasterization | |
| # Other dependencies | |
| conda env create -f environment.yml | |
| conda activate LucidFusion | |
| ``` | |
| ## Pretrained Weights | |
| Our pre-trained weights will be released soon, please check back! | |
| Our current model loads pre-trained diffusion model weights ([Stable Diffusion 2.1] (https://huggingface.co/stabilityai/stable-diffusion-2-1/tree/main) in default) for training purposes. | |
| ## Inference | |
| A shell script is provided with example files. | |
| To run, you first need to setup the pretrained weights as follows: | |
| ``` | |
| cd LucidFusion | |
| mkdir output/demo | |
| # Download the pretrained weights and name it as best.ckpt | |
| # Place the pretrained weights in LucidFusion/output/demo/best.ckpt | |
| # The result will be saved in the LucidFusion/output/demo/<obj name>/ | |
| bash scripts/demo.sh | |
| ``` | |
| More example files will be released soon | |
| You can also try your own example! To do that: | |
| First obtain images and place them in the examples folder | |
| ``` | |
| LucidFusion | |
| ├── examples/ | |
| | ├── "your obj name"/ | |
| | | ├── "image_01.png" | |
| | | ├── "image_02.png" | |
| | | ├── ... | |
| ``` | |
| Then you need to run preprocess.py to extract recentered image and its mask: | |
| ``` | |
| # Run the following will create two folders (images, masks) in "your obj name" folder. | |
| # You can check to see if the extract mask is corrected. | |
| python preprocess.py "examples/"you obj name" --outdir examples/"your obj name" | |
| # Modify demo.sh to DEMO='examples/"you obj name" ' then run the file | |
| bash scripts/demo.sh | |
| ``` | |