File size: 3,409 Bytes
4248044 0e99353 4248044 0e99353 e02e4d8 0e99353 e02e4d8 0e99353 4248044 70375cc 43de88c 70375cc 43de88c 254f7e3 70375cc 4248044 70375cc 4248044 43de88c 6d3e94d 70375cc 70cd3cf 70375cc 254f7e3 70375cc 6d3e94d 70375cc 4248044 70375cc d94960f ff66300 43de88c ff66300 43de88c ff66300 307e9e0 4248044 ff66300 70375cc ff66300 70375cc 43de88c 70375cc 4248044 70375cc 4248044 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 |
---
license: openrail
pipeline_tag: image-to-3d
---
# Overview
This is a duplicate of [ashawkey/imagedream-ipmv-diffusers](https://huggingface.co/ashawkey/imagedream-ipmv-diffusers).
It is hosted here for the purpose of persistence and reproducibility for the ML for 3D course.
### Usage
This project can be used from other projects as follows.
```
import torch
from diffusers import DiffusionPipeline
# Text to Multi-View Diffusion
text_pipeline = DiffusionPipeline.from_pretrained(
"ashawkey/mvdream-sd2.1-diffusers",
custom_pipeline="dylanebert/multi_view_diffusion",
torch_dtype=torch.float16,
trust_remote_code=True,
).to("cuda")
# Image to Multi-View Diffusion
image_pipeline = DiffusionPipeline.from_pretrained(
"ashawkey/imagedream-ipmv-diffusers",
custom_pipeline="dylanebert/multi_view_diffusion",
torch_dtype=torch.float16,
trust_remote_code=True,
).to("cuda")
```
Original model card below.
---
# MVDream-diffusers
A **unified** diffusers implementation of [MVDream](https://github.com/bytedance/MVDream) and [ImageDream](https://github.com/bytedance/ImageDream).
We provide converted `fp16` weights on huggingface:
- [MVDream](https://huggingface.co/ashawkey/mvdream-sd2.1-diffusers)
- [ImageDream](https://huggingface.co/ashawkey/imagedream-ipmv-diffusers)
### Install
```bash
# dependency
pip install -r requirements.txt
# xformers is required! please refer to https://github.com/facebookresearch/xformers
pip install ninja
pip install -v -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
```
### Usage
```bash
python run_mvdream.py "a cute owl"
python run_imagedream.py data/anya_rgba.png
```
### Convert weights
MVDream:
```bash
# download original ckpt (we only support the SD 2.1 version)
mkdir models
cd models
wget https://huggingface.co/MVDream/MVDream/resolve/main/sd-v2.1-base-4view.pt
wget https://raw.githubusercontent.com/bytedance/MVDream/main/mvdream/configs/sd-v2-base.yaml
cd ..
# convert
python convert_mvdream_to_diffusers.py --checkpoint_path models/sd-v2.1-base-4view.pt --dump_path ./weights_mvdream --original_config_file models/sd-v2-base.yaml --half --to_safetensors --test
```
ImageDream:
```bash
# download original ckpt (we only support the pixel-controller version)
cd models
wget https://huggingface.co/Peng-Wang/ImageDream/resolve/main/sd-v2.1-base-4view-ipmv.pt
wget https://raw.githubusercontent.com/bytedance/ImageDream/main/extern/ImageDream/imagedream/configs/sd_v2_base_ipmv.yaml
cd ..
# convert
python convert_mvdream_to_diffusers.py --checkpoint_path models/sd-v2.1-base-4view-ipmv.pt --dump_path ./weights_imagedream --original_config_file models/sd_v2_base_ipmv.yaml --half --to_safetensors --test
```
### Acknowledgement
- The original papers:
```bibtex
@article{shi2023MVDream,
author = {Shi, Yichun and Wang, Peng and Ye, Jianglong and Mai, Long and Li, Kejie and Yang, Xiao},
title = {MVDream: Multi-view Diffusion for 3D Generation},
journal = {arXiv:2308.16512},
year = {2023},
}
@article{wang2023imagedream,
title={ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation},
author={Wang, Peng and Shi, Yichun},
journal={arXiv preprint arXiv:2312.02201},
year={2023}
}
```
- This codebase is modified from [mvdream-hf](https://github.com/KokeCacao/mvdream-hf).
|