|
--- |
|
license: other |
|
license_name: server-side-public-license |
|
license_link: https://www.mongodb.com/legal/licensing/server-side-public-license |
|
tags: |
|
- diffusion |
|
- virtual try-on |
|
- virtual try-off |
|
- image generation |
|
- fashion |
|
- e-commerce |
|
base_model: |
|
- CompVis/stable-diffusion-v1-4 |
|
pipeline_tag: image-to-image |
|
library_name: diffusers |
|
--- |
|
|
|
## TryOffDiff |
|
|
|
The models proposed in the paper _"TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models"_ |
|
[[paper]][paper_arxiv] [[project page]][project_page]: |
|
- `tryoffdiff.pth`: The pre-trained StableDiffusion-v1.4 fine-tuned on `VITON-HD-train` dataset. |
|
- `.pth`: A U-Net trained from scratch on `VITON-HD-train` dataset. |
|
- `.pth`: |
|
|
|
|
|
## Usage |
|
|
|
```python |
|
from huggingface_hub import hf_hub_download |
|
|
|
class TryOffDiff(nn.Module): |
|
def __init__(self): |
|
super().__init__() |
|
self.unet = UNet2DConditionModel.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="unet") |
|
self.transformer = torch.nn.TransformerEncoderLayer(d_model=768, nhead=8, batch_first=True) |
|
self.proj = nn.Linear(1024, 77) |
|
self.norm = nn.LayerNorm(768) |
|
|
|
def adapt_embeddings(self, x): |
|
x = self.transformer(x) |
|
x = self.proj(x.permute(0, 2, 1)).permute(0, 2, 1) |
|
return self.norm(x) |
|
|
|
def forward(self, noisy_latents, t, cond_emb): |
|
cond_emb = self.adapt_embeddings(cond_emb) |
|
return self.unet(noisy_latents, t, encoder_hidden_states=cond_emb).sample |
|
|
|
path_model = hf_hub_download( |
|
repo_id="rizavelioglu/tryoffdiff", |
|
filename="tryoffdiff.pth", # or one of ablations ["ldm-1", "ldm-2", "ldm-3", ...] |
|
) |
|
net = TryOffDiff() |
|
net.load_state_dict(torch.load(path_model, weights_only=False)) |
|
net.eval().to(device) |
|
``` |
|
|
|
> Check out the demo code on [HuggingFace Spaces][hf_spaces] for the full running example. |
|
|
|
> Also, check out [GitHub repository][github] to get more information on |
|
> training, inference, and evaluation. |
|
|
|
### License |
|
TL;DR: Not available for commercial use, unless the FULL source code is shared! \ |
|
This project is intended solely for academic research. No commercial benefits are derived from it. |
|
Models are licensed under [Server Side Public License (SSPL)][license] |
|
|
|
### Citation |
|
If you find this repository useful in your research, please consider giving a star ⭐ and a citation: |
|
``` |
|
@article{velioglu2024tryoffdiff, |
|
title = {TryOffDiff: Virtual-Try-Off via High-Fidelity Garment Reconstruction using Diffusion Models}, |
|
author = {Velioglu, Riza and Bevandic, Petra and Chan, Robin and Hammer, Barbara}, |
|
journal = {arXiv}, |
|
year = {2024}, |
|
note = {\url{https://doi.org/nt3n}} |
|
} |
|
``` |
|
|
|
[hf_spaces]: https://huggingface.co/spaces/rizavelioglu/tryoffdiff/blob/main/app.py |
|
[project_page]: https://rizavelioglu.github.io/tryoffdiff/ |
|
[paper_arxiv]: https://arxiv.org/abs/2411.18350 |
|
[github]: https://github.com/rizavelioglu/tryoffdiff |
|
[license]: https://www.mongodb.com/legal/licensing/server-side-public-license |
|
|
|
|
