Spaces:
Runtime error
Runtime error
modify readme.py
Browse files
README.md
CHANGED
|
@@ -1,71 +0,0 @@
|
|
| 1 |
-
# ___***FilterPrompt: Guiding Image Transfer in Diffusion Models***___
|
| 2 |
-
|
| 3 |
-
<a href='https://meaoxixi.github.io/FilterPrompt/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
|
| 4 |
-
<a href='https://arxiv.org/pdf/2404.13263'><img src='https://img.shields.io/badge/Paper-blue'></a>
|
| 5 |
-
<a href='https://arxiv.org/pdf/2404.13263'><img src='https://img.shields.io/badge/Demo-orange'></a>
|
| 6 |
-
|
| 7 |
-
We propose FilterPrompt, an approach to enhance the model control effect. It can be universally applied to any diffusion model, allowing users to adjust the representation of specific image features in accordance with task requirements, thereby facilitating more precise and controllable generation outcomes. In particular, our designed experiments demonstrate that the FilterPrompt optimizes feature correlation, mitigates content conflicts during the generation process, and enhances the model's control capability.
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
---
|
| 12 |
-
# Getting Started
|
| 13 |
-
## Prerequisites
|
| 14 |
-
- We recommend running this repository using [Anaconda](https://docs.anaconda.com/anaconda/install/).
|
| 15 |
-
- NVIDIA GPU (Available memory is greater than 20GB)
|
| 16 |
-
- CUDA CuDNN (version β₯ 11.1, we actually use 11.7)
|
| 17 |
-
- Python 3.11.3 (Gradio requires Python 3.8 or higher)
|
| 18 |
-
- PyTorch: [Find the torch version that is suitable for the current cuda](https://pytorch.org/get-started/previous-versions/)
|
| 19 |
-
- γexampleγ:`pip install torch==2.0.0+cu117 torchvision==0.15.1+cu117 torchaudio==2.0.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117`
|
| 20 |
-
|
| 21 |
-
## Installation
|
| 22 |
-
Specifically, inspired by the concept of decoupled cross-attention in [IP-Adapter](https://ip-adapter.github.io/), we apply a similar methodology.
|
| 23 |
-
Please follow the instructions below to complete the environment configuration required for the code:
|
| 24 |
-
- Cloning this repo
|
| 25 |
-
```
|
| 26 |
-
git clone --single-branch --branch main https://github.com/Meaoxixi/FilterPrompt.git
|
| 27 |
-
```
|
| 28 |
-
- Dependencies
|
| 29 |
-
|
| 30 |
-
All dependencies for defining the environment are provided in `requirements.txt`.
|
| 31 |
-
```
|
| 32 |
-
cd FilterPrompt
|
| 33 |
-
conda create --name fp_env python=3.11.3
|
| 34 |
-
conda activate fp_env
|
| 35 |
-
pip install torch==2.0.0+cu117 torchvision==0.15.1+cu117 torchaudio==2.0.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
|
| 36 |
-
pip install -r requirements.txt
|
| 37 |
-
```
|
| 38 |
-
- Download the necessary modules in the relative path `models/` from the following links
|
| 39 |
-
|
| 40 |
-
| Path | Description |
|
| 41 |
-
|:---------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------|
|
| 42 |
-
| `models/` | root path |
|
| 43 |
-
| βββ `ControlNet/` | Place the pre-trained model of [ControlNet](https://huggingface.co/lllyasviel) |
|
| 44 |
-
| βββ `control_v11f1p_sd15_depth ` | [ControlNet_depth](https://huggingface.co/lllyasviel/control_v11f1p_sd15_depth/tree/main) |
|
| 45 |
-
| βββ `control_v11p_sd15_softedge` | [ControlNet_softEdge](https://huggingface.co/lllyasviel/control_v11p_sd15_softedge/tree/main) |
|
| 46 |
-
| βββ `IP-Adapter/` | [IP-Adapter](https://huggingface.co/h94/IP-Adapter/tree/main/models) |
|
| 47 |
-
| βββ `image_encoder ` | image_encoder of IP-Adapter |
|
| 48 |
-
| βββ `other needed configuration files` | |
|
| 49 |
-
| βββ `sd-vae-ft-mse/` | Place the model of [sd-vae-ft-mse](https://huggingface.co/stabilityai/sd-vae-ft-mse/tree/main) |
|
| 50 |
-
| βββ `stable-diffusion-v1-5/` | Place the model of [stable-diffusion-v1-5](https://huggingface.co/runwayml/stable-diffusion-v1-5) |
|
| 51 |
-
| βββ `Realistic_Vision_V4.0_noVAE/` | Place the model of [Realistic_Vision_V4.0_noVAE](https://huggingface.co/SG161222/Realistic_Vision_V4.0_noVAE/tree/main) |
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
## Demo on Gradio
|
| 57 |
-
|
| 58 |
-
After installation and downloading the models, you can use `python app.py` to perform code in gradio. We have designed four task types to facilitate you to experience the application scenarios of FilterPrompt.
|
| 59 |
-
|
| 60 |
-
## Citation
|
| 61 |
-
If you find [FilterPrompt](https://arxiv.org/abs/2404.13263) helpful in your research/applications, please cite using this BibTeX:
|
| 62 |
-
```bibtex
|
| 63 |
-
@misc{wang2024filterprompt,
|
| 64 |
-
title={FilterPrompt: Guiding Image Transfer in Diffusion Models},
|
| 65 |
-
author={Xi Wang and Yichen Peng and Heng Fang and Haoran Xie and Xi Yang and Chuntao Li},
|
| 66 |
-
year={2024},
|
| 67 |
-
eprint={2404.13263},
|
| 68 |
-
archivePrefix={arXiv},
|
| 69 |
-
primaryClass={cs.CV}
|
| 70 |
-
}
|
| 71 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|