|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- CaptionEmporium/coyo-hd-11m-llavanext |
|
- CortexLM/midjourney-v6 |
|
language: |
|
- en |
|
base_model: |
|
- black-forest-labs/FLUX.1-dev |
|
pipeline_tag: image-to-image |
|
library_name: diffusers |
|
--- |
|
|
|
<img src="assets/banner-dark.png?raw=true" alt="Banner Picture 1" style="width:1024px;"/> |
|
<a href="https://discord.gg/FHY2guThfy"> |
|
<img src="https://github.com/XLabs-AI/x-flux/blob/main/assets/readme/light/join-our-discord-rev1.png?raw=true" style="width:1024px;"/> |
|
</a> |
|
<img src="assets/ip_adapter_0.jpg?raw=true" alt="example_0" style="width:1024px;"/> |
|
<img src="assets/mona_workflow.jpg?raw=true" alt="Mona Anime Workflow 1" style="width:1024px;"/> |
|
|
|
This repository provides a IP-Adapter checkpoint for |
|
[FLUX.1-dev model](https://huggingface.co/black-forest-labs/FLUX.1-dev) by Black Forest Labs |
|
|
|
[See our github](https://github.com/XLabs-AI/x-flux-comfyui) for comfy ui workflows. |
|
|
|
# Models |
|
The IP adapter is trained on a resolution of 512x512 for 150k steps and 1024x1024 for 350k steps while maintaining the aspect ratio. |
|
We release **v2 version** - which can be used directly in ComfyUI! |
|
|
|
Please, see our [ComfyUI custom nodes installation guide](https://github.com/XLabs-AI/x-flux-comfyui) |
|
|
|
# Examples |
|
|
|
See examples of our models results below. |
|
Also, some generation results with input images are provided in "Files and versions" |
|
|
|
# Inference |
|
|
|
To try our models, you have 3 options: |
|
1. Use main.py from our [official repo](https://github.com/XLabs-AI/x-flux) |
|
2. Use our custom nodes for ComfyUI and test it with provided workflows (check out folder /workflows) |
|
3. Diffusers 🧨 |
|
|
|
## Instruction for ComfyUI |
|
1. Go to ComfyUI/custom_nodes |
|
2. Clone [x-flux-comfyui](https://github.com/XLabs-AI/x-flux-comfyui.git), path should be ComfyUI/custom_nodes/x-flux-comfyui/*, where * is all the files in this repo |
|
3. Go to ComfyUI/custom_nodes/x-flux-comfyui/ and run python setup.py |
|
4. Update x-flux-comfy with `git pull` or reinstall it. |
|
5. Download Clip-L `model.safetensors` from [OpenAI VIT CLIP large](https://huggingface.co/openai/clip-vit-large-patch14), and put it to `ComfyUI/models/clip_vision/*`. |
|
6. Download our IPAdapter from [huggingface](https://huggingface.co/XLabs-AI/flux-ip-adapter/tree/main), and put it to `ComfyUI/models/xlabs/ipadapters/*`. |
|
7. Use `Flux Load IPAdapter` and `Apply Flux IPAdapter` nodes, choose right CLIP model and enjoy your genereations. |
|
8. You can find example workflow in folder workflows in this repo. |
|
|
|
## Diffusers 🧨 |
|
1. Install Diffusers 🧨 `pip install -U diffusers` |
|
2. Run the example |
|
```python |
|
import torch |
|
from diffusers import FluxPipeline |
|
from diffusers.utils import load_image |
|
|
|
pipe: FluxPipeline = FluxPipeline.from_pretrained( |
|
"black-forest-labs/FLUX.1-dev", |
|
torch_dtype=torch.bfloat16, |
|
).to("cuda") |
|
|
|
image = load_image("monalisa.jpg").resize((1024, 1024)) |
|
|
|
pipe.load_ip_adapter( |
|
"XLabs-AI/flux-ip-adapter-v2", |
|
weight_name="ip_adapter.safetensors", |
|
image_encoder_pretrained_model_name_or_path="openai/clip-vit-large-patch14" |
|
) |
|
|
|
def LinearStrengthModel(start, finish, size): |
|
return [ |
|
(start + (finish - start) * (i / (size - 1))) for i in range(size) |
|
] |
|
|
|
ip_strengths = LinearStrengthModel(0.4, 1.0, 19) |
|
pipe.set_ip_adapter_scale(ip_strengths) |
|
|
|
image = pipe( |
|
width=1024, |
|
height=1024, |
|
prompt='wearing red sunglasses, golden chain and a green cap', |
|
negative_prompt="", |
|
true_cfg_scale=1.0, |
|
generator=torch.Generator().manual_seed(0), |
|
ip_adapter_image=image, |
|
).images[0] |
|
|
|
image.save('result.jpg') |
|
``` |
|
|
|
|
|
|
|
If you get bad results, try to set to play with ip strength |
|
### Limitations |
|
The IP Adapter is currently in beta. |
|
We do not guarantee that you will get a good result right away, it may take more attempts to get a result. |
|
<img src="assets/ip_adapter_2.jpg?raw=true" alt="example_2" style="width:1024px;"/> |
|
<img src="assets/ip_adapter_3.jpg?raw=true" alt="example_3" style="width:1024px;"/> |
|
<img src="assets/ip_adapter_1.jpg?raw=true" alt="example_1" style="width:1024px;"/> |
|
<img src="assets/ip_adapter_4.jpg?raw=true" alt="example_4" style="width:1024px;"/> |
|
<img src="assets/ip_adapter_5.jpg?raw=true" alt="example_5" style="width:1024px;"/> |
|
<img src="assets/ip_adapter_6.jpg?raw=true" alt="example_6" style="width:1024px;"/> |
|
|
|
|
|
## License |
|
Our weights fall under the [FLUX.1 [dev]](https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/LICENSE.md) Non-Commercial License<br/> |
