|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
- zh |
|
base_model: stabilityai/stable-diffusion-xl-base-1.0 |
|
library_name: diffusers |
|
tags: |
|
- Text-to-Image |
|
- SDXL |
|
- Stable Diffusion |
|
- ID Customization |
|
pipeline_tag: text-to-image |
|
--- |
|
<div style="display: flex; justify-content: center; align-items: center;"> |
|
<img src="./images/images_alibaba.png" alt="alibaba" style="width: 20%; height: auto; margin-right: 5%;"> |
|
<img src="./images/images_alimama.png" alt="alimama" style="width: 20%; height: auto;"> |
|
</div> |
|
|
|
[中文版Readme](./README_ZH.md) |
|
|
|
EcomID aims to generate customized images from a single reference ID image, ensuring strong semantic consistency while being controlled by keypoints. |
|
|
|
This repository provides the EcomID method and model, combining the strengths of [PuLID](https://github.com/ToTheBeginning/PuLID) and [InstantID](https://github.com/instantX-research/InstantID) for better background consistency, facial keypoint control, and realistic facial representation with improved similarity. |
|
|
|
# EcomID Overview |
|
|
|
## EcomID Structure |
|
<img src="./images/overflow.png" alt="alibaba" style="width: 100%; height: auto; margin-right: 5%;"> |
|
|
|
|
|
- **IP-Adapter of PuLID**: EcomID incorporates the ID-Encoder and cross-attention components from PuLID, trained with alignment loss. |
|
This method effectively reduces the interference of ID embeddings on text embeddings within the cross-attention part, minimizing disruption to the underlying model's text-to-image capabilities. |
|
- **InstantID’s IdentityNet Architecture**: Utilizing **a dataset of 2 million aesthetically pleasing portrait images**, IdentityNet enhances keypoint control, improving ID consistency and facial realism. During training, the IP-adapter is frozen, and only the IdentityNet is trained. Facial landmarks are used as conditional inputs, while face embeddings are integrated into IdentityNet via cross-attention. |
|
|
|
# Show Cases |
|
## Comparison with Other Methods |
|
### 1、Preserved Text-to-Image Capability |
|
|
|
<table> |
|
<tr> |
|
<th style="width: 28%;">Prompt</th> |
|
<th style="width: 24%;">Reference Image</th> |
|
<th style="width: 24%;">EcomID</th> |
|
<th style="width: 24%;">InstantID</th> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">girl, white skin, black hair, long wavy hair, <span style="color:red"><strong>in European style living room, Retro tone, decorations</strong></span>, depth of field.</td> |
|
<td><img src="images/show_case/50.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/49.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/48.png" alt="InstantID图像" width="100%"></td> |
|
</tr> |
|
<table> |
|
|
|
As shown above, EcomID ***preserves background generation abilities while minimizing stylization, greatly enhancing realism***. |
|
The visualizations highlight more authentic portraits with improved background semantic consistency, showcasing EcomID's advantage in generating realistic images. |
|
|
|
### 2、Improved Facial Control and Consistency |
|
<table> |
|
<tr> |
|
<th style="width: 24%;">Prompt</th> |
|
<th style="width: 19%;">Reference Image</th> |
|
<th style="width: 19%;">EcomID</th> |
|
<th style="width: 19%;">InstantID</th> |
|
<th style="width: 19%;">PuLID</th> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">A close-up portrait of a man standing in the library, holding <span style="color:red"><strong>two smiling toddlers</strong></span> next to him.</td> |
|
<td><img src="images/show_case/20.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/17.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/18.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/19.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<table> |
|
|
|
As shown above, EcomID employs keypoints as conditional inputs for training, ***allowing for precise adjustments of facial positions, sizes, and orientations***. This capability ensures that the generated portraits are more controllable while further enhancing facial similarity and the overall quality of the images. |
|
|
|
### More showcases |
|
EcomID enhances portrait representation, delivering a more authentic and aesthetically pleasing appearance while ensuring semantic consistency and greater internal ID similarity (i.e., traits that do not vary with age, hairstyle, glasses, or other physical changes). |
|
|
|
<table> |
|
<tr> |
|
<th style="width: 24%;">Prompt</th> |
|
<th style="width: 19%;">Reference Image</th> |
|
<th style="width: 19%;">EcomID</th> |
|
<th style="width: 19%;">InstantID</th> |
|
<th style="width: 19%;">PuLID</th> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">A close-up portrait of a <span style="color:red"><strong>little girl with double braids</strong></span>, wearing a white dress, standing on the beach during sunset.</td> |
|
<td><img src="images/show_case/21.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/22.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/23.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/24.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">A close-up portrait of a <span style="color:red"><strong>very little girl</strong></span> with double braids, wearing <span style="color:red"><strong>a hat</strong></span> and white dress, standing on the beach during sunset.</td> |
|
<td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/47.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/46.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/45.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">Agrizzled detective, <span style="color:red"><strong>fedora</strong></span> casting a shadow over his square jaw, a <span style="color:red"><strong>cigar dangling from his lips</strong></span>, his trench coat evocative of film noir, in a <span style="color:red"><strong>rainy alley</strong></span>.</td> |
|
<td><img src="images/show_case/25.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/26.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/27.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/28.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">A smiling girl with <span style="color:red"><strong>bangs and long hair</strong></span> in a school uniform stands under cherry trees, holding a book.</td> |
|
<td><img src="images/show_case/29.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/30.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/31.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/32.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">A <span style="color:red"><strong>very old</strong></span> witch, wearing a black cloak, with a pointed hat, holding a magic wand, against a background of a misty forest.</td> |
|
<td><img src="images/show_case/33.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/34.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/35.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/36.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td style="font-size: 12px;">A man clad in cyberpunk fashion: <span style="color:red"><strong>neon accents, reflective sunglasses，</strong></span> and a leather jacket with glowing circuit patterns. He stands stoically amidst a soaked cityscape.</td> |
|
<td><img src="images/show_case/37.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/38.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/39.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/40.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
|
|
</table> |
|
|
|
### More Base Models, Resolutions, and Styles |
|
<table> |
|
<tr> |
|
<th style="width: 12%;">SDXL models</th> |
|
<th style="width: 24%;">Prompt</th> |
|
<th style="width: 16%;">Reference Image</th> |
|
<th style="width: 16%;">EcomID</th> |
|
<th style="width: 16%;">InstantID</th> |
|
<th style="width: 16%;">PuLID</th> |
|
</tr> |
|
<tr> |
|
<td>sd-xl-base-1.0</td> |
|
<td style="font-size: 12px;">girl, solo, brown hair, holding a little teddy bear on her hands, wearing a school uniform, standing in the library, <span style="color:red"><strong>cartoon style</strong></span>.</td> |
|
<td><img src="images/show_case/1.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/2.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/3.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/4.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td>EcomXL</td> |
|
<td style="font-size: 12px;">A close-up portrait of a <span style="color:red"><strong>very little girl</strong></span> with double braids, wearing <span style="color:red"><strong>a hat</strong></span> and white dress, standing on the beach during sunset.</td> |
|
<td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/47.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/46.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/45.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td>DreamShaperXL</td> |
|
<td style="font-size: 12px;">solo, looking_at_viewer, smile, brown_hair, upper_body, open_clothes, teeth, open_jacket, black_jacket, blurry_background, realistic</td> |
|
<td><img src="images/show_case/44.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/6.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/7.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/8.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
<tr> |
|
<td>leosam_xl_v7</td> |
|
<td style="font-size: 12px;">A close-up portrait of a girl, solo, dress, jewelry, beach and sea, pink_dress, realistic.</td> |
|
<td><img src="images/show_case/9.png" alt="参考图像" width="100%"></td> |
|
<td><img src="images/show_case/15.png" alt="EcomID图像" width="100%"></td> |
|
<td><img src="images/show_case/14.png" alt="InstantID图像" width="100%"></td> |
|
<td><img src="images/show_case/16.png" alt="PuLID图像" width="100%"></td> |
|
</tr> |
|
</table> |
|
|
|
### Notes |
|
- Unless otherwise specified, the showcases are generated using the base model EcomXL, which is also highly compatible with various other SDXL-based models, such as [leosams-helloworld-xl](https://civitai.com/models/43977/leosams-helloworld-xl), [dreamshaper-xl](https://civitai.com/models/112902/dreamshaper-xl), [stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and so on. |
|
- It works very well with SDXL Turbo/Lighting, [EcomXL Inpainting ControlNet](https://huggingface.co/alimama-creative/EcomXL_controlnet_inpaint) and [EcomXL Softedge ControlNet](https://huggingface.co/alimama-creative/EcomXL_controlnet_softedge). |
|
|
|
# How to use |
|
|
|
## ComfyUI |
|
|
|
- The EcomID_ComfyUI node has been released: [click here](https://github.com/alimama-creative/SDXL_EcomID_ComfyUI) |
|
|
|
# Training Details |
|
|
|
The model is trained on 2M Taobao images, where the proportion of human faces is greater than 3%. The images have a resolution greater than 800, and the aesthetic score is above 5.5. |
|
|
|
Mixed precision: fp16 |
|
|
|
Learning rate: 1e-4 |
|
|
|
Batch size: 2 |
|
|
|
Image size: 1024x1024 |
