|
--- |
|
language: en |
|
thumbnail: ./assets/example.png |
|
tags: |
|
- text-to-image |
|
- diffusion models |
|
- LoRA fine-tuning |
|
- animagine-xl-3.0 |
|
- stable_diffusion_xl |
|
- kohya_ss |
|
- waifu2x |
|
license: apache-2.0 |
|
model: |
|
name: SDXL-LoRA-Text2Image |
|
description: A model based on SDXL with LoRA fine-tuning for Text-To-Image generation. |
|
pipeline_tag: image-generation |
|
repo: https://huggingface.co/TfiyuenLau/GirlsFrontline_SDXL_LoRA |
|
library: huggingface |
|
framework: pytorch |
|
version: 1.0.0 |
|
pretrained_model: stable_diffusion_xl |
|
base_model: animagine-xl-3.0 |
|
fine_tuner: kohya_ss |
|
data_augmentation: waifu2x |
|
task: text-to-image |
|
--- |
|
|
|
# 基于SDXL模型LoRA微调实现Text-To-Image文生图 |
|
|
|
 |
|
|
|
## 一、Model Library |
|
1. 微调数据集：[基于SDXL模型的《少女前线2：追放》LoRA微调数据集](https://www.kaggle.com/datasets/yukikonata/sdxl2lora) |
|
2. 预训练模型：[stable_diffusion_xl](https://huggingface.co/docs/diffusers/api/pipelines/stable_diffusion/stable_diffusion_xl) |
|
3. 底模：[animagine-xl-3.0](https://huggingface.co/cagliostrolab/animagine-xl-3.0) |
|
4. SDXL LoRA微调训练器：[kohya_ss](https://github.com/bmaltais/kohya_ss) |
|
5. 数据集画质增强：[waifu2x](https://github.com/nagadomi/waifu2x) |
|
|
|
## 二、Prompt Dict |
|
1. 少前2追放角色5 |
|
* 佩里缇亚: PKPSP |
|
* 塞布丽娜: SPAS12 |
|
* 托洛洛: AKAlfa |
|
* 桑朵莱希: G36 |
|
* 琼玖: QBZ191 |
|
* 维普雷: Vepr12 |
|
* 莫辛纳甘: MosinNagant |
|
* 黛烟: QBZ95 |
|
* 克罗丽科: Kroliko |
|
* 夏克里: XCRL |
|
* 奇塔: MP7 |
|
* 寇尔芙: TaurusCurve |
|
* 科谢尼娅: APS |
|
* 纳甘: Nagant1895 |
|
* 纳美西丝: OM50 |
|
* 莉塔拉: GalilARM |
|
* 闪电: OTs14 |
|
|
|
2. Pixiv画师风格 |
|
* おにねこ（鬼猫）: Onineko26 |
|
* 麻生: AsouAsabu |
|
* mignon: Mignon |
|
* migolu: Migolu |
|
|
|
## 三、使用方式 |
|
1. 安装部分环境（默认已安装pytorch等必要环境） |
|
~~~sh |
|
pip install diffusers --upgrade |
|
pip install transformers accelerate safetensors |
|
~~~ |
|
|
|
2. 使用Hugging Face下载并使用底模（animagine-xl-3.0）和LoRA模型 |
|
~~~python |
|
import torch |
|
import datetime |
|
from PIL import Image |
|
import matplotlib.pyplot as plt |
|
from diffusers import ( |
|
StableDiffusionXLPipeline, |
|
EulerAncestralDiscreteScheduler, |
|
AutoencoderKL |
|
) |
|
|
|
# LoRA Hugging Face ID |
|
lora_id = "TfiyuenLau/GirlsFrontline_SDXL_LoRA" |
|
|
|
# Load VAE component |
|
vae = AutoencoderKL.from_pretrained( |
|
"madebyollin/sdxl-vae-fp16-fix", |
|
torch_dtype=torch.float16 |
|
) |
|
|
|
# Configure the pipeline |
|
pipe = StableDiffusionXLPipeline.from_pretrained( |
|
"cagliostrolab/animagine-xl-3.0", |
|
vae=vae, |
|
torch_dtype=torch.float16, |
|
use_safetensors=True, |
|
) |
|
pipe.load_lora_weights(lora_id) |
|
pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config) |
|
pipe.to('cuda') |
|
~~~ |
|
|
|
3. 生成图像 |
|
~~~python |
|
# Define Prompt |
|
output = "./output.png" |
|
prompt = "1girl, OTs14, gloves, looking at viewer, smile, food, holding, solo, closed mouth, sitting, yellow eyes, black gloves, masterpiece, best quality" |
|
negative_prompt = "nsfw, lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry, artist name" |
|
|
|
# Generate Image |
|
image = pipe( |
|
prompt, |
|
negative_prompt=negative_prompt, |
|
width=1024, |
|
height=1024, |
|
guidance_scale=7, |
|
num_inference_steps=28 |
|
).images[0] |
|
|
|
# Save & Show |
|
image.save(output) |
|
image = Image.open(output) |
|
plt.axis('off') |
|
plt.imshow(image) |
|
image.close() |
|
~~~ |
|
|
