|
--- |
|
license: creativeml-openrail-m |
|
tags: |
|
- text-to-image |
|
- stable-diffusion |
|
- lora |
|
- diffusers |
|
base_model: stabilityai/stable-diffusion-xl-base-1.0 |
|
pivotal_tuning: true |
|
textual_embeddings: embeddings.pti |
|
instance_prompt: <s0><s1> |
|
inference: false |
|
--- |
|
# sdxl-tron LoRA by [fofr](https://replicate.com/fofr) |
|
### A fine-tuned SDXL lora based on Tron Legacy |
|
|
|
 |
|
> |
|
|
|
## Inference with Replicate API |
|
Grab your replicate token [here](https://replicate.com/account) |
|
```bash |
|
pip install replicate |
|
export REPLICATE_API_TOKEN=r8_************************************* |
|
``` |
|
|
|
```py |
|
import replicate |
|
|
|
output = replicate.run( |
|
"sdxl-tron@sha256:fd920825e12db2a942f8a9cac40ad4f624a34a06faba3ac1b44a5305df8c6e2d", |
|
input={"prompt": "A futuristic close-up portrait photo in the style of TRN"} |
|
) |
|
print(output) |
|
``` |
|
You may also do inference via the API with Node.js or curl, and locally with COG and Docker, [check out the Replicate API page for this model](https://replicate.com/fofr/sdxl-tron/api) |
|
|
|
## Inference with 🧨 diffusers |
|
Replicate SDXL LoRAs are trained with Pivotal Tuning, which combines training a concept via Dreambooth LoRA with training a new token with Textual Inversion. |
|
As `diffusers` doesn't yet support textual inversion for SDXL, we will use cog-sdxl `TokenEmbeddingsHandler` class. |
|
|
|
The trigger tokens for your prompt will be `<s0><s1>` |
|
|
|
```shell |
|
pip install diffusers transformers accelerate safetensors huggingface_hub |
|
git clone https://github.com/replicate/cog-sdxl cog_sdxl |
|
``` |
|
|
|
```py |
|
import torch |
|
from huggingface_hub import hf_hub_download |
|
from diffusers import DiffusionPipeline |
|
from cog_sdxl.dataset_and_utils import TokenEmbeddingsHandler |
|
from diffusers.models import AutoencoderKL |
|
|
|
pipe = DiffusionPipeline.from_pretrained( |
|
"stabilityai/stable-diffusion-xl-base-1.0", |
|
torch_dtype=torch.float16, |
|
variant="fp16", |
|
).to("cuda") |
|
|
|
pipe.load_lora_weights("fofr/sdxl-tron", weight_name="lora.safetensors") |
|
|
|
text_encoders = [pipe.text_encoder, pipe.text_encoder_2] |
|
tokenizers = [pipe.tokenizer, pipe.tokenizer_2] |
|
|
|
embedding_path = hf_hub_download(repo_id="fofr/sdxl-tron", filename="embeddings.pti", repo_type="model") |
|
embhandler = TokenEmbeddingsHandler(text_encoders, tokenizers) |
|
embhandler.load_embeddings(embedding_path) |
|
prompt="A futuristic close-up portrait photo in the style of TRN" |
|
images = pipe( |
|
prompt, |
|
cross_attention_kwargs={"scale": 0.8}, |
|
).images |
|
#your output image |
|
images[0] |
|
``` |
|
|
