🍰 Hybrid-sd-small-vae for Stable Diffusion

Hybrid-sd-small-vae is a pruned-finetuned version VAE which uses the same "latent API" as the base model SD-VAE. It has smaller size, faster inference speed, as well as well-performed image generation in image saturation and image clarity compared to SD1.5. Specifically，we decreses parameters from original 83.65M to 62.01M, inferece time from 186.58ms to 135.58ms, roughly save up to 43.7% memory usage (12987MiB -> 9087MiB) without lossing T2I generation quality. The model is useful for real-time previewing of the SD1.x generation process, and you are very welcome to try it !!!!!!

Index Table

Model	Params (M)	Decoder inference time (ms)	Decoder GPU Memory Usage (MiB)
SD1.5	83.65	186.58	12987
Hybrid-sd-small-vae	62.014 ↓	135.58 ↓	9087 ↓

T2I Comparison using one A100 GPU, The image order from left to right : SD-VAE -> Hybrid-sd-small-vae

This repo contains .safetensors versions of the Hybrid-sd-small-vae weights. For SDXL, use Hybrid-sd-small-vae-xl instead (the SD and SDXL VAEs are incompatible).

Using in 🧨 diffusers

Firstly download our repository to load the AutoencoderKL

git clone https://github.com/bytedance/Hybrid-SD/tree/main

from bytenn_autoencoder_kl import AutoencoderKL
import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-2-1-base", torch_dtype=torch.float16
)

vae = AutoencoderKL.from_pretrained('cqyan/hybrid-sd-small-vae', torch_dtype=torch.float16)
pipe.vae = vae
pipe = pipe.to("cuda")
prompt = "A warm and loving family portrait, highly detailed, hyper-realistic, 8k resolution, photorealistic, soft and natural lighting"
image = pipe(prompt, num_inference_steps=25).images[0]
image.save("family.png")