|
--- |
|
license: creativeml-openrail-m |
|
tags: |
|
- text-to-video |
|
- stable-diffusion |
|
- animatediff |
|
library_name: diffusers |
|
inference: false |
|
--- |
|
# AnimateDiff-Lightning |
|
|
|
<video src='https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/animatediff_lightning_samples.mp4' width="100%" autoplay muted loop></video> |
|
|
|
AnimateDiff-Lightning is a lightning-fast text-to-video generation model. It can generate 16-frame 512px videos in a few steps. For more information, please refer to our research paper: [AnimateDiff-Lightning: Cross-Model Diffusion Distillation](https://huggingface.co/ByteDance/AnimateDiff-Lightning/resolve/main/animatediff_lightning_report.pdf). We release the model as part of the research. |
|
|
|
Our models are distilled from [AnimateDiff SD1.5 v2](https://huggingface.co/guoyww/animatediff). This repository contains checkpoints for 1-step, 2-step, 4-step, and 8-step distilled models. The generation quality of our 2-step, 4-step, and 8-step model is great. Our 1-step model is only provided for research purposes. |
|
|
|
|
|
## Recommendation |
|
|
|
AnimateDiff-Lightning produces the best results when used with stylized base models. We recommend using the following base models: |
|
|
|
Realistic |
|
- [epiCRealism](https://civitai.com/models/25694) |
|
- [Realistic Vision](https://civitai.com/models/4201) |
|
- [DreamShaper](https://civitai.com/models/4384) |
|
- [AbsoluteReality](https://civitai.com/models/81458) |
|
- [MajicMix Realistic](https://civitai.com/models/43331) |
|
|
|
Anime & Cartoon |
|
- [ToonYou](https://civitai.com/models/30240) |
|
- [IMP](https://civitai.com/models/56680) |
|
- [Mistoon Anime](https://civitai.com/models/24149) |
|
- [DynaVision](https://civitai.com/models/75549) |
|
- [RCNZ Cartoon 3d](https://civitai.com/models/66347) |
|
- [MajicMix Reverie](https://civitai.com/models/65055) |
|
|
|
Additionally, feel free to explore different settings. We find using 3 inference steps on the 2-step model produces great results. We find certain base models produces better results with CFG. We also recommend using [Motion LoRAs](https://huggingface.co/guoyww/animatediff/tree/main) as they produce stronger motion. We use Motion LoRAs with strength 0.7~0.8 to avoid watermark. |
|
|
|
## Diffusers Usage |
|
|
|
```python |
|
import torch |
|
from diffusers import AnimateDiffPipeline, MotionAdapter, EulerDiscreteScheduler |
|
from diffusers.utils import export_to_gif |
|
from huggingface_hub import hf_hub_download |
|
from safetensors.torch import load_file |
|
|
|
device = "cuda" |
|
dtype = torch.float16 |
|
|
|
step = 4 # Options: [1,2,4,8] |
|
repo = "AnimateDiff-Lightning" |
|
ckpt = f"animatediff_lightning_{step}step_diffusers.safetensors" |
|
base = "SG161222/Realistic_Vision_V5.1_noVAE" # Choose to your favorite base model. |
|
|
|
adapter = MotionAdapter().to(device, dtype) |
|
adapter.load_state_dict(load_file(hf_hub_download(repo ,ckpt), device=device)) |
|
pipe = AnimateDiffPipeline.from_pretrained(base, motion_adapter=adapter, torch_dtype=dtype).to(device) |
|
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing", beta_schedule="linear") |
|
|
|
output = pipe(prompt="A girl smiling", guidance_scale=1.0, num_inference_steps=step) |
|
export_to_gif(output.frames[0], "animation.gif") |
|
``` |
|
|
|
## ComfyUI Usage |
|
|
|
1. Download [AnimateDiff-Lightning ComfyUI Workflow](https://huggingface.co/ByteDance/AnimateDiff-Lightning/raw/main/comfyui/animatediff_lightning_workflow.json) |
|
|
|
2. Install nodes. You can install them manually or use [ComfyUI-Manager](https://github.com/ltdrdata/ComfyUI-Manager). |
|
* [ComfyUI-AnimateDiff-Evolved](https://github.com/Kosinkadink/ComfyUI-AnimateDiff-Evolved) |
|
* [ComfyUI-VideoHelperSuite](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite) |
|
|
|
3. Download your favorite base model checkpoint and put them under `/models/checkpoints/` |
|
|
|
4. Download AnimateDiff-Lightning checkpoint `animatediff_lightning_Nstep_comfyui.safetensors` and put them under `/custom_nodes/ComfyUI-AnimateDiff-Evolved/models/` |
|
|
|
|
|
 |
