a-r-r-o-w/HunyuanVideo-tuxemons

LoRA Finetune

Prompt

Style of snomexut, A fluffy, brown-haired tuxemon with a curious expression, sporting an orange and black striped harness and unique patterned tail, explores its surroundings.

Prompt

Style of snomexut, A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.

Prompt

Style of snomexut, A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.

Prompt

Style of snomexut, A menacing blue tuxemon with sharp claws and fierce fangs roars against a cosmic backdrop, its body adorned with glowing orbs and sharp pink details.

Model description

This is a lora finetune of https://huggingface.co/hunyuanvideo-community/HunyuanVideo.

The model was trained using finetrainers. Training progress over time (4000 steps, 6000 steps, 8000 steps, 10000 steps):

Download model

Download LoRA in the Files & Versions tab.

Usage

Requires the 🧨 Diffusers library installed.

Trigger phrase: "Style of snomexut,"

import torch
from diffusers import HunyuanVideoPipeline, HunyuanVideoTransformer3DModel
from diffusers.utils import export_to_video

model_id = "hunyuanvideo-community/HunyuanVideo"
transformer = HunyuanVideoTransformer3DModel.from_pretrained(
    model_id, subfolder="transformer", torch_dtype=torch.bfloat16
)
pipe = HunyuanVideoPipeline.from_pretrained(model_id, transformer=transformer, torch_dtype=torch.float16)
pipe.vae.enable_tiling()
pipe.to("cuda")

pipe.load_lora_weights("a-r-r-o-w/HunyuanVideo-tuxemons", adapter_name="hunyuanvideo-lora")
pipe.set_adapters("hunyuanvideo-lora", 1.2)

output = pipe(
    prompt="Style of snomexut, a cat-like Tuxemon creature walks in alien-world grass, and observes its surroundings.",
    height=768,
    width=768,
    num_frames=33,
    num_inference_steps=30,
    generator=torch.Generator().manual_seed(73),
).frames[0]
export_to_video(output, "output-tuxemon.mp4", fps=15)

For more details, including weighting, merging and fusing LoRAs, check the documentation on loading LoRAs in diffusers.

Training params

Dataset contained 250 images from diffusers/tuxemon and 150 samples generated from a Flux LoRA trained on the same dataset.

id_token: "Style of snomexut,"
target_modules: to_q to_k to_v to_out.0 add_q_proj add_k_proj add_v_proj to_add_out
lr: 1e-5
training_steps: 10000
video_resolution_buckets: 1x768x768 1x512x512
rank: 128
lora_alpha: 128
optimizer: adamw
weight_decay: 0.01
flow_weighting_scheme: logit_normal
flow_shift: 7.0
validation_prompts: $ID_TOKEN A fluffy, brown-haired tuxemon with a curious expression, sporting an orange and black striped harness and unique patterned tail, explores its surroundings.@@@1x768x768:::$ID_TOKEN A curious yellow Tuxemon bird with large, expressive eyes and a distinctive golden beak, set against a starry black background, looks gently to its left.@@@1x768x768:::$ID_TOKEN A menacing Tuxemon with the appearance of a green and white aquatic shark, featuring sharp teeth and red eyes, emerges from the dark abyss, ready to unleash its fury.@@@1x768x768:::$ID_TOKEN A cheerful light blue tuxemon with four tentacle-like arms and sharp ears smiles playfully against a dark background.@@@1x768x768:::$ID_TOKEN A cheerful white Tuxemon with large ears and a playful expression joyfully floats in a starry outer space backdrop.@@@1x768x768:::$ID_TOKEN A finely detailed, silver-toned pin shaped like a futuristic double-barreled gun, adorned with orange textured panels, popularly associated with the Tuxemon universe.@@@1x768x768:::$ID_TOKEN A menacing blue tuxemon with sharp claws and fierce fangs roars against a cosmic backdrop, its body adorned with glowing orbs and sharp pink details.@@@1x768x768:::$ID_TOKEN A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.@@@1x768x768:::$ID_TOKEN A fluffy, brown-haired tuxemon with a curious expression, sporting an orange and black striped harness and unique patterned tail, explores its surroundings.@@@49x768x768:::$ID_TOKEN A curious yellow Tuxemon bird with large, expressive eyes and a distinctive golden beak, set against a starry black background, looks gently to its left.@@@49x768x768:::$ID_TOKEN A menacing Tuxemon with the appearance of a green and white aquatic shark, featuring sharp teeth and red eyes, emerges from the dark abyss, ready to unleash its fury.@@@49x768x768:::$ID_TOKEN A cheerful light blue tuxemon with four tentacle-like arms and sharp ears smiles playfully against a dark background.@@@49x768x768:::$ID_TOKEN A finely detailed, silver-toned pin shaped like a futuristic double-barreled gun, adorned with orange textured panels, popularly associated with the Tuxemon universe.@@@49x768x768:::$ID_TOKEN A menacing blue tuxemon with sharp claws and fierce fangs roars against a cosmic backdrop, its body adorned with glowing orbs and sharp pink details.@@@49x768x768:::$ID_TOKEN A vibrant yellow Tuxemon, wielding a fiery blue staff, joyfully manipulates swirling flames around its cheerful form.@@@49x768x768