Spaces:
Running
on
Zero
Running
on
Zero
chores
Browse files
README.md
CHANGED
|
@@ -11,6 +11,7 @@ preload_from_hub:
|
|
| 11 |
- dashtoon/hunyuan-video-keyframe-control-lora
|
| 12 |
license: apache-2.0
|
| 13 |
short_description: Precise control over the start & end frames, and in between.
|
|
|
|
| 14 |
---
|
| 15 |
Original concept and lora by dashtoon. Further optimization goals and full HF / Gradio support in the works by [AI Without Borders](https://huggingface.co/aiwithoutborders-xyz).
|
| 16 |
|
|
|
|
| 11 |
- dashtoon/hunyuan-video-keyframe-control-lora
|
| 12 |
license: apache-2.0
|
| 13 |
short_description: Precise control over the start & end frames, and in between.
|
| 14 |
+
suggested_hardware: l4x1
|
| 15 |
---
|
| 16 |
Original concept and lora by dashtoon. Further optimization goals and full HF / Gradio support in the works by [AI Without Borders](https://huggingface.co/aiwithoutborders-xyz).
|
| 17 |
|
app.py
CHANGED
|
@@ -40,7 +40,10 @@ video_transforms = transforms.Compose(
|
|
| 40 |
)
|
| 41 |
|
| 42 |
model_id = "hunyuanvideo-community/HunyuanVideo"
|
| 43 |
-
lora_path = hf_hub_download("dashtoon/hunyuan-video-keyframe-control-lora", "i2v.sft")
|
|
|
|
|
|
|
|
|
|
| 44 |
|
| 45 |
transformer = HunyuanVideoTransformer3DModel.from_pretrained(model_id, subfolder="transformer", torch_dtype=torch.bfloat16)
|
| 46 |
global pipe
|
|
@@ -287,11 +290,21 @@ def main():
|
|
| 287 |
# Define the interface inputs
|
| 288 |
|
| 289 |
with gr.Blocks() as demo:
|
| 290 |
-
gr.
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 295 |
|
| 296 |
with gr.Row():
|
| 297 |
with gr.Column():
|
|
@@ -315,31 +328,36 @@ def main():
|
|
| 315 |
|
| 316 |
generate_button = gr.Button("Generate Video")
|
| 317 |
generate_button.click(generate_video, inputs=[prompt_textbox, frame1, frame2, resolution, guidance_scale, num_frames, num_inference_steps], outputs=outputs)
|
|
|
|
|
|
|
| 318 |
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
|
|
|
|
|
|
| 324 |
|
| 325 |
-
|
| 326 |
-
|
| 327 |
-
|
| 328 |
-
|
| 329 |
-
|
| 330 |
-
|
| 331 |
-
|
| 332 |
-
|
| 333 |
|
| 334 |
-
|
| 335 |
-
|
| 336 |
-
|
| 337 |
-
|
| 338 |
-
|
| 339 |
-
|
| 340 |
|
| 341 |
-
|
| 342 |
-
|
|
|
|
| 343 |
|
| 344 |
demo.launch(show_error=True)
|
| 345 |
|
|
|
|
| 40 |
)
|
| 41 |
|
| 42 |
model_id = "hunyuanvideo-community/HunyuanVideo"
|
| 43 |
+
lora_path = hf_hub_download("dashtoon/hunyuan-video-keyframe-control-lora", "i2v.sft")
|
| 44 |
+
# lora_path = "./cache/i2v.sft"
|
| 45 |
+
|
| 46 |
+
# Replace with the actual LORA path
|
| 47 |
|
| 48 |
transformer = HunyuanVideoTransformer3DModel.from_pretrained(model_id, subfolder="transformer", torch_dtype=torch.bfloat16)
|
| 49 |
global pipe
|
|
|
|
| 290 |
# Define the interface inputs
|
| 291 |
|
| 292 |
with gr.Blocks() as demo:
|
| 293 |
+
with gr.Group():
|
| 294 |
+
gr.Markdown("""# HunyuanVideo Keyframe Control Lora for Video Generation
|
| 295 |
+
**Generate videos using the HunyuanVideo model with a prompt and two (or more) frames as conditions. Gradio / HF Spaces implementation demo.**
|
| 296 |
+
---
|
| 297 |
+
For more technical information check out the [original repo by dashtoon.](https://huggingface.co/dashtoon/hunyuan-video-keyframe-control-lora) Special shoutout to @pftq for work on optimization and ideas. Gradio Implementation by [AI Without Borders](https://huggingface.co/aiwithoutborders-xyz); this repo will be moved to the org's namespace once billing is sorted.
|
| 298 |
+
|
| 299 |
+
* Unfortunately, it's still difficult to run on a ZeroGPU space, but we're getting closer. Until then, or until we are granted a GPU allocation, this space was created for you to **DUPLICATE** and begin generating on your own hardware..
|
| 300 |
+
|
| 301 |
+
I will fill out a request for GPU allocation for the demo with HF soon.
|
| 302 |
+
|
| 303 |
+
""")
|
| 304 |
+
gr.Markdown("")
|
| 305 |
+
gr.Markdown("")
|
| 306 |
+
gr.Markdown("---")
|
| 307 |
+
gr.Markdown("")
|
| 308 |
|
| 309 |
with gr.Row():
|
| 310 |
with gr.Column():
|
|
|
|
| 328 |
|
| 329 |
generate_button = gr.Button("Generate Video")
|
| 330 |
generate_button.click(generate_video, inputs=[prompt_textbox, frame1, frame2, resolution, guidance_scale, num_frames, num_inference_steps], outputs=outputs)
|
| 331 |
+
with gr.Accordion():
|
| 332 |
+
gr.Markdown("""
|
| 333 |
|
| 334 |
+
## dashtoon: HunyuanVideo Keyframe Control Lora is an adapter for HunyuanVideo T2V model for keyframe-based video generation.
|
| 335 |
+
---
|
| 336 |
+
**Our architecture builds upon existing models, introducing key enhancements to optimize keyframe-based video generation**:
|
| 337 |
+
|
| 338 |
+
* We modify the input patch embedding projection layer to effectively incorporate keyframe information. By adjusting the convolutional input parameters, we enable the model to process image inputs within the Diffusion Transformer (DiT) framework.
|
| 339 |
+
* We apply Low-Rank Adaptation (LoRA) across all linear layers and the convolutional input layer. This approach facilitates efficient fine-tuning by introducing low-rank matrices that approximate the weight updates, thereby preserving the base model's foundational capabilities while reducing the number of trainable parameters.
|
| 340 |
+
* The model is conditioned on user-defined keyframes, allowing precise control over the generated video's start and end frames. This conditioning ensures that the generated content aligns seamlessly with the specified keyframes, enhancing the coherence and narrative flow of the video.
|
| 341 |
|
| 342 |
+
## Example Results
|
| 343 |
+
|
| 344 |
+
| Image 1 | Image 2 | Generated Video |
|
| 345 |
+
|---------|---------|-----------------|
|
| 346 |
+
|  |  | <video controls autoplay src="https://content.dashtoon.ai/stability-images/14b7dd1a-1f46-4c4c-b4ec-9d0f948712af.mp4"></video> |
|
| 347 |
+
|  |  | <video controls autoplay src="https://content.dashtoon.ai/stability-images/b00ba193-b3b7-41a1-9bc1-9fdaceba6efa.mp4"></video> |
|
| 348 |
+
|  |  | <video controls autoplay src="https://content.dashtoon.ai/stability-images/0cb84780-4fdf-4ecc-ab48-12e7e1055a39.mp4"></video> |
|
| 349 |
+
|  |  | <video controls autoplay src="https://content.dashtoon.ai/stability-images/ce12156f-0ac2-4d16-b489-37e85c61b5b2.mp4"></video> |
|
| 350 |
|
| 351 |
+
## Recommended Settings
|
| 352 |
+
1. The model works best on human subjects. Single subject images work slightly better.
|
| 353 |
+
2. It is recommended to use the following image generation resolutions `720x1280`, `544x960`, `1280x720`, `960x544`.
|
| 354 |
+
3. It is recommended to set frames from 33 upto 97. Can go upto 121 frames as well (but not tested much).
|
| 355 |
+
4. Prompting helps a lot but works even without. The prompt can be as simple as just the name of the object you want to generate or can be detailed.
|
| 356 |
+
5. `num_inference_steps` is recommended to be 50, but for fast results you can use 30 as well. Anything less than 30 is not recommended.
|
| 357 |
|
| 358 |
+
## FINAL THOUGHTS: This ZeroGPU space, while successfully loaded, has its memory packed to the rim. If you're lucky you may be able to sneak in a small demo inference here and there, but you will most definitely not be using the recommended settings listed above. Help, of course, is not only welcome but very much appreciated. Learn more about our non-profit initiative, [AI Without Borders](https://huggingface.co/aiwithoutborders-xyz), by following us on Huggingface or on [X](http://x.com/borderlesstools), where we will be announcing a handful of exciting developments.
|
| 359 |
+
|
| 360 |
+
""", sanitize_html=False, elem_id="md_footer", container=True)
|
| 361 |
|
| 362 |
demo.launch(show_error=True)
|
| 363 |
|