Image to Video Generation with Stable Video Diffusion

Stable Video Diffusion (SVD) Image-to-Video is a diffusion model that takes in a still image as a conditioning frame, and generates a video from it. In this tutorial we consider how to convert and run Stable Video Diffusion using OpenVINO. We will use stable-video-diffusion-img2video-xt model as example. Additionally, to speedup video generation process we apply AnimateLCM LoRA weights and run optimization with NNCF.

Notebook Contents

This notebook demonstrates how to convert and run stable video diffusion using OpenVINO.

Notebook contains the following steps:

Create PyTorch model pipeline using Diffusers
Convert Stable Video Diffusion Pipeline models to OpenVINO
- Convert Image Encoder
- Convert U-Net
- Convert VAE Encoder and Decoder
Create Stable Video Diffusion Pipeline with OpenVINO
Optimize pipeline with NNCF
Compare results of original and optimized pipelines
Interactive Demo

Installation Instructions

This is a self-contained example that relies solely on its own code.
We recommend running the notebook in a virtual environment. You only need a Jupyter server to start. For details, please refer to Installation Guide.