maxin-cn/Cinemo

Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models

This repo contains pre-trained weights for our paper exploring image animation with motion diffusion models (Cinemo). You can find more visualizations on our project page.

In this project, we propose a novel method called Cinemo, which can perform motion-controllable image animation with strong consistency and smoothness. To improve motion smoothness, Cinemo learns the distribution of motion residuals, rather than directly generating subsequent frames. Additionally, a structural similarity index-based method is proposed to control the motion intensity. Furthermore, we propose a noise refinement technique based on discrete cosine transformation to ensure temporal consistency. These three methods help Cinemo generate highly consistent, smooth, and motion-controlled image animation results. Compared to previous methods, Cinemo offers simpler and more precise user control and better generative performance.

News

• (🔥 New) Jun. 2, 2024. 💥 The inference code is released. The checkpoint can be found here.

Setup

First, download and set up the repo:

git clone https://github.com/maxin-cn/Cinemo
cd Cinemo

We provide an environment.yml file that can be used to create a Conda environment. If you only want to run pre-trained models locally on CPU, you can remove the cudatoolkit and pytorch-cuda requirements from the file.

conda env create -f environment.yml
conda activate cinemo

Animation

You can sample from our pre-trained Cinemo models with animation.py. Weights for our pre-trained Cinemo model can be found here. The script has various arguments to adjust sampling steps, change the classifier-free guidance scale, etc:

bash pipelines/animation.sh

Other Applications

You can also utilize Cinemo for other applications, such as motion transfer and video editing:

bash pipelines/video_editing.sh

Acknowledgments

Cinemo has been greatly inspired by the following amazing works and teams: LaVie and SEINE, we thank all the contributors for open-sourcing.

Bibtex citation

@misc{ma2024cinemoconsistentcontrollableimage,
      title={Cinemo: Consistent and Controllable Image Animation with Motion Diffusion Models}, 
      author={Xin Ma and Yaohui Wang and Gengyun Jia and Xinyuan Chen and Yuan-Fang Li and Cunjian Chen and Yu Qiao},
      year={2024},
      eprint={2407.15642},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.15642}, 
}