Cinematographic Camera Diffusion Model

This repo provides PyTorch implementation of our paper :

Cinematographic Camera Diffusion Model

Hongda Jiang, Xi Wang, Marc Christie, Libin Liu,Baoquan Chen

Eurographics 2024

The homepage and paper will release after published.

Prerequisites

The environment requirement for this repo is simple.

Linux
NVIDIA GPU + CUDA CuDNN
Python 3.8
Pytorch, torchvision, tqdm, matplotlib, numpy, CLIP

Dataset

We provide dataset in link. The dataset is a numpy dict, where the key 'cam' includes the camera trajectories and 'info' includes the text descriptions.

Pretrained Model

We provide weights with text only training results. Please create an empty folder weight and put the weight file into the folder.

Tips: If you want to use the pretrained weight, please use zooms in and zooms out when you want to generate sequence with pushes in and pulls out since in the training we use these two prompts.

Inference

Simply run python main.py gen and the generated sequences will be put in folder gen.

We provide a Unity Scene for visualize the result link, the version of the project is 2018.2.13f1. You need to set the file path, shooting target (head), shooting character. Here we provide an example of 'pan' motion with prompt 'The camera pans to the character. The camera switches from right front view to right back view. The character is at the middle center of the screen. The camera shoots at close shot.'.

Evaluation

We provide the code of classifier classify.py, metric metric.py, and LSTM based camera motion generator LSTM.py. The training and testing dataset are separated with 9:1 ratio randomly.

Acknowledgement

This code is standing on the shoulders of giants. We want to thank the following contributors that our code is based on:

Conditional Diffusion MNIST, MDM: Human Motion Diffusion Model.