Papers
arxiv:2408.06070

ControlNeXt: Powerful and Efficient Control for Image and Video Generation

Published on Aug 12
· Submitted by akhaliq on Aug 13
#3 Paper of the day
Authors:
,
,
,

Abstract

Diffusion models have demonstrated remarkable and robust abilities in both image and video generation. To achieve greater control over generated results, researchers introduce additional architectures, such as ControlNet, Adapters and ReferenceNet, to integrate conditioning controls. However, current controllable generation methods often require substantial additional computational resources, especially for video generation, and face challenges in training or exhibit weak control. In this paper, we propose ControlNeXt: a powerful and efficient method for controllable image and video generation. We first design a more straightforward and efficient architecture, replacing heavy additional branches with minimal additional cost compared to the base model. Such a concise structure also allows our method to seamlessly integrate with other LoRA weights, enabling style alteration without the need for additional training. As for training, we reduce up to 90% of learnable parameters compared to the alternatives. Furthermore, we propose another method called Cross Normalization (CN) as a replacement for Zero-Convolution' to achieve fast and stable training convergence. We have conducted various experiments with different base models across images and videos, demonstrating the robustness of our method.

Community

Paper submitter
Paper author

Thanks for sharing!
Project Page:https://pbihao.github.io/projects/controlnext/index.html

·

Hi, I noticed that there might be a mistake in Eq. 8 and Eq. 9. \mu_c should be \mu_m and \sigma_c should be \sigma_m.

Thanks for sharing our work! :D

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Sign up or log in to comment

Models citing this paper 6

Browse 6 models citing this paper

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2408.06070 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 16