TransDiff--The Simplest AR Transformer + Diffusion Image Generation Method

Hello, I’m excited to introduce our new work—Marrying Autoregressive Transformer and Diffusion with Multi-Reference Autoregression, which we’ll refer to as TransDiff.

TL;DR

First of all, TransDiff is currently the simplest method to combine AR Transformer and Diffusion for image generation. TransDiff encodes discrete inputs (such as categories, text, etc.) and continuous inputs (such as images) through an AR Transformer into image semantic representations, and then decodes the representations into images through a smaller Diffusion Decoder.

Additionally, we propose a new autoregressive paradigm—MRAR (Multi-Reference Autoregression). This paradigm is similar to In-context Learning (ICL) in NLP: by learning from previous images of the same category, better and more diverse images are generated. The only difference is that the previous images are generated by the model itself.

Detailed Introduction

To save the readers' time, we’ve abandoned the traditional paper structure and introduced TransDiff in a more 'conversational' Q&A format. These questions are also the motivation for our research.

Q: Why use Transformer? What information does the AR Transformer encode in our work?

A: Early CLIP work and the subsequent large models in the VL domain have already proven the advantages of Transformers in image understanding. Especially in the CLIP work, the ViT model can align the image representation to a semantic space (the cosine similarity between text BERT representation and image ViT representation).
Similarly, experiments have shown that the AR Transformer in TransDiff also encodes categories and images into the image's high-level (contrastive pixel)semantic space. The following demonstrates the generation of images by randomly concatenating 256-dimensional features of different categories. Unlike other models (VAR, LlamaGen, etc.) that edit pixels, qualitative experiments show the model's semantic editing ability.

Q: Does the use of a smaller Diffusion Decoder in TransDiff have limitations? Is it better than pure Diffusion and AR Transformer methods?

A: TransDiff's decoder uses the DiT structure and follows the Flow Matching paradigm. The diffusion component accounts for one-third of the total parameters, significantly fewer than mainstream diffusion models. However, compared to all available pure Diffusion and AR Transformer methods, TransDiff still shows certain advantages in benchmark tests, at least demonstrating a "back-and-forth" competitive performance.

Q: TransDiff looks like MAR, is it just a simple imitation of MAR?

A: Although TransDiff and MAR are structurally similar, the characteristics exhibited by the models are quite different. First, MAR generates on the pixel (or patch) level without explicit semantic representation. Additionally, MAR uses a very simple Diffusion Decoder (with n layers of MLP) which limits the decoder's expressive power. Therefore, as shown in the figure below: MAR cannot "generate images in one step", and image patches are gradually refined through autoregressive iterations.

Q: What’s good about MRAR? Does it have advantages over the commonly used Token-Level AR and Scale-Level AR in AR Transformers?

A: First, compared to Token-Level AR and Scale-Level AR, TransDiff with MRAR has a significant advantage in benchmarks. Secondly, we found that the higher the diversity of semantic representations, the higher the image quality. MRAR can significantly improve the diversity of semantic representations compared to Scale-Level AR.

Finally, some demos

One More Thing

TransDiff with MRAR has demonstrated the potential for generating continuous frames without training on video data. Therefore, we will also apply TransDiff to the video generation field in the future, so stay tuned.