Note 1. The intra-image bidirectional attention is important, and replacing it with causal attention hurts text-to-image generation. 2. There is a clear advantage to using the U-Net up and down blocks instead of a simple linear layer for modality mapping.

Note 1. Extract the feature of token from the last hidden layer of LLM and project to SAM decoder. 2. Joint train with pixel-level understanding data often leads to decreased image-level capability.

Note 1. Image Encoder: a ConvNeXt-L-based CLIP model to reach high resolution. 2. Directly combining a frozen perception module with LLM doesn’t perform well. 3. Use a simple MLP to map the LLM output’s hidden states of the [SEG] token to the visual space. 4. Propose a good Region Encoder Design adapted from a pre-trained Image-Encoder. 5. “ Expression [SEG]." Since the “Expression" is flexible and variable, the LLM is less likely to overfit to a fixed response

Note 1. A Slow pathway extracts features at a low frame rate while keeping as many spatial details as possible. 2. A Fast pathway operates on a high frame rate but uses a larger spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. 3. Concate them together and bang, here we have a good video features even without training.

Note 1. Scale to 512 input video frames with the Token Turning Machine Combiner. 2. The ‘Process’ is implemented with a standard Transformer with layers of MHA and MLPs. The functions ‘Read’, ‘Write’, and ‘Output’ is implemented with Attention Pooling.

Note 1. Training methods: 1.1 progressively higher-quality data, the maximum image resolution gradually increases, and more model parts are unfrozen. 2. Dataset 2.1 Apply image deduplication, it is possible to train on just half of the LAION dataset with only a minimal reduction in performance compared to using the full dataset

Note 1. Unfreezing the CLIP encoder significantly improves when interpolating to a higher MLLM input resolution that differs from the pre-training resolution. 2. Introduce a Pre-Alignment training stage: 2.1 Traini each pre-trained vision expert with their own projector on SFT data, while keeping the language model frozen,

Note 1. Arrange slow and fast frames in an interleaving pattern. p × p pooling and 2p × 2p pooling for slow and fast frames, respectively 2. Use a tagging model to categorize the video content; InsTag (https://arxiv.org/pdf/2308.07074)