Feature Extraction
unitok_tokenizer / README.md
nielsr's picture
nielsr HF staff
Improve model card
fe59873 verified
|
raw
history blame
4 kB
metadata
license: apache-2.0
pipeline_tag: feature-extraction

UniTok: A Unified Tokenizer for Visual Generation and Understanding

This repository contains UniTok, a unified visual tokenizer for both image generation and understanding tasks, as presented in UniTok: A Unified Tokenizer for Visual Generation and Understanding.

Project Page: https://foundationvision.github.io/UniTok/

Code: https://github.com/FoundationVision/UniTok

teaser

UniTok encodes fine-grained details for generation and captures high-level semantics for understanding. It's compatible with autoregressive generative models (e.g., LlamaGen), multimodal understanding models (e.g., LLaVA), and unified MLLMs (e.g., Chameleon and Liquid).

Built upon UniTok, we construct an MLLM capable of both multimodal generation and understanding, which sets a new state-of-the-art among unified autoregressive MLLMs. The weights of our MLLM will be released soon.

teaser

Performance

Method #Tokens rFID ↓ Accuracy
VQVAE Model
VQ-GAN 256 4.98 --
RQ-VAE 256 1.30 --
VAR 680 0.90 --
CLIP Model
CLIP 256 -- 76.2
SigLIP 256 -- 80.5
ViTamin 256 -- 81.2
Unified Model
TokenFlow † 680 1.37 --
VILA-U † 256 1.80 73.3
UniTok 256 0.39 70.5
UniTok † 256 0.38 78.6

† indicates the model uses pretrained CLIP weights for initialization. Although CLIP weight initialization boosts ImageNet zero-shot accuracy, we notice that random initialization leads to better downstream understanding performance. We thus release the model checkpoint of UniTok that is trained from scratch.

Model Weights

Model Res. #Token Code Shape rFID Checkpoint
UniTok-Large 256 256 16 $\times$ 16 $\times$ 8 0.39 Download

Usage

(... rest of README content ...)

Citation

@article{unitok,
  title={UniTok: A Unified Tokenizer for Visual Generation and Understanding},
  author={Ma, Chuofan and Jiang, Yi and Wu, Junfeng and Yang, Jihan and Yu, Xin and Yuan, Zehuan and Peng, Bingyue and Qi, Xiaojuan},
  journal={arXiv preprint arXiv:2502.20321},
  year={2025}
}