Hugging Face's logo

FoundationVision
/
unitok_tokenizer

Feature Extraction
Model card Files Community
1
unitok_tokenizer
File size: 4,002 Bytes 
fe59873
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
 
---
license: apache-2.0
pipeline_tag: feature-extraction
---

# UniTok: A Unified Tokenizer for Visual Generation and Understanding

This repository contains UniTok, a unified visual tokenizer for both image generation and understanding tasks, as presented in [UniTok: A Unified Tokenizer for Visual Generation and Understanding](https://hf.co/papers/2502.20321).

Project Page: https://foundationvision.github.io/UniTok/

Code: https://github.com/FoundationVision/UniTok

![teaser](assets/teaser.png)

UniTok encodes fine-grained details for generation and captures high-level semantics for understanding. It's compatible with autoregressive generative models (e.g., LlamaGen), multimodal understanding models (e.g., LLaVA), and unified MLLMs (e.g., Chameleon and Liquid).


Built upon UniTok, we construct an MLLM capable of both multimodal generation and understanding, which sets a new state-of-the-art among unified autoregressive MLLMs. The weights of our MLLM will be released soon.

![teaser](assets/samples.png)

## Performance

<table>
    <thead>
        <tr>
            <th>Method</th>
            <th>#Tokens</th>
            <th>rFID &darr;</th>
            <th>Accuracy</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td colspan="4"><i>VQVAE Model</i></td>
        </tr>
        <tr align="center">
            <td>VQ-GAN</td>
            <td>256</td>
            <td>4.98</td>
            <td>--</td>
        </tr>
        <tr align="center">
            <td>RQ-VAE</td>
            <td>256</td>
            <td>1.30</td>
            <td>--</td>
        </tr>
        <tr align="center">
            <td>VAR</td>
            <td>680</td>
            <td>0.90</td>
            <td>--</td>
        </tr>
        <tr>
            <td colspan="4"><i>CLIP Model</i></td>
        </tr>
        <tr align="center">
            <td>CLIP</td>
            <td>256</td>
            <td>--</td>
            <td>76.2</td>
        </tr>
        <tr align="center">
            <td>SigLIP</td>
            <td>256</td>
            <td>--</td>
            <td>80.5</td>
        </tr>
        <tr align="center">
            <td>ViTamin</td>
            <td>256</td>
            <td>--</td>
            <td>81.2</td>
        </tr>
        <tr>
            <td colspan="4"><i>Unified Model</i></td>
        </tr>
        <tr align="center">
            <td>TokenFlow &dagger;</td>
            <td>680</td>
            <td>1.37</td>
            <td>--</td>
        </tr>
        <tr align="center">
            <td>VILA-U &dagger;</td>
            <td>256</td>
            <td>1.80</td>
            <td>73.3</td>
        </tr>
        <tr align="center">
            <td>UniTok</td>
            <td>256</td>
            <td>0.39</td>
            <td>70.5</td>
        </tr>
        <tr align="center">
            <td>UniTok &dagger;</td>
            <td>256</td>
            <td>0.38</td>
            <td>78.6</td>
        </tr>
    </tbody>
</table>


&dagger; indicates the model uses pretrained CLIP weights for initialization. Although CLIP weight initialization boosts ImageNet zero-shot accuracy,
we notice that random initialization leads to better downstream understanding performance.
We thus release the model checkpoint of UniTok that is trained from scratch.



## Model Weights

|    Model     | Res. | #Token |        Code Shape         | rFID |  Checkpoint  |
|:------------:|:----:|:------:|:-------------------------:|:----:|:------------:|
| UniTok-Large | 256  |  256   | 16 $\times$ 16 $\times$ 8 | 0.39 | [Download](https://huggingface.co/FoundationVision/UniTok/blob/main/unitok_tokenizer.pth) |


## Usage

(... rest of README content ...)

## Citation

```bibtex
@article{unitok,
  title={UniTok: A Unified Tokenizer for Visual Generation and Understanding},
  author={Ma, Chuofan and Jiang, Yi and Wu, Junfeng and Yang, Jihan and Yu, Xin and Yuan, Zehuan and Peng, Bingyue and Qi, Xiaojuan},
  journal={arXiv preprint arXiv:2502.20321},
  year={2025}
}
```