Upload model

.gitattributes

@@ -8,6 +8,7 @@
 *.h5 filter=lfs diff=lfs merge=lfs -text
 *.joblib filter=lfs diff=lfs merge=lfs -text
 *.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.lz4 filter=lfs diff=lfs merge=lfs -text
 *.mlmodel filter=lfs diff=lfs merge=lfs -text
 *.model filter=lfs diff=lfs merge=lfs -text
 *.msgpack filter=lfs diff=lfs merge=lfs -text
@@ -33,3 +34,23 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+# Audio files - uncompressed
+*.pcm filter=lfs diff=lfs merge=lfs -text
+*.sam filter=lfs diff=lfs merge=lfs -text
+*.raw filter=lfs diff=lfs merge=lfs -text
+# Audio files - compressed
+*.aac filter=lfs diff=lfs merge=lfs -text
+*.flac filter=lfs diff=lfs merge=lfs -text
+*.mp3 filter=lfs diff=lfs merge=lfs -text
+*.ogg filter=lfs diff=lfs merge=lfs -text
+*.wav filter=lfs diff=lfs merge=lfs -text
+# Image files - uncompressed
+*.bmp filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.tiff filter=lfs diff=lfs merge=lfs -text
+# Image files - compressed
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.webp filter=lfs diff=lfs merge=lfs -text
+

README.md

@@ -1,3 +1,215 @@
 ---
 license: apache-2.0
+datasets:
+- projecte-aina/CA-ZH_Parallel_Corpus
+language:
+- zh
+- ca
+base_model:
+- facebook/m2m100_1.2B
 ---
+## Projecte Aina’s Catalan-Chinese machine translation model
+
+## Table of Contents
+<details>
+<summary>Click to expand</summary>
+
+- [Model description](#model-description)
+- [Intended uses and limitations](#intended-uses-and-limitations)
+- [How to use](#how-to-use)
+- [Limitations and bias](#limitations-and-bias)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Additional information](#additional-information)
+
+</details>
+
+ 
+## Model description
+
+This machine translation model is built upon the M2M100 1.2B, fine-tuned specifically for Catalan-Chinese translation. 
+It is trained on a combination of Catalan-Chinese datasets 
+totalling 94,187,858 sentence pairs. 113,305 sentence pairs were parallel data collected from the web, while the remaining 94,074,553 sentence pairs 
+were parallel synthetic data created using the 
+[Aina Project's Spanish-Catalan machine translation model](https://huggingface.co/projecte-aina/aina-translator-es-ca) and the [Aina Project's English-Catalan machine translation model](https://huggingface.co/projecte-aina/aina-translator-en-ca). 
+
+Following the fine-tuning phase, Contrastive Preference Optimization (CPO) was applied to further refine the model's outputs. CPO training involved pairs of "chosen" and "rejected" translations for a total of 4,006 sentences. These sentences were sourced from the Flores development set (997 sentences), the Flores devtest set (1,012 sentences), and the NTREX set (1,997 sentences).
+
+The model was evaluated on the Projecte Aina's Catalan-Chinese evaluation dataset, which contains 1022 sentences. 
+
+## Intended uses and limitations
+
+You can use this model for machine translation from Catalan to simplified Chinese.
+
+## How to use
+
+### Usage
+
+Translate a sentence using python
+```python
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+
+model_id = "projecte-aina/aina-translator-ca-zh-v2"
+
+model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+sentence = "Benvingut al projecte Aina!"
+
+input_ids = tokenizer(sentence, return_tensors="pt").input_ids
+output_ids = model.generate(input_ids, max_length=200, num_beams=5)
+
+generated_translation= tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()
+print(generated_translation)
+#欢迎来到 Aina 项目！
+```
+
+
+## Limitations and bias
+At the time of submission, no measures have been taken to estimate the bias and toxicity embedded in the model. 
+However, we are well aware that our models may be biased. We intend to conduct research in these areas in the future, and if completed, this model card will be updated. 
+
+## Training
+
+### Training data
+
+The Catalan-Chinese data collected from the web was a combination of the following datasets:
+
+| Dataset       	| Sentences before cleaning	|
+|-------------------|----------------|
+| OpenSubtitles  	| 139.300	|
+| WikiMatrix | 90.643 |
+| Wikipedia 	| 68.623|
+| **Total**     	| **298.566** |
+
+94.074.553 sentence pairs of synthetic parallel data were created from the following Spanish-Chinese datasets and English-Chinese datasets:
+
+**Spanish-Chinese:**
+
+| Dataset       	| Sentences before cleaning	|
+|-------------------|----------------|
+| NLLB 	|24.051.233|
+| UNPC | 17.599.223 |
+| MultiUN | 9.847.770 |
+| OpenSubtitles | 9.319.658 |
+| MultiParaCrawl | 3.410.087 |
+| MultiCCAligned | 3.006.694 |
+| WikiMatrix | 1.214.322 |
+| News Commentary | 375.982 |
+| Tatoeba | 9.404 |
+| **Total**     	| **68.834.373** |
+
+**English-Chinese:**
+
+| Dataset       	| Sentences before cleaning	|
+|-------------------|----------------|
+| NLLB 	|71.383.325|
+| CCAligned | 15.181.415 |
+| Paracrawl | 14.170.869|
+| WikiMatrix | 2.595.119|
+| **Total**     	| **103.330.728** |
+
+
+
+### Training procedure
+
+### Data preparation
+
+**Catalan-Chinese parallel data**
+
+The Chinese side of all datasets were first processed using the [Hanzi Identifier](https://github.com/tsroten/hanzidentifier) to detect Traditional Chinese, which was subsequently converted to Simplified Chinese using [OpenCC](https://github.com/BYVoid/OpenCC).
+
+All data was then filtered according to two specific criteria:
+
+- Alignment: sentence level alignments were calculated using [LaBSE](https://huggingface.co/sentence-transformers/LaBSE) and sentence pairs with a score below 0.75 were discarded.
+
+- Language identification: the probability of being the target language was calculated using [Lingua.py](https://github.com/pemistahl/lingua-py) and sentences with a language probability score below 0.5 were discarded.
+
+Next, Spanish data was translated into Catalan using the Aina Project's [Spanish-Catalan machine translation model](https://huggingface.co/projecte-aina/aina-translator-es-ca), while English data was translated into Catalan using the Aina Project's [English-Catalan machine translation model](https://huggingface.co/projecte-aina/aina-translator-en-ca).
+
+The filtered and translated datasets are then concatenated and deduplicated to form a final corpus of 94.187.858.
+
+**Catalan-Chinese Contrastive Preference Optimization dataset**
+
+The CPO dataset is built by comparing the quality of translations across four distinct sources:
+
+- Reference translation: Chinese sentences from Flores test set, Flores devtest set, and NTREX dataset.
+- [aina-translator-ca-zh](https://huggingface.co/projecte-aina/aina-translator-ca-zh): A specialized bilingual model for Catalan-Chinese translations.
+- Google Translate: A widely-used general-purpose machine translation system.
+- OpenAI GPT-4: A large-scale language model capable of performing a wide range of tasks in conversational settings, including high-quality translation.
+
+To evaluate the quality of translations without relying on human annotations, we employ two reference-free evaluation models:
+
+- [Unbabel/wmt23-cometkiwi-da-xxl](https://huggingface.co/Unbabel/wmt23-cometkiwi-da-xxl)
+- [Unbabel/XCOMET-XXL](https://huggingface.co/Unbabel/XCOMET-XXL)
+
+These models provide direct assessment scores for each translation. The scores from both models are averaged to determine the relative quality of each translation. Based on this evaluation, the highest-scoring ("chosen") and lowest-scoring ("rejected") translations are identified for each source sentence, forming contrastive pairs. The CPO dataset comprises a total of 4,006 such pairs of "chosen" and "rejected" translations.
+
+
+#### Training
+
+The training was executed on NVIDIA GPUs utilizing the Hugging Face Transformers framework. 
+The model was trained for 245.000 updates.
+
+Following fine-tuning on the M2M100 1.2B model, Contrastive Preference Optimization (CPO) was performed using our CPO dataset and the Hugging Face CPO Trainer. This phase involved 1,500 updates.
+
+## Evaluation
+
+### Variable and metrics
+
+Below are the evaluation results on the Projecte Aina's Catalan-Chinese test set, compared to Google Translate for the CA-ZH direction. The evaluation was conducted using [`tower-eval`](https://github.com/deep-spin/tower-eval) following the standard setting (beam search with beam size 5, limiting the translation length to 200 tokens). We report the following metrics:
+
+- BLEU: Sacrebleu implementation, version:2.4.0
+- ChrF: Sacrebleu implementation.
+- Comet: Model checkpoint: "Unbabel/wmt22-comet-da".
+- Comet-kiwi: Model checkpoint: "Unbabel/wmt22-cometkiwi-da".
+
+
+### Evaluation results
+
+Below are the evaluation results on the machine translation from Chinese to Catalan compared to [Google Translate](https://translate.google.com/):
+
+
+#### Projecte Aina's Catalan-Chinese evaluation dataset
+
+|              |   Bleu ↑  |   ChrF ↑ |   Comet ↑ |   Comet-kiwi ↑ |
+|:-----------------------|-------:|------:|-------:|--------:|-------------:|---------:|
+| aina-translator-zh-ca-v2 |  **28.55** |  **57.64** |    **0.87** |         **0.82** |   
+| Google Translate         |  26.84     |   55.7     |    0.86     |         **0.82** | 
+
+
+
+## Additional information
+
+### Author
+The Language Technologies Unit from Barcelona Supercomputing Center.
+
+### Contact
+For further information, please send an email to <[email protected]>.
+
+### Copyright
+Copyright(c) 2023 by Language Technologies Unit, Barcelona Supercomputing Center.
+
+### License
+[Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+### Funding
+This work has been promoted and financed by the Generalitat de Catalunya through the [Aina project](https://projecteaina.cat/).
+
+### Disclaimer
+
+<details>
+<summary>Click to expand</summary>
+
+The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0. 
+
+Be aware that the model may have biases and/or any other undesirable distortions.
+
+When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it) 
+or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and, 
+in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
+
+In no event shall the owner and creator of the model (Barcelona Supercomputing Center) 
+be liable for any results arising from the use made by third parties.
+
+</details>

added_tokens.json

@@ -0,0 +1,112 @@
+{
+  "__af__": 128004,
+  "__am__": 128005,
+  "__ar__": 128006,
+  "__ast__": 128007,
+  "__az__": 128008,
+  "__ba__": 128009,
+  "__be__": 128010,
+  "__bg__": 128011,
+  "__bn__": 128012,
+  "__br__": 128013,
+  "__bs__": 128014,
+  "__ca__": 128015,
+  "__ceb__": 128016,
+  "__cs__": 128017,
+  "__cy__": 128018,
+  "__da__": 128019,
+  "__de__": 128020,
+  "__el__": 128021,
+  "__en__": 128022,
+  "__es__": 128023,
+  "__et__": 128024,
+  "__fa__": 128025,
+  "__ff__": 128026,
+  "__fi__": 128027,
+  "__fr__": 128028,
+  "__fy__": 128029,
+  "__ga__": 128030,
+  "__gd__": 128031,
+  "__gl__": 128032,
+  "__gu__": 128033,
+  "__ha__": 128034,
+  "__he__": 128035,
+  "__hi__": 128036,
+  "__hr__": 128037,
+  "__ht__": 128038,
+  "__hu__": 128039,
+  "__hy__": 128040,
+  "__id__": 128041,
+  "__ig__": 128042,
+  "__ilo__": 128043,
+  "__is__": 128044,
+  "__it__": 128045,
+  "__ja__": 128046,
+  "__jv__": 128047,
+  "__ka__": 128048,
+  "__kk__": 128049,
+  "__km__": 128050,
+  "__kn__": 128051,
+  "__ko__": 128052,
+  "__lb__": 128053,
+  "__lg__": 128054,
+  "__ln__": 128055,
+  "__lo__": 128056,
+  "__lt__": 128057,
+  "__lv__": 128058,
+  "__mg__": 128059,
+  "__mk__": 128060,
+  "__ml__": 128061,
+  "__mn__": 128062,
+  "__mr__": 128063,
+  "__ms__": 128064,
+  "__my__": 128065,
+  "__ne__": 128066,
+  "__nl__": 128067,
+  "__no__": 128068,
+  "__ns__": 128069,
+  "__oc__": 128070,
+  "__or__": 128071,
+  "__pa__": 128072,
+  "__pl__": 128073,
+  "__ps__": 128074,
+  "__pt__": 128075,
+  "__ro__": 128076,
+  "__ru__": 128077,
+  "__sd__": 128078,
+  "__si__": 128079,
+  "__sk__": 128080,
+  "__sl__": 128081,
+  "__so__": 128082,
+  "__sq__": 128083,
+  "__sr__": 128084,
+  "__ss__": 128085,
+  "__su__": 128086,
+  "__sv__": 128087,
+  "__sw__": 128088,
+  "__ta__": 128089,
+  "__th__": 128090,
+  "__tl__": 128091,
+  "__tn__": 128092,
+  "__tr__": 128093,
+  "__uk__": 128094,
+  "__ur__": 128095,
+  "__uz__": 128096,
+  "__vi__": 128097,
+  "__wo__": 128098,
+  "__xh__": 128099,
+  "__yi__": 128100,
+  "__yo__": 128101,
+  "__zh__": 128102,
+  "__zu__": 128103,
+  "……": 128113,
+  "！": 128108,
+  "（": 128109,
+  "）": 128110,
+  "，": 128104,
+  "：": 128106,
+  "；": 128105,
+  "？": 128107,
+  "［": 128112,
+  "］": 128111
+}

config.json

@@ -0,0 +1,37 @@
+{
+  "_name_or_path": "/gpfs/scratch/bsc88/bsc088657/m2m/checkpoints/checkpoint-245000",
+  "activation_dropout": 0.0,
+  "activation_function": "relu",
+  "architectures": [
+    "M2M100ForConditionalGeneration"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": 0,
+  "d_model": 1024,
+  "decoder_attention_heads": 16,
+  "decoder_ffn_dim": 8192,
+  "decoder_layerdrop": 0.05,
+  "decoder_layers": 24,
+  "decoder_start_token_id": 2,
+  "dropout": 0.1,
+  "early_stopping": true,
+  "encoder_attention_heads": 16,
+  "encoder_ffn_dim": 8192,
+  "encoder_layerdrop": 0.05,
+  "encoder_layers": 24,
+  "eos_token_id": 2,
+  "gradient_checkpointing": false,
+  "init_std": 0.02,
+  "is_encoder_decoder": true,
+  "max_length": 200,
+  "max_position_embeddings": 1024,
+  "model_type": "m2m_100",
+  "num_beams": 5,
+  "num_hidden_layers": 24,
+  "pad_token_id": 1,
+  "scale_embedding": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.42.3",
+  "use_cache": true,
+  "vocab_size": 128114
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "decoder_start_token_id": 2,
+  "early_stopping": true,
+  "eos_token_id": 2,
+  "max_length": 200,
+  "num_beams": 5,
+  "pad_token_id": 1,
+  "transformers_version": "4.42.3"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e34fa98252340a82379911956474ff7787045ab668ab8b0d4b9e4f037a29bb5
+size 4958009000

sentencepiece.bpe.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d8f7c76ed2a5e0822be39f0a4f95a55eb19c78f4593ce609e2edbc2aea4d380a
+size 2423393

special_tokens_map.json

@@ -0,0 +1,139 @@
+{
+  "additional_special_tokens": [
+    "__af__",
+    "__am__",
+    "__ar__",
+    "__ast__",
+    "__az__",
+    "__ba__",
+    "__be__",
+    "__bg__",
+    "__bn__",
+    "__br__",
+    "__bs__",
+    "__ca__",
+    "__ceb__",
+    "__cs__",
+    "__cy__",
+    "__da__",
+    "__de__",
+    "__el__",
+    "__en__",
+    "__es__",
+    "__et__",
+    "__fa__",
+    "__ff__",
+    "__fi__",
+    "__fr__",
+    "__fy__",
+    "__ga__",
+    "__gd__",
+    "__gl__",
+    "__gu__",
+    "__ha__",
+    "__he__",
+    "__hi__",
+    "__hr__",
+    "__ht__",
+    "__hu__",
+    "__hy__",
+    "__id__",
+    "__ig__",
+    "__ilo__",
+    "__is__",
+    "__it__",
+    "__ja__",
+    "__jv__",
+    "__ka__",
+    "__kk__",
+    "__km__",
+    "__kn__",
+    "__ko__",
+    "__lb__",
+    "__lg__",
+    "__ln__",
+    "__lo__",
+    "__lt__",
+    "__lv__",
+    "__mg__",
+    "__mk__",
+    "__ml__",
+    "__mn__",
+    "__mr__",
+    "__ms__",
+    "__my__",
+    "__ne__",
+    "__nl__",
+    "__no__",
+    "__ns__",
+    "__oc__",
+    "__or__",
+    "__pa__",
+    "__pl__",
+    "__ps__",
+    "__pt__",
+    "__ro__",
+    "__ru__",
+    "__sd__",
+    "__si__",
+    "__sk__",
+    "__sl__",
+    "__so__",
+    "__sq__",
+    "__sr__",
+    "__ss__",
+    "__su__",
+    "__sv__",
+    "__sw__",
+    "__ta__",
+    "__th__",
+    "__tl__",
+    "__tn__",
+    "__tr__",
+    "__uk__",
+    "__ur__",
+    "__uz__",
+    "__vi__",
+    "__wo__",
+    "__xh__",
+    "__yi__",
+    "__yo__",
+    "__zh__",
+    "__zu__"
+  ],
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,1032 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128004": {
+      "content": "__af__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128005": {
+      "content": "__am__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128006": {
+      "content": "__ar__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128007": {
+      "content": "__ast__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128008": {
+      "content": "__az__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128009": {
+      "content": "__ba__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128010": {
+      "content": "__be__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128011": {
+      "content": "__bg__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128012": {
+      "content": "__bn__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128013": {
+      "content": "__br__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128014": {
+      "content": "__bs__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128015": {
+      "content": "__ca__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128016": {
+      "content": "__ceb__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128017": {
+      "content": "__cs__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128018": {
+      "content": "__cy__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128019": {
+      "content": "__da__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128020": {
+      "content": "__de__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128021": {
+      "content": "__el__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128022": {
+      "content": "__en__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128023": {
+      "content": "__es__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128024": {
+      "content": "__et__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128025": {
+      "content": "__fa__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128026": {
+      "content": "__ff__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128027": {
+      "content": "__fi__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128028": {
+      "content": "__fr__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128029": {
+      "content": "__fy__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128030": {
+      "content": "__ga__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128031": {
+      "content": "__gd__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128032": {
+      "content": "__gl__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128033": {
+      "content": "__gu__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128034": {
+      "content": "__ha__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128035": {
+      "content": "__he__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128036": {
+      "content": "__hi__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128037": {
+      "content": "__hr__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128038": {
+      "content": "__ht__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128039": {
+      "content": "__hu__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128040": {
+      "content": "__hy__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128041": {
+      "content": "__id__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128042": {
+      "content": "__ig__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128043": {
+      "content": "__ilo__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128044": {
+      "content": "__is__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128045": {
+      "content": "__it__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128046": {
+      "content": "__ja__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128047": {
+      "content": "__jv__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128048": {
+      "content": "__ka__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128049": {
+      "content": "__kk__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128050": {
+      "content": "__km__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128051": {
+      "content": "__kn__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128052": {
+      "content": "__ko__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128053": {
+      "content": "__lb__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128054": {
+      "content": "__lg__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128055": {
+      "content": "__ln__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128056": {
+      "content": "__lo__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128057": {
+      "content": "__lt__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128058": {
+      "content": "__lv__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128059": {
+      "content": "__mg__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128060": {
+      "content": "__mk__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128061": {
+      "content": "__ml__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128062": {
+      "content": "__mn__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128063": {
+      "content": "__mr__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128064": {
+      "content": "__ms__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128065": {
+      "content": "__my__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128066": {
+      "content": "__ne__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128067": {
+      "content": "__nl__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128068": {
+      "content": "__no__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128069": {
+      "content": "__ns__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128070": {
+      "content": "__oc__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128071": {
+      "content": "__or__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128072": {
+      "content": "__pa__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128073": {
+      "content": "__pl__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128074": {
+      "content": "__ps__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128075": {
+      "content": "__pt__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128076": {
+      "content": "__ro__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128077": {
+      "content": "__ru__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128078": {
+      "content": "__sd__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128079": {
+      "content": "__si__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128080": {
+      "content": "__sk__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128081": {
+      "content": "__sl__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128082": {
+      "content": "__so__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128083": {
+      "content": "__sq__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128084": {
+      "content": "__sr__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128085": {
+      "content": "__ss__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128086": {
+      "content": "__su__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128087": {
+      "content": "__sv__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128088": {
+      "content": "__sw__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128089": {
+      "content": "__ta__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128090": {
+      "content": "__th__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128091": {
+      "content": "__tl__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128092": {
+      "content": "__tn__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128093": {
+      "content": "__tr__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128094": {
+      "content": "__uk__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128095": {
+      "content": "__ur__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128096": {
+      "content": "__uz__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128097": {
+      "content": "__vi__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128098": {
+      "content": "__wo__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128099": {
+      "content": "__xh__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128100": {
+      "content": "__yi__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128101": {
+      "content": "__yo__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128102": {
+      "content": "__zh__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128103": {
+      "content": "__zu__",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "128104": {
+      "content": "，",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128105": {
+      "content": "；",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128106": {
+      "content": "：",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128107": {
+      "content": "？",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128108": {
+      "content": "！",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128109": {
+      "content": "（",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128110": {
+      "content": "）",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128111": {
+      "content": "］",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128112": {
+      "content": "［",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "128113": {
+      "content": "……",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "additional_special_tokens": [
+    "__af__",
+    "__am__",
+    "__ar__",
+    "__ast__",
+    "__az__",
+    "__ba__",
+    "__be__",
+    "__bg__",
+    "__bn__",
+    "__br__",
+    "__bs__",
+    "__ca__",
+    "__ceb__",
+    "__cs__",
+    "__cy__",
+    "__da__",
+    "__de__",
+    "__el__",
+    "__en__",
+    "__es__",
+    "__et__",
+    "__fa__",
+    "__ff__",
+    "__fi__",
+    "__fr__",
+    "__fy__",
+    "__ga__",
+    "__gd__",
+    "__gl__",
+    "__gu__",
+    "__ha__",
+    "__he__",
+    "__hi__",
+    "__hr__",
+    "__ht__",
+    "__hu__",
+    "__hy__",
+    "__id__",
+    "__ig__",
+    "__ilo__",
+    "__is__",
+    "__it__",
+    "__ja__",
+    "__jv__",
+    "__ka__",
+    "__kk__",
+    "__km__",
+    "__kn__",
+    "__ko__",
+    "__lb__",
+    "__lg__",
+    "__ln__",
+    "__lo__",
+    "__lt__",
+    "__lv__",
+    "__mg__",
+    "__mk__",
+    "__ml__",
+    "__mn__",
+    "__mr__",
+    "__ms__",
+    "__my__",
+    "__ne__",
+    "__nl__",
+    "__no__",
+    "__ns__",
+    "__oc__",
+    "__or__",
+    "__pa__",
+    "__pl__",
+    "__ps__",
+    "__pt__",
+    "__ro__",
+    "__ru__",
+    "__sd__",
+    "__si__",
+    "__sk__",
+    "__sl__",
+    "__so__",
+    "__sq__",
+    "__sr__",
+    "__ss__",
+    "__su__",
+    "__sv__",
+    "__sw__",
+    "__ta__",
+    "__th__",
+    "__tl__",
+    "__tn__",
+    "__tr__",
+    "__uk__",
+    "__ur__",
+    "__uz__",
+    "__vi__",
+    "__wo__",
+    "__xh__",
+    "__yi__",
+    "__yo__",
+    "__zh__",
+    "__zu__"
+  ],
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "language_codes": "m2m100",
+  "model_max_length": 1000000000000000019884624838656,
+  "num_madeup_words": 8,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "sp_model_kwargs": {},
+  "src_lang": "ca",
+  "tgt_lang": "zh",
+  "tokenizer_class": "M2M100Tokenizer",
+  "unk_token": "<unk>",
+  "use_fast": false
+}