Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 rednote-hilab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,3 +1,119 @@
----
-license: mit
----
+---
+license: mit
+license_link: https://huggingface.co/rednote-hilab/dots.llm1.inst-FP8-dynamic/blob/main/LICENSE
+pipeline_tag: text-generation
+base_model: rednote-hilab/dots.llm1.inst
+tags:
+- chat
+library_name: transformers
+language:
+- en
+- zh
+---
+
+# dots1
+
+<p align="center">
+    <img src="figures/new_logo2.png" width="300"/>
+<p>
+
+<p align="center">
+    &nbsp&nbsp🤗 <a href="https://huggingface.co/rednote-hilab">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp 📑 <a href="https://www.arxiv.org/abs/2506.05767">Paper</a> &nbsp&nbsp 
+<br>
+🖥️ <a href="https://huggingface.co/spaces/rednote-hilab/dots-demo">Demo</a>&nbsp&nbsp | &nbsp&nbsp💬 <a href="figures/wechat.png">WeChat (微信)</a>&nbsp&nbsp | &nbsp&nbsp📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>&nbsp&nbsp
+</p>
+
+
+Visit our Hugging Face (click links above), search checkpoints with names starting with `dots.llm1` or visit the [dots1 collection](https://huggingface.co/collections/rednote-hilab/dotsllm1-68246aaaaba3363374a8aa7c), and you will find all you need! Enjoy!
+
+
+## News
+
+- 2025.06.06: We released the `dots.llm1` series. Check our [report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf) for more details!
+
+
+## 1. Introduction
+
+
+The `dots.llm1` model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models. 
+Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B after pretrained on high-quality corpus without synthetic data. To foster further research, we open-source intermediate training checkpoints spanning the entire training process, providing valuable insights into the learning dynamics of large language models.
+
+
+<p align="center">
+  <img width="90%" src="./figures/performance.png">
+</p>
+
+## 2. Model Summary
+
+**This repo contains the base and instruction-tuned `dots.llm1` model**. which has the following features:
+
+- Type: A MoE model with 14B activated and 142B total parameters trained on high-quality corpus.
+- Training Stages: Pretraining and SFT.
+- Architecture: Multi-head Attention with QK-Norm in attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.
+- Number of Layers: 62
+- Number of Attention Heads: 32
+- Supported Languages: English, Chinese
+- Context Length: 32,768 tokens
+- License: MIT
+
+The highlights from `dots.llm1` include:
+
+- **Enhanced Data Processing**: We propose a scalable and fine-grained *three-stage* data processing framework designed to generate large-scale, high-quality and diverse data for pretraining.
+- **No Synthetic Data during Pretraining**: High-quality non-synthetic tokens was used in base model pretraining.
+- **Performance and Cost Efficiency**: `dots.llm1` is an open-source model that activates only *14B* parameters at inference, delivering both comprehensive capabilities and high computational efficiency.
+- **Infrastructure**: We introduce an innovative MoE all-to-all communication and computation overlapping recipe based on interleaved 1F1B pipeline scheduling and an efficient grouped GEMM implementation to boost computational efficiency.
+- **Open Accessibility to Model Dynamics**: Intermediate model checkpoints are released spanning the entire training process, facilitating future research into the learning dynamics of large language models.
+
+## 3. dots.llm1.inst.FP8-dynamic 
+
+### Docker (vllm)
+
+We release the quantized `dots.llm1.inst.FP8-dynamic` model, which retains approximately 98% of the original performance after quantization. 
+
+run vllm inference using docker `rednote-hilab/vllm-openai-v0.9.1`. The docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/rednotehilab/dots1/tags), based on the official images.
+
+```bash
+python3 -m vllm.entrypoints.openai.api_server --model rednote-hilab/dots.llm1.inst.FP8-dynamic --tensor-parallel-size 4 --pipeline-parallel-size 1 --trust-remote-code --served-model-name dots1
+```
+
+
+### Inference with huggingface
+
+We are working to merge it into Transformers ([PR #38143](https://github.com/huggingface/transformers/pull/38143)).
+
+#### Chat Completion
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
+
+model_name = "rednote-hilab/dots.llm1.inst-FP8-dynamic"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto")
+
+messages = [
+    {"role": "user", "content": "Write a piece of quicksort code in C++"}
+]
+input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
+outputs = model.generate(input_tensor.to(model.device), max_new_tokens=200)
+
+result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
+print(result)
+```
+
+## Citation
+
+If you find `dots.llm1` is useful or want to use in your projects, please kindly cite our paper:
+
+```
+@misc{huo2025dotsllm1technicalreport,
+      title={dots.llm1 Technical Report}, 
+      author={Bi Huo and Bin Tu and Cheng Qin and Da Zheng and Debing Zhang and Dongjie Zhang and En Li and Fu Guo and Jian Yao and Jie Lou and Junfeng Tian and Li Hu and Ran Zhu and Shengdong Chen and Shuo Liu and Su Guang and Te Wo and Weijun Zhang and Xiaoming Shi and Xinxin Peng and Xing Wu and Yawen Liu and Yuqiu Ji and Ze Wen and Zhenhai Liu and Zichao Li and Zilong Liao},
+      year={2025},
+      eprint={2506.05767},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2506.05767}, 
+}
+```

added_tokens.json

@@ -0,0 +1,18 @@
+{
+  "<|endofexecution|>": 151655,
+  "<|endofobservation|>": 151653,
+  "<|endofresponse|>": 151649,
+  "<|endofsystem|>": 151651,
+  "<|endoftext|>": 151643,
+  "<|endofuserprompt|>": 151647,
+  "<|execution|>": 151654,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|observation|>": 151652,
+  "<|reject-unknown|>": 151656,
+  "<|response|>": 151648,
+  "<|sec-cot|>": 151657,
+  "<|sec-end-cot|>": 151658,
+  "<|system|>": 151650,
+  "<|userprompt|>": 151646
+}

chat_template.jinja

@@ -0,0 +1 @@
+{% if messages[0]['role'] == 'system' %}<|system|>{{ messages[0]['content'] }}<|endofsystem|>{% set start_idx = 1 %}{% else %}<|system|>You are a helpful assistant.<|endofsystem|>{% set start_idx = 0 %}{% endif %}{% for idx in range(start_idx, messages|length) %}{% if messages[idx]['role'] == 'user' %}<|userprompt|>{{ messages[idx]['content'] }}<|endofuserprompt|>{% elif messages[idx]['role'] == 'assistant' %}<|response|>{{ messages[idx]['content'] }}<|endofresponse|>{% endif %}{% endfor %}{% if add_generation_prompt and messages[-1]['role'] == 'user' %}<|response|>{% endif %}

config.json

@@ -0,0 +1,83 @@
+{
+  "architectures": [
+    "Dots1ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "eos_token_id": 151645,
+  "first_k_dense_replace": 1,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 10944,
+  "max_position_embeddings": 32768,
+  "max_window_layers": 62,
+  "model_type": "dots1",
+  "moe_intermediate_size": 1408,
+  "moe_layer_freq": 1,
+  "n_group": 1,
+  "n_routed_experts": 128,
+  "n_shared_experts": 2,
+  "norm_topk_prob": true,
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 62,
+  "num_key_value_heads": 32,
+  "pretraining_tp": 1,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "strategy": "token",
+          "symmetric": true,
+          "type": "float"
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "minmax",
+          "observer_kwargs": {},
+          "strategy": "channel",
+          "symmetric": true,
+          "type": "float"
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000000,
+  "routed_scaling_factor": 2.5,
+  "scoring_func": "sigmoid",
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "topk_method": "noaux_tc",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.0.dev0",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}

generation_config.json

@@ -0,0 +1,12 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151643,
+    151649
+  ],
+  "temperature": 0.7,
+  "top_p": 0.8,
+  "transformers_version": "4.53.0.dev0"
+}

model-00001-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ab1f1fc578ea20548e29a5f1ff6932256a72659026c6e4919fefa0eaca70621
+size 4996875472

model-00002-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce4b176ee00241f9572c364a56af1e7fa314bd8b45d3e09952dce00036d972ac
+size 4994592496

model-00003-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a05e21bef9f58317374a171679d3043a557adefd62fb091c8e3379b170105829
+size 4994592544

model-00004-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d9099b5df18510058851059e3b74d8583ad723c4da51123bbff22245470e728b
+size 4995652432

model-00005-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c741564d9251a64263fe0fb30e7330d9a966699bd50c2bd260757d7d858c08b2
+size 4998222344

model-00006-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9b97849b6d3ebc0370f72c8b4d03ff7c6606e8af1db44a222ec923a10fd9c0de
+size 4994604952

model-00007-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef12b060e0e56790fa8b31f5fd01e36097b54cd82fb70d34690a7ff4f2057a1c
+size 4994594192

model-00008-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d12900e2778b1340b109fa44c750cb582af52b6bdc1be31dfd6828adbb330c2
+size 4994594192

model-00009-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ddfa7d05519635cf297171ae1394fccbbd8855f80fae7be07cafb3c4aa0073d
+size 4994604952

model-00010-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abcd9f8ff2d57475d032afd00bd86248064e660724c40a256b53624231e69421
+size 4994594304

model-00011-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3e0422c6bae38ccf3f0b6d4f3324aca9781db2f04a87ec12897b1c1bb916467c
+size 4999283072

model-00012-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9876344fe8335b47579a3e7ec29cf5fc5cb7b9c4f4792859c9da063a030171b9
+size 4994594152

model-00013-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e2b765c56c5d7b4aba5982dc9dc9e01f9bb5ab6cf5dd5b7811f8cbc5219b5982
+size 4994594192

model-00014-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad594c5d811737d4c30462eec90ed886cf5620cde5cbbcaa5eec11ea6883b294
+size 4994604952

model-00015-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7f691deec14d78ea8fdee35f30d10928fb78e34372fb5d20df7a76180394d61
+size 4994594192

model-00016-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:aa3f03a37da6525e616df47d49f9b7bc311bc4a00f670f98dd64a1a4a21184ad
+size 4994594224

model-00017-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:832b011f298ba71f113aa515f7ecc7969bc3b1ae0730114f9bdec3b7252c26f6
+size 4994605080

model-00018-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef09dd1f595775dd03ea6760f1e4beba5eed4bcf68bb672c14d5588e7f4d62fc
+size 4999272232

model-00019-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1cbdad6f4d2719ce6488f472383905fabcde7748166f4f7f2091ab115129c47
+size 4994604952

model-00020-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2946e4f4e962fd18ea2c69ed72bbbbbcd968990b8eccad63c73b368879996151
+size 4994594192

model-00021-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:384e894a3e16bb96ea578df909338538109e2403bdfff2b09058cdba57698a70
+size 4994594192

model-00022-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fc87eda93850f44dab1ffbea4167f8d1feac577b9d1b06e3fa216e0ba20f5c4a
+size 4994604952

model-00023-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d596d4066330d4f55260ef5b79b1c5b4bc6cfa30693364ce0ca7cc6d310f633d
+size 4994594280

model-00024-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e5bc96b7a71ac1e92a0e62f14b9d346eedeed346bf0b346f6f8e0afdd442345e
+size 4999283112

model-00025-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:90c2c4f20a32c79542384c4c2a134ad557ffd887169d26e73fe3ec301b071fb4
+size 4994594136

model-00026-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ba89ef52e3ec217842ad83bea244455bb0a8769d05ff588b357af697b10a1514
+size 4994594192

model-00027-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bee99c440239515e9f7db4b72de0cc88349ae3c68452b8463f99bb22dcdf3f30
+size 4994604952

model-00028-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:505152e94e861cf3207ddcbff5cf86c85e18e92b74d68c2be3c6256a089a891f
+size 4994594192

model-00029-of-00029.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:933ecc46966f99923332dd4f2b477292300b5e6e49f86d4ec217b315f0a874ff
+size 4411269184

special_tokens_map.json

@@ -0,0 +1,33 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|userprompt|>",
+    "<|endofuserprompt|>",
+    "<|response|>",
+    "<|endofresponse|>",
+    "<|system|>",
+    "<|endofsystem|>",
+    "<|observation|>",
+    "<|endofobservation|>",
+    "<|execution|>",
+    "<|endofexecution|>",
+    "<|reject-unknown|>",
+    "<|sec-cot|>",
+    "<|sec-end-cot|>"
+  ],
+  "eos_token": {
+    "content": "<|endofresponse|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endofresponse|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe750f7b1f42fb8f760ee7ee91fa5fb3974795b3dc14c269819b83a086f5e98d
+size 11420764

tokenizer_config.json

@@ -0,0 +1,160 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|userprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|endofuserprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|response|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|endofresponse|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|endofsystem|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|observation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|endofobservation|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|execution|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|endofexecution|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|reject-unknown|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<|sec-cot|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151658": {
+      "content": "<|sec-end-cot|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|userprompt|>",
+    "<|endofuserprompt|>",
+    "<|response|>",
+    "<|endofresponse|>",
+    "<|system|>",
+    "<|endofsystem|>",
+    "<|observation|>",
+    "<|endofobservation|>",
+    "<|execution|>",
+    "<|endofexecution|>",
+    "<|reject-unknown|>",
+    "<|sec-cot|>",
+    "<|sec-end-cot|>"
+  ],
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endofresponse|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 131072,
+  "pad_token": "<|endofresponse|>",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}