neuralmagic

/

DeepSeek-R1-Distill-Llama-70B-FP8-dynamic

like 4

Follow

Neural Magic 337

can you please go back to DeepSeek's official chat template that includes <think> to force it to start with a <think> otherwise it often skips thinking, or mixes thinking in with the response

https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama-70B/blob/main/tokenizer_config.json#L34

Thanks @erichartford for pointing this out. Did DeepSeek change the tokenizer? Just want to make sure we understand why it diverged

@erichartford Great catch, thanks a lot for pointing it out! Indeed, DeepSeek updated tokenizer_config.json with this diff 14 days ago, and we forked their model for quantization on the initial release day; therefore, we haven't picked it up. We will update all of our quantized models.

Nice thank you

Note a lot of ui's are dependent on the opening <think> which the model will not longer generate - providers may want to insert a <think> at the beginning of the response