internlm2-7b / tokenization_internlm2_fast.py

Commit History

fix no white space when using stream_chat with fast tokenizer
3a7c99d

x54-729 commited on

fast tokenizer and stream_chat fix (#2)
4acf7da
verified

RangiLyu x54-729 commited on