bigcode

/

santacoder

like 332

Follow

BigCode 1.15k

Here is some sample code that does stop tokens:

https://github.com/arjunguha/BigCode-demos/blob/main/bigcode.ipynb

(See Cell 4, the stop_at_stop_tokens function.)

SantaCoder has an <|endoftext|> token which marks the end/beginning of a file, it is now by default in the tokenizer along with the FIM special tokens