Changing the call function.
4e98ce2
nioushasadjadi
commited on
Commit History
Change _encode function to encode.
4a303bd
nioushasadjadi
commited on
Adding max_length and padding to tokenizer and encoder.
156a2ea
nioushasadjadi
commited on
Fixing tokenize return bug
3888382
nioushasadjadi
commited on
Fixing tokenizer return type
bcd9e56
nioushasadjadi
commited on
Fixing encoder and tokenize functions.
82681b6
nioushasadjadi
commited on
Adding automap to the tokenizer config
92d46e2
nioushasadjadi
commited on
Download the tokenizer from Hugging Face Hub
eb1e311
nioushasadjadi
commited on
Add tokenizer.py
1c6d85d
nioushasadjadi
commited on