xgen-7b-8k-base / tokenization_xgen.py

Commit History

fix issue in get vocab
c2f5b28

tianxie-sf commited on

_convert_id_to_token (#12)
3987e09

tianxie-sf commited on

add pad token and default eos token (#6)
e9a901f

rooa tianxie-sf commited on

.
d2cc612

rooa commited on

.
b9f4adf

rooa commited on

.
04d6513

rooa commited on