athnlp2025_tokenization / requirements.txt
gpantaz
Add application file
79660f1
raw
history blame contribute delete
137 Bytes
gradio>=4.38.1
transformers>4.40.0
sentencepiece
tiktoken
icetk
torch
nltk
boto3
protobuf==4.25.3
ai2-olmo
ipadic
fugashi
datasets
loguru