change hebrew code from iw (deprecated) to he (#21) d71546d ArthurZ HF staff altryne commited on Jan 31, 2023
Allow single quotes "'" and hyphens "-" (#12) ed97120 ArthurZ HF staff sanchit-gandhi HF staff commited on Nov 28, 2022
Add pad token to tokenizer (#9) 5c551b8 ArthurZ HF staff sanchit-gandhi HF staff commited on Nov 3, 2022