hotchpotch
/

xlm-roberta-japanese-tokenizer-16k

Model card Files Files and versions Community

xlm-roberta-japanese-tokenizer-16k / README.md

hotchpotch's picture

Update README.md

08b017e verified about 1 month ago

|

history blame contribute delete

167 Bytes

vocab_size が 16384 サイズになるように作成した日本語 tokenizer
学習元は wikpedia (日本語のみ)
unidic + sentencepiece(unigram) で学習