iioSnail
/

ChineseBERT-base

feature-extraction

Model card Files Files and versions Community

iioSnail commited on Jul 9, 2023

Commit

181e9b5

·

1 Parent(s): e795962

Update bert_tokenizer.py

Files changed (1) hide show

bert_tokenizer.py +1 -2

bert_tokenizer.py CHANGED Viewed

@@ -43,8 +43,7 @@ class ChineseBertTokenizer(BertTokenizerFast):
         self.max_length = 512
         download_file('vocab.txt')
-        print("vocab_file:", vocab_file)
-        self.tokenizer = BertWordPieceTokenizer(vocab_file)
         # load pinyin map dict
         download_file('config/pinyin_map.json')

         self.max_length = 512
         download_file('vocab.txt')
+        self.tokenizer = BertWordPieceTokenizer(str(vocab_file))
         # load pinyin map dict
         download_file('config/pinyin_map.json')