dewdev
/

language_detection

Text Classification

Model card Files Files and versions Community

dewdev commited on Feb 26

Commit

86c0047

·

verified ·

1 Parent(s): 7262f79

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +3 -0

tokenizer.json CHANGED Viewed

@@ -52,6 +52,9 @@
   "normalizer": {
     "type": "Sequence",
     "normalizers": [
       {
         "type": "BertNormalizer",
         "clean_text": true,

   "normalizer": {
     "type": "Sequence",
     "normalizers": [
+      {
+        "type": "NFD"
+      },
       {
         "type": "BertNormalizer",
         "clean_text": true,