dewdev
/

language_detection

Text Classification

Model card Files Files and versions Community

dewdev commited on Feb 26

Commit

7a25b89

·

verified ·

1 Parent(s): 623e05a

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +2 -2

tokenizer.json CHANGED Viewed

@@ -53,7 +53,7 @@
     "type": "Sequence",
     "normalizers": [
       {
-        "type": "NFD"
       },
       {
         "type": "BertNormalizer",
@@ -50348,4 +50348,4 @@
       "erg": 50256
     }
   }
-}

     "type": "Sequence",
     "normalizers": [
       {
+        "type": "NFKD"
       },
       {
         "type": "BertNormalizer",
       "erg": 50256
     }
   }
+}