hf-internal-testing
/

llama-tokenizer

Model card Files Files and versions Community

Narsil HF staff commited on Apr 4, 2023

Commit

0917162

•

1 Parent(s): 41c492a

Upload tokenizer.json

Files changed (1) hide show

tokenizer.json +3 -3

tokenizer.json CHANGED Viewed

@@ -9,7 +9,7 @@
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
-      "normalized": false,
       "special": true
     },
     {
@@ -27,7 +27,7 @@
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
-      "normalized": false,
       "special": true
     }
   ],
@@ -93382,4 +93382,4 @@
       "▁ ▁"
     ]
   }
-}

       "single_word": false,
       "lstrip": false,
       "rstrip": false,
+      "normalized": true,
       "special": true
     },
     {
       "single_word": false,
       "lstrip": false,
       "rstrip": false,
+      "normalized": true,
       "special": true
     }
   ],
       "▁ ▁"
     ]
   }
+}