miscovery
/

tokenizer_v2

Model card Files Files and versions Community

mahrnoud commited on May 21

Commit

cd155f9

·

1 Parent(s): 065f052

Update Readme.md file

Files changed (2) hide show

.DS_Store +0 -0
README.md +2 -2

.DS_Store CHANGED Viewed

Binary files a/.DS_Store and b/.DS_Store differ

README.md CHANGED Viewed

@@ -26,10 +26,10 @@ This tokenizer was trained on:
 ```python
 from transformers import AutoTokenizer
-tokenizer = AutoTokenizer.from_pretrained("miscovery/arabic-english-tokenizer")
 # Example usage
-text = "بسم الله الرحمن الرحيم Hello World"
 encoded = tokenizer(text)
 print(encoded)
 ```

 ```python
 from transformers import AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("miscovery/tokenizer")
 # Example usage
+text = "بسم الله الرحمن الرحيم"
 encoded = tokenizer(text)
 print(encoded)
 ```