Jonathan1909 commited on
Commit
59685c3
1 Parent(s): d34f045

Fix example usage

Browse files
Files changed (1) hide show
  1. README.md +9 -0
README.md CHANGED
@@ -21,6 +21,7 @@ The original tokenizer is supposed to be used (i.e. `tokenizer.model` in [GitHub
21
  ```python
22
  import torch
23
  from transformers import AutoModelForCausalLM
 
24
 
25
  torch.set_default_dtype(torch.bfloat16)
26
  model = AutoModelForCausalLM.from_pretrained(
@@ -44,5 +45,13 @@ inputs = {
44
  outputs = model.generate(**inputs)
45
  ```
46
 
 
 
 
 
 
 
 
 
47
 
48
  Note: A multi-GPU machine is required to test the model with the example code (For now, a 8x80G multi-GPU machine is required).
 
21
  ```python
22
  import torch
23
  from transformers import AutoModelForCausalLM
24
+ from sentencepiece import SentencePieceProcessor
25
 
26
  torch.set_default_dtype(torch.bfloat16)
27
  model = AutoModelForCausalLM.from_pretrained(
 
45
  outputs = model.generate(**inputs)
46
  ```
47
 
48
+ You could also use the transformers-compatible version of the tokenizer [Xenova/grok-1-tokenizer](https://huggingface.co/Xenova/grok-1-tokenizer)
49
+ ```python
50
+ from transformers import LlamaTokenizerFast
51
+
52
+ tokenizer = LlamaTokenizerFast.from_pretrained('Xenova/grok-1-tokenizer')
53
+ inputs = tokenizer('hello world')
54
+ ```
55
+
56
 
57
  Note: A multi-GPU machine is required to test the model with the example code (For now, a 8x80G multi-GPU machine is required).