Feature Extraction
Transformers
Safetensors
ModularStarEncoder
custom_code
andreagurioli1995 commited on
Commit
a8dac07
·
verified ·
1 Parent(s): 9e7ec4e

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +4 -0
README.md CHANGED
@@ -11,6 +11,10 @@ tags: []
11
 
12
  ## Model Details
13
  Input should take this format when tokenized f"{tokenizer.sep_token}{code_snippet}{tokenizer.cls_token}"
 
 
 
 
14
  ### Model Description
15
 
16
  <!-- Provide a longer summary of what this model is. -->
 
11
 
12
  ## Model Details
13
  Input should take this format when tokenized f"{tokenizer.sep_token}{code_snippet}{tokenizer.cls_token}"
14
+
15
+ You can download the tokenizer following:
16
+ from transformers import AutoTokenizer
17
+ tokenizer = AutoTokenizer.from_pretrained("andreagurioli1995/ModularStarEncoder")
18
  ### Model Description
19
 
20
  <!-- Provide a longer summary of what this model is. -->