Spaces:
Sleeping
Sleeping
File size: 135 Bytes
c4eb0c2 |
1 2 3 4 5 6 |
import tiktoken
def tiktoken_len(text):
tokens = tiktoken.encoding_for_model("gpt-3.5-turbo").encode(text)
return len(tokens)
|