Why is vocab_size larget than tokens?

#6
by wjq521168 - opened

The README.md shows: vocab_size = 32000, but tokens is 65534, i am not sure there's an error here.

Your need to confirm your account before you can post a new comment.

Sign up or log in to comment