Are decode and encode in the tokenizer reversed?
#9
by
pootow
- opened
class RWKVWorldTokenizer(PreTrainedTokenizer):
....
self.encoder[idx] = x
self.decoder = {}
for k, v in self.encoder.items():
self.decoder[v] = int(k)
I think encoder should convert strings into ids, and decoder should convert ids back to strings.