why eos_token_id is different between config.json and generation_config.json

by xjpang85 - opened Nov 30, 2023

Discussion

xjpang85

Nov 30, 2023

In config.json eos_token_id is 2 and in generation_config.json is 7.

Kai01ai

Nov 30, 2023

Duplicate of #5

Annorita

Dec 19, 2023

Can we just modify the bos and eos setting in config.json and tokenizer_config.json and make it align with generation_config.json? If yes, I can make a PR.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment