Ensure tensor alignment is preserved by adding dummy padding metadata if necessary 37e11f9 verified CISCai commited on Sep 19, 2024
Added new Help section with common use-case examples 73fd84b verified CISCai commited on Sep 13, 2024
Ensure token names are listed for _token_id even when tokens metadata comes afterwards 6ca68af verified CISCai commited on Sep 9, 2024
Implemented edit support for tokenizer.ggml.scores and token_types ad78086 verified CISCai commited on Sep 6, 2024