ocisd4
/

llama-2-tokenizer-dataprep

Model card Files Files and versions Community

AlexHung29629 commited on Jul 25, 2023

Commit

96e482f

·

1 Parent(s): aa58c34

Update README.md

Files changed (1) hide show

README.md +2 -1

README.md CHANGED Viewed

	@@ -1 +1,2 @@
1	- 關閉自動添加`<s>`，方便產生megatron-deepspeed訓練用檔案


1	+ - 關閉自動添加`<s>`，方便產生megatron-deepspeed訓練用檔案
2	+ - 指定pad token為`<unk>`，訓練時token數才會正確，以及finetune_t0.py才能正確pack_sample