Hugging Face
Models
Datasets
Spaces
Posts
Docs
Enterprise
Pricing
Log In
Sign Up
ocisd4
/
llama-2-tokenizer-dataprep
like
0
Follow
ocisd4
23
Model card
Files
Files and versions
Community
YAML Metadata Warning:
empty or missing yaml metadata in repo card (
https://huggingface.co/docs/hub/model-cards#model-card-metadata
)
關閉自動添加
<s>
,方便產生megatron-deepspeed訓練用檔案
指定pad token為
<unk>
,訓練時token數才會正確,以及finetune_t0.py才能正確pack_sample
Downloads last month
-
Downloads are not tracked for this model.
How to track
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and HF Inference API was unable to determine this model's library.
Collection including
ocisd4/llama-2-tokenizer-dataprep
tokenizers
Collection
12 items
•
Updated
Jun 11, 2024