Hugging Face's logo

chchen
/
Llama-3.1-8B-Instruct-SFT-600

PEFT
Safetensors
llama-factory
lora
Generated from Trainer
Model card Files Community
Llama-3.1-8B-Instruct-SFT-600 / README.md
chchen's picture
chchen
End of training
9279660 verified
preview code
|
raw
history blame contribute delete
1.81 kB
metadata
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
license: llama3.1
tags:
  - llama-factory
  - lora
  - generated_from_trainer
model-index:
  - name: Llama-3.1-8B-Instruct-SFT-600
    results: []

Llama-3.1-8B-Instruct-SFT-600

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the bct_non_cot_sft_600 dataset. It achieves the following results on the evaluation set:

  • Loss: 0.0599

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-06
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 10.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
0.996 1.4815 50 0.7778
0.1326 2.9630 100 0.0880
0.0931 4.4444 150 0.0636
0.0915 5.9259 200 0.0614
0.0888 7.4074 250 0.0600
0.0653 8.8889 300 0.0599

Framework versions

  • PEFT 0.12.0
  • Transformers 4.45.2
  • Pytorch 2.3.0
  • Datasets 2.19.0
  • Tokenizers 0.20.0