Hugging Face's logo

chchen
/
Llama-3.1-8B-Instruct-SFT-900

PEFT
Safetensors
llama-factory
lora
Generated from Trainer
Model card Files Community
Llama-3.1-8B-Instruct-SFT-900 / README.md
chchen's picture
chchen
End of training
e80aaa6 verified
preview code
|
raw
history blame contribute delete
2.02 kB
metadata
base_model: meta-llama/Llama-3.1-8B-Instruct
library_name: peft
license: llama3.1
tags:
  - llama-factory
  - lora
  - generated_from_trainer
model-index:
  - name: Llama-3.1-8B-Instruct-SFT-900
    results: []

Llama-3.1-8B-Instruct-SFT-900

This model is a fine-tuned version of meta-llama/Llama-3.1-8B-Instruct on the bct_non_cot_sft_900 dataset. It achieves the following results on the evaluation set:

  • Loss: 0.1053

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-06
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 16
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 10.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss
1.201 0.9877 50 1.0016
0.1407 1.9753 100 0.1513
0.0885 2.9630 150 0.1082
0.0743 3.9506 200 0.1068
0.0855 4.9383 250 0.1062
0.0571 5.9259 300 0.1058
0.063 6.9136 350 0.1054
0.0597 7.9012 400 0.1057
0.0694 8.8889 450 0.1053
0.0593 9.8765 500 0.1053

Framework versions

  • PEFT 0.12.0
  • Transformers 4.45.2
  • Pytorch 2.3.0
  • Datasets 2.19.0
  • Tokenizers 0.20.0