Hugging Face's logo

tyzhu
/
lmind_nq_train6000_eval6489_v1_qa_Qwen_Qwen1.5-4B_lora2

PEFT
Safetensors
Generated from Trainer
Eval Results
Model card Files Community
lmind_nq_train6000_eval6489_v1_qa_Qwen_Qwen1.5-4B_lora2 / README.md
tyzhu's picture
tyzhu
End of training
22cabb7 verified
preview code
|
raw
history blame contribute delete
3.21 kB
metadata
license: other
base_model: Qwen/Qwen1.5-4B
tags:
  - generated_from_trainer
datasets:
  - tyzhu/lmind_nq_train6000_eval6489_v1_qa
metrics:
  - accuracy
model-index:
  - name: lmind_nq_train6000_eval6489_v1_qa_Qwen_Qwen1.5-4B_lora2
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: tyzhu/lmind_nq_train6000_eval6489_v1_qa
          type: tyzhu/lmind_nq_train6000_eval6489_v1_qa
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.5578974358974359
library_name: peft

lmind_nq_train6000_eval6489_v1_qa_Qwen_Qwen1.5-4B_lora2

This model is a fine-tuned version of Qwen/Qwen1.5-4B on the tyzhu/lmind_nq_train6000_eval6489_v1_qa dataset. It achieves the following results on the evaluation set:

  • Loss: 2.4726
  • Accuracy: 0.5579

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0001
  • train_batch_size: 1
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • num_devices: 4
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 32
  • total_eval_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: constant
  • lr_scheduler_warmup_ratio: 0.05
  • num_epochs: 20.0

Training results

Training Loss Epoch Step Accuracy Validation Loss
1.7657 0.9973 187 0.5738 1.6215
1.497 2.0 375 0.5742 1.6180
1.2345 2.9973 562 0.5713 1.6951
1.0084 4.0 750 0.5659 1.8059
0.8397 4.9973 937 0.5647 1.9245
0.7186 6.0 1125 0.5614 2.0345
0.6421 6.9973 1312 0.5608 2.1148
0.5968 8.0 1500 0.5585 2.1779
0.5417 8.9973 1687 0.5568 2.2654
0.5356 9.9733 1870 0.5594 2.2527
0.5261 10.9973 2057 2.3376 0.5585
0.5179 12.0 2245 2.3704 0.5595
0.5116 12.9973 2432 2.3617 0.5589
0.5056 14.0 2620 2.4022 0.5581
0.5063 14.9973 2807 2.3861 0.5587
0.4796 16.0 2995 2.3658 0.5585
0.4757 16.9973 3182 2.4195 0.5577
0.4779 18.0 3370 2.4573 0.5573
0.4782 18.9973 3557 2.4896 0.5589
0.4784 19.9733 3740 2.4726 0.5579

Framework versions

  • PEFT 0.5.0
  • Transformers 4.40.2
  • Pytorch 2.3.0
  • Datasets 2.19.1
  • Tokenizers 0.19.1