eshachar's picture
End of training
4d8debf verified
metadata
base_model: bigcode/starcoderbase-7b
library_name: peft
license: bigcode-openrail-m
tags:
  - generated_from_trainer
model-index:
  - name: peft-starcoder-lora-cutlass
    results: []

peft-starcoder-lora-cutlass

This model is a fine-tuned version of bigcode/starcoderbase-7b on an unknown dataset. It achieves the following results on the evaluation set:

  • Loss: 0.4322

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0005
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_steps: 30
  • training_steps: 4000

Training results

Training Loss Epoch Step Validation Loss
0.5409 0.025 100 0.4729
0.3672 0.05 200 0.4051
0.3432 0.075 300 0.3876
0.312 0.1 400 0.3548
0.2594 0.125 500 0.3507
0.2821 0.15 600 0.3428
0.1958 0.175 700 0.3391
0.2362 0.2 800 0.3405
0.209 0.225 900 0.3421
0.2114 0.25 1000 0.3481
0.1877 0.275 1100 0.3447
0.1903 0.3 1200 0.3533
0.1723 0.325 1300 0.3578
0.1465 0.35 1400 0.3643
0.1751 0.375 1500 0.3639
0.1213 0.4 1600 0.3749
0.119 0.425 1700 0.3692
0.1199 0.45 1800 0.3820
0.0986 0.475 1900 0.3816
0.1232 0.5 2000 0.3868
0.0819 0.525 2100 0.3923
0.096 0.55 2200 0.3964
0.0873 0.575 2300 0.3977
0.0892 0.6 2400 0.4007
0.0802 0.625 2500 0.4078
0.066 0.65 2600 0.4119
0.0863 0.675 2700 0.4125
0.0722 0.7 2800 0.4189
0.0888 0.725 2900 0.4201
0.0757 0.75 3000 0.4263
0.0666 0.775 3100 0.4234
0.0757 0.8 3200 0.4275
0.053 0.825 3300 0.4289
0.0772 0.85 3400 0.4290
0.0546 0.875 3500 0.4297
0.0601 0.9 3600 0.4311
0.0606 0.925 3700 0.4317
0.0544 0.95 3800 0.4314
0.065 0.975 3900 0.4320
0.0503 1.0 4000 0.4322

Framework versions

  • PEFT 0.11.1
  • Transformers 4.42.4
  • Pytorch 2.4.0a0+07cecf4168.nv24.05
  • Datasets 2.20.0
  • Tokenizers 0.19.1