Hugging Face's logo

triniaguilar
/
xlm-roberta-peft-biobert-batch-size-16

PEFT
TensorBoard
Safetensors
Generated from Trainer
Model card Files Metrics Community
xlm-roberta-peft-biobert-batch-size-16 / README.md
triniaguilar's picture
triniaguilar
End of training
10b96ca verified
preview code
|
raw
history blame
21.6 kB
metadata
library_name: peft
license: mit
base_model: FacebookAI/xlm-roberta-large
tags:
  - generated_from_trainer
datasets:
  - biobert_json
metrics:
  - precision
  - recall
  - f1
  - accuracy
model-index:
  - name: xlm-roberta-peft-biobert-batch-size-16
    results: []

xlm-roberta-peft-biobert-batch-size-16

This model is a fine-tuned version of FacebookAI/xlm-roberta-large on the biobert_json dataset. It achieves the following results on the evaluation set:

  • Loss: 0.0864
  • Precision: 0.9510
  • Recall: 0.9696
  • F1: 0.9602
  • Accuracy: 0.9812

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0004
  • train_batch_size: 16
  • eval_batch_size: 16
  • seed: 42
  • optimizer: Use paged_adamw_8bit with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • training_steps: 4282
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Precision Recall F1 Accuracy
2.3363 0.0327 20 1.1420 0.9730 0.0063 0.0126 0.6885
0.9396 0.0654 40 0.4876 0.7521 0.7189 0.7351 0.8868
0.4691 0.0980 60 0.2740 0.8289 0.8190 0.8239 0.9238
0.3834 0.1307 80 0.2312 0.7956 0.8816 0.8364 0.9349
0.2974 0.1634 100 0.3061 0.7659 0.9015 0.8282 0.9096
0.2836 0.1961 120 0.2028 0.8521 0.9040 0.8773 0.9449
0.2603 0.2288 140 0.1505 0.8951 0.9228 0.9087 0.9582
0.201 0.2614 160 0.1596 0.8936 0.8894 0.8915 0.9502
0.2219 0.2941 180 0.1522 0.8977 0.9239 0.9106 0.9583
0.199 0.3268 200 0.1571 0.8836 0.9115 0.8973 0.9566
0.1878 0.3595 220 0.1263 0.9048 0.9404 0.9223 0.9651
0.2207 0.3922 240 0.1415 0.8829 0.9594 0.9195 0.9613
0.1687 0.4248 260 0.1414 0.8970 0.9246 0.9106 0.9553
0.1934 0.4575 280 0.1131 0.9186 0.9506 0.9343 0.9695
0.1318 0.4902 300 0.1076 0.9269 0.9533 0.9399 0.9720
0.1523 0.5229 320 0.1116 0.9179 0.9365 0.9271 0.9681
0.1648 0.5556 340 0.1445 0.8873 0.9595 0.9220 0.9608
0.1386 0.5882 360 0.1313 0.8627 0.9127 0.8870 0.9557
0.1411 0.6209 380 0.0999 0.9240 0.9550 0.9392 0.9718
0.1266 0.6536 400 0.1008 0.9207 0.9575 0.9387 0.9717
0.1203 0.6863 420 0.1021 0.9231 0.9604 0.9414 0.9716
0.131 0.7190 440 0.0979 0.9312 0.9445 0.9378 0.9715
0.1242 0.7516 460 0.1121 0.9114 0.9673 0.9386 0.9704
0.1343 0.7843 480 0.0989 0.9257 0.9585 0.9418 0.9725
0.1413 0.8170 500 0.0921 0.9390 0.9568 0.9478 0.9756
0.1044 0.8497 520 0.1186 0.9069 0.9611 0.9332 0.9688
0.1239 0.8824 540 0.1302 0.9119 0.9616 0.9361 0.9653
0.1479 0.9150 560 0.1211 0.9094 0.9607 0.9344 0.9662
0.1513 0.9477 580 0.1021 0.9223 0.9579 0.9397 0.9721
0.1329 0.9804 600 0.1032 0.9356 0.9371 0.9363 0.9722
0.1439 1.0131 620 0.1014 0.9266 0.9656 0.9457 0.9742
0.1118 1.0458 640 0.1143 0.9098 0.9630 0.9357 0.9704
0.0979 1.0784 660 0.0920 0.9408 0.9614 0.9510 0.9768
0.0974 1.1111 680 0.0877 0.9401 0.9577 0.9488 0.9756
0.0932 1.1438 700 0.1008 0.9189 0.9740 0.9456 0.9739
0.1027 1.1765 720 0.1066 0.9220 0.9666 0.9437 0.9711
0.0992 1.2092 740 0.1110 0.9412 0.9457 0.9434 0.9744
0.1096 1.2418 760 0.1052 0.9316 0.9526 0.9420 0.9734
0.1492 1.2745 780 0.0952 0.9393 0.9522 0.9457 0.9748
0.0866 1.3072 800 0.0898 0.9445 0.9595 0.9520 0.9771
0.0975 1.3399 820 0.1042 0.9207 0.9588 0.9393 0.9735
0.1265 1.3725 840 0.0882 0.9319 0.9502 0.9410 0.9750
0.1086 1.4052 860 0.0939 0.9315 0.9619 0.9465 0.9747
0.0985 1.4379 880 0.0967 0.9282 0.9664 0.9469 0.9752
0.1223 1.4706 900 0.0924 0.9298 0.9676 0.9483 0.9746
0.0833 1.5033 920 0.0994 0.9406 0.9215 0.9310 0.9674
0.1043 1.5359 940 0.0958 0.9275 0.9680 0.9473 0.9743
0.137 1.5686 960 0.0831 0.9363 0.9644 0.9502 0.9769
0.1156 1.6013 980 0.0825 0.9434 0.9660 0.9546 0.9784
0.0963 1.6340 1000 0.0940 0.9314 0.9684 0.9495 0.9749
0.0669 1.6667 1020 0.1094 0.9291 0.9693 0.9487 0.9731
0.0984 1.6993 1040 0.0921 0.9331 0.9658 0.9492 0.9761
0.0831 1.7320 1060 0.1035 0.9279 0.9686 0.9478 0.9727
0.1059 1.7647 1080 0.0903 0.9372 0.9643 0.9506 0.9775
0.0971 1.7974 1100 0.1004 0.9304 0.9461 0.9382 0.9721
0.106 1.8301 1120 0.1155 0.9090 0.9734 0.9401 0.9712
0.0894 1.8627 1140 0.0910 0.9340 0.9698 0.9516 0.9773
0.0925 1.8954 1160 0.0949 0.9247 0.9577 0.9409 0.9726
0.0989 1.9281 1180 0.0916 0.9339 0.9683 0.9508 0.9769
0.0987 1.9608 1200 0.0918 0.9360 0.9681 0.9518 0.9768
0.0919 1.9935 1220 0.0948 0.9314 0.9727 0.9516 0.9767
0.0773 2.0261 1240 0.0825 0.9377 0.9622 0.9498 0.9775
0.0846 2.0588 1260 0.0900 0.9331 0.9676 0.9500 0.9769
0.0858 2.0915 1280 0.0886 0.9324 0.9706 0.9511 0.9770
0.067 2.1242 1300 0.0965 0.9246 0.9732 0.9483 0.9742
0.0691 2.1569 1320 0.1031 0.9444 0.9478 0.9461 0.9739
0.1099 2.1895 1340 0.0903 0.9339 0.9625 0.9480 0.9753
0.0613 2.2222 1360 0.0964 0.9322 0.9667 0.9491 0.9764
0.0758 2.2549 1380 0.0976 0.9306 0.9727 0.9512 0.9762
0.096 2.2876 1400 0.0982 0.9357 0.9598 0.9476 0.9747
0.0778 2.3203 1420 0.0800 0.9442 0.9685 0.9562 0.9798
0.0546 2.3529 1440 0.0838 0.9456 0.9655 0.9554 0.9796
0.0858 2.3856 1460 0.0979 0.9422 0.9582 0.9502 0.9746
0.0809 2.4183 1480 0.0945 0.9303 0.9675 0.9485 0.9739
0.1069 2.4510 1500 0.1209 0.8963 0.9704 0.9319 0.9669
0.0935 2.4837 1520 0.0900 0.9370 0.9690 0.9528 0.9769
0.0791 2.5163 1540 0.0803 0.9477 0.9687 0.9581 0.9804
0.0759 2.5490 1560 0.0822 0.9443 0.9715 0.9577 0.9799
0.0817 2.5817 1580 0.0801 0.9511 0.9697 0.9603 0.9812
0.0721 2.6144 1600 0.0906 0.9352 0.9756 0.9550 0.9778
0.0661 2.6471 1620 0.0883 0.9394 0.9696 0.9542 0.9774
0.0682 2.6797 1640 0.0855 0.95 0.9568 0.9534 0.9780
0.0779 2.7124 1660 0.0814 0.9463 0.9701 0.9580 0.9804
0.0696 2.7451 1680 0.0792 0.9406 0.9646 0.9525 0.9780
0.0579 2.7778 1700 0.0834 0.9413 0.9704 0.9556 0.9782
0.1837 2.8105 1720 0.0841 0.9393 0.9723 0.9555 0.9794
0.0993 2.8431 1740 0.0816 0.9422 0.9743 0.9580 0.9797
0.0645 2.8758 1760 0.0909 0.9367 0.9633 0.9498 0.9770
0.0771 2.9085 1780 0.0918 0.9277 0.9639 0.9455 0.9754
0.104 2.9412 1800 0.1006 0.9237 0.9636 0.9433 0.9731
0.0883 2.9739 1820 0.0864 0.9412 0.9678 0.9543 0.9779
0.0653 3.0065 1840 0.0939 0.9290 0.9701 0.9491 0.9758
0.1244 3.0392 1860 0.1186 0.9132 0.9614 0.9367 0.9703
0.0524 3.0719 1880 0.0928 0.9355 0.9683 0.9516 0.9771
0.0756 3.1046 1900 0.0897 0.9403 0.9687 0.9543 0.9781
0.0664 3.1373 1920 0.0875 0.9426 0.9676 0.9549 0.9777
0.0653 3.1699 1940 0.1014 0.9265 0.9741 0.9497 0.9749
0.0614 3.2026 1960 0.0892 0.9461 0.9657 0.9558 0.9782
0.0705 3.2353 1980 0.0979 0.9242 0.9626 0.9430 0.9737
0.064 3.2680 2000 0.0845 0.9491 0.9643 0.9567 0.9799
0.059 3.3007 2020 0.0818 0.9442 0.9718 0.9578 0.9803
0.0769 3.3333 2040 0.0788 0.9451 0.9682 0.9565 0.9801
0.0595 3.3660 2060 0.0811 0.9493 0.9698 0.9594 0.9801
0.0542 3.3987 2080 0.0906 0.9367 0.9698 0.9530 0.9777
0.0613 3.4314 2100 0.0819 0.9452 0.9657 0.9553 0.9786
0.0745 3.4641 2120 0.1052 0.9211 0.9691 0.9445 0.9725
0.0654 3.4967 2140 0.0822 0.9469 0.9680 0.9573 0.9794
0.0645 3.5294 2160 0.0798 0.9439 0.9683 0.9559 0.9792
0.0641 3.5621 2180 0.0830 0.9398 0.9725 0.9559 0.9774
0.0517 3.5948 2200 0.0795 0.9503 0.9700 0.9600 0.9801
0.0532 3.6275 2220 0.0877 0.9415 0.9714 0.9562 0.9787
0.0576 3.6601 2240 0.0829 0.9492 0.9670 0.9581 0.9797
0.0609 3.6928 2260 0.0822 0.9517 0.9662 0.9589 0.9796
0.0577 3.7255 2280 0.0850 0.9487 0.9728 0.9606 0.9804
0.0704 3.7582 2300 0.0829 0.9514 0.9724 0.9618 0.9815
0.052 3.7908 2320 0.0870 0.9447 0.9711 0.9577 0.9798
0.0794 3.8235 2340 0.0837 0.9493 0.9709 0.9600 0.9804
0.0693 3.8562 2360 0.0943 0.9428 0.9685 0.9555 0.9769
0.0727 3.8889 2380 0.0928 0.9370 0.9735 0.9549 0.9775
0.0729 3.9216 2400 0.0914 0.9352 0.9698 0.9522 0.9775
0.0805 3.9542 2420 0.0804 0.9452 0.9712 0.9580 0.9804
0.0737 3.9869 2440 0.0807 0.9450 0.9701 0.9574 0.9802
0.0556 4.0196 2460 0.0814 0.9484 0.9723 0.9602 0.9812
0.0533 4.0523 2480 0.0832 0.9424 0.9738 0.9578 0.9801
0.0435 4.0850 2500 0.0824 0.9453 0.9708 0.9579 0.9801
0.0356 4.1176 2520 0.0869 0.9414 0.9688 0.9549 0.9795
0.0565 4.1503 2540 0.0857 0.9536 0.9657 0.9596 0.9806
0.0466 4.1830 2560 0.0884 0.9509 0.9661 0.9584 0.9789
0.0692 4.2157 2580 0.0849 0.9435 0.9742 0.9586 0.9803
0.0388 4.2484 2600 0.0844 0.9412 0.9714 0.9560 0.9795
0.0471 4.2810 2620 0.0780 0.9511 0.9674 0.9592 0.9810
0.0666 4.3137 2640 0.0800 0.9511 0.9663 0.9586 0.9806
0.0574 4.3464 2660 0.0975 0.9253 0.9721 0.9481 0.9751
0.0597 4.3791 2680 0.0821 0.9489 0.9611 0.9550 0.9796
0.045 4.4118 2700 0.0837 0.9457 0.9691 0.9572 0.9799
0.0427 4.4444 2720 0.0871 0.9392 0.9710 0.9548 0.9792
0.0389 4.4771 2740 0.0816 0.9475 0.9712 0.9592 0.9810
0.0481 4.5098 2760 0.0865 0.9465 0.9708 0.9585 0.9805
0.0649 4.5425 2780 0.0827 0.9484 0.9637 0.9560 0.9795
0.0533 4.5752 2800 0.0850 0.9398 0.9698 0.9546 0.9792
0.0431 4.6078 2820 0.0776 0.9478 0.9752 0.9613 0.9815
0.0519 4.6405 2840 0.0813 0.9434 0.9731 0.9580 0.9797
0.0416 4.6732 2860 0.0804 0.9444 0.9673 0.9557 0.9793
0.0405 4.7059 2880 0.0826 0.9504 0.9701 0.9602 0.9799
0.0522 4.7386 2900 0.0833 0.9436 0.9708 0.9570 0.9799
0.0595 4.7712 2920 0.0835 0.9434 0.9733 0.9581 0.9799
0.0362 4.8039 2940 0.0809 0.9484 0.9719 0.9600 0.9805
0.0436 4.8366 2960 0.0835 0.9490 0.9695 0.9592 0.9802
0.059 4.8693 2980 0.0894 0.9363 0.9721 0.9539 0.9776
0.0395 4.9020 3000 0.0793 0.9475 0.9724 0.9598 0.9807
0.0413 4.9346 3020 0.0818 0.9476 0.9690 0.9582 0.9809
0.0666 4.9673 3040 0.0816 0.9449 0.9692 0.9569 0.9794
0.0452 5.0 3060 0.0811 0.9466 0.9739 0.9601 0.9806
0.0367 5.0327 3080 0.0811 0.9453 0.9712 0.9581 0.9807
0.0521 5.0654 3100 0.0749 0.9506 0.9710 0.9607 0.9817
0.0308 5.0980 3120 0.0852 0.9406 0.9728 0.9564 0.9794
0.0434 5.1307 3140 0.0804 0.9501 0.9712 0.9605 0.9808
0.0414 5.1634 3160 0.0799 0.9497 0.9714 0.9604 0.9806
0.0343 5.1961 3180 0.0826 0.9477 0.9712 0.9593 0.9803
0.0355 5.2288 3200 0.0882 0.9436 0.9693 0.9562 0.9792
0.0351 5.2614 3220 0.0869 0.9489 0.9704 0.9595 0.9805
0.0377 5.2941 3240 0.0864 0.9503 0.9669 0.9586 0.9802
0.0331 5.3268 3260 0.0865 0.9508 0.9698 0.9602 0.9809
0.0386 5.3595 3280 0.0850 0.9511 0.9692 0.9601 0.9807
0.0312 5.3922 3300 0.0856 0.9500 0.9665 0.9582 0.9801
0.0382 5.4248 3320 0.0844 0.9551 0.9625 0.9588 0.9807
0.0559 5.4575 3340 0.0841 0.9506 0.9693 0.9598 0.9811
0.0295 5.4902 3360 0.0872 0.9480 0.9671 0.9575 0.9800
0.0448 5.5229 3380 0.0832 0.9537 0.9676 0.9606 0.9814
0.0348 5.5556 3400 0.0874 0.9493 0.9686 0.9588 0.9801
0.0396 5.5882 3420 0.0893 0.9470 0.9677 0.9572 0.9793
0.0477 5.6209 3440 0.0924 0.9455 0.9677 0.9565 0.9782
0.0535 5.6536 3460 0.0824 0.9490 0.9695 0.9592 0.9804
0.0343 5.6863 3480 0.0823 0.9502 0.9690 0.9595 0.9808
0.0409 5.7190 3500 0.0845 0.9480 0.9695 0.9586 0.9805
0.0345 5.7516 3520 0.0847 0.9499 0.9686 0.9592 0.9807
0.0337 5.7843 3540 0.0863 0.9488 0.9707 0.9597 0.9806
0.0379 5.8170 3560 0.0824 0.9534 0.9694 0.9613 0.9814
0.0399 5.8497 3580 0.0820 0.9521 0.9722 0.9620 0.9812
0.038 5.8824 3600 0.0825 0.9497 0.9698 0.9596 0.9803
0.0287 5.9150 3620 0.0812 0.9530 0.9687 0.9608 0.9810
0.0333 5.9477 3640 0.0843 0.9496 0.9705 0.9599 0.9807
0.0448 5.9804 3660 0.0861 0.9485 0.9678 0.9580 0.9793
0.0312 6.0131 3680 0.0863 0.9449 0.9694 0.9570 0.9790
0.037 6.0458 3700 0.0832 0.9519 0.9688 0.9603 0.9811
0.0274 6.0784 3720 0.0841 0.9522 0.9680 0.9600 0.9808
0.0341 6.1111 3740 0.0843 0.9489 0.9704 0.9595 0.9806
0.0247 6.1438 3760 0.0872 0.9448 0.9674 0.9559 0.9799
0.0225 6.1765 3780 0.0883 0.9457 0.9656 0.9556 0.9798
0.0244 6.2092 3800 0.0912 0.9447 0.9679 0.9562 0.9791
0.0357 6.2418 3820 0.0898 0.9492 0.9685 0.9588 0.9805
0.0435 6.2745 3840 0.0895 0.9505 0.9690 0.9596 0.9808
0.0289 6.3072 3860 0.0903 0.9507 0.9684 0.9594 0.9807
0.0299 6.3399 3880 0.0907 0.9487 0.9674 0.9579 0.9803
0.0302 6.3725 3900 0.0898 0.9508 0.9681 0.9594 0.9806
0.0277 6.4052 3920 0.0899 0.9504 0.9680 0.9591 0.9808
0.0296 6.4379 3940 0.0895 0.9497 0.9670 0.9582 0.9808
0.0214 6.4706 3960 0.0909 0.9479 0.9691 0.9584 0.9808
0.0259 6.5033 3980 0.0895 0.9505 0.9702 0.9602 0.9813
0.0316 6.5359 4000 0.0880 0.9517 0.9694 0.9605 0.9812
0.0307 6.5686 4020 0.0873 0.9520 0.9700 0.9609 0.9814
0.0339 6.6013 4040 0.0875 0.9515 0.9699 0.9606 0.9812
0.0202 6.6340 4060 0.0876 0.9511 0.9700 0.9605 0.9812
0.0243 6.6667 4080 0.0877 0.9511 0.9690 0.9600 0.9811
0.0384 6.6993 4100 0.0870 0.9515 0.9690 0.9602 0.9812
0.0276 6.7320 4120 0.0871 0.9508 0.9690 0.9598 0.9811
0.0381 6.7647 4140 0.0867 0.9510 0.9690 0.9599 0.9811
0.035 6.7974 4160 0.0861 0.9521 0.9688 0.9604 0.9813
0.0277 6.8301 4180 0.0863 0.9515 0.9690 0.9602 0.9812
0.0219 6.8627 4200 0.0863 0.9515 0.9693 0.9603 0.9812
0.0232 6.8954 4220 0.0864 0.9509 0.9694 0.9601 0.9811
0.0266 6.9281 4240 0.0866 0.9508 0.9695 0.9601 0.9812
0.0269 6.9608 4260 0.0865 0.9510 0.9695 0.9602 0.9812
0.0237 6.9935 4280 0.0864 0.9510 0.9696 0.9602 0.9812

Framework versions

  • PEFT 0.13.2
  • Transformers 4.46.3
  • Pytorch 2.5.1+cu121
  • Datasets 3.2.0
  • Tokenizers 0.20.3