README.md

---
license: other
library_name: peft
tags:
- llama-factory
- lora
- generated_from_trainer
base_model: meta-llama/Meta-Llama-3-8B
model-index:
- name: sft_trained_woaqa_llama3
  results: []
datasets:
- jiazhengli/Synthetic_Rationale
- jiazhengli/Rationale_MCTS
language:
- en
metrics:
- accuracy
- f1
---


# Meta-Llama-3-8B-QLoRA-Assessment-Rationale-sft

The model trained with w/o private data from the EMNLP 2024 Paper: Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring. 

- **Paper:** [Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring](https://arxiv.org/abs/2406.19949) (EMNLP 2024 Findings)
- **GitHub Repository:** [Thought Tree Assessment Repository](https://github.com/lijiazheng99/thought_tree_assessment)

## Intended uses & limitations

This model offers a valuable resource for research in explainable AI within educational technology. The model is trained with **noisy** response-level rationales. This makes them **unsuitable** for direct application in high-stakes assessments without additional verification.

## Training and evaluation data

We trained and evaluated the model on the [Synthetic Rationale data](https://huggingface.co/datasets/jiazhengli/Synthetic_Rationale), which was generated from the [Rationale MCTS data](https://huggingface.co/datasets/jiazhengli/Rationale_MCTS).

To extract scores from rationales, please use the [jiazhengli/deberta-v3-large-Rationale-to-Score](https://huggingface.co/jiazhengli/deberta-v3-large-Rationale-to-Score).

## Citation

Please cite the following work if you utilize this model:

**BibTeX:**

```bibtex
@misc{li2024calibratingllmspreferenceoptimization,
      title={Calibrating LLMs with Preference Optimization on Thought Trees for Generating Rationale in Science Question Scoring}, 
      author={Jiazheng Li and Hainiu Xu and Zhaoyue Sun and Yuxiang Zhou and David West and Cesare Aloisi and Yulan He},
      year={2024},
      eprint={2406.19949},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2406.19949}, 
}
```

## Training procedure

Please refer to our [paper](https://arxiv.org/abs/2406.19949).

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- num_epochs: 4.0
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:-----:|:----:|:---------------:|
| 0.9813        | 0.63  | 100  | 0.9671          |
| 0.9108        | 1.26  | 200  | 0.9250          |
| 0.8976        | 1.9   | 300  | 0.9091          |
| 0.8687        | 2.53  | 400  | 0.9005          |
| 0.8548        | 3.16  | 500  | 0.8958          |
| 0.8468        | 3.79  | 600  | 0.8952          |

### Framework versions

- PEFT 0.10.0
- Transformers 4.38.2
- Pytorch 2.2.1+cu121
- Datasets 2.18.0
- Tokenizers 0.15.2