README.md

---
library_name: transformers
tags:
  - llama
  - fine-tuned
  - physics
  - smolLM
  - LoRA
license: apache-2.0
---


# fine-tuned-smolLM2-135M-with-LoRA-on-camel-ai-physics

This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) on the dataset [akhilfau/physics_decontaminated_2](https://huggingface.co/datasets/akhilfau/physics_decontaminated_2). This dataset was created by decontaminating the [camel-ai/physics](https://huggingface.co/datasets/camel-ai/physics) dataset from [mmlu:college_physics](https://huggingface.co/datasets/lighteval/mmlu).

---

## Model Performance

This model was evaluated on **MMLU: college_physics** using **LightEval**. The evaluation compared the base model (HuggingFaceTB/SmolLM2-135M) and the fine-tuned model (akhilfau/fine-tuned-smolLM2-135M-with-LoRA-on-camel-ai-physics). Results are as follows:


## Model Description

The fine-tuned model leverages **LoRA (Low-Rank Adaptation)** for parameter-efficient fine-tuning. The base model is SmolLM2-135M, which uses the **LlamaForCausalLM** architecture, and it was fine-tuned to enhance its understanding of physics-related questions and answers using the **akhilfau/physics_decontaminated_2** dataset.

---

## Training and Evaluation Data

### Dataset Details:

- **Training Dataset:** [akhilfau/physics_decontaminated_2](https://huggingface.co/datasets/akhilfau/physics_decontaminated_2)
- **Evaluation Dataset:** [mmlu:college_physics](https://huggingface.co/datasets/lighteval/mmlu/viewer/college_physics)

The training dataset was decontaminated to ensure no overlap with the evaluation dataset for fair performance testing.

---

## Training Procedure

### Training Hyperparameters

| Hyperparameter           | Value              |
|---------------------------|--------------------|
| Learning Rate            | 0.0005            |
| Train Batch Size         | 4                 |
| Eval Batch Size          | 4                 |
| Seed                     | 42                |
| Optimizer                | AdamW with betas=(0.9, 0.999), epsilon=1e-8 |
| LR Scheduler Type        | Cosine            |
| Number of Epochs         | 8                 |

### Training Results

| Training Loss | Epoch | Step  | Validation Loss |
|---------------|-------|-------|-----------------|
| 1.0151        | 1.0   | 4000  | 1.0407          |
| 1.0234        | 2.0   | 8000  | 1.0087          |
| 0.9995        | 3.0   | 12000 | 0.9921          |
| 0.9528        | 4.0   | 16000 | 0.9824          |
| 0.9353        | 5.0   | 20000 | 0.9755          |
| 0.9121        | 6.0   | 24000 | 0.9720          |
| 0.9175        | 7.0   | 28000 | 0.9707          |
| 0.9197        | 8.0   | 32000 | 0.9706          |

---

## Intended Use

This model is specifically fine-tuned for physics-related reasoning tasks and QA tasks. It may perform well on datasets that require understanding physics-related problems and concepts. Evaluation results show a measurable improvement compared to the base model on MMLU college physics tasks.

---

## Framework Versions

- **PEFT**: 0.13.2
- **Transformers**: 4.46.2
- **Pytorch**: 2.4.1+cu121
- **Datasets**: 3.1.0
- **Tokenizers**: 0.20.3