File size: 3,714 Bytes
3c85c02 d17d9e4 caad5a8 d17d9e4 3c85c02 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 d17d9e4 caad5a8 3c85c02 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
---
base_model: microsoft/phi-2
library_name: peft
license: apache-2.0
datasets:
- neil-code/dialogsum-test
language:
- en
metrics:
- bleu
pipeline_tag: question-answering
tags:
- QuestionAnswering
- legal
- finan
- chem
- biology
---
license: apache-2.0
language:
- en
metrics:
- rouge
base_model:
- microsoft/phi-2
pipeline_tag: question-answering
---
This repo containes the last checkpoint of my fine tuned model. Click this link to go the final model
https://huggingface.co/JamieAi33/Phi-2_PEFT
# Model Card for PEFT-Fine-Tuned Model
This model card documents a PEFT-fine-tuned version of `microsoft/phi-2` for question-answering tasks. The PEFT fine-tuning improved the model's performance, as detailed in the evaluation section.
## Model Details
### Model Description
- **Developed by:** JamieAi33
- **Finetuned from model:** `microsoft/phi-2`
- **Model type:** PEFT fine-tuned transformer
- **Language(s) (NLP):** English
- **License:** Apache 2.0
The base model `microsoft/phi-2` was adapted using Parameter-Efficient Fine-Tuning (PEFT) for question-answering tasks. The training process focused on improving performance metrics while keeping computational costs low.
---
### Model Sources
- **Repository:** https://huggingface.co/JamieAi33/Phi-2-QLora
- **Paper:** [Optional: Add a reference to PEFT or any relevant paper]
- **Demo:** [Optional: Link to your Hugging Face Space or demo]
---
## Uses
### Direct Use
This model can be used out-of-the-box for question-answering tasks.
### Downstream Use
The model can be fine-tuned further on domain-specific datasets for improved performance.
### Out-of-Scope Use
Avoid using this model for tasks outside question-answering or where fairness, bias, and ethical considerations are critical without further validation.
---
## Bias, Risks, and Limitations
Users should be aware that:
- The model is trained on publicly available data and may inherit biases present in the training data.
- It is optimized for English and may perform poorly in other languages.
---
## How to Get Started with the Model
Here鈥檚 an example of loading the model:
```python
from transformers import AutoModel
from peft import PeftModel
base_model = AutoModel.from_pretrained("microsoft/phi-2")
adapter_model = PeftModel.from_pretrained(base_model, "JamieAi33/Phi-2-QLora")
# Model Name: PEFT Fine-Tuned `microsoft/phi-2`
This repository contains a PEFT fine-tuned version of the `microsoft/phi-2` model for question-answering tasks. The fine-tuning process leveraged Parameter-Efficient Fine-Tuning (PEFT) techniques to achieve improved performance.
---
## Metrics
The model's performance was evaluated using the ROUGE metric. Below are the results:
| **Metric** | **Original Model** | **PEFT Model** | **Absolute Improvement** |
|-----------------|--------------------|----------------|---------------------------|
| **ROUGE-1** | 29.76% | 44.51% | +14.75% |
| **ROUGE-2** | 10.76% | 15.68% | +4.92% |
| **ROUGE-L** | 21.69% | 30.95% | +9.25% |
| **ROUGE-Lsum** | 22.75% | 31.49% | +8.74% |
---
## Training Configuration
| Hyperparameter | Value |
|-----------------------|-------------------------|
| **Batch Size** | 1 |
| **Learning Rate** | 2e-4 |
| **Max Steps** | 1000 |
| **Optimizer** | Paged AdamW (8-bit) |
| **Logging Steps** | 25 |
| **Evaluation Steps** | 25 |
| **Gradient Checkpointing** | Enabled | |