|
--- |
|
base_model: microsoft/phi-2 |
|
library_name: peft |
|
license: apache-2.0 |
|
datasets: |
|
- neil-code/dialogsum-test |
|
language: |
|
- en |
|
metrics: |
|
- bleu |
|
pipeline_tag: question-answering |
|
tags: |
|
- QuestionAnswering |
|
- legal |
|
- finan |
|
- chem |
|
- biology |
|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
metrics: |
|
- rouge |
|
base_model: |
|
- microsoft/phi-2 |
|
pipeline_tag: question-answering |
|
--- |
|
|
|
|
|
|
|
This repo containes the last checkpoint of my fine tuned model. Click this link to go the final model |
|
https://huggingface.co/JamieAi33/Phi-2_PEFT |
|
# Model Card for PEFT-Fine-Tuned Model |
|
|
|
This model card documents a PEFT-fine-tuned version of `microsoft/phi-2` for question-answering tasks. The PEFT fine-tuning improved the model's performance, as detailed in the evaluation section. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
- **Developed by:** JamieAi33 |
|
- **Finetuned from model:** `microsoft/phi-2` |
|
- **Model type:** PEFT fine-tuned transformer |
|
- **Language(s) (NLP):** English |
|
- **License:** Apache 2.0 |
|
|
|
The base model `microsoft/phi-2` was adapted using Parameter-Efficient Fine-Tuning (PEFT) for question-answering tasks. The training process focused on improving performance metrics while keeping computational costs low. |
|
|
|
--- |
|
|
|
### Model Sources |
|
|
|
- **Repository:** https://huggingface.co/JamieAi33/Phi-2-QLora |
|
- **Paper:** [Optional: Add a reference to PEFT or any relevant paper] |
|
- **Demo:** [Optional: Link to your Hugging Face Space or demo] |
|
|
|
--- |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
This model can be used out-of-the-box for question-answering tasks. |
|
|
|
### Downstream Use |
|
The model can be fine-tuned further on domain-specific datasets for improved performance. |
|
|
|
### Out-of-Scope Use |
|
Avoid using this model for tasks outside question-answering or where fairness, bias, and ethical considerations are critical without further validation. |
|
|
|
--- |
|
|
|
## Bias, Risks, and Limitations |
|
|
|
Users should be aware that: |
|
- The model is trained on publicly available data and may inherit biases present in the training data. |
|
- It is optimized for English and may perform poorly in other languages. |
|
|
|
--- |
|
|
|
## How to Get Started with the Model |
|
|
|
Here鈥檚 an example of loading the model: |
|
|
|
```python |
|
from transformers import AutoModel |
|
from peft import PeftModel |
|
|
|
base_model = AutoModel.from_pretrained("microsoft/phi-2") |
|
adapter_model = PeftModel.from_pretrained(base_model, "JamieAi33/Phi-2-QLora") |
|
|
|
|
|
|
|
# Model Name: PEFT Fine-Tuned `microsoft/phi-2` |
|
|
|
This repository contains a PEFT fine-tuned version of the `microsoft/phi-2` model for question-answering tasks. The fine-tuning process leveraged Parameter-Efficient Fine-Tuning (PEFT) techniques to achieve improved performance. |
|
|
|
--- |
|
|
|
## Metrics |
|
|
|
The model's performance was evaluated using the ROUGE metric. Below are the results: |
|
|
|
| **Metric** | **Original Model** | **PEFT Model** | **Absolute Improvement** | |
|
|-----------------|--------------------|----------------|---------------------------| |
|
| **ROUGE-1** | 29.76% | 44.51% | +14.75% | |
|
| **ROUGE-2** | 10.76% | 15.68% | +4.92% | |
|
| **ROUGE-L** | 21.69% | 30.95% | +9.25% | |
|
| **ROUGE-Lsum** | 22.75% | 31.49% | +8.74% | |
|
|
|
--- |
|
## Training Configuration |
|
|
|
| Hyperparameter | Value | |
|
|-----------------------|-------------------------| |
|
| **Batch Size** | 1 | |
|
| **Learning Rate** | 2e-4 | |
|
| **Max Steps** | 1000 | |
|
| **Optimizer** | Paged AdamW (8-bit) | |
|
| **Logging Steps** | 25 | |
|
| **Evaluation Steps** | 25 | |
|
| **Gradient Checkpointing** | Enabled | |
