README.md

---
library_name: peft
base_model: Sakonii/distilgpt2-nepali
license: apache-2.0
datasets:
- Bibek1129/nepali_SQuAD_single_qsn
language:
- ne
pipeline_tag: text-generation
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->



## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->
The model is finetuned on Sakonii/distilgpt2-nepali with Bibek1129/nepali_SQuAD_multiple_qsns dataset.The dataset is converted to nepali using Nepali_nlp library using SQuAD dataset.


- **Model type:** distilgpt2
- **Language(s) (NLP):** ne(Nepali)
- **Finetuned from model :** https://huggingface.co/Sakonii/distilgpt2-nepali

### Model Sources 

<!-- Provide the basic links for the model. -->
For training snippets and inference check the following repository.
- **Repository:** https://github.com/HordesOfGhost/Nepali_LLMs/]


## How to Get Started with the Model

Use the code below to get started with the model.
```python
!pip install peft 
!pip install transformers
!pip install sentencepiece
```
```python
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM,AutoTokenizer
from transformers import pipeline

base_model = "Sakonii/distilgpt2-nepali"
adapter_model = "Bibek1129/distilgpt2-nepali-single-qs-generator"

tokenizer = AutoTokenizer.from_pretrained(base_model)

config = PeftConfig.from_pretrained(adapter_model)
model = AutoModelForCausalLM.from_pretrained(base_model)
model = PeftModel.from_pretrained(model, adapter_model)
model = model.merge_and_unload()

prompt = """तपाईं एउटा प्रश्न उत्पन्न गर्ने मोडेल हुनुहुन्छ। तपाइँलाई एक सन्दर्भ दिइएको हुन्छ र तपाइँ त्यसमा आधारित एउटा प्रश्न उत्पन्न गर्नुहुन्छ।

### सन्दर्भ:
राजनीति 'शहरका मामिलाहरू') गतिविधिहरूको सेट हो जुन समूहहरूमा निर्णय गर्न वा व्यक्तिहरू बीचको शक्ति सम्बन्धका अन्य रूपहरू, जस्तै स्रोत वा स्थितिको वितरणसँग सम्बन्धित छ। राजनीति र सरकारको अध्ययन गर्ने सामाजिक विज्ञानको शाखालाई राजनीति विज्ञान भनिन्छ।
यसलाई "राजनीतिक समाधान" को सन्दर्भमा सकारात्मक रूपमा प्रयोग गर्न सकिन्छ जुन सम्झौता र अहिंसात्मक छ, वा वर्णनात्मक रूपमा "सरकारको कला वा विज्ञान" को रूपमा, तर प्राय: नकारात्मक अर्थ पनि बोक्छ। अवधारणालाई विभिन्न तरिकामा परिभाषित गरिएको छ, र यसलाई
व्यापक रूपमा प्रयोग गर्ने वा सीमित रूपमा, प्रायोगिक वा सामान्य रूपमा, र यसको लागि द्वन्द्व वा सहयोग बढी आवश्यक छ कि छैन भन्ने बारेमा विभिन्न दृष्टिकोणहरूमा मौलिक रूपमा फरक फरक विचारहरू छन्।

### प्रश्न:
"""
pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=64)

def format_output(prompt,pipe):
  inference = pipe(prompt)[0]["generated_text"]
  
  # Select after प्रश्नहरू: and break line after each ?
  inference = inference.split("प्रश्न:")[-1].replace("?","?\n")
  
  # only take first question
  index = inference.find("?")
  inference = inference[:index+1] 
  return inference

print(format_output(prompt, pipe))
'''
  Output:
        राजनीतिक आन्दोलनमा, राजनीतिक कार्यसूचीको सन्दर्भमा कुन प्रकारको राजनीति महत्वपूर्ण छ?
'''
```

## Training Details

### Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
The dataset is created by converting SQuAD dataset to nepali using Nepali_nlp using PEFT.

https://huggingface.co/datasets/Bibek1129/nepali_SQuAD_single_qsn

### Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
The model is trained with the lora config (rank=32,lora_alpha=64,target_modules="c_fc","c_attn","c_proj","lm_head");with 512 tokens per instance, 4 instances per batch, and around 118.1K training steps.

#### Training Hyperparameters
Following are the training hyperparameters.
<li>learning_rate:2e-4</li>
<li>fp16:True</li>
<li>optim:"paged_adamw_32bit"</li>
<li>lr_scheduler_type:"constant"</li>
<li>num_train_epochs:15</li>
Lora Config:

```python
config={
  "alpha_pattern": {},
  "auto_mapping": null,
  "base_model_name_or_path": "Sakonii/distilgpt2-nepali",
  "bias": "none",
  "fan_in_fan_out": false,
  "inference_mode": true,
  "init_lora_weights": true,
  "layers_pattern": null,
  "layers_to_transform": null,
  "lora_alpha": 64,
  "lora_dropout": 0.05,
  "modules_to_save": null,
  "peft_type": "LORA",
  "r": 32,
  "rank_pattern": {},
  "revision": null,
  "target_modules": [
    "c_proj",
    "lm_head",
    "c_fc",
    "c_attn"
  ],
  "task_type": "CAUSAL_LM"
}

```

### Results
  <li>train/loss:3.1028</li>

### Framework versions

- PEFT 0.9.0
-