---
license: apache-2.0
datasets:
- NIEXCHE/turkish_agriculture_QA_llama2_22.6k
language:
- tr
- en
---

# Model Card for LLaMA-2-7B-NIEXCHE

This model was fine-tuned from LLaMA-2-7B on a Turkish agriculture QA dataset. It supports both Turkish and English languages and was trained for use in agriculture-related natural language processing (NLP) tasks.

## Model Details

### Model Description

- **Developed by:** NIEXCHE (Fevzi KILAS)
- **Finetuned from model:** [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)
- **License:** Apache-2.0
- **Language(s) (NLP):** Turkish, English
- **Model type:** LLaMA-2-based model
- **Training Dataset:** [NIEXCHE/turkish_agriculture_QA_llama2_22.6k](https://huggingface.co/datasets/NIEXCHE/turkish_agriculture_QA_llama2_22.6k)

### Model Sources

- **Repository:** [Model Repository (TBA)](#)
- **Demo:** [TBA](#)

## Uses

### Direct Use

The model can be used directly for question-answering tasks related to agriculture in Turkish and English. It is fine-tuned specifically for agricultural Q&A, making it suitable for similar domains and use cases.

### Out-of-Scope Use

The model might not perform well on general knowledge questions outside of the agriculture domain.

## Training Details

### Training Data

The training data was a custom dataset created by translating and cleaning agricultural QA data from [this source](https://huggingface.co/datasets/KisanVaani/agriculture-qa-english-only). The dataset contains 22.6k question-answer pairs in Turkish.

### Training Procedure

The model was trained using the following frameworks and libraries:
- **Frameworks:** PyTorch, `transformers`, `accelerate==0.21.0`, `peft==0.4.0`, `bitsandbytes==0.40.2`, `trl==0.4.7`
- **Precision:** The model was trained using 4-bit quantization (BNB) with mixed precision (`float16`) to optimize memory usage.

#### Training Hyperparameters

- **Base Model:** `meta-llama/Llama-2-7b`
- **Batch Size:** 4 (per device)
- **Learning Rate:** 2e-4
- **LoRA Parameters:** 
  - lora_r = 64
  - lora_alpha = 16
  - lora_dropout = 0.1
- **Epochs:** 1
- **Optimizer:** Paged AdamW (32-bit)
- **Gradient Accumulation Steps:** 1
- **Scheduler:** Cosine
- **Max Gradient Norm:** 0.3
- **Gradient Checkpointing:** Enabled
- **Warmup Ratio:** 0.03
- **Group by Length:** Enabled
- **Max Sequence Length:** None

### Hardware

- **Training Hardware:** Google Colab Pro (A100 GPU) and 53 GB system RAM.
- **Training Time:** Approximately 1 hour 40 minutes.

Training output:
`TrainOutput(global_step=5654, training_loss=0.7829279924898043, metrics={'train_runtime': 6029.996, 'train_samples_per_second': 3.75, 'train_steps_per_second': 0.938, 'total_flos': 5.516196145999872e+16, 'train_loss': 0.7829279924898043, 'epoch': 1.0})`



## Evaluation

The same dataset (`NIEXCHE/turkish_agriculture_QA_llama2_22.6k`) was used for evaluation purposes.

## Environmental Impact

Carbon emissions were estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute).

- **Hardware Type:** Google Colab (A100 GPU)
- **Hours used:** 1 hour 40 minutes
- **Compute Region:** Google Cloud (Colab)
- **Carbon Emitted:** Estimations pending

## Citation

If you use this model in your research or applications, please cite it as:

```bibtex
@misc{Fevzi2024LLaMA-2-7B-NIEXCHE,
  author = {Fevzi KILAS},
  title = {LLaMA-2-7B-NIEXCHE: A Turkish Agriculture QA Model},
  year = {2024},
  howpublished = {https://huggingface.co/NIEXCHE/turkish_agriculture_QA_llama2_22.6k}
}
```
## Contact:

[NIEXCHE (Fevzi KILAS)](https://niexche.github.io/)