README.md

---
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
model_name: llama-8b-south-africa
languages:
  - Xhosa
  - Zulu 
  - Tswana
  - Northern Sotho
  - Afrikaans
license: apache-2.0
tags:
  - african-languages
  - multilingual
  - instruction-tuning
  - transfer-learning
library_name: peft

model_description: |
  This model is a fine-tuned version of Meta's LLaMA-3.1-8B-Instruct model, specifically adapted for South African languages. The training data consists of the Alpaca Cleaned dataset translated into five South African languages: Xhosa, Zulu, Tswana, Northern Sotho, and Afrikaans using machine translation techniques.
  
  Key Features:
  - Base architecture: LLaMA-3.1-8B-Instruct
  - Training approach: Instruction tuning via translated datasets
  - Target languages: 5 South African languages
  - Cost-efficient: Total cost ~$1,870 ($370/language for translation + $15 for training)

training_details:
  hyperparameters:
    learning_rate: 0.0002
    train_batch_size: 4
    eval_batch_size: 8
    gradient_accumulation_steps: 2
    total_train_batch_size: 8
    optimizer: "Adam with betas=(0.9,0.999) and epsilon=1e-08"
    lr_scheduler_type: cosine
    lr_scheduler_warmup_ratio: 0.1
    num_epochs: 1
    seed: 42
    distributed_type: multi-GPU
  
  results:
    final_loss: 1.0959
    validation_loss: 0.0571
    total_steps: 5596
    completed_epochs: 0.9999

model_evaluation:
  xhosa:
    afrimgsm:
      accuracy: 0.02
    afrimmlu:
      accuracy: 0.29
    afrixnli:
      accuracy: 0.44
  zulu:
    afrimgsm:
      accuracy: 0.045
    afrimmlu:
      accuracy: 0.29
    afrixnli:
      accuracy: 0.43

limitations: |
  - Current evaluation metrics are limited to Xhosa and Zulu due to Iroko language availability
  - Machine translation was used for training data generation, which may impact quality
  - Low performance on certain tasks (particularly AfriMGSM) suggests room for improvement

framework_versions:
  pytorch: 2.4.1+cu121
  transformers: 4.44.2
  peft: 0.12.0
  datasets: 3.0.0
  tokenizers: 0.19.1

resources:
  benchmark_visualization: assets/Benchmarks_(1).pdf
  training_dataset: https://huggingface.co/datasets/yahma/alpaca-cleaned
---

# LLaMA-3.1-8B South African Languages Model

This model card provides detailed information about the LLaMA-3.1-8B model fine-tuned for South African languages. The model demonstrates cost-effective cross-lingual transfer learning for African language processing.

## Model Overview

The model is based on Meta's LLaMA-3.1-8B-Instruct architecture and has been fine-tuned on translated versions of the Alpaca Cleaned dataset. The training approach leverages machine translation to create instruction-tuning data in five South African languages, making it a cost-effective solution for multilingual AI development.

## Training Methodology

### Dataset Preparation
The training data was created by translating the Alpaca Cleaned dataset into five target languages:
- Xhosa
- Zulu
- Tswana
- Northern Sotho
- Afrikaans

Machine translation was used to generate the training data, with a cost of $370 per language.

### Training Process
The model was trained using the PEFT (Parameter-Efficient Fine-Tuning) library on the Akash Compute Network. Key aspects of the training process include:
- Single epoch training
- Multi-GPU distributed training setup
- Cosine learning rate schedule with 10% warmup
- Adam optimizer with β1=0.9, β2=0.999, ε=1e-08
- Total training cost: $15

## Performance Evaluation

### Evaluation Scope
Current evaluation metrics are available for two languages:
1. Xhosa (xho)
2. Zulu (zul)

Evaluation was conducted using three benchmark datasets:

### AfriMGSM Results
- Xhosa: 2.0% accuracy
- Zulu: 4.5% accuracy

### AfriMMIU Results
- Xhosa: 29.0% accuracy
- Zulu: 29.0% accuracy

### AfriXNLI Results
- Xhosa: 44.0% accuracy
- Zulu: 43.0% accuracy

## Limitations and Considerations

1. **Evaluation Coverage**
   - Only Xhosa and Zulu could be evaluated due to limitations in available benchmarking tools
   - Performance on other supported languages remains unknown

2. **Training Data Quality**
   - Reliance on machine translation may impact the quality of training data
   - Potential artifacts or errors from the translation process could affect model performance

3. **Performance Gaps**
   - Notably low performance on AfriMGSM tasks indicates room for improvement
   - Further investigation needed to understand performance disparities across tasks

## Technical Requirements

The model requires the following framework versions:
- PyTorch: 2.4.1+cu121
- Transformers: 4.44.2
- PEFT: 0.12.0
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Usage Example

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "meta-llama/llama-8b-south-africa"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Example usage for text generation
text = "Translate to Xhosa: Hello, how are you?"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
result = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(result)
```

## License

This model is released under the Apache 2.0 license. The full license text can be found at https://www.apache.org/licenses/LICENSE-2.0.txt

## Acknowledgments

- Meta AI for the base LLaMA-3.1-8B-Instruct model
- Akash Network for providing computing resources
- Contributors to the Alpaca Cleaned dataset
- The African NLP community for benchmark datasets and evaluation tools