---
license: apache-2.0
datasets:
- Hailay/TigQA
- masakhane/masakhaner2
language:
- am
- ti
metrics:
- accuracy
- f1
base_model:
- FacebookAI/xlm-roberta-base
pipeline_tag: zero-shot-classification
library_name: transformers
---

# XLM-R and EXLMR Model
We introduce the EXLMR model, an extension of XLM-R, which expands its tokenizer vocabulary to incorporate new languages and alleviate out-of-vocabulary (OOV) issues. We initialize the embeddings for the newly added vocabulary in a way that allows the model to leverage this newly added vocabularies effectively. Our approach not only benefits low-resource languages but also improves performance on high-resource languages, that were part of the original XLM-R model. 
## Model Overview

The **XLM-R** (Cross-lingual Language Model - RoBERTa) is a multilingual model trained on 100 languages. The **EXLMR** (Extended XLM-RoBERTa) is an extended version designed to improve performance on low-resource languages spoken in Ethiopia, including Amharic, Tigrinya, and Afaan Oromo.

## Model Details

- **Base Model**: XLM-R
- **Extended Version**: EXLMR
- **Languages Supported**: Amharic, Tigrinya, Afaan Oromo, and more
- **Training Data**: Trained on a large multilingual corpus

## Usage


EXLMR addresses tokenization issues inherent to the XLM-R model, such as out-of-vocabulary (OOV) tokens and over-tokenization, especially for low-resource languages.
Fine-tuning on specific datasets will help adapt the model to particular tasks and improve its performance. You can use this model with the `transformers` library for various NLP tasks. 
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
# Define the model checkpoint
checkpoint = "Hailay/EXLMR"  

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)


EXLMR has been designed to support underrepresented languages, particularly those spoken in Ethiopia (such as Amharic, Tigrinya, and Afaan Oromo). Like XLM-RoBERTa, EXLMR can be finetuned to handle multiple languages simultaneously, making it effective for cross-lingual tasks such as machine translation, multilingual text classification, and question answering. EXLMR-base follows the same architecture as RoBERTa-base, with 12 layers, 768 hidden dimensions, and 12 attention heads, totaling approximately 270M parameters.

|Model|Vocabulary Size|
|---|---|
|XLM-Roberta|250002|
|EXLMR|280147|