README.md

metadata

license: cc-by-sa-4.0
base_model:
  - facebook/nllb-200-3.3B
pipeline_tag: translation

AraDiCE-msa-to-lev: An MSA to Levantine Machine Translation Model Based on NLLB-3.3B

This repository includes an MSA-to-LEV machine translation model. This model was used to curate dialectal benchmarks for the AraDiCE paper. The human post-edited benchmarks can be foundhere.

Sample Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("QCRI/AraDiCE-msa-to-lev")
model = AutoModelForSeq2SeqLM.from_pretrained("QCRI/AraDiCE-msa-to-lev")

article = "يظهر سلف الأدب المكسيكي في آداب المستعمرات الأصلية في أمريكا الوسطى"
inputs = tokenizer(article, return_tensors="pt")

translated_tokens = model.generate(
    **inputs, forced_bos_token_id=tokenizer.convert_tokens_to_ids("ajp_Arab"), max_length=30
)
translation = tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0]
print(translation)

License

The model is distributed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). The full license text can be found in the accompanying licenses_by-nc-sa_4.0_legalcode.txt file.

Citation

Please find the paperhere.

@article{mousi2024aradicebenchmarksdialectalcultural,
      title={{AraDiCE}: Benchmarks for Dialectal and Cultural Capabilities in LLMs},
      author={Basel Mousi and Nadir Durrani and Fatema Ahmad and Md. Arid Hasan and Maram Hasanain and Tameem Kabbani and Fahim Dalvi and Shammur Absar Chowdhury and Firoj Alam},
      year={2024},
      publisher={arXiv:2409.11404},
      url={https://arxiv.org/abs/2409.11404},
}