CAMeL-Lab/arabart-qalb15-gec-ged-13

AraBART+Morph+GEC¹³ QALB-2015 Model

Model description

AraBART+Morph+GEC¹³ is a Modern Standard Arabic (MSA) grammatical error correction (GEC) model that was built by fine-tuning the AraBART model. For the fine-tuning, we used the QALB-2015 and QALB-2015 datasets. Please note that this model was fine-tuned on morphologically preprocessed text. Our fine-tuning procedure and the hyperparameters we used can be found in our paper "Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation." Our fine-tuning code and data can be found here.

Intended uses

You can use the AraBART+Morph+GEC¹³ model as part of an extended version of the transformers that we make publicly available. The GEC model is intended to be used with this GED model as we outlined in the example below. We used this GEC model to report results on the QALB-2015-L2 dev and test sets in our paper.

How to use

To use the model with our extended version of transformers:

from transformers import AutoTokenizer, BertForTokenClassification, MBartForConditionalGeneration
from camel_tools.disambig.bert import BERTUnfactoredDisambiguator
from camel_tools.utils.dediac import dediac_ar
import torch.nn.functional as F
import torch

bert_disambig = BERTUnfactoredDisambiguator.pretrained()

ged_tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/camelbert-msa-qalb15-ged-13')
ged_model = BertForTokenClassification.from_pretrained('CAMeL-Lab/camelbert-msa-qalb15-ged-13')

gec_tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/arabart-qalb15-gec-ged-13')
gec_model = MBartForConditionalGeneration.from_pretrained('CAMeL-Lab/arabart-qalb15-gec-ged-13')

text = 'و قال له انه يحب اكل الطعام بكثره .'

# morph processing the input text
text_disambig = bert_disambig.disambiguate(text.split())
morph_pp_text = [dediac_ar(w_disambig.analyses[0].analysis['diac']) for w_disambig in text_disambig]
morph_pp_text = ' '.join(morph_pp_text)

# GED tagging
inputs = ged_tokenizer([morph_pp_text], return_tensors='pt')
logits = ged_model(**inputs).logits
preds = F.softmax(logits, dim=-1).squeeze()[1:-1]
pred_ged_labels = [ged_model.config.id2label[p.item()] for p in torch.argmax(preds, -1)]

# Extending GED label to GEC-tokenized input
ged_label2ids = gec_model.config.ged_label2id
tokens, ged_labels = [], []

for word, label in zip(morph_pp_text.split(), pred_ged_labels):
    word_tokens = gec_tokenizer.tokenize(word)
    if len(word_tokens) > 0:
         tokens.extend(word_tokens)
         ged_labels.extend([label for _ in range(len(word_tokens))])


input_ids = gec_tokenizer.convert_tokens_to_ids(tokens)
input_ids = [gec_tokenizer.bos_token_id] + input_ids + [gec_tokenizer.eos_token_id]

label_ids = [ged_label2ids.get(label, ged_label2ids['<pad>']) for label in ged_labels]
label_ids = [ged_label2ids['UC']] + label_ids + [ged_label2ids['UC']]
attention_mask = [1 for _ in range(len(input_ids))]


gen_kwargs = {'num_beams': 5, 'max_length': 100,
              'num_return_sequences': 1,
              'no_repeat_ngram_size': 0, 'early_stopping': False,
              'ged_tags': torch.tensor([label_ids]),
              'attention_mask': torch.tensor([attention_mask])
              }

# GEC generation
generated = gec_model.generate(torch.tensor([input_ids]), **gen_kwargs)

generated_text = gec_tokenizer.batch_decode(generated,
                                            skip_special_tokens=True,
                                            clean_up_tokenization_spaces=False
                                            )[0]
  
print(generated_text) # وقال له أنه يحب أكل الطعام بكثرة .

Citation

@inproceedings{alhafni-etal-2023-advancements,
    title = "Advancements in {A}rabic Grammatical Error Detection and Correction: An Empirical Investigation",
    author = "Alhafni, Bashar  and
      Inoue, Go  and
      Khairallah, Christian  and
      Habash, Nizar",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.396",
    doi = "10.18653/v1/2023.emnlp-main.396",
    pages = "6430--6448",
    abstract = "Grammatical error correction (GEC) is a well-explored problem in English with many existing models and datasets. However, research on GEC in morphologically rich languages has been limited due to challenges such as data scarcity and language complexity. In this paper, we present the first results on Arabic GEC using two newly developed Transformer-based pretrained sequence-to-sequence models. We also define the task of multi-class Arabic grammatical error detection (GED) and present the first results on multi-class Arabic GED. We show that using GED information as auxiliary input in GEC models improves GEC performance across three datasets spanning different genres. Moreover, we also investigate the use of contextual morphological preprocessing in aiding GEC systems. Our models achieve SOTA results on two Arabic GEC shared task datasets and establish a strong benchmark on a recently created dataset. We make our code, data, and pretrained models publicly available.",
}