luel/gpt2-tigrinya-medium

Model Card for GPT-2 Tigrinya Medium

Model Summary

This is a GPT-2 model trained from scratch on Tigrinya text data. It was trained on 20.6 million tokens, primarily from news sources.

Model Description

• Model type: GPT-2
• Language: Tigrinya (ትግርኛ)
• Finetuned from model: Trained from scratch (no pre-training)

Model Architecture

• Parameters: 51.9M
• Context Window: 128 tokens
• Vocabulary Size: 52,000

Training Details

• Training regime: fp16 mixed precision
• Number of Epochs: 12
• Batch Size: 6 (with gradient accumulation steps of 8)
• Learning Rate: 5e-4

Evaluation

• Training Perplexity: 28.6
• Training Loss: 3.12

Usage

from transformers import pipeline
# Load the model
generator = pipeline('text-generation', model='luel/gpt2-tigrinya-medium')

prompt = "ክልል ትግራይ"
# Generate text
text = generator(prompt, max_length=100)[0]['generated_text']
print(text)

Limitations

• Limited context window of 128 tokens.
• Best suited for medium-length Tigrinya text generation.
• Outputs should be reviewed for accuracy.