|
--- |
|
language: |
|
- ms |
|
- id |
|
tags: |
|
- roberta |
|
- fine-tuned |
|
- transformers |
|
- bert |
|
- masked-language-model |
|
license: apache-2.0 |
|
model_type: roberta |
|
metrics: |
|
- accuracy |
|
base_model: |
|
- mesolitica/roberta-base-bahasa-cased |
|
pipeline_tag: token-classification |
|
--- |
|
|
|
# Fine-tuned RoBERTa on Malay Language |
|
|
|
This model is a fine-tuned version of the `mesolitica/roberta-base-bahasa-cased` model, specifically trained on a custom Malay dataset. The model is fine-tuned for **Masked Language Modeling (MLM)** on normalized Malay sentences. |
|
|
|
## Model Description |
|
|
|
This model is based on the **RoBERTa** architecture, a robustly optimized version of BERT. It was pre-trained on a large corpus of text in the Malay language and then fine-tuned on a specialized dataset consisting of normalized Malay sentences. The fine-tuning task involved predicting masked tokens in sentences, which is typical for masked language modeling tasks. |
|
|
|
### Training Details |
|
|
|
- **Pre-trained Model**: `mesolitica/roberta-base-bahasa-cased` |
|
- **Task**: Masked Language Modeling (MLM) |
|
- **Training Dataset**: Custom dataset of Malay sentences |
|
- **Training Duration**: 3 epochs |
|
- **Batch Size**: 16 per device |
|
- **Learning Rate**: 1e-6 |
|
- **Optimizer**: AdamW |
|
- **Evaluation**: Evaluated every 200 steps |
|
|
|
## Training and Validation Loss |
|
|
|
The following table shows the training and validation loss at each evaluation step during the fine-tuning process: |
|
|
|
| Step | Training Loss | Validation Loss | |
|
|-------|---------------|-----------------| |
|
| 200 | 0.069000 | 0.069317 | |
|
| 800 | 0.070100 | 0.067430 | |
|
| 1400 | 0.069000 | 0.066185 | |
|
| 2000 | 0.037900 | 0.066657 | |
|
| 2600 | 0.040200 | 0.066858 | |
|
| 3200 | 0.041800 | 0.066634 | |
|
| 3800 | 0.023700 | 0.067717 | |
|
| 4400 | 0.024500 | 0.068275 | |
|
| 5000 | 0.024500 | 0.068108 | |
|
|
|
|
|
### Observations |
|
- The training loss consistently decreased over time, with notable reductions in the earlier steps. |
|
- The validation loss showed slight fluctuations, but overall, it remained relatively stable after the first few thousand steps. |
|
- The model demonstrated good convergence as training progressed, with a sharp drop in the training loss after the first few steps. |
|
|
|
|
|
## Intended Use |
|
This model is intended for tasks such as: |
|
- **Masked Language Modeling (MLM)**: Fill in the blanks for masked tokens in a Malay sentence. |
|
- **Text Generation**: Generate plausible text given a context. |
|
- **Text Understanding**: Extract contextual meaning from Malay sentences. |
|
|
|
## Updated News |
|
- This model is used for the research paper : **"Mitigating Linguistic Bias between Malay and Indonesian Languages using Masked Language Models"** which been accepted as a short paper (poster presentation) for the **Research Track** at **DASFAA 2025**. |
|
- **Author**: Ferdinand Lenchau Bit, Iman Khaleda binti Zamri, Amzine Toushik Wasi, Taki Hasan Rafi, and Dong-Kyu Chae (Department of Computer Science, Hanyang University, Seoul, South Korea) |
