---
datasets:
- cobrayyxx/FLEURS_ID-EN
language:
- id
- en
metrics:
- bleu
- chrf
base_model:
- facebook/nllb-200-distilled-600M
pipeline_tag: translation
---
## Model description

This model is a fine-tuned version of [facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) on an Indonesian-English [CoVoST2](https://huggingface.co/datasets/cobrayyxx/COVOST2_ID-EN) dataset.

## Intended uses & limitations

This model is used to predict the translation of Indonesian Transcription.

## How to Use
This is how to use the model with Faster-Whisper.
1. Convert the model into the CTranslate2 format with float16 quantization.
   ```
   !ct2-transformers-converter --model cobrayyxx/nllb-indo-en-covost2 --quantization float16 --output_dir ct2/ct2-nllb-indo-en-float16
   ```
2. Load the converted model using `ctranslate2` library.
   ```
    from faster_whisper import WhisperModel
    import os
   
    ct2_model_name = "ct2-nllb-indo-en-float16"
    
    ct_model_path = os.path.join("ct2", ct2_model_name)
    translator = ctranslate2.Translator(ct_model_path, device=device)
   ```
3. Download the SentencePiece model
    ```
    !wget https://s3.amazonaws.com/opennmt-models/nllb-200/flores200_sacrebleu_tokenizer_spm.model
    ```
4. Load the SentencePiece model
    ```
    import sentencepiece as spm

    sp_model_path = os.path.join(directory, "flores200_sacrebleu_tokenizer_spm.model")

    sp = spm.SentencePieceProcessor()
    sp.load(sp_model_path)
    ```
5. Now, the loaded model can be used.
   ```
    src_lang = "ind_Latn"
    tgt_lang = "eng_Latn"
    
    beam_size = 5
    
    source_sentences = lst_of_sentences
    
    source_sents = [sent.strip() for sent in source_sentences]
    target_prefix = [[tgt_lang]] * len(source_sents)
    
    # Chunk source sentences into subword
    source_sents_subworded = sp.encode_as_pieces(source_sents)
    source_sents_subworded = [[src_lang] + sent + ["</s>"] for sent in source_sents_subworded]
    
    # Translate the source sentences
    translations = translator.translate_batch(source_sents_subworded,
                                              batch_type="tokens",
                                              max_batch_size=2024,
                                              beam_size=beam_size,
                                              target_prefix=target_prefix)
    translations = [translation.hypotheses[0] for translation in translations]
    
    # Merge all of the subword in the target sentences
    translations_desubword = sp.decode(translations)
    translations_desubword = [sent[len(tgt_lang):].strip() for sent in translations_desubword]
   ```
  
    Note: If you faced the kernel error everytime running the code above. You have to install `nvidia-cublas` and `nvidia-cudnn`
    
    ```
    apt update
    apt install libcudnn9-cuda-12
    ```
  
    and Install the library using pip. [Read The Documentation for more.](https://github.com/SYSTRAN/faster-whisper?tab=readme-ov-file#gpu)
    ```
    pip install nvidia-cublas-cu12 nvidia-cudnn-cu12==9.*
  
    export LD_LIBRARY_PATH=`python3 -c 'import os; import nvidia.cublas.lib; import nvidia.cudnn.lib; print(os.path.dirname(nvidia.cublas.lib.__file__) + ":" + os.path.dirname(nvidia.cudnn.lib.__file__))'`
    ```
    Special thanks to [Yasmin Moslem](https://huggingface.co/ymoslem) for her help in resolving this.
   
## Training procedure

### Training Results

| Epoch | Training Loss | Validation Loss | BLEU |
|-------|--------------|----------------|------|
| 1     | 0.119100     | 0.048539       | 60.267190 |
| 2     | 0.020900     | 0.044844       | 59.821654 |
| 3     | 0.014600     | 0.048637       | 60.185481 |
| 4     | 0.007200     | 0.052005       | 60.150045 |
| 5     | 0.005100     | 0.054909       | 59.938441 |
| 6     | 0.004500     | 0.056668       | 60.032409 |
| 7     | 0.003800     | 0.058903       | 60.176242 |
| 8     | 0.002900     | 0.059880       | 60.168394 |
| 9     | 0.002400     | 0.060914       | 60.280851 |

## Model Evaluation

The performance of the baseline and fine-tuned model were evaluated using the BLEU and CHRF++ metrics on the validation dataset. 
This fine-tuned model shows some improvement over the baseline model.
| Model                 | BLEU  | ChrF++ |
|-----------------------|------:|-------:|
| Baseline             | 50.91 | 68.1  |
| Fine-Tuned          |58.3 |  73.62 |
### Evaluation details
- BLEU: Measures the overlap between predicted and reference text based on n-grams.
- CHRF: Uses character n-grams for evaluation, making it particularly suitable for morphologically rich languages.

# Credits
Huge thanks to [Yasmin Moslem](https://huggingface.co/ymoslem) for mentoring me.