|
--- |
|
license: apache-2.0 |
|
language: |
|
- ar |
|
- bn |
|
- cs |
|
- de |
|
- es |
|
- en |
|
- el |
|
- fr |
|
- id |
|
- it |
|
- he |
|
- hu |
|
- ja |
|
- kk |
|
- ko |
|
- ro |
|
- ru |
|
- az |
|
- uk |
|
- ur |
|
- vi |
|
- zh |
|
- ms |
|
- nl |
|
- ne |
|
- th |
|
- tr |
|
- pt |
|
- pl |
|
base_model: |
|
- Qwen/Qwen2-7B |
|
--- |
|
|
|
|
|
# Marco-LLM-GLO |
|
|
|
## Introduction |
|
|
|
Marco-LLM is a series of advanced multilingual language models designed to bridge the performance gap between high-resource languages and low-resource languages. This repository contains the Marco-LLM base language model with 7 billion parameters. |
|
|
|
The model has undergone extensive multilingual continual pretraining on a diverse dataset containing over 5 trillion tokens, with a particular focus on enhancing performance in low-resource languages while maintaining strong capabilities in high-resource languages like English and Chinese. |
|
|
|
Compared to state-of-the-art open-source language models, Marco-LLM demonstrates significant improvements in multilingual tasks, including machine translation, question answering, and reasoning across multiple languages. |
|
For more details, please refer to our [Hugging Face page](https://huggingface.co/AIDC-AI/Marco-LLM-GLO). |
|
|
|
## Model Details |
|
|
|
Marco-LLM includes a 7B parameter model based on the Transformer architecture. The key features of Marco-LLM are: |
|
|
|
- Multilingual Training: The model is trained on a large-scale multilingual dataset covering 29 languages, including both high-resource languages (e.g., English, Chinese) and low-resource languages (e.g., Kazakh, Nepali). |
|
|
|
- Enhanced Tokenizer: An improved tokenizer is used to better handle multilingual data, ensuring higher efficiency and accuracy in tokenization. |
|
|
|
- Post-Training: Marco-LLM supports various post-training methods, such as Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO), to further enhance performance for specific tasks and languages. |
|
## Usage |
|
|
|
It is not advised to use the base language models for direct text generation tasks. Instead, it is recommended to apply post-training methods such as Supervised Fine-tuning (SFT), Reinforcement Learning with Human Feedback (RLHF), or continued pretraining to adapt the models for specific use cases. |
|
|
|
|
|
## Citation |
|
|
|
If you find our work helpful, please give us a citation. |
|
``` |
|
@article{unique_identifier, |
|
|
|
title={Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement}, |
|
|
|
journal={arXiv}, |
|
|
|
volume={}, |
|
|
|
number={2412.04003}, |
|
|
|
year={2024}, |
|
|
|
url={https://arxiv.org/abs/2412.04003} |
|
|
|
} |
|
``` |
