| --- |
| license: apache-2.0 |
| language: |
| - ar |
| - bn |
| - cs |
| - de |
| - es |
| - en |
| - el |
| - fr |
| - id |
| - it |
| - he |
| - hu |
| - ja |
| - kk |
| - ko |
| - ro |
| - ru |
| - az |
| - uk |
| - ur |
| - vi |
| - zh |
| - ms |
| - nl |
| - ne |
| - th |
| - tr |
| - pt |
| - pl |
| base_model: |
| - Qwen/Qwen2-7B |
| --- |
| |
|
|
| # Marco-LLM-GLO |
|
|
| ## Introduction |
|
|
| Marco-LLM is a series of advanced multilingual language models designed to bridge the performance gap between high-resource languages and low-resource languages. This repository contains the Marco-LLM base language model with 7 billion parameters. |
|
|
| The model has undergone extensive multilingual continual pretraining on a diverse dataset containing over 5 trillion tokens, with a particular focus on enhancing performance in low-resource languages while maintaining strong capabilities in high-resource languages like English and Chinese. |
|
|
| Compared to state-of-the-art open-source language models, Marco-LLM demonstrates significant improvements in multilingual tasks, including machine translation, question answering, and reasoning across multiple languages. |
| For more details, please refer to our [Hugging Face page](https://huggingface.co/AIDC-AI/Marco-LLM-GLO). |
|
|
| ## Model Details |
|
|
| Marco-LLM includes a 7B parameter model based on the Transformer architecture. The key features of Marco-LLM are: |
|
|
| - Multilingual Training: The model is trained on a large-scale multilingual dataset covering 29 languages, including both high-resource languages (e.g., English, Chinese) and low-resource languages (e.g., Kazakh, Nepali). |
|
|
| - Enhanced Tokenizer: An improved tokenizer is used to better handle multilingual data, ensuring higher efficiency and accuracy in tokenization. |
|
|
| - Post-Training: Marco-LLM supports various post-training methods, such as Supervised Fine-tuning (SFT) and Direct Preference Optimization (DPO), to further enhance performance for specific tasks and languages. |
| ## Usage |
|
|
| It is not advised to use the base language models for direct text generation tasks. Instead, it is recommended to apply post-training methods such as Supervised Fine-tuning (SFT), Reinforcement Learning with Human Feedback (RLHF), or continued pretraining to adapt the models for specific use cases. |
|
|
|
|
| ## Citation |
|
|
| If you find our work helpful, please give us a citation. |
| ``` |
| @article{unique_identifier, |
| |
| title={Marco-LLM: Bridging Languages via Massive Multilingual Training for Cross-Lingual Enhancement}, |
| |
| journal={arXiv}, |
| |
| volume={}, |
| |
| number={2412.04003}, |
| |
| year={2024}, |
| |
| url={https://arxiv.org/abs/2412.04003} |
| |
| } |
| ``` |
