|
|
---
|
|
|
license: apache-2.0
|
|
|
datasets:
|
|
|
- faur-ai/fulg
|
|
|
language:
|
|
|
- ro
|
|
|
---
|
|
|
|
|
|
# LLMic Model Card
|
|
|
|
|
|
[LLMic: Romanian Foundation Language Model](https://arxiv.org/abs/2501.07721)
|
|
|
|
|
|
## Model Summary
|
|
|
|
|
|
LLMic is a bilingual Romanian-English foundation model. LLmic is a 3B
|
|
|
parameters dense decoder-only Transformer model based on Llama2.
|
|
|
|
|
|
## Architecture
|
|
|
|
|
|
| Parameter | Value |
|
|
|
|-----------|---------|
|
|
|
| Sequence Length | 2048 |
|
|
|
| Number of Layers | 24 |
|
|
|
| Embedding Size | 2,560 |
|
|
|
| FFN Hidden Size | 10,240 |
|
|
|
| Number of Heads | 20 |
|
|
|
| Number of KV Heads | 5 |
|
|
|
| Activation Function | SiLU |
|
|
|
| Position Encodings | RoPE (Θ=500,000) |
|
|
|
| Layer Norm | RMSNorm (ε=10⁻⁵) |
|
|
|
| Tied Embeddings | No |
|
|
|
|
|
|
## Intended Use
|
|
|
|
|
|
Our model is designed to accelerate research on Romanian language models, serving as a building block for generative AI applications.
|
|
|
|
|
|
## Use with transformers
|
|
|
|
|
|
```python
|
|
|
from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer
|
|
|
|
|
|
device = "cuda"
|
|
|
model_id = "faur-ai/LLMic"
|
|
|
prompt = "Capitala României este"
|
|
|
|
|
|
model = AutoModelForCausalLM.from_pretrained(model_id).to(device)
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
|
|
streamer = TextStreamer(tokenizer)
|
|
|
|
|
|
inputs = tokenizer.encode(
|
|
|
prompt,
|
|
|
add_special_tokens=False,
|
|
|
return_tensors='pt',
|
|
|
).to(device)
|
|
|
|
|
|
outputs = model.generate(
|
|
|
streamer=streamer,
|
|
|
input_ids=inputs,
|
|
|
temperature=0.8,
|
|
|
do_sample=True
|
|
|
)
|
|
|
```
|
|
|
|
|
|
## Data Overview
|
|
|
|
|
|
### Training Datasets
|
|
|
|
|
|
| Source | Size |
|
|
|
|---------|------|
|
|
|
| *Romanian (300B)* | |
|
|
|
| Web Sources | 621 GB |
|
|
|
| Discussions, Curated & Parallel | 10 GB |
|
|
|
| *English (700B)* | |
|
|
|
| FineWebEdu | -- |
|
|
|
| Dolma Subset | 109 GB |
|
|
|
|
|
|
#### Benchmark datasets
|
|
|
|
|
|
We evaluated LLMic on the WMT16 language translation benchmark for English-to-Romanian.
|
|
|
|
|
|
| Model | Score |
|
|
|
|--------|--------|
|
|
|
| LLMIC | 41.01 |
|
|
|
| mBART | 38.50 |
|
|
|
| Llama-3.1-8B-Instruct | 29.02 |
|
|
|
| RoMistral-7b-Instruct | 27.70 |
|
|
|
| RoLlama3-8b-Instruct | 27.31 |
|
|
|
| Mistral-7B-Instruct-v0.2 | 26.19 |
|
|
|
| RoGemma-7b-Instruct | 25.96 |
|
|
|
| Gemma-1.1-7b-it | 25.48 |
|
|
|
|
|
|
|
|
|
## Citation
|
|
|
|
|
|
**BibTeX:**
|
|
|
|
|
|
```
|
|
|
@misc{bădoiu2025llmicromanianfoundationlanguage,
|
|
|
title={LLMic: Romanian Foundation Language Model},
|
|
|
author={Vlad-Andrei Bădoiu and Mihai-Valentin Dumitru and Alexandru M. Gherghescu and Alexandru Agache and Costin Raiciu},
|
|
|
year={2025},
|
|
|
eprint={2501.07721},
|
|
|
archivePrefix={arXiv},
|
|
|
primaryClass={cs.CL},
|
|
|
url={https://arxiv.org/abs/2501.07721},
|
|
|
}
|
|
|
```
|
|
|
|
