You need to agree to share your Hugging Face information to access this model
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
Lara Mini base is released under the CC BY-NC-SA 4.0 license, for non-commercial use only. Access is granted for non-commercial purposes, such as research and reproducibility studies. Please share your name and affiliation: each request is reviewed by our team.
Log in or Sign Up to review the conditions and access this model content.
Lara Mini Base
Read more in our blog post.
Lara Mini base is a multilingual mixture-of-experts base model by Translated, with 7.2B total and 0.6B active parameters per token, pretrained from scratch on 6T tokens and extended to a context of 128k tokens.
Overall and domain averages of Lara Mini base and of the four best other models with up to 2B active parameters on
each average.
- Multilingual by design. About a third of the training data (2.18T tokens) is in 44 languages besides English (counting Simplified and Traditional Chinese separately), including all 24 official languages of the European Union.
- Strong for its size. Among the 36 open base models with up to 2B active parameters that we evaluated, it has the highest average over English, multilingual, math and code benchmarks (57.0), ahead of models trained on 4 to 6 times as many tokens.
- Fast. It needs the compute per token of a 0.6B dense model. For a single user on one A100 GPU, it generates about 1,100 characters per second, at least 1.8 times as fast as every model in our comparison with a higher average.
- Long context with a small state. Only 7 of its 34 layers use global attention; the others are Gated DeltaNet and sliding-window attention layers, whose state does not grow with the input.
This is a base model: it continues text. It is not tuned to follow instructions or to chat, and it has no safety tuning.
Model details
| Developer | Translated SRL, the copyright holder and licensor |
| Model type | Decoder-only mixture of experts base model; hybrid of Gated DeltaNet, sliding-window and global attention |
| Parameters | 7.2B total, 0.6B active per token (embeddings included) |
| Layers | 34: 14 Gated DeltaNet (GDN), 13 sliding-window attention (window 512, RoPE), 7 global attention (no positional encoding) |
| Layer pattern | GDN, sliding, GDN, global, then (sliding, GDN, sliding, GDN, global) × 6 |
| Hidden size | 1,024 |
| Attention | 16 query heads, 4 key-value heads, head dimension 128, QK-norm |
| Gated DeltaNet | 8 key heads, 16 value heads, head dimension 128, short convolution of width 4 |
| Mixture of experts | In every layer: 128 routed experts (top-4, fp32 router) and 1 shared expert, all SwiGLU with hidden size 512 |
| Vocabulary | 151,936 tokens; input and output embeddings tied |
| Context length | 4,096 tokens in the first three stages; 131,072 after the long-context stage |
| Precision | bf16 |
| Training data | 6T tokens of public data: English and multilingual text, code, math and SFT-like data, then 100B tokens in the long-context stage |
| Languages | English and 44 others (Simplified and Traditional Chinese counted separately), which together cover all 24 official EU languages (see below) |
| License | CC BY-NC-SA 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/) |
| Intended use | Research, and a starting point for fine-tuning and post-training; a base model, not tuned to follow instructions or for safety |
| Limitations | See Limitations |
Languages. English, Arabic, Azerbaijani, Bengali, Bosnian, Bulgarian, Catalan, Chinese (Simplified and Traditional), Croatian, Czech, Danish, Dutch, Estonian, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Irish, Italian, Japanese, Korean, Latvian, Lithuanian, Malay, Maltese, Norwegian, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Thai, Turkish, Ukrainian and Vietnamese.
Training
Data. All training data come from publicly available datasets (some of them released under access agreements), which we deduplicated and filtered into a pool of 13.8T tokens. Training read 6T tokens from this pool, 97% of them unique. The anneal stage skips the training sequences that contain documents matching items of our evaluation benchmarks.
| Domain | Main sources | Tokens | Share |
|---|---|---|---|
| English | Nemotron-CC and its synthetic data, FineWeb-Edu, FinePDFs, FineWiki | 2.49T | 41.4% |
| Multilingual | FineWeb-2, FineWeb2-HQ, HPLT 3.0, CulturaX, FinePDFs, FineWiki, synthetic question-answer data | 2.18T | 36.4% |
| Code | Nemotron-CC code pages, The Stack v3 (permissive files), Nemotron synthetic code | 0.77T | 12.8% |
| Math | FineMath, MegaMath, Nemotron-CC-Math | 0.35T | 5.9% |
| SFT-like | Nemotron pretraining collections (question-answer, math and code examples) | 0.21T | 3.5% |
We release no training data with this model.
Stages. Pretraining has three stages (main, transition and anneal) over 6T tokens at a sequence length of 4,096. Each stage moves toward higher-quality, question-answer, code and math data, while multilingual text stays at about a third. A fourth stage of 100B tokens, in sequences of 32,768 and then 131,072 tokens, extends the context to 128k.
Recipe. Muon for the weight matrices and AdamW for the other parameters, peak learning rate 8e-4, global batch of about 8.4M tokens, constant learning rate for 90% of training and then a decay to zero.
Evaluation
We evaluated Lara Mini base and 56 open base models ourselves, all with the same prompts, settings and answer extraction (lm-evaluation-harness, few-shot or zero-shot, no chat template).
- English: NLI, PAWS, Belebele, COPA, PIQA, HellaSwag, WinoGrande, ARC-Challenge, MMLU, MMLU-Pro
- Multilingual: XNLI, PAWS-X, Belebele, XCOPA, Global PIQA, T-HellaSwag, INCLUDE, Global MMLU, averaged over the languages of Lara Mini base
- Math: GSM8K, MATH, MGSM
- Code: MBPP, MBPP+, CRUXEval-I, CRUXEval-O
Overall is the mean of the four domain averages. Models are grouped by active parameters.
| Model | Active | Total | Tokens | English | Multilingual | Math | Code | Overall |
|---|---|---|---|---|---|---|---|---|
| Lara Mini base | 0.6B | 7.2B | 6T | 66.4 | 56.7 | 59.9 | 45.1 | 57.0 |
| granite-4.0-h-tiny | 1.0B | 6.9B | 22.5T | 66.2 | 51.8 | 49.4 | 48.4 | 53.9 |
| Qwen3-1.7B | 1.7B | 1.7B | 36T | 63.8 | 50.3 | 54.6 | 46.6 | 53.8 |
| Falcon-H1-1.5B-Deep | 1.6B | 1.6B | 3T | 65.3 | 45.2 | 55.2 | 47.2 | 53.2 |
| Hunyuan-1.8B | 1.8B | 1.8B | -- | 62.7 | 44.3 | 52.0 | 49.7 | 52.2 |
| LFM2.5-8B-A1B | 1.7B | 8.5B | 38T† | 65.7 | 48.6 | 52.3 | 36.6 | 50.8 |
| Trinity-Nano | 1.0B | 6.1B | 10T | 68.4 | 47.4 | 45.0 | 42.0 | 50.7 |
| Qwen3.5-2B | 1.9B | 1.9B | -- | 63.3 | 50.0 | 45.6 | 36.7 | 48.9 |
| Gemma-4-E2B | 2.0B | 5.1B | -- | 63.3 | 53.3 | 18.0 | 32.8 | 41.9 |
| Larger models | ||||||||
| Apertus-v1.5-8B | 8.0B | 8.0B | 19T | 69.0 | 59.2 | 50.6 | 44.3 | 55.8 |
| Gemma-4-E4B | 4.0B | 7.9B | -- | 70.1 | 61.1 | 44.4 | 46.0 | 55.4 |
| Ministral-3-3B | 3.4B | 3.4B | 1-3T† | 67.7 | 55.4 | 49.9 | 42.3 | 53.8 |
| Llama-3.1-8B | 8.0B | 8.0B | 15T | 69.6 | 55.2 | 36.6 | 43.6 | 51.2 |
Tokens: pretraining tokens reported by each developer (--: not reported; †: distilled from a larger model during pretraining, wholly or in part). For Lara Mini base, the 6T of the first three stages; the long-context stage adds 100B. Gemma-4 models are listed at their effective sizes.
Speed. We measured the generation speed of 45 of these models on one A100 GPU, in bf16 without quantization. Speed is in characters per second, so that models with tokenizers of different density compare fairly. Every model with a higher Overall average than Lara Mini base generates text at least 1.8 times more slowly.
Overall average against generation speed for a single user (characters per second, log scale) for Lara Mini base
and 45 open base models; the line is the Pareto frontier.
Long context. RULER (13 English tasks), ONERULER (4 retrieval tasks in 26 languages) and HELMET (4 categories), with tasks whose score does not depend on the text a base model generates after its answer.
| Model | Active | RULER 32k | RULER 128k | ONERULER 32k | ONERULER 128k | HELMET 32k | HELMET 128k |
|---|---|---|---|---|---|---|---|
| Lara Mini base | 0.6B | 76.6 | 63.7 | 90.1 | 80.9 | 63.0 | 54.7 |
| Trinity-Nano | 1.0B | 71.0 | 61.8 | 73.9 | 62.1 | 54.9 | 51.8 |
| granite-4.0-h-tiny | 1.0B | 29.5 | 26.4 | 19.5 | 14.1 | 41.0 | 36.3 |
| Falcon-H1-1.5B-Deep | 1.6B | 59.7 | 38.7 | 60.5 | 38.8 | 45.5 | 34.4 |
| Qwen3.5-2B | 1.9B | 83.7 | 73.8 | 87.7 | 77.4 | 64.0 | 56.8 |
| Gemma-4-E2B | 2.0B | 79.2 | 71.0 | 89.3 | 84.7 | 66.0 | 58.6 |
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
path = "translated/Lara-4-Mini-Base"
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda")
inputs = tokenizer("La Divina Commedia è un poema scritto da", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))
- Requires
transformers>=5.14(tested up to 5.22.2). - Install flash-linear-attention for fast Gated DeltaNet kernels on GPU. Without it, a plain PyTorch path is used, which is exact but slow on long inputs.
- The model was trained without a BOS token: do not prepend one. The tokenizer already does not add it.
- To fine-tune, load with
output_router_logits=True. The loss then includes the router load-balancing loss (router_aux_loss_coef, 0.001 as in pretraining), which keeps the experts balanced.
Limitations
- A base model: it continues text. It does not follow instructions or hold a conversation, and it has no safety tuning. It can produce text that is wrong, biased or offensive.
- Behavior beyond 131,072 tokens is untested.
Third-party terms that may apply
Part of the training data is synthetic text that NVIDIA generated with third-party models. The cards of these datasets state that a model trained on them "may be subject to redistribution and use requirements" in the licenses of the models that generated the data:
- Qwen. Nemotron-CC v2 and
Nemotron-Pretraining-SFT v1 contain data
generated with Qwen2.5 and Qwen3 models, and point to the
Qwen License Agreement (copy in
LICENSES/Qwen-LICENSE-AGREEMENT.txt). Built with Qwen. - DeepSeek. Nemotron-CC v2, Nemotron-Pretraining-SFT v1 and the Multiple-Choice and Generative subsets of
Nemotron-Pretraining-Specialized v1.1
and v1.2 contain data generated
with DeepSeek models, and point to the
DeepSeek License Agreement (copy in
LICENSES/DeepSeek-LICENSE-MODEL.txt). Do not use Lara Mini base, or models derived from it, for the uses listed in Attachment A of that agreement, and pass this restriction on when you distribute them. - Nemotron-CC-Math v1 contains data generated with Phi-4, whose license is MIT.
License
Lara Mini base is released under the
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
license (CC BY-NC-SA 4.0); see LICENSE. You may share and adapt it for non-commercial purposes, giving
appropriate credit, and must distribute adaptations under the same license. The code and the tokenizer are
under the Apache License 2.0 (see NOTICE and LICENSE-APACHE-2.0).
For commercial use or other uses not covered by this license, we are open to licensing Lara Mini base under more permissive terms. If you are interested, please contact us at research@translated.com.
Citation
@techreport{translated2026laraminibase,
title = {Lara Mini Base},
author = {{Translated, Lara Team}},
institution = {Translated},
year = {2026},
month = nov,
type = {Technical Report},
url = {https://huggingface.co/translated/Lara-4-Mini-Base}
}
- Downloads last month
- -