README.md

---
license: apache-2.0
language:
- sv
- da
- "no"
- nn
- fi
- is
- en
tags:
- translation
- nordic
- scandinavian
- machine-translation
- bifrost
- nodenestor
pipeline_tag: translation
metrics:
- chrf
---

# Bifrost 1.2B

A from-scratch 1.2B-parameter translation model for the **Nordic languages ↔ English**:
Swedish (`sv`), Danish (`da`), Norwegian Bokmål (`nb`), Norwegian Nynorsk (`nn`),
Finnish (`fi`), and Icelandic (`is`), plus cross-Nordic directions.

On FLORES-200 devtest it **beats NLLB-200-3.3B and TranslateGemma-12B on the
English→Nordic average** — at a fraction of their size.

The teacher of the **Bifrost** Nordic-translation family from
[NodeNestor](https://nodenestor.com) — for a ~3× smaller/faster distilled option see
[Bifrost Flash 430M](https://huggingface.co/NodeNestor/bifrost-flash-430m).

## Results — FLORES-200 devtest, chrF++ (sacrebleu, `word_order=2`, n=500)

**Headline — English→Nordic average:**

| Model | Params | en→Nordic chrF++ |
|---|---|---|
| **Nordic Translator (this model)** | **1.2B** | **57.4** |
| NLLB-200-3.3B | 3.3B | 56.1 |
| TranslateGemma-12B | 12B | 55.7 |

**Group averages:**

| Direction group | chrF++ |
|---|---|
| English → Nordic | 57.4 |
| Nordic → English | 63.6 |
| Nordic ↔ Nordic | 54.5 |
| **Overall** | **58.1** |

**Per-direction (chrF++):**

| Dir | score | | Dir | score |
|---|---|---|---|---|
| en→sv | 63.8 | | sv→en | 67.4 |
| en→da | 65.4 | | da→en | 69.3 |
| en→nb | 58.2 | | nb→en | 64.9 |
| en→nn | 57.8 | | nn→en | 68.9 |
| en→fi | 50.1 | | fi→en | 55.3 |
| en→is | 49.2 | | is→en | 55.8 |
| sv→da | 62.5 | | da→sv | 62.7 |
| sv→fi | 51.0 | | fi→sv | 50.5 |
| nb→nn | 53.0 | | nn→nb | 55.5 |
| fi→da | 50.9 | | is→sv | 50.1 |

Strongest relative to the references on the low-resource directions (Nynorsk,
Icelandic). NLLB-3.3B still leads on several →English directions and Finnish.

## Usage

The model expects a control-token prompt and is decoded greedily:

```
[BOS] [<2{tgt_lang}>] {source_token_ids} [<eos_src>]   →   generate until [EOS]
```

The **target-language control token placed right after `[BOS]` selects the output
language** — the source language is inferred. Control-token IDs (above the 65000
SentencePiece vocab):

| token | id | | token | id |
|---|---|---|---|---|
| `<2en>` | 65000 | | `<2nn>` | 65004 |
| `<2sv>` | 65001 | | `<2fi>` | 65005 |
| `<2da>` | 65002 | | `<2is>` | 65006 |
| `<2nb>` | 65003 | | `<eos_src>` | 65007 |

`[BOS]=1`, `[EOS]=2`. Tokenizer: `nordic_unigram_65k.model` (SentencePiece, 65000
pieces + 8 control tokens = vocab 65008).

The weights ship as `model.safetensors`, with a self-contained pure-PyTorch
implementation in `modeling_nordic.py` (no training-stack dependencies). Three ways
to run it:

**1. Standalone (pure torch, KV-cached):**

```python
import torch, sentencepiece as spm
from modeling_nordic import NordicTranslator

sp = spm.SentencePieceProcessor(); sp.load("nordic_unigram_65k.model")
LANG = {"en":65000,"sv":65001,"da":65002,"nb":65003,"nn":65004,"fi":65005,"is":65006}

model = NordicTranslator.from_checkpoint("model.safetensors", device="cuda")
ids = model.translate(sp.encode("Hello, how are you?", out_type=int), LANG["sv"])
print(sp.decode(ids))     # -> Hej, hur är det med er?
```

**2. HuggingFace (`trust_remote_code`):**

```python
from transformers import AutoModelForCausalLM
import torch, sentencepiece as spm
sp = spm.SentencePieceProcessor(); sp.load("nordic_unigram_65k.model")
m = AutoModelForCausalLM.from_pretrained(".", trust_remote_code=True,
                                         dtype=torch.bfloat16).cuda().eval()
ids = [1, 65001] + sp.encode("Hello, how are you?", out_type=int) + [65007]   # 65001=<2sv>
out = m.generate(torch.tensor([ids]).cuda(), max_new_tokens=128, do_sample=False, eos_token_id=2)
print(sp.decode([t for t in out[0, len(ids):].tolist() if t < 65000]))
```

**3. vLLM** (custom architecture — register the included plugin): see
`vllm_nordic.py` + `vllm_pkg/` and `example_vllm.py`. Install the plugin
(`pip install -e vllm_pkg`) inside a vLLM environment, then serve with
`--skip-tokenizer-init` and feed control-token prompts.

The control-token prompt is `[BOS] [<2{tgt}>] {source_ids} [<eos_src>]` → generate
until `[EOS]`; decode only ids `< 65000`. The FLORES numbers above were produced with
the batched, KV-cached standalone path.

## Model details

- **Architecture:** a grouped-query-attention (GQA) decoder. 18 layers,
  hidden 2048, FFN 6144 (SwiGLU), 16 query heads / 4 KV heads, head dim 128, RoPE
  (θ=500000, partial 0.25), RMSNorm, parallel residual, fused QKV. ~1.2B params.
- **Context length:** 4096 tokens (trained and evaluated at 4096; longer inputs truncate).
- **Precision:** bf16.
- **Vocab:** 65008 (`nordic_unigram_65k` SentencePiece + 8 control tokens).

## Training

- **From scratch.** A ~**120B-token run**: a ~19B-token trunk, then **+~100B
  tokens of continued training** (clean data, cosine schedule with a monolingual
  floor + anneal). The **released checkpoint is ~96B into the long run (~115B
  cumulative)** — on the cosine tail, so quality ≈ the 100B point.
- **Data:** parallel + monolingual Nordic/English (Wikipedia parallel, DCLM en↔Nordic,
  Aya cross-lingual, FineWeb-Edu, Nemotron-CC), balanced en↔Nordic blend.
- **Objective:** next-token cross-entropy on the target side.

## Limitations

- Trained at **4096-token context**; longer inputs are truncated.
- **Finnish and Icelandic** (en→) are the weakest directions — lower-resource,
  morphologically hard.
- Greedy decoding; no built-in length/formatting control beyond the prompt.
- Not instruction-tuned — it is a dedicated translation model, not a chat model.
- May produce occasional off-target output on the hardest low-resource pairs.

## Acknowledgments

- **Tokenizer** (`nordic_unigram_65k`) developed by a collaborator; included here with
  permission.

## Citation

```bibtex
@misc{nodenestor_bifrost_1.2b_2026,
  title  = {Bifrost 1.2B},
  author = {Nilsson, Ludvig},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/NodeNestor/bifrost-1.2b}},
  note   = {NodeNestor}
}
```