README.md

---
license: cc-by-sa-4.0
language:
- nb
- nn
- 'no'
pipeline_tag: token-classification
tags:
- part-of-speech-tagging
- morphological-analysis
- lemmatization
- universal-dependencies
- executorch
- norwegian
- bokmal
- nynorsk
- on-device
---

# Prism Norwegian (prism-no)

Norwegian UPOS tagging, morphological features, and lemmatization with
**calibrated confidences**, built for **on-device, fully offline** use.
One compact model (17.6 M parameters) covers both written standards —
Bokmål (`nb`) and Nynorsk (`nn`) — in a single set of weights; mixed
input is fine. Since 0.2.3 the artifact manifest also declares the
BCP 47 macrolanguage `no`, so hosts that match languages strictly
against `language_tags` accept plain-`no` documents without aliases
(0.2.3 is a manifest-only release: the model bytes are identical to
0.2.2, and all published quality and speed numbers apply unchanged).

**It beats UDPipe 2.17 on UPOS and lemmas on the official UD test
splits — at about one twentieth of UDPipe's model size** (fast
artifact; one tenth for fp32), running fully offline on a laptop CPU.

This repository mirrors the versioned release artifacts of the
[Prism project](https://github.com/dmlux/Prism). Prism ships native
runtimes for **Swift, C++, C, and Java/Kotlin** that read these
artifacts directly.

## Which folder to use

| Folder | Size | When to use |
| --- | ---: | --- |
| `prism-no-0.2.3-fast/` | ≈ 45 MB | **Recommended.** int8; up to 2× faster, development-split quality within 0.014 pp of fp32 |
| `prism-no-0.2.3/` | ≈ 94 MB | Bit-exact fp32 reference behind the published benchmark |

An application bundles exactly one folder. The folder is everything a
Prism runtime needs; point the tagger API at its local path:

```swift
let tagger = try PrismTagger(artifactURL: artifactFolder)                // Swift
```
```cpp
prism::tagger::Tagger tagger("prism-no-0.2.3-fast");                     // C++
```
```java
try (var tagger = PrismTagger.load(Path.of("prism-no-0.2.3-fast"))) {}   // Java
```

Quick starts for every binding: the
[project README](https://github.com/dmlux/Prism#quick-start). The
artifact contract (programs, `model.ptd` weights, tokenizer and label
schemas, checksums) is documented in
[docs/INTEGRATION.md](https://github.com/dmlux/Prism/blob/main/docs/INTEGRATION.md).

**Note:** these are [ExecuTorch](https://pytorch.org/executorch)
programs with the decoding policy and calibration baked in — not
`transformers`-loadable checkpoints. `vocabulary.json` is a standard
Hugging Face `tokenizer.json` and loads with the `tokenizers` library.

## Quality

Evaluated exactly once on the untouched official UD test splits against
UDPipe 2.17 (gold tokenization, official CoNLL definitions):

| Test F1 | Prism | UDPipe 2.17 |
| --- | ---: | ---: |
| Bokmål UPOS | **98.76%** | 98.57% |
| Bokmål Lemmas | **98.98%** | 98.87% |
| Bokmål UFeats | 97.20% | **97.59%** |
| Nynorsk UPOS | **98.77%** | 98.60% |
| Nynorsk Lemmas | **98.68%** | 98.56% |
| Nynorsk UFeats | 96.94% | **97.38%** |

Prism wins UPOS and lemmas on both written standards and stays behind
only on exact morphology bundles — from a model a twentieth of
UDPipe's size.

### fast versus fp32

The frozen test evaluation above is fp32; the fast artifact is
quality-gated on the development split (67,619 tokens across both
standards — the test splits are evaluated exactly once and stay
reserved for the fp32 benchmark). Accuracy with the identical
production decoding policy:

| Task | Standard | fp32 | fast | Delta |
| --- | --- | ---: | ---: | ---: |
| UPOS | nb | 99.1724% | 99.1641% | -0.0082 pp |
| UPOS | nn | 98.8384% | 98.8448% | +0.0064 pp |
| UFeats exact | nb | 97.9021% | 97.8883% | -0.0137 pp |
| UFeats exact | nn | 95.3408% | 95.3312% | -0.0096 pp |
| Lemma | nb | 99.2301% | 99.2246% | -0.0055 pp |
| Lemma | nn | 98.8672% | 98.8608% | -0.0064 pp |

Every delta is at most 0.014 percentage points — an order of magnitude
below seed-to-seed training variance. int8 costs no measurable
quality.

### Speed

Measured with the **reproducible benchmark suite** on the checked-in
CC0 example texts that ship inside the Prism repository
([`data/examples/`](https://github.com/dmlux/Prism/tree/main/data/examples))
— anyone can rerun the identical protocol on their own hardware.
Apple M4 Max, CPU only, release builds, median of 3 repetitions. The
benchmark document is the Bokmål text repeated seven times:
**385 sentences / 6,335 tokens**, roughly a book chapter:

| C++ end-to-end, 6,335-token document | fp32 | fast (int8) |
| --- | ---: | ---: |
| Raw text in, tagged sentences out | 3.63 s (1,813 tokens/s) | **1.94 s (3,543 tokens/s)** |
| Pretokenized input | 3.58 s (1,832 tokens/s) | **1.91 s (3,590 tokens/s)** |
| Tagger load (cold start) | 33 ms | 35 ms |

So the fast artifact tags a full chapter in under two seconds, a
single ~900-token text in well under half a second (Swift cross-check,
single pass: 387 ms wall, 2,101 tokens/s at lower batch saturation).
The model-independent stages are effectively free next to the forward
pass: runtime segmentation ≈ 8 M tokens/s, byte-level BPE
≈ 2.2–2.6 M tokens/s.

Reproduce it:

```bash
git clone https://github.com/dmlux/Prism.git
cmake -S cpp -B cpp/build -DCMAKE_BUILD_TYPE=Release -DPRISM_BENCHMARKS=ON
cmake --build cpp/build --target prism_benchmarks --parallel
cpp/build/prism_benchmarks --benchmark_repetitions=3 --benchmark_report_aggregates_only=true
```

Recorded run and protocol:
[docs/benchmarks/v0.3.0.md](https://github.com/dmlux/Prism/blob/main/docs/benchmarks/v0.3.0.md);
earlier records (measured on a private text before the reproducible
protocol existed):
[docs/benchmarks/prism-no-0.2.2.md](https://github.com/dmlux/Prism/blob/main/docs/benchmarks/prism-no-0.2.2.md).

## Model description

A 17.6 M-parameter encoder student (16-layer NorBERT4-xsmall backbone,
hidden 192) distilled from a NorBERT4-large teacher, with a character
CNN feeding
morphology and lemma heads, a structured morphology decoder, and
per-head temperature calibration (UPOS ECE 0.0017). Trained on the
official UD gold treebanks plus teacher-labeled silver text. Full
technical reference:
[docs/ARCHITECTURE.md](https://github.com/dmlux/Prism/blob/main/docs/ARCHITECTURE.md).

## Training data and attribution

This model exists thanks to openly licensed Norwegian resources:

- [UD Norwegian-Bokmaal](https://github.com/UniversalDependencies/UD_Norwegian-Bokmaal)
  and [UD Norwegian-Nynorsk](https://github.com/UniversalDependencies/UD_Norwegian-Nynorsk)
  treebanks (Universal Dependencies contributors, based on the
  Norwegian Dependency Treebank by the National Library of Norway) —
  CC BY-SA 4.0
- NBdigital ([`sbr-43`](https://www.nb.no/sprakbanken/en/resource-catalogue/oai-nb-no-sbr-43/))
  and municipal documents ([`sbr-60`](https://www.nb.no/sprakbanken/en/resource-catalogue/oai-nb-no-sbr-60/)),
  National Library of Norway, Språkbanken — CC0
- Nynorsk Wikipedia, Wikimedia contributors — CC BY-SA 4.0 (text never
  redistributed)
- Backbone: [`ltg/norbert4-xsmall`](https://huggingface.co/ltg/norbert4-xsmall);
  distillation teacher and silver labeler:
  [`ltg/norbert4-large`](https://huggingface.co/ltg/norbert4-large)
  (Language Technology Group, University of Oslo) — Apache 2.0

Pinned revisions and checksums travel inside each artifact
(`manifest.json`, `LICENSES/`).

## License

Model weights: **CC BY-SA 4.0.** Using or bundling the unmodified
artifact — including commercially, in closed-source applications — is
fine (keep the attribution); redistributed modified weights must stay
open. Prism source code is Apache 2.0.