--- license: cc-by-sa-4.0 language: - nb - nn - 'no' pipeline_tag: token-classification tags: - part-of-speech-tagging - morphological-analysis - lemmatization - universal-dependencies - executorch - norwegian - bokmal - nynorsk - on-device --- # Prism Norwegian (prism-no) Norwegian UPOS tagging, morphological features, and lemmatization with **calibrated confidences**, built for **on-device, fully offline** use. One compact model (17.6 M parameters) covers both written standards — Bokmål (`nb`) and Nynorsk (`nn`) — in a single set of weights; mixed input is fine. Since 0.2.3 the artifact manifest also declares the BCP 47 macrolanguage `no`, so hosts that match languages strictly against `language_tags` accept plain-`no` documents without aliases (0.2.3 is a manifest-only release: the model bytes are identical to 0.2.2, and all published quality and speed numbers apply unchanged). **It beats UDPipe 2.17 on UPOS and lemmas on the official UD test splits — at about one twentieth of UDPipe's model size** (fast artifact; one tenth for fp32), running fully offline on a laptop CPU. This repository mirrors the versioned release artifacts of the [Prism project](https://github.com/dmlux/Prism). Prism ships native runtimes for **Swift, C++, C, and Java/Kotlin** that read these artifacts directly. ## Which folder to use | Folder | Size | When to use | | --- | ---: | --- | | `prism-no-0.2.3-fast/` | ≈ 45 MB | **Recommended.** int8; up to 2× faster, development-split quality within 0.014 pp of fp32 | | `prism-no-0.2.3/` | ≈ 94 MB | Bit-exact fp32 reference behind the published benchmark | An application bundles exactly one folder. The folder is everything a Prism runtime needs; point the tagger API at its local path: ```swift let tagger = try PrismTagger(artifactURL: artifactFolder) // Swift ``` ```cpp prism::tagger::Tagger tagger("prism-no-0.2.3-fast"); // C++ ``` ```java try (var tagger = PrismTagger.load(Path.of("prism-no-0.2.3-fast"))) {} // Java ``` Quick starts for every binding: the [project README](https://github.com/dmlux/Prism#quick-start). The artifact contract (programs, `model.ptd` weights, tokenizer and label schemas, checksums) is documented in [docs/INTEGRATION.md](https://github.com/dmlux/Prism/blob/main/docs/INTEGRATION.md). **Note:** these are [ExecuTorch](https://pytorch.org/executorch) programs with the decoding policy and calibration baked in — not `transformers`-loadable checkpoints. `vocabulary.json` is a standard Hugging Face `tokenizer.json` and loads with the `tokenizers` library. ## Quality Evaluated exactly once on the untouched official UD test splits against UDPipe 2.17 (gold tokenization, official CoNLL definitions): | Test F1 | Prism | UDPipe 2.17 | | --- | ---: | ---: | | Bokmål UPOS | **98.76%** | 98.57% | | Bokmål Lemmas | **98.98%** | 98.87% | | Bokmål UFeats | 97.20% | **97.59%** | | Nynorsk UPOS | **98.77%** | 98.60% | | Nynorsk Lemmas | **98.68%** | 98.56% | | Nynorsk UFeats | 96.94% | **97.38%** | Prism wins UPOS and lemmas on both written standards and stays behind only on exact morphology bundles — from a model a twentieth of UDPipe's size. ### fast versus fp32 The frozen test evaluation above is fp32; the fast artifact is quality-gated on the development split (67,619 tokens across both standards — the test splits are evaluated exactly once and stay reserved for the fp32 benchmark). Accuracy with the identical production decoding policy: | Task | Standard | fp32 | fast | Delta | | --- | --- | ---: | ---: | ---: | | UPOS | nb | 99.1724% | 99.1641% | -0.0082 pp | | UPOS | nn | 98.8384% | 98.8448% | +0.0064 pp | | UFeats exact | nb | 97.9021% | 97.8883% | -0.0137 pp | | UFeats exact | nn | 95.3408% | 95.3312% | -0.0096 pp | | Lemma | nb | 99.2301% | 99.2246% | -0.0055 pp | | Lemma | nn | 98.8672% | 98.8608% | -0.0064 pp | Every delta is at most 0.014 percentage points — an order of magnitude below seed-to-seed training variance. int8 costs no measurable quality. ### Speed Measured with the **reproducible benchmark suite** on the checked-in CC0 example texts that ship inside the Prism repository ([`data/examples/`](https://github.com/dmlux/Prism/tree/main/data/examples)) — anyone can rerun the identical protocol on their own hardware. Apple M4 Max, CPU only, release builds, median of 3 repetitions. The benchmark document is the Bokmål text repeated seven times: **385 sentences / 6,335 tokens**, roughly a book chapter: | C++ end-to-end, 6,335-token document | fp32 | fast (int8) | | --- | ---: | ---: | | Raw text in, tagged sentences out | 3.63 s (1,813 tokens/s) | **1.94 s (3,543 tokens/s)** | | Pretokenized input | 3.58 s (1,832 tokens/s) | **1.91 s (3,590 tokens/s)** | | Tagger load (cold start) | 33 ms | 35 ms | So the fast artifact tags a full chapter in under two seconds, a single ~900-token text in well under half a second (Swift cross-check, single pass: 387 ms wall, 2,101 tokens/s at lower batch saturation). The model-independent stages are effectively free next to the forward pass: runtime segmentation ≈ 8 M tokens/s, byte-level BPE ≈ 2.2–2.6 M tokens/s. Reproduce it: ```bash git clone https://github.com/dmlux/Prism.git cmake -S cpp -B cpp/build -DCMAKE_BUILD_TYPE=Release -DPRISM_BENCHMARKS=ON cmake --build cpp/build --target prism_benchmarks --parallel cpp/build/prism_benchmarks --benchmark_repetitions=3 --benchmark_report_aggregates_only=true ``` Recorded run and protocol: [docs/benchmarks/v0.3.0.md](https://github.com/dmlux/Prism/blob/main/docs/benchmarks/v0.3.0.md); earlier records (measured on a private text before the reproducible protocol existed): [docs/benchmarks/prism-no-0.2.2.md](https://github.com/dmlux/Prism/blob/main/docs/benchmarks/prism-no-0.2.2.md). ## Model description A 17.6 M-parameter encoder student (16-layer NorBERT4-xsmall backbone, hidden 192) distilled from a NorBERT4-large teacher, with a character CNN feeding morphology and lemma heads, a structured morphology decoder, and per-head temperature calibration (UPOS ECE 0.0017). Trained on the official UD gold treebanks plus teacher-labeled silver text. Full technical reference: [docs/ARCHITECTURE.md](https://github.com/dmlux/Prism/blob/main/docs/ARCHITECTURE.md). ## Training data and attribution This model exists thanks to openly licensed Norwegian resources: - [UD Norwegian-Bokmaal](https://github.com/UniversalDependencies/UD_Norwegian-Bokmaal) and [UD Norwegian-Nynorsk](https://github.com/UniversalDependencies/UD_Norwegian-Nynorsk) treebanks (Universal Dependencies contributors, based on the Norwegian Dependency Treebank by the National Library of Norway) — CC BY-SA 4.0 - NBdigital ([`sbr-43`](https://www.nb.no/sprakbanken/en/resource-catalogue/oai-nb-no-sbr-43/)) and municipal documents ([`sbr-60`](https://www.nb.no/sprakbanken/en/resource-catalogue/oai-nb-no-sbr-60/)), National Library of Norway, Språkbanken — CC0 - Nynorsk Wikipedia, Wikimedia contributors — CC BY-SA 4.0 (text never redistributed) - Backbone: [`ltg/norbert4-xsmall`](https://huggingface.co/ltg/norbert4-xsmall); distillation teacher and silver labeler: [`ltg/norbert4-large`](https://huggingface.co/ltg/norbert4-large) (Language Technology Group, University of Oslo) — Apache 2.0 Pinned revisions and checksums travel inside each artifact (`manifest.json`, `LICENSES/`). ## License Model weights: **CC BY-SA 4.0.** Using or bundling the unmodified artifact — including commercially, in closed-source applications — is fine (keep the attribution); redistributed modified weights must stay open. Prism source code is Apache 2.0.