| --- |
| license: cc-by-sa-4.0 |
| language: |
| - en |
| pipeline_tag: token-classification |
| tags: |
| - part-of-speech-tagging |
| - morphological-analysis |
| - lemmatization |
| - universal-dependencies |
| - executorch |
| - english |
| - modernbert |
| - on-device |
| --- |
| |
| # Prism English (prism-en) |
|
|
| English UPOS tagging, morphological features, and lemmatization with |
| **calibrated confidences**, built for **on-device, fully offline** use. |
| A compact 18.0 M-parameter model on a ModernBERT-lineage (Ettin) |
| backbone — the same artifact contract and native runtimes as |
| [`prism-no`](https://huggingface.co/dmlux/prism-no), now for English. |
|
|
| This repository mirrors the versioned release artifacts of the |
| [Prism project](https://github.com/dmlux/Prism). Prism ships native |
| runtimes for **Swift, C++, C, and Java/Kotlin** that read these |
| artifacts directly. **Requires the language-independent Prism runtimes, |
| library 0.6.0 or later.** |
|
|
| ## Which folder to use |
|
|
| | Folder | Size | What it is | |
| | --- | ---: | --- | |
| | `prism-en-0.1.0/` | ≈ 94 MB | fp32 reference model | |
|
|
| This first English release ships a single precision (fp32). There is |
| no int8 `-fast` variant yet: ModernBERT's rotary-embedding indexing |
| does not lower cleanly through the int8 path, and a fast variant is |
| planned. |
|
|
| An application bundles the folder as-is. It is everything a Prism |
| runtime needs; point the tagger API at its local path: |
|
|
| ```swift |
| let tagger = try PrismTagger(artifactURL: artifactFolder) // Swift |
| ``` |
| ```cpp |
| prism::tagger::Tagger tagger("prism-en-0.1.0"); // C++ |
| ``` |
| ```java |
| try (var tagger = PrismTagger.load(Path.of("prism-en-0.1.0"))) {} // Java |
| ``` |
|
|
| Quick starts for every binding: the |
| [project README](https://github.com/dmlux/Prism#quick-start). The |
| artifact contract (programs, `model.ptd` weights, tokenizer and label |
| schemas, checksums) is documented in |
| [docs/INTEGRATION.md](https://github.com/dmlux/Prism/blob/main/docs/INTEGRATION.md). |
|
|
| **Note:** these are [ExecuTorch](https://pytorch.org/executorch) |
| programs with the decoding policy and calibration baked in — not |
| `transformers`-loadable checkpoints. `vocabulary.json` is a standard |
| Hugging Face `tokenizer.json` (the ModernBERT tokenizer) and loads with |
| the `tokenizers` library. |
|
|
| ## Quality |
|
|
| Measured on the UD English-EWT **test** split with the official UD |
| metrics, against UDPipe 2.17: |
|
|
| | Test F1 | prism-en 0.1.0 | UDPipe 2.17 | |
| | --- | ---: | ---: | |
| | UPOS | 97.29% | **97.65%** | |
| | UFeats | 97.71% | **98.16%** | |
| | Lemmas | 97.36% | **97.99%** | |
|
|
| Competitive on EWT web text — within ~0.4–0.6 pp of UDPipe — from an |
| 18 M-parameter model that runs fully offline on a laptop CPU. The |
| Ettin-400m teacher beats UDPipe on all three metrics; distilling down |
| to the 18 M student gives up ~0.5 pp, a deliberate later-optimization |
| target (0.1.0 ships the current student so English is available now). |
|
|
| ## Model description |
|
|
| An 18.0 M-parameter encoder student on |
| [`jhu-clsp/ettin-encoder-17m`](https://huggingface.co/jhu-clsp/ettin-encoder-17m) |
| (a 7-layer ModernBERT-lineage encoder, hidden 256), distilled from the |
| Ettin-encoder-400m teacher, with a character CNN feeding the morphology |
| and lemma heads, a structured morphology decoder, and per-head |
| temperature calibration. Label spaces: **18 UPOS · 21 morphology |
| features · 1,632 lemma edit rules**; **50,368** byte-level BPE subwords. |
| Full technical reference: |
| [docs/ARCHITECTURE.md](https://github.com/dmlux/Prism/blob/main/docs/ARCHITECTURE.md). |
|
|
| ## Training data and attribution |
|
|
| - **Gold treebank:** [UD English-EWT](https://github.com/UniversalDependencies/UD_English-EWT) |
| (revision `c5baffde`, UD 2.17) by the Universal Dependencies |
| contributors — CC BY-SA 4.0. (The GUM treebank is deliberately |
| **excluded**: its CC BY-NC-SA license is incompatible with this |
| release.) |
| - **Silver corpora** (unlabeled running text, labeled offline by the |
| Prism teacher; the text itself is not redistributed): Project |
| Gutenberg (public domain) and English Wikipedia (Wikimedia |
| contributors — CC BY-SA 4.0). |
| - **Backbone:** [`jhu-clsp/ettin-encoder-17m`](https://huggingface.co/jhu-clsp/ettin-encoder-17m) |
| and its Ettin-encoder-400m teacher by the JHU CLSP group — MIT. |
|
|
| Pinned revisions and checksums travel inside the artifact |
| (`manifest.json`, `LICENSES/`). |
|
|
| ## License |
|
|
| Model weights: **CC BY-SA 4.0** — the same share-alike license as the |
| UD English-EWT gold annotations. Using or bundling the unmodified |
| artifact — including commercially, in closed-source applications — is |
| fine (keep the `LICENSES/` attribution); redistributed modified weights |
| must stay open. Prism source code is Apache 2.0. |
|
|
