| --- |
| language: |
| - fi |
| license: other |
| license_name: mit-and-cc-by-4.0-and-gpl-3.0 |
| license_link: https://huggingface.co/samuli/juurio/blob/main/ATTRIBUTION.md |
| library_name: pytorch |
| tags: |
| - finnish |
| - morphology |
| - inflection |
| - lemmatization |
| - dependency-parsing |
| --- |
| |
| # Juurio |
|
|
| Open-vocabulary Finnish morphology over raw text. Give it a sentence and an operation (`past`, `passive`, `plural`, …) and it returns the transformed sentence or an explicit abstention. It also inflects out-of-vocabulary and loan words that a closed-lexicon morphology cannot (`striimata` → `striimannee`, `striimaaja` → `striimaajillanikin`). |
|
|
| ```python |
| from juurio import transform, realize_np, realize_verbform |
| |
| transform("past", "Koira haukkuu pihalla.").text # 'Koira haukkui pihalla.' |
| transform("passive", "Mies osti talon.").text # 'Ostettiin talo.' |
| transform("pluperfect","Hän lukee kirjan.").text # 'Hän oli lukenut kirjan.' (compound tense) |
| transform("neg_imperative","Osta talo.").text # 'Älä osta taloa.' (imperative/2nd-person source) |
| transform("potential", "Nuori striimaa peliä.", lemma="striimata").text # 'Nuori striimannee peliä.' (OOV verb) |
| transform("plural+poss_1sg+clitic_kin", "Koira haukkuu pihalla.").text # 'Koiranikin haukkuvat pihalla.' |
| |
| # phrase-level morphology utilities |
| realize_np("talo", case="Ine", number="Plur", px="1Sg").text # 'taloissani' |
| realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text # 'striimaajillanikin' (OOV) |
| realize_verbform("mennä", form="inf3", case="Ade").text # 'menemällä' |
| realize_verbform("striimata", form="va").text # 'striimaava' (OOV participle) |
| ``` |
|
|
| Every call returns a `Result` (`ok`, `text`, `reason`, `trace`, `warnings`). Input that is out of scope or that the operation does not apply to abstains, with `ok=False` and a documented reason. Forms produced by a neural model rather than the exception lexicon are listed in `trace["neural_unverified"]` and in `warnings`. |
|
|
| ## Install |
|
|
| ```bash |
| pip install https://huggingface.co/samuli/juurio/resolve/main/juurio-0.1.0-py3-none-any.whl |
| juurio setup # download and verify the model checkpoints and data |
| ``` |
|
|
| Python 3.12 or newer. The install pulls in `torch`, `transformers`, `numpy` and `huggingface-hub`. |
|
|
| On a machine with a GPU, that installs the CUDA build of PyTorch. On a CPU-only machine the default PyTorch wheel also bundles CUDA, a download of several GB you can't use. To get the small CPU-only build instead, install it first: |
|
|
| ```bash |
| pip install torch --index-url https://download.pytorch.org/whl/cpu |
| pip install https://huggingface.co/samuli/juurio/resolve/main/juurio-0.1.0-py3-none-any.whl |
| ``` |
|
|
| The wheel contains only the code. `juurio setup` downloads the model checkpoints and data from this repository (about 190 MB), checks their sha256 hashes and caches them. Without it, they are downloaded on first use. `juurio doctor` shows what is present and what is missing. |
|
|
| ## Background |
|
|
| Finnish is richly inflecting, and the standard open-source tools for it are excellent but closed-vocabulary and heavy at runtime. A finite-state morphology ([Giella `fin`](https://github.com/giellalt/lang-fin) and [Omorfi](https://github.com/flammie/omorfi), also the engine behind [Voikko](https://voikko.puimula.org/)) inflects only words already in its lexicon, and a neural dependency parser ([Stanza](https://stanfordnlp.github.io/stanza/)) brings a large dependency stack. A loan or slang word the lexicon has never seen (`striimata`, `striimaaja`) cannot be inflected. |
|
|
| Juurio distills those tools into small neural models and a compact exception lexicon, so at runtime it needs only PyTorch and it can inflect words the finite-state morphology does not know. The heavy tools were used only to build the models and data: Giella `fin` is the morphology the neural models learned from, Stanza produced silver parses for the parser, and Voikko was used to check generated forms. |
|
|
| ## Architecture |
|
|
| | concern | component | |
| |---|---| |
| | structure and scope (which clause, subject, object) | neural UD dependency parser | |
| | morphology of known words | exception lexicon (the forms the neural realizers get wrong) | |
| | open-vocabulary morphology (OOV and loan words) | small neural realizers: nominal, verb-form and finite-verb | |
|
|
| The rule engine handles all non-local structure (agreement, object case, subject drop). The neural realizers are called per word, only when the lexicon has no entry for the form. |
|
|
| ## Entry points |
|
|
| - `transform(op, text, lemma=None)`: finite-clause transforms that preserve grammar, such as negation, tense and mood (including the potential and the compound tenses), passive, question, the full imperative paradigm (with the negative `älä` series), number with agreement, degree, enclitics, possessives and `+`-composed nominal morphology. |
| - `realize_np(np, …)`: phrase-level nominal morphology, with the full case system, possessive suffixes, clitics and modifier agreement. Arbitrary case lives here because it is syntactically governed. |
| - `realize_verbform(verb, …)`: phrase-level non-finite verb morphology. Participles (VA, NUT, TAVA, TU and the agent participle MA) and infinitives, inflected in case and number with an optional possessive suffix. |
| - `inflect(word, …)`, `analyze(text)` and `lemmatize(text)`: single-word inflection, analysis and lemmatization. |
|
|
| Anything semantic, such as existential framing, is deliberately out of scope. |
|
|
| ## CLI |
|
|
| Every library function has a command of the same name. The CLI is a thin wrapper over the same `Transformer`. |
|
|
| ``` |
| # one command per library function |
| juurio transform <op> "<text>" [--lemma L] [--trace] [--json] # transform() |
| juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json] # inflect() |
| juurio analyze "<text>" [--json] # analyze() |
| juurio lemmatize "<text>" [--json] # lemmatize() |
| juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] # realize_np() |
| juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--px 3] # realize_verbform() |
| |
| # infrastructure (no library counterpart) |
| juurio doctor # read-only environment diagnostics (dependencies, checkpoints + sha256) |
| juurio setup # download and verify the checkpoints and data |
| juurio list-ops # documented operation catalog |
| juurio batch <op> < lines # or per-line "op<TAB>text[<TAB>lemma]" |
| juurio demo | bench | repl |
| ``` |
|
|
| `inflect` takes a case, number and degree phrase positionally (`juurio inflect talo mon gen` → `talojen`) or the equivalent `--case`, `--number` and `--degree` flags. |
|
|
| Exit status is `0` on success, `2` if any transform abstained (useful for scripting) and `1` on setup failure. |
|
|
| ## Files |
|
|
| | path | what | |
| |---|---| |
| | `juurio-0.1.0-py3-none-any.whl` | the Python package | |
| | `juurio/`, `pyproject.toml` | source code of the package | |
| | `docs/api.md` | API and CLI contract, scope boundary, reason codes | |
| | `checkpoints/` | parser, analyser and realizer models with their tokenizer and vocabularies | |
| | `data/` | exception lexicon, morphology tag table and vocabularies | |
| | `ATTRIBUTION.md` | licences and sources, file by file | |
|
|
| ## Training data |
|
|
| - The realizers learned from inflected forms generated with the Giella `fin` transducer, for lemmas found in a Finnish web corpus. |
| - The exception lexicon holds the Giella forms the realizers get wrong. |
| - The parser was trained on [UD_Finnish-TDT](https://universaldependencies.org/treebanks/fi_tdt/index.html) and on silver parses of Finnish web text made with Stanza. |
| - The morphology tag table lists Giella's tags for the 100,000 most frequent web-text word forms where the neural analyser disagrees with Giella. E-mail addresses, @handles, URLs, phone numbers and hashtags were removed from it. |
|
|
| ## Limitations |
|
|
| - `transform` handles one finite main clause in the active indicative, within limits of about twelve words and one finite verb. Subordinate and coordinated clauses, already negated clauses and multi-verb fragments abstain. |
| - Forms from the neural realizers are model guesses. They are flagged in `warnings` but not verified, and some are wrong: `realize_np("bugisovellus", case="Ine", px="1Pl", clitic="kin")` gives *bugisovelludessammekin* instead of *bugisovelluksessammekin*. |
| - The parser makes mistakes, and a wrong lemma or analysis can lead to a wrong form or an abstention. The `lemma=` argument overrides the lemma of the main verb. |
|
|
| ## License |
|
|
| Three parts with separate licences (details in [`ATTRIBUTION.md`](ATTRIBUTION.md)): |
|
|
| | part | licence | |
| |---|---| |
| | code (`juurio/`, the wheel) | [MIT](LICENSE) | |
| | models (`checkpoints/`, `data/char_vocab.json`) | [CC BY 4.0](LICENSE-CC-BY-4.0) | |
| | data derived from Giella `fin` (`data/wordlist.json`, `data/fst_table.jsonl`, `data/fst_tag_vocab.json`) | [GPL-3.0](LICENSE-GPL-3.0), the licence of [GiellaLT `lang-fin`](https://github.com/giellalt/lang-fin) (© UiT The Arctic University of Norway / Giellatekno) | |
|
|
