Blame for docs/api.md

# juurio — API & CLI contract

Open-vocabulary Finnish morphological transforms over raw text. You hand it a sentence and an
operation (`past`, `passive`, `plural`, …); it parses, transforms the finite main clause, and
returns the result or an explicit abstention. The differentiator is **OOV/loan words**: words the
exception lexicon does not cover (`striimata`, `tubettaja`) are inflected by small neural realizers.

## Architecture (the "split thesis")

Three components, each owning what it is best at:

| Concern | Component |
|---|---|
| structure / scope (which clause, which subject/object) | neural UD dependency parser (Giella/Stanza-free) |
| idiosyncrasy-free morphology (known words) | net + lexicon exception table (byte-exact to the Giella FST) |
| open-vocabulary morphology (OOV/loan words) | tiny lemma-anchored neural realizers |

The serving runtime needs only torch/numpy/transformers — no Stanza, no Giella FST, no Voikko (those
are used only to build the models and data). The rule engine owns all non-local structure (agreement propagation, object
case, subject drop); the neural realizers are called **per token**, only when the lexicon has no
entry for the cell.

## Scope boundary (a feature, not a limitation)

Two entry points with deliberately different promises:

- **`transform(op, text)` — finite-clause transforms** that preserve (or deliberately rewrite) the
  clause's grammar: negation, tense/mood, passive, imperative, question, number agreement, possessive
  on the subject head. Output is a grammatical sentence or an explicit abstain.
- **`realize_np(np, …)` — phrase-level nominal morphology**: inflect an NP into a Case/Number/Px(+
  clitic). This is where *arbitrary case* lives, because case is **syntactically governed** — making
  it a clause op would produce ungrammatical sentences (`Talo on iso` → `Talossa on iso`).

Anything **semantic** — e.g. existential/locative framing (`Taloissani on koira`), which introduces
and reorders arguments — is **out of scope** for both; it would be a separate future *generator* op,
not a morphological transform. Keeping these apart is what lets each give a clean guarantee.

## Python API

```python
from juurio import Transformer, transform, SUPPORTED_OPS, OP_CATALOG, REASONS, ABSTAIN_REASONS

t = Transformer()                                 # loads parser + models once (heavy; reuse it)
r = t.transform("past", "Koira haukkuu pihalla.")
r.ok            # True
r.text          # 'Koira haukkui pihalla.'
r.reason        # 'ok'

# one-shot convenience (reuses a process-wide Transformer):
transform("plural", "Striimaaja nukkuu.").text     # 'Striimaajat nukkuvat.'  (OOV noun, neural)
```

`import juurio` is cheap; torch + the neural parser/realizers load lazily on the first
`Transformer()` / `transform()` (no Stanza/Giella at serving time).

### `Transformer.transform(op, text, lemma=None, sent_index=0) -> Result`

- **op** — one of `SUPPORTED_OPS` (else `ValueError`).
- **text** — raw text; a leading dialogue dash/quote is handled and re-prepended.
- **lemma** — override the parser's lemma for the main verb. Use when a loan stem is mis-lemmatized
  (`striimaa` → parser says `striä`); a confidence warning is raised automatically otherwise.
- **sent_index** — which parsed sentence to operate on (default 0).

### `Result`

| field | meaning |
|---|---|
| `text` | transformed sentence, or `None` if abstained |
| `ok` | `True` iff a form was produced |
| `reason` | one of `REASONS` (`"ok"` on success, else in `ABSTAIN_REASONS`) |
| `trace` | provenance dict (see below) — `juurio transform --trace` prints it |
| `warnings` | low-confidence-lemma notes + **neural realization** notices (model guess, unverified against the lexicon) |

`trace` fields: `op`, `root_verb`, `parser_lemma`, `lemma_source` (`override`/`parser`); and, when
relevant, `neural_oov` (forms the realizer produced — model guesses), `oov_unrealizable` /
`oov_unverified` (words it could not/did-not-trust). `realize_np` traces `util`, `case`, `number`,
`px`, `clitic`, `head`. Inspect on the CLI with `--trace`, or as JSON with `--json`.

### Performance

Models load once per process; the first call also loads the neural parser (the cold cost).
Indicative numbers (`juurio bench`, CPU parser + GPU realizer):

| | time |
|---|---|
| `Transformer()` construct | ~2 s |
| first `transform` (cold — parser load) | ~9 s |
| `transform` (warm) | ~70 ms |
| `realize_np` (warm) | ~30 ms |

Reuse one `Transformer` (or the module-level `transform`/`realize_np`, which cache a singleton).

## Scope contract

`transform` operates on a **single finite Act/Ind main clause** (≈≤12 words, one finite verb).
Anything else **abstains** (`ok=False`) — it never guesses. Reason codes:

| reason | meaning |
|---|---|
| `ok` | a transformation was produced |
| `parse_empty` | the parser returned no usable sentence at `sent_index` |
| `no_clause` | no finite Act/Ind main clause (subordinate-only, fragment, or already-negated) |
| `out_of_scope` | exceeds scope limits (too long / >1 finite verb / coordination) |
| `op_not_applicable` | op does not apply to this clause's base state (e.g. `past` on a past clause) |
| `oov_unrealizable` | an open-class word needed inflecting but **neither** the lexicon **nor** the neural realizer could produce the target form |
| `oov_unverified` | the neural realizer produced a candidate for an OOV word but it failed validation (stem-inconsistent or near-tie) and was **suppressed**, not emitted |
| `conflicting_features` | `inflect` got two valid but contradictory inputs (spec `mon gen` + `number="sing"`; `proper=True` + a degree; `like=` + an explicit case/number/degree; a single-word control on a phrase) |
| `unanalyzable_model` | `inflect(like=…)` could not analyze the model word into a nominal case/number cell to copy |

**Neural OOV is surfaced, never silent.** When a form is produced by the neural realizer (not the
lexicon), it appears in `trace["neural_unverified"]` and a `warnings` entry. Treat it as a model guess.

## Operations

`SUPPORTED_OPS` is the flat tuple; `OP_CATALOG` groups them with descriptions. Summary:

- **finite verb** — `neg`, `past`, `pres`, `cond`, `perfect`, `potential`, `passive`
- **compound tenses** (analytic olla auxiliary + the perfect participle) — `pluperfect`
  (`olin sanonut`), `cond_perfect` (`olisin sanonut`), `pot_perfect` (`lienen sanonut`); OOV verbs
  reuse the trained `perfect` participle + the lexicon's `olla` auxiliary
- **composites** (applied atomically) — `neg+past`, `neg+cond`, `neg+perfect`, `passive+past`,
  `passive+cond`, `passive+perfect`, `neg+passive`, `neg+passive+past`, `neg+passive+cond`,
  `neg+passive+perfect`
- **clause** — `quest` (polar question)
- **imperative paradigm** (drop subject, total object → Nom; negative forces partitive; trailing
  punctuation preserved) — `imperative` (Sg2, OOV-capable), `imperative_pl2/sg3/pl1/pl3`, and the
  negative `älä`-series `neg_imperative` (Sg2, OOV-capable), `neg_imperative_pl2/sg3/pl1/pl3`.
  **Scope:** applies only when the command's addressee is in the clause — a subjectless/already-imperative
  source (`osta talo.`) or a 2nd-person subject (`sinä/te`); a 3rd/1st-person subject abstains
  (`out_of_scope`), since turning *"the man buys a house"* into a command would invent an addressee.
- **number** (re-agrees subject NP, modifiers, finite verb, copula predicate) — `plural`, `singular`
- **degree** (predicate adjective) — `comp`, `super`
- **enclitics** (on the finite verb) — `clitic_kin`, `clitic_kaan`, `clitic_han`, `clitic_pa`
- **possessive suffix** on the subject head, preserving its case/number — `poss_1sg`, `poss_2sg`,
  `poss_3`, `poss_1pl`, `poss_2pl` (`talo`→`taloni`, `talossa`→`talossani`; OOV `striimaaja`→`striimaajani`)

`juurio list-ops` prints the live catalog.

### Combined nominal morphology

Nominal ops compose with `+`: a `+`-join of **nominal atoms** (`plural`/`singular`, `poss_*`,
`clitic_*`) is applied to the subject head as **one realization** with the final
Number+Case+Px, then the clitic is attached — deterministic surface order `stem+case+number+Px+clitic`.
This is correct-by-construction (you cannot re-pluralize the already-realized string `taloni`).
Number changes also re-agree modifiers and the finite verb.

```
poss_1sg+clitic_kin        Koira haukkuu pihalla.  -> Koiranikin haukkuu pihalla.
plural+poss_1sg            Koira haukkuu pihalla.  -> Koirani haukkuvat pihalla.
plural+poss_1sg+clitic_kin Koira haukkuu pihalla.  -> Koiranikin haukkuvat pihalla.
plural+poss_1sg            Striimaaja nukkuu.      -> Striimaajani nukkuvat.   (OOV head, neural)
```

Each dimension may appear at most once. `+`-composed **verb** ops (`neg+passive+past`) are separate,
pre-registered atoms in the catalog. Mixed verb+nominal composition is not supported.

## Phrase-level nominal morphology: `realize_np`

Finnish case is **syntactically governed**, so arbitrary case is *not* a clause transform (a
subject-case change would be ungrammatical). Instead it is a phrase utility, separate from
`transform`, which keeps grammaticality guarantees:

```python
from juurio import realize_np
realize_np("talo", case="Ine", number="Plur", px="1Sg").text                 # 'taloissani'
realize_np("iso talo", case="Ine", number="Plur").text                       # 'isoissa taloissa'
realize_np("Helsinki", case="Ine").text                                      # 'Helsingissä'
realize_np("striimaaja", case="Ade", number="Plur", px="1Sg", clitic="kin").text  # 'striimaajillanikin' (OOV)
```

`realize_np(np, case, number, px, clitic)` inflects an NP — a bare lemma or a short `adj* noun`
phrase — into the target Case/Number with an optional possessive and enclitic, agreeing modifiers,
and per-token lexicon→neural fallback (surface order `stem+case+number+Px+clitic`). It returns a `Result`
(`ok` with the NP, or an abstain reason). This is the showcase for open-vocabulary nominal
morphology; `transform` is for grammaticality-preserving clause operations. Sentence-level locatives
(`Taloissani on koira`) are a separate future generator op, not a case transform.

## Phrase-level verb morphology: `realize_verbform`

The non-finite counterpart of `realize_np`: inflect a VERB into a **participle** or **infinitive**.
The four participles inflect like adjectives in case+number; the 2nd/3rd infinitives are case-marked.
A second open-vocabulary surface — the lexicon handles known verbs, a small **verbform realizer**
inflects OOV/loan verbs (`striimata`), flagged `unverified`.

```python
from juurio import realize_verbform           # form ∈ {va, nut, tava, tu, inf2, inf2_pass, inf3, ma}
realize_verbform("lukea", form="va").text                    # 'lukeva'    (present active participle)
realize_verbform("lukea", form="va", case="Ine").text        # 'lukevassa'
realize_verbform("lukea", form="tu").text                    # 'luettu'    (past passive participle)
realize_verbform("mennä", form="inf3", case="Ade").text      # 'menemällä' (3rd inf adessive)
realize_verbform("mennä", form="inf3", case="Ine").text      # 'menemässä'
realize_verbform("tehdä", form="ma", case="Ine").text        # 'tekemässä' (agent participle)
realize_verbform("tehdä", form="inf2_pass").text             # 'tehtäessä' (passive 2nd inf, temporal)
realize_verbform("kuulla", form="tu", case="Par", px="3").text  # 'kuultuansa' (temporal: 'after hearing')
realize_verbform("striimata", form="va").text                # 'striimaava' (OOV verb, neural)
```

`va`/`nut`/`tava`/`tu` = present/past active/passive participle; `inf2`/`inf3` = 2nd/3rd infinitive;
`inf2_pass` = passive 2nd infinitive (`-ttaessa` temporal, "while X is being done"; inessive-only,
case-invariant); `ma` = agent participle (-ma/-mä, "the X that someone does", inflects in case/number).
`px` (one of `None`/`1Sg`/`2Sg`/`3`/`1Pl`/`2Pl`) attaches a possessive suffix — participles inflect
like nominals, so `tu`+`Par`+`px` realizes the **temporal construction of anteriority** (`kuultuaan` /
the `-nsa` variant `kuultuansa` = "after hearing"). For OOV verbs the bare participle comes from the
verbform realizer and the case+Px from the noun realizer (both neural stages flagged unverified).
Returns a `Result` (`ok` with the form, or `oov_unrealizable`/`oov_unverified` when the morphology
engine cannot produce/trust it). Like `realize_np`, this is a phrase utility, not a clause transform.

## Friendly inflection: `inflect`

The ergonomic single-word front door over `realize_np` — human-readable case/number/degree names, a
proper-noun toggle, and **inflect-by-example**. Returns a `Result` (never a bare string: abstention
stays first-class).

```python
from juurio import inflect
inflect("talo", "gen").text                         # 'talon'    (alias: gen/genitive/genetiivi)
inflect("talo", "mon gen").text                     # 'talojen'  (combined spec, taivutin-style)
inflect("talo", case="inessive", number="plural").text          # 'taloissa'
inflect("talo", px="1sg").text                      # 'taloni'
inflect("kaunis", "gen", degree="comp").text        # 'kauniimman'  (degree => adjective path)
inflect("meri", "gen").text                         # 'meren'    (common noun)
inflect("Meri", "gen", proper=True).text            # proper-noun reading (≠ 'meren')
inflect("makaronilaatikko", like="talojen").text    # 'makaronilaatikkojen'  (copy a model word's cell)
inflect("iso talo", "mon gen").text                 # 'isojen talojen'  (phrase -> realize_np)
```

`inflect(word, case="nom", number="sing", *, degree="pos", px="none", clitic=None, proper=None,
pos=None, like=None)`:

- **case** — a case alias, *or* a combined spec (`"mon gen"`, `"plural genitive comparative"`).
  Accepts English abbrev/full and Finnish (`gen`/`genitive`/`genetiivi`, `mon`/`monikko`).
- **number / degree** — aliases too; an explicit value that contradicts the combined spec abstains
  (`conflicting_features`).
- **px / clitic** — possessive suffix (`1sg`…/`None`) and enclitic, as in `realize_np`. `px=None` is
  accepted (= no suffix). Rejected on a single adjective target.
- **proper** — `True`/`False` forces PROPN/NOUN; `None` (default) infers PROPN from a leading capital.
- **pos** — explicit `'noun'`/`'proper'`/`'adjective'` (wins over inference). PRON/NUM are not yet a
  first-class `pos`. A degree implies the adjective path when `pos` is unset.
- **like** — a model word; its analyzed case (+number, +degree if adjectival) is copied onto `word`.
  Mutually exclusive with `case`/`number`/`degree`; an unanalyzable model abstains
  (`unanalyzable_model`).

An **unknown alias** raises `ValueError` (a typo fails fast); a **contradiction** abstains
(`conflicting_features`). A single token is realized **parser-free** (so `proper`/`pos` are honored);
a whitespace phrase delegates to `realize_np` (which applies `px`/`clitic` to the head but rejects the
single-word controls `proper`/`pos`/`degree`). An empty `word` raises `ValueError`.

## Analysis: `analyze` / `lemmatize`

The inverse of generation (what a generate-only inflector cannot do): the serving neural parser
exposed directly.

```python
from juurio import analyze, lemmatize, TokenAnalysis
sents = analyze("Koirat haukkuvat pihalla.")        # list[list[TokenAnalysis]] (one list per sentence)
tok = sents[0][0]
tok.lemma, tok.upos, tok.feats                      # 'koira', 'NOUN', {'Case': 'Nom', 'Number': 'Plur'}
tok.asdict()                                         # plain dict (id/form/lemma/upos/head/deprel/feats)
lemmatize("Koirat haukkuvat pihalla.")              # ['koira', 'haukkua', 'piha', '.']  (flat)
```

`TokenAnalysis` is a frozen record (`id`, `form`, `lemma`, `upos`, `head` — 1-based, 0=root —,
`deprel`, `feats` as a read-only UD mapping). `analyze` keeps sentence boundaries; `lemmatize`
returns a flat lemma list.

## CLI

Each library function has a like-named command (the CLI is a thin wrapper over the same
`Transformer`); `realize_np`/`realize_verbform` keep the hyphenated spelling.

```
# one command per library function
juurio transform <op> "<text>" [--lemma L] [--trace] [--json]      # transform()
juurio inflect "<word>" [spec…] [--case C] [--number N] [--degree D] [--px P] [--clitic K] [--pos P] [--like W] [--proper] [--json]   # inflect()
juurio analyze "<text>" [--json]                                   # analyze()
juurio lemmatize "<text>" [--json]                                 # lemmatize()
juurio realize-np "<np>" [--case Ine] [--number Plur] [--px 1Sg] [--clitic kin] [--json]   # realize_np()
juurio realize-verbform "<verb>" [--form va|nut|tava|tu|inf2|inf2_pass|inf3|ma] [--case Ine] [--number Plur] [--px 3]   # realize_verbform()

# infrastructure (no library counterpart)
juurio list-ops
juurio batch <op> [--lemma L] [--json]  < lines        # one op for every stdin line
juurio batch [--json]                   < tsv          # per-line "op<TAB>text[<TAB>lemma]"
juurio demo                                            # curated tour (incl. OOV) — also a smoke test
juurio repl                                            # interactive shell; model loaded once
```

`inflect` accepts a human case/number/degree phrase positionally (`juurio inflect talo mon gen`
→ `talojen`, `juurio inflect kaunis gen --degree comp`), or as `--case/--number/--degree` flags;
a positional phrase wins over the flags. `--proper`/`--no-proper` force/forbid a proper-noun reading.

`juurio repl` is a read-eval loop over every feature (model loaded once): `inflect talo mon gen`,
`inflect makaronilaatikko like=talojen`, `transform passive Mies osti talon.`, `analyze …`,
`lemmatize …`, `np …`, `verb …`, `ops`, `help`. Options are `key=value` (`px=1sg proper=true
pos=adjective clitic=kin like=talojen`).

Also `python -m juurio …`. Without `--json`, prints the result (warnings appended after `⚠`) and
sends abstentions to stderr as `[abstain: <reason>] <input>`. With `--json`, one JSON object per
line (`text/ok/reason/warnings/trace`). **Exit status:** `0` if every transform produced a form,
`2` if any abstained — so scripts can branch on it.

```console
$ juurio transform plural "Striimaaja nukkuu."
Striimaajat nukkuvat.  ⚠ neural realization (unverified by FST): ['striimaaja->Striimaajat (NOUN)']
```

## Quality

The package is designed to **abstain rather than guess**: out-of-scope inputs return an explicit
abstention (`ok=False`) with a reason code instead of a speculative form, and neural OOV realizations
are always surfaced (in `trace["neural_unverified"]` and `warnings`) rather than emitted silently. Known
words are inflected to be byte-exact to the reference FST (through the exception lexicon); OOV/loan words are inflected by the neural
realizers, with stem-inconsistent or near-tie candidates suppressed (`oov_unverified`) rather than
returned.

Run `juurio demo` for a curated tour (including OOV cases) that doubles as a smoke test.