docs/model_load_bearing_work_items.md

# Model load-bearing work items

Date: 2026-06-07

This note turns the hosted Omni eval result into four implementation workstreams. The goal is to make the configured model more visibly load-bearing without weakening deterministic safety controls.

Current hosted evidence:

- Hosted eval follow-up trace: `traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl`
- Whole-output hosted competence: 31/50
- Full deterministic fallback use: 8/50
- Field-level model retention: 480/650 fields, or 73.8%
- Deterministic patches: 170/650 fields
- Final validation: 50/50

Use these follow-up metrics for current README, checklist, and submission scorecard wording. Keep the baseline below only as comparison evidence. Final validation is application safety; whole-output competence and field retention are the model-load-bearing metrics.

Baseline evidence:

- Hosted eval trace: `traces/hosted_omni_eval_20260607T194833Z.jsonl`
- Model: `nvidia/nemotron-3-nano-omni-30b-a3b-reasoning`
- Corpus: 50 synthetic cases across `data/eval/*.jsonl`
- Whole-output hosted competence: 28/50
- Deterministic fallback use: 22/50
- Final validation: 50/50

Safety principle: do not loosen validators to raise the score. Increase model contribution by narrowing the model's tasks, validating its outputs field-by-field, and preserving deterministic fallback for any unsafe, missing, or ungrounded field.

## 1. Field-level model acceptance and provenance

Current problem: a single invalid field can force full deterministic fallback, even when other model-generated fields are useful and safe.

Implementation direction:

- Validate navigator output at the field or section level where possible.
- Retain validated model-generated fields.
- Deterministically patch or replace only failing fields.
- Record field provenance as `model_raw`, `model_repaired`, or `deterministic_fallback`.
- Surface provenance in trace and UI so fallback is never counted as model competence.

Done when:

- A trace can show which bounded navigator fields were model-generated versus repaired or fallback-filled.
- Eval metrics can distinguish whole-output competence from field-level model contribution.
- Existing final safety validation still passes.

## 2. More constrained prompt contract

Current problem: hosted outputs commonly miss SBAR fields, omit required observations, introduce unsupported handoff facts, or use forbidden clinical phrasing.

Implementation direction:

- Give the model a literal JSON skeleton with every required key.
- Provide an explicit allowed-facts inventory derived from confirmed intake, deterministic rules, and retrieved cards.
- Provide a required-observations inventory derived from retrieved protocol cards.
- Add concise examples for routine or negated cases where emergency cards are nearby but red flags are absent.
- Keep diagnosis, prescription, dosing, discharge, and autonomous routing prohibitions explicit.

Done when:

- Raw hosted output pass rate improves without validator relaxation.
- Missing SBAR key failures decrease.
- Missing-observation grounding failures decrease.
- Negated routine cases are less likely to be escalated or contaminated by nearby emergency card language.

## 3. Focused field repair before fallback

Current problem: repair currently asks the model to regenerate the whole navigator output after validation failure. That is larger than necessary and can reintroduce unrelated defects.

Implementation direction:

- Classify validation failures by field or section.
- For isolated SBAR failures, repair only `handoff_note_sbar`.
- For missing-observation failures, repair only `missing_info_to_collect` and `next_observations_to_collect`.
- For source-card or pathway failures, repair only citations and candidate pathways.
- Revalidate repaired fields before merging them into the final output.
- Fall back deterministically for fields that still fail after focused repair.

Done when:

- Repair attempts are smaller and traceable by field.
- Successful focused repair increases retained model contribution.
- Repeated repair failure still produces safe deterministic final output.

## 4. Eval metrics for model load-bearing behavior

Current problem: the eval has a strict whole-output competence metric, but it cannot yet quantify partial model contribution.

Implementation direction:

- Add field-level metrics such as `model_field_pass_rate`, `model_visible_fields_retained`, `deterministic_patch_count`, and per-field provenance counts.
- Keep existing metrics: raw model success, repair success, fallback use, final validation success.
- Report field-level contribution separately from whole-output competence.
- Update the hosted eval result note after running the new metrics.

Done when:

- The eval can report how much of the visible navigator output came from the model.
- Fallback cannot inflate model competence metrics.
- The project can honestly claim partial or improved model load-bearing behavior with trace evidence.

## Implementation checkpoint

Checkpoint trace: `traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl`

Implemented changes:

- `figment/prompt_builder.py` now includes a literal required JSON skeleton, an allowed-facts inventory, required-observations inventory, and routine/negated-case guidance.
- `figment/field_provenance.py` provides schema-bounded field merging and provenance labels.
- `figment/focused_repair.py` classifies validation failures into scoped repair prompts.
- `figment/eval_metrics.py` reports field-level model retention and deterministic patch counts.
- `figment/navigator.py` and `scripts/run_eval.py` now retain validated model fields, patch unsafe or missing fields deterministically, and record `field_provenance`.

Measured result:

- Whole-output hosted competence improved from 28/50 to 31/50.
- Full deterministic fallback dropped from 22/50 to 8/50.
- Final validation stayed at 50/50.
- Field-level model retention is 480/650 fields, or 73.8%.
- Deterministic patches remain visible and are not counted as model competence.

Open follow-up:

- Focused repair increases latency; consider capping repair scopes or batching repair prompts.
- SBAR and referral cases still need targeted prompt or validator UX work.