docs/model_load_bearing_work_items.md

Model load-bearing work items

Date: 2026-06-07

This note turns the hosted Omni eval result into four implementation workstreams. The goal is to make the configured model more visibly load-bearing without weakening deterministic safety controls.

Current hosted evidence:

• Hosted eval follow-up trace: traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl
• Whole-output hosted competence: 31/50
• Full deterministic fallback use: 8/50
• Field-level model retention: 480/650 fields, or 73.8%
• Deterministic patches: 170/650 fields
• Final validation: 50/50

Use these follow-up metrics for current README, checklist, and submission scorecard wording. Keep the baseline below only as comparison evidence. Final validation is application safety; whole-output competence and field retention are the model-load-bearing metrics.

Baseline evidence:

• Hosted eval trace: traces/hosted_omni_eval_20260607T194833Z.jsonl
• Model: nvidia/nemotron-3-nano-omni-30b-a3b-reasoning
• Corpus: 50 synthetic cases across data/eval/*.jsonl
• Whole-output hosted competence: 28/50
• Deterministic fallback use: 22/50
• Final validation: 50/50

Safety principle: do not loosen validators to raise the score. Increase model contribution by narrowing the model's tasks, validating its outputs field-by-field, and preserving deterministic fallback for any unsafe, missing, or ungrounded field.

1. Field-level model acceptance and provenance

Current problem: a single invalid field can force full deterministic fallback, even when other model-generated fields are useful and safe.

Implementation direction:

• Validate navigator output at the field or section level where possible.
• Retain validated model-generated fields.
• Deterministically patch or replace only failing fields.
• Record field provenance as model_raw, model_repaired, or deterministic_fallback.
• Surface provenance in trace and UI so fallback is never counted as model competence.

Done when:

• A trace can show which bounded navigator fields were model-generated versus repaired or fallback-filled.
• Eval metrics can distinguish whole-output competence from field-level model contribution.
• Existing final safety validation still passes.

2. More constrained prompt contract

Current problem: hosted outputs commonly miss SBAR fields, omit required observations, introduce unsupported handoff facts, or use forbidden clinical phrasing.

Implementation direction:

• Give the model a literal JSON skeleton with every required key.
• Provide an explicit allowed-facts inventory derived from confirmed intake, deterministic rules, and retrieved cards.
• Provide a required-observations inventory derived from retrieved protocol cards.
• Add concise examples for routine or negated cases where emergency cards are nearby but red flags are absent.
• Keep diagnosis, prescription, dosing, discharge, and autonomous routing prohibitions explicit.

Done when:

• Raw hosted output pass rate improves without validator relaxation.
• Missing SBAR key failures decrease.
• Missing-observation grounding failures decrease.
• Negated routine cases are less likely to be escalated or contaminated by nearby emergency card language.

3. Focused field repair before fallback

Current problem: repair currently asks the model to regenerate the whole navigator output after validation failure. That is larger than necessary and can reintroduce unrelated defects.

Implementation direction:

• Classify validation failures by field or section.
• For isolated SBAR failures, repair only handoff_note_sbar.
• For missing-observation failures, repair only missing_info_to_collect and next_observations_to_collect.
• For source-card or pathway failures, repair only citations and candidate pathways.
• Revalidate repaired fields before merging them into the final output.
• Fall back deterministically for fields that still fail after focused repair.

Done when:

• Repair attempts are smaller and traceable by field.
• Successful focused repair increases retained model contribution.
• Repeated repair failure still produces safe deterministic final output.

4. Eval metrics for model load-bearing behavior

Current problem: the eval has a strict whole-output competence metric, but it cannot yet quantify partial model contribution.

Implementation direction:

• Add field-level metrics such as model_field_pass_rate, model_visible_fields_retained, deterministic_patch_count, and per-field provenance counts.
• Keep existing metrics: raw model success, repair success, fallback use, final validation success.
• Report field-level contribution separately from whole-output competence.
• Update the hosted eval result note after running the new metrics.

Done when:

• The eval can report how much of the visible navigator output came from the model.
• Fallback cannot inflate model competence metrics.
• The project can honestly claim partial or improved model load-bearing behavior with trace evidence.

Implementation checkpoint

Checkpoint trace: traces/hosted_omni_eval_load_bearing_20260607T210047Z.jsonl

Implemented changes:

• figment/prompt_builder.py now includes a literal required JSON skeleton, an allowed-facts inventory, required-observations inventory, and routine/negated-case guidance.
• figment/field_provenance.py provides schema-bounded field merging and provenance labels.
• figment/focused_repair.py classifies validation failures into scoped repair prompts.
• figment/eval_metrics.py reports field-level model retention and deterministic patch counts.
• figment/navigator.py and scripts/run_eval.py now retain validated model fields, patch unsafe or missing fields deterministically, and record field_provenance.

Measured result:

• Whole-output hosted competence improved from 28/50 to 31/50.
• Full deterministic fallback dropped from 22/50 to 8/50.
• Final validation stayed at 50/50.
• Field-level model retention is 480/650 fields, or 73.8%.
• Deterministic patches remain visible and are not counted as model competence.

Open follow-up:

• Focused repair increases latency; consider capping repair scopes or batching repair prompts.
• SBAR and referral cases still need targeted prompt or validator UX work.