INTEGRATION.md

Integrating the three phases into the live slug

All modules are dependency-light, frozen-model, and run on the Space. They reuse the existing extraction and the OpenRouter/HF completer you already have.

Files

• artifact_meta.py Phase 1: lifecycle metadata (provenance/confidence/usage/validation)
• rule_phrasing.py Phase 1: RuleShaping negative-constraint phrasing (LLM + deterministic fallback)
• session_store.py Phase 2: store of past sessions (JSON; use the Modal volume in prod)
• gotcha_cluster.py Phase 2: cluster recurring gotchas (embedding or LLM-judge backend)
• promotion_engine.py Phase 2: promote recurring clusters -> validated L3 rules; demote on failure
• multi_analyst_extract.py Phase 3: Trace2Skill-style N-analyst recurrence merge (higher-fidelity extraction)

Wiring (minimal)

1. After each session's extraction, persist it: store = SessionStore("/vol/gallery/sessions.json") # Modal volume in prod store.add(session_id, extraction)

2. Phase 3 (optional, improves fidelity): replace single-pass gotcha extraction with gotchas = multi_analyst_gotchas(session_text, completer, n_analysts=4, had_errors=bool(extraction.get("dead_ends")), agree_min=2, judge_same=same_gotcha_judge) Cost knob: n_analysts (3–5 cheap; larger = higher fidelity, higher cost).

3. Phase 2 (the headline): run a promotion pass (cheap; do it after add, or on idle): rules = promote( store, k_threshold=3, cluster_fn=lambda items: cluster_by_judge(items, same_gotcha_judge), reshape_fn=lambda g: to_negative_constraint_llm(g, completer), validate_fn=rule_validates, # reuse the distractor/novel eval harness require_human=False, # set True for the 10/10 human-gated path ) Persist promoted rules (with their ArtifactMeta) alongside sessions.

4. Surface a new lens in the app: "the rules the slug has learned across your sessions" — list each L3 rule with its provenance (which sessions), confidence, and last-validated time. This is the lifecycle-governance story made visible.

The validation gate (important — don't skip)

validate_fn(rule_text, cluster) should reuse the held-out check you already built (distractor_eval / novel_eval): does an answer that APPLIES the rule beat one that ignores it, on a task derived from the cluster? Promoting without this is how you promote "compact noise" (the spectrum paper's exact warning). Gate-only is the default; require_human=True adds confirmation for the 10/10 path.

What to claim (honestly)

"Single self-generated skills give a frontier model ~0.0 uplift (our eval, matching SkillsBench as cited in arXiv:2604.15877). We implement a working slice of that paper's proposed solution — the 'missing diagonal' of adaptive upward compression: the slug consolidates gotchas that recur across sessions into validated, negative-constraint-phrased L3 rules (RuleShaping) carrying provenance/confidence/ lifecycle metadata, with demotion when a rule stops validating, and Trace2Skill-style multi-analyst recurrence-merging for extraction fidelity. Honest limits: recurrence is a proxy for value (mitigated by the validation gate + optional human confirmation); small analyst-N is a cheaper approximation of Trace2Skill's 128."

---

CORRECTIONS after reading the actual papers (not just the survey)

• Phase 3 was rebuilt as trace2skill_faithful.py after reading arXiv:2603.25158 in full. The first version (multi_analyst_extract.py) was a simplified guess that MISSED the paper's central mechanism: the error analyst is an AGENTIC, VALIDATION- GATED loop (it must produce a verified causal cause or the trace is EXCLUDED), not a single call. The paper shows the single-call version over-attributes parse errors as root cause in 57% of cases and hallucinates causes — i.e. the leak/false-cause bug. Consolidation is an LLM merge operator applied hierarchically with a prevalent- pattern bias; success patches are lower-trust; low-support items route to references/. Use trace2skill_faithful.py; keep multi_analyst_extract.py only as the simpler fallback if the agentic loop is too costly on the Space.

• RuleShaping verified against arXiv:2604.11088 ("Do Agent Rules Shape or Distort?"). Confirmed: +7–14pp, negative constraints the only beneficial type, positive directives hurt. Added nuance: rules work via CONTEXT PRIMING (random ~ curated), state-dependent process guardrails ("don't X until Y") are the highest-benefit category (63.8%), and rules are collectively helpful even when individually inert (no degradation up to 50). rule_phrasing.py now biases toward state-dependent negative guardrails.

REMAINING GAPS (do not overclaim)

1. CLUSTERING IS UNVALIDATED ON REAL DATA. The promotion engine works when the "same underlying gotcha" judge is good. On arbitrary messy sessions, clustering quality is unknown — a bad judge promotes noise or misses recurrences. This is the biggest open risk and needs a real embedding/LLM judge tested on labeled session data.

2. THE ERROR-ANALYST VALIDATION GATE is faithful in STRUCTURE but ADAPTED: the paper runs a minimal fix against ground truth in a sandbox. A narrated coding session fed to the slug usually has no ground truth / sandbox, so we use strict self-verification (cite trace evidence) instead. This is weaker than the paper's executable gate and should be labeled as an adaptation, not a reproduction.

3. NO END-TO-END RUN ON REAL SLUG SESSIONS YET. Everything is validated on controlled/ realistic-but-synthetic data. Whether real slug sessions cluster usefully is untested.