APP_HANDOFF.md

App-level handoff — composite operation of the SP model as a local assistant

This is the operational handoff: how the bounded-memory soft-prompt (SP) reasoner is wired into a usable on-device app, what was built, what's verified, and where the edges are. For the research handoff (training, deepKL, architecture) see HANDOFF.md; for the runbook see OPERATING.md; for the portfolio writeup README.md.

One line: a 1.5B reasoning distill runs with O(1) bounded KV (distant context → 32 SP vectors); a tiered retrieval + memory layer (same-session → prior-session → web) feeds verbatim facts into the raw window so the lossy SP is bypassed, with a groundedness guard that resamples contaminated answers. Runs at ~36 tok/s on an 8 GB Mac (MLX 4-bit).

---

0. FINAL SPEC (current behaviour, after all iterations)

What tiered_rag_mlx.ChatSession.turn() does for each user message, end to end:

A. Classify the turn (intent_of) — ONE learned 6-class router: a LogisticRegression on a pure BGE-small embedding (evals/intent_clf.joblib, ~95% CV) routes each turn to exactly one action:

• recall — a question about the USER → personal memory only (L1/L2), never web.
• lookup — a general/WORLD question → web (so "who is the emperor of Japan" goes to web, not personal L1).
• math — a self-contained problem → solve.
• command — "recompute the total" → recall-log + compute.
• fact — a stated value/attribute → log.
• chitchat — greeting / conversational remark / open-ended request → respond.

This replaces all the regex gates (_is_question, _is_factlike, _looks_mathy, compute_ref) — one embedding model does all routing. Training = hand-labelled core + open-model-generated examples (Qwen2.5-3B-Instruct-4bit, evals/intent_gen.py). Regex gates are only a low-confidence/no-model fallback. (History: regex → BGE+syntactic hybrid for small data → pure BGE once data grew → split question→recall/ lookup after a realistic session showed world questions hitting personal memory and hallucinating. The recall/lookup split was the key fix for that — a personal vs world question is a routing distinction, not a retrieval-threshold one. intent_features.py is deprecated.)

B. Write to memory (so future turns can recall it):

• explicit save ("remember my … is X") → persist to disk (L2) + reply instantly "Got it — saved." (no generation).
• a fact-like statement (asserts a value/attribute: "8 guests", "paint it blue") → auto-log to L1 (same-session).
• chit-chat / knowledge-requests / questions → NOT logged (logging them poisons recall).
• specific info stated INSIDE a non-recall turn (e.g. "$120/day" inside a math question) → pinned to working memory by the independent specificity head (see B′), even though the turn isn't a logged fact.

B′. Specificity head (independent, pin working memory) — closes the gap where a specific VALUE is stated inside a question/math turn ("if I spend $120 a day…") and is therefore never logged, so a later "recompute" / "what was the rate" misses it. The head is one more linear probe on the SAME BGE vector already computed for routing/retrieval (evals/specificity_clf.joblib, 5-fold CV AUC 0.996), chosen via a bake-off (evals/specificity_probe.py) where it beat tokenizer-fragmentation/tpc (AUC 0.87) and a form-frequency surprisal proxy (0.30 — dead). specific_spans(text) enumerates cheap candidates (money, codes, times, measures, proper-noun runs) and keeps those the probe scores ≥0.6. In turn(): for any intent except recall/lookup (there the specific span is what's being asked about, e.g. "who is Naruhito" — not user info), a turn with specific spans is pinned (TieredMemory.pins, rolling cap 12). Pins feed: command/compute as primary context (the recompute needs the value), and recall/lookup as a strict (sim≥0.5) fallback only — genuine logged facts win, so a pinned "$120 a day for 3 days" can't derail "how long am I staying". Verified evals/pin_test.py (detector 9/9; pin→recall flow pass) and the realistic session: recompute the food total now = 480 (was un-recoverable before, the $120 lived only in a math turn). Train/extend: python3.12 evals/specificity_train.py (+ optional specificity_gen.jsonl). Follow-up math turns also get WM pins injected (excluding the current turn's own pin) so a multi-turn question that references earlier context ("she pays with $50, how much change" needs turn-1's $24, which scrolled out of the raw window) can recover it — this was a real bug (the designed RAG didn't fire on math follow-ups, only on command). The recall/lookup pin-fallback intercepts only pins NOT already logged in the session (p not in session) — a LOGGED fact ("going to Kyoto") must not shadow a genuine world question ("emperor of Japan", which is geographically near "Kyoto"); logged facts are recall-reachable anyway. POLICY COUPLING (important): this "working-memory-only" filter assumes the app logs ONLY facts to the session — i.e. non-fact turns are called with store="none" (cf. app_session.py / app_eval2.py). If a math/chitchat turn were logged to the session, its pinned value would stop being WM-only and the lookup→pins safety net could not fire for it. Evaluated end-to-end in evals/app_eval2.py (PHASE 1 architecture scorecard 15/15: routing + pin + retrieval; PHASE 2 real generation). PHASE 1 caught the Kyoto/Japan over-interception bug above.

C. Retrieve context (only for questions / referential imperatives) — _match = BGE-small semantic cosine (runtime/rag.py, 33M, CPU), lexical-overlap fallback if BGE unavailable. Handles paraphrase ("which spot did I leave the car?" ≈ "parked in bay 12", "how many people?" ≈ "guests"):

1. L1 same-session — semantic top-K (cap 4) of the logged facts, kept within 0.12 cosine of the best (a strong single match drops distractors), ties broken by recency so a correction wins.
2. L2 prior-session — same semantic match over the on-disk profile (survives a fresh process).
3. self-contained math (has digits) → solve it, don't retrieve.
4. referential imperative ("recompute the total") → inject the recent fact log + compute.
5. else → L3 web (DuckDuckGo → Wikipedia, keyless).

D′. Recall = clean-context quote — for a recall turn that retrieved a fact chunk, the answer is generated by ChatSession._clean_quote() from an ISOLATED context (the retrieved chunk + question only, temp 0.2, no soft prompt, no conversation history), with the groundedness re-roll. Why: in a long multi-turn chat the distractor turns' CoTs compressed into the SP can hijack a recall into computing/confabulating (e.g. quoting a distractor's number); an isolated context removes that bias. Validated: deep recall after 6 distractors went from a stochastic 0/3–3/3 to a reliable 3/3 across repeats (evals/recall_distract.py). The retrieval (pins/L1) was already firing; this fixed the recall generation discipline. (Raw SP alone confabulates 0/3, and even full-KV base fails to self-recall — so the external retrieval injection is essential, not a crutch.)

D. Inject + instruct — matched facts go verbatim into the raw window:

• recall → "quote the value verbatim; if it was corrected, use the most recent."
• compute → "use the fact(s) and compute." (and the groundedness check is skipped for math).

E. Generate — free generation (no think-budget), temp 0.6, rw 1024; bounded only by a 240 s wall-clock timeout + 2000-token per-turn cap + a 6-identical-token loop guard.

F. Make a real chat answer (2-pass) — think freely; if the post-</think> answer is weak (empty / a "…:" preamble / just a \boxed{}), inject "Final answer:" and continue, so a clean answer always surfaces (e.g. "4000 square meters.", not an empty reply).

G. Quality gate — if the answer is empty/timed-out, or (for a recall turn) not grounded in the retrieved context, delete the CoT and re-roll (retries=2; cheap — the hypernet recomputes the SP).

Memory tiers: L1 same-session (in-RAM facts) → L2 prior-session (disk jsonl) → L3 web. Verified: app_demo recall 3/3, routing 5/5, web+math hit; 5- and 10-rally consistency (corrections persist; the oldest fact fades after ~many turns). Knobs: temp 0.6, rw 1024, retries 2, TIMEOUT_S 240, MAX_NEW 2000. Limits: bounded-recency (turn-1 facts fade in a long chat — the specificity head pins specific values but a rolling cap-12 buffer still ages out; broad summary-memory not built), and the 1.5B base ceiling (mis-reads/mis-computes hard problems).

---

0.5 Performance vs the full-KV base (GSM8K, 2026-06-10)

Does bounded SP-evict compression cost accuracy vs keeping the whole CoT in KV? Measured base (rw=8000, full-KV) vs SP-evict (rw=1024, the shipping window) on GSM8K, interleaved, identical decoding (evals/ gsm8k_eval.py, evals/gsm8k_multiturn.py). Yardstick = the base model's OWN accuracy in-harness (NOT 100%).

• Single-turn (n=12): base 6/12 vs SP 8/12; base-only losses = 0 (SP never lost a problem to base). SP CoT is shorter (med 1378 vs 1894 tok) and hits the 2000-tok cap less (2 vs 6) → compression keeps generation focused. (Caveat: short problems fit the 1024 window → SP barely compresses → not a hard test.)
• Multi-turn (n=10, turn-2 depends on turn-1 which scrolled out of the window → the real compression test): base 6/10 vs SP 6/10 (TIED) once the math-turn pin injection is in. 9/10 failures are None = TERMINATION (240s timeout / 2000-tok cap hit mid-CoT) — and they hit BASE (4×) as much as SP (3×). base has full-KV verbatim context and fails the SAME way → the bottleneck is generation TERMINATION, NOT compression. Captured None cases re-run correctly → the model can solve them; it just often doesn't stop in budget (R1 CoT is long).
• Verdict: on the fair in-harness comparison there is no clear accuracy loss from compression — SP ≈ base; what's left is the 1.5B's own ceiling + a termination artifact shared by base. (Low absolute % is the cap + 4-bit, not an SP deficiency.)
• Termination fix (_gen_once, 2026-06-10): on a mid-CoT timeout/cap with no usable answer, close </think> and force "Final answer:" on a small own budget (64 tok / 45 s) to salvage an answer from the partial CoT instead of returning None. Helps base & SP equally.

1. The composite pipeline (per user turn)

 user turn
    │
    ▼
 classify(intent)  ──► chit-chat ───────────────► answer (no retrieval)
    │                  fact ──────────────────────► save (instant "Got it — saved."), persist?→disk
    ▼ question
 TieredMemory.retrieve  (first tier over the bar wins)
    ├─ L1 same-session   (in-memory turns)
    ├─ L2 prior-session  (disk jsonl — survives a fresh process)
    ├─ self-contained math? → solve, DON'T web-search
    └─ L3 web            (DuckDuckGo → Wikipedia, urllib+bs4, no headless browser)
    │
    ▼ inject matched text verbatim into the raw window ("answer from Context only")
 bounded SP-evict generation  (force-<think>, FREE gen; 2000-tok cap + 240s timeout + loop guard — see §6.5)
    │
    ▼ answer empty / timed-out / (grounded) contaminated?  ── yes ──► delete CoT, re-roll (retries=2)
    │                                            (gen/kept hold the convo; SP recomputed each step)
    ▼ no
 answer  (only post-</think> shown; <think> scratchpad hidden)

2. App-level file map

file	role
tiered_rag_mlx.py	operational core — TieredMemory (L1/L2/L3 cascade + pins working memory, fuzzy/semantic match), ChatSession (bounded SP-evict gen + injection + groundedness retry), learned router (intent_of), specificity head (specific_spans, pin gate), is_grounded. Run directly → L1/L2/L3 composite test.
evals/specificity_train.py · evals/specificity_clf.joblib	independent specificity head — train + the ~20 KB BGE linear probe that detects pin-worthy specific/proper info (CV AUC 0.996).
evals/specificity_probe.py	the bake-off that selected BGE-probe over tpc-fragmentation / surprisal-proxy for specificity detection.
evals/pin_test.py	operational test for the specificity head + pinning (detector unit 9/9; value-in-question → later-recall flow).
app_demo_mlx.py	two-session application simulation + readiness report (onboarding→disk, then a fresh "today" session over a mixed 8-turn conversation).
chat_mlx2.py	recommended interactive chat (recipe: temp 0.6, no system prompt, force-<think>, free generation + empty-answer re-roll, large rw).
mem_rag_mlx.py	conversation-memory RAG A/B (recall a fact the SP compressed away; 0/2 → 2/2).
rag_web_mlx.py	web-grounded answer (DuckDuckGo → Wikipedia) windowed-RAG.
runtime/web_search.py	keyless search backends — DuckDuckGoSearch (urllib+bs4, recommended), WikipediaSearch, PlaywrightBingSearch (brittle, optional), Tavily/Brave (API).
pooler_mlx.py · gen_mlx.py · sp_mlx.py · mt_mlx.py · build_fft_hf.py	MLX port of the pooler + SP-evict rollout, single-turn / battery / multi-turn runners, FFT→HF builder.

3. Run it

pip install mlx mlx-lm transformers torch huggingface_hub beautifulsoup4 lxml   # no Playwright needed
python build_fft_hf.py && python -m mlx_lm convert --hf-path ./fft_hf --mlx-path ./fft_mlx4 -q --q-bits 4 --q-group-size 64

python app_demo_mlx.py            # full two-session app simulation + readiness report
python tiered_rag_mlx.py         # L1/L2/L3 tier-routing composite test
python chat_mlx2.py 1000         # interactive chat

Minimal integration (the whole app loop):

import tiered_rag_mlx as T
mem  = T.TieredMemory("user_profile.jsonl")
chat = T.ChatSession(mem, rw=512)
def on_message(text):
    if T.classify(text) == "fact":                                    # store + instant ack, no generation
        return chat.turn(text, store="persist" if T.wants_persist(text) else "session", ack_only=True)[0]
    return chat.turn(text, store="none")[0]                           # free gen; turn() retries empty/contaminated

4. What's verified (composite tests)

test	result
Tier routing (app_demo, tiered_rag)	5/5 deterministic — chit-chat / fact-save / math / L1 / L2 / L3 all dispatch correctly, every run
MEMORY recall hit-rate (L1/L2, app_demo) — the architecture's metric	reliable — the SP/memory's job (hold & surface a fact across turns) lands consistently
web (L3) / math — orthogonal, reported for reference	stochastic: web = retrieval quality + base extraction; math = pure base-1.5B arithmetic with the query fully visible (no compression) — NOT a memory test, out of scope

Scoring note (don't repeat my mistake): the old "answer correctness X/5" lumped three unrelated things. What this system is measured by is MEMORY recall (L1/L2). A math miss is the base model's arithmetic, not an architecture failure — the math query is fully visible, nothing is compressed/recalled. Also: "HIT" = gold substring present (a hit-rate, a proxy), not a verified correctness rate; and any single app_demo run is 1-sample/turn (2–4/5 swing) — use N-run averages for real numbers. | same-session recall (L1) | ✅ "B3, spot 47" | | prior-session recall (L2, from disk, fresh process) | ✅ "POL-55821", "Aki" | | web grounding (L3) | ✅ "Sam Altman" (after contamination retry); Mt Fuji "3,776.24 m" | | conversation-memory RAG (mem_rag) | ✅ 0/2 → 2/2 (recover SP-compressed facts) | | reasoning correctness (5 Dolphin-R1 word problems) | SP-evict ≈ 3/5 vs base/full-KV ≈ 4/5 — same ballpark; the ceiling is the base 1.5B, not compression (§6.5 #7). Chat-answer fix surfaces clean answers ("4000 square meters.", "2 loads.") | | typical miss | base-model mis-read / mis-compute, or verbose / weak instruction-following (not a memory or compression failure) |

5. Engineering findings & fixes (the bugs the composite tests caught)

1. Headless Bing is brittle → it serves a JS shell / dictionary bot-detection page. Switched the default web backend to DuckDuckGo via plain urllib+bs4 (server-rendered results, no Chromium).
2. Non-questions web-searched & contaminated ("my room is 1408" → "1408 Film Locations" → "140813") → question-gate: only information-seeking turns retrieve.
3. Self-contained math got web-searched (would contaminate the calc) → math-gate: solve, don't search.
4. park ≠ parked, 1-word questions missed → fuzzy prefix overlap + trusted-local threshold.
5. Fact-saves produced rambly "(no answer)" → instant ack (store without generating).
6. Recency bleed ("Aki" answered for the CEO) → inject "answer from the Context block only, ignore earlier conversation" + a groundedness guard (is_grounded) that resamples when the answer's salient novel token isn't in the retrieved context (catches out-of-context / empty / question-echo).

6.5 Generation control — the long investigation (read before touching _gen_once)

How to bound/terminate generation was the hardest sub-problem. Final state and the reasoning, so the next person doesn't re-run the same dead ends:

Current committed behaviour (tiered_rag_mlx.ChatSession):

• Free generation — no in-loop "think-budget" (we do not force </think> after N tokens).
• 2-pass chat answer (_gen_once) — pass 0 thinks freely; if the post-</think> answer is weak (empty / ends in ':' / just a \boxed{}), pass 1 injects "Final answer: " and continues, forcing a clean chat answer. This fixed the dominant chat failure: the R1-distill routinely traps its answer inside <think> or emits only a preamble, so the user saw an empty reply even when the answer was computed. After the fix, math word-problems surface clean answers ("4000 square meters.", "2 loads.", "72 female guests…") instead of empties. _extract_answer is the fallback (surfaces a boxed-in-think).
• Backstops only: a 2000-token per-turn output cap (state hygiene, below) + a 240 s wall-clock timeout + a 6-identical-token degeneration guard.
• Delete-CoT re-roll (turn(retries=2)): if an answer is empty / timed-out / (for grounded turns) contaminated, roll back the turn's tokens and resample. Cheap because the hypernet recomputes the SP from the token list each step — no KV surgery.

What we learned (and retracted):

1. Termination is temperature-stochastic, NOT a model incapacity. The same prompt sometimes EOSes in ~365 tokens, sometimes rambles. So delete-CoT re-roll genuinely works for short-reasoning turns (recall/chat): a fresh sample usually terminates.
2. The per-turn output cap is load-bearing for multi-turn. Fully-uncapped free-gen lets one runaway turn (a greeting hit 3000+ tokens) flood the bounded window/SP and poison later turns (recall bled wrong values, app dropped to 1/5). The 2000 cap stops that. It is state hygiene, not think-shaping (it doesn't force </think>).
3. DO NOT re-add <think>-compaction via re-tokenization. An attempt to strip prior <think> spans from history by decode → regex → re-encode each turn corrupted the context and caused reliable non-termination (recall turns went 13–52 s HIT → 70–153 s FAIL). It was reverted. If you want clean history, track think-span token ranges at generation time and slice ids — never round-trip through text.
4. Hard math fails by OVER-thinking, not non-termination. On "$1000 at 5% compound, 3 yrs" the model computes the correct 1157 mid-<think>, then second-guesses itself over ~4000 tokens into a wrong \boxed{1000} (and the box lands inside </think>, so the post-</think> answer is empty). The 2000 cap also truncates it before it closes. So: short-reasoning → free+re-roll is right; long-reasoning → the model self-degrades.
5. Temperature is load-bearing — and 0.4 was too low. A controlled single-turn sweep of the compound-interest problem: temp 0.3 → 5632 tok, no </think>, timeout (deterministic over-think rut); temp 0.9 → 4843 tok, no </think>, timeout + wrong \boxed{220} (wanders); temp 0.6 → 1730 tok, closes </think> in 71 s, correct \boxed{1157.63}. Both extremes over-think; 0.6 (DeepSeek-R1's recommended 0.5–0.7) is the sweet spot. TEMP was 0.4 (set for "faithful copying"), which contributed to the math over-thinking — now 0.6. So #4's "math self-degrades" was partly a too-low-temp artifact, like the compaction bug. (Caveat: this is the isolated result; the multi-turn app_demo score is high-variance run-to-run — 2–4/5 — and temp is not a silver bullet there.)
6. No clean automatic fix for the long-reasoning tail. A fixed think-budget would catch the early-correct answer — but you can't tell a priori whether a long chain is over-thinking or genuinely necessary, so a global budget would harm problems that truly need long reasoning. Left as free-gen. Real fix = a calculator/tool for arithmetic, or a better base.
7. The accuracy ceiling is the base 1.5B, NOT the SP compression. Eval on 5 Dolphin-R1 reasoning problems (grade-school word problems), 2-pass chat-answer on: SP-evict (rw=512, compressed) ≈ 3/5 vs base (rw=8000, nothing compressed, full-KV) ≈ 4/5 — same ballpark, within 1-sample noise. So the modest ~60-80% correctness is the 1.5B's own level on this corpus, not a compression artifact; misses are the base model's mis-reads / mis-computes, not lost context. (Single sample/turn — use N-run averages for a real gap. The chat-answer fix in _gen_once is what makes both surface clean answers.)

6.7 Organic multi-turn consistency (_matches, auto-log, compute-vs-quote)

A flowing conversation with dependencies and a correction — "bookshelf, 5 shelves" → "each 80 cm, total width?" → "paint it blue" → "make it 6 shelves, redo the width" → "remind me the colour and shelf count" — totally failed before (the model lost the specifics to the SP and hallucinated: "75 books", "gray", "12 m wall", "15 shelves"). Root cause: organic conversational state flows only through the lossy SP-compressed KV; it was never written to the retrievable memory. Fix (all in tiered_rag_mlx):

1. Auto-log statements — turn() writes every non-question user turn to mem.session (L1) even when store="none", so the conversation's facts become retrievable verbatim.
2. Multi-fact retrieval — _matches returns the top-cap (≈3) near-top-scoring chunks (not just the single best), chronological, so a 2-fact question ("colour AND count") gets BOTH, and a later correction ("6 shelves instead of 5") is read after the original. A strong single match (a unique code) still drops weak distractors (relative-score threshold).
3. Compute-vs-quote — a math turn that pulls a fact from context gets a "use the fact and compute" instruction (and skips the groundedness check), while a recall turn keeps "quote verbatim, use the most recent if corrected".
4. rw default 1024 — wider window keeps more recent dialogue verbatim.

After the fix the same conversation: width computed from the recalled count (400, then 480 after the 5→6 correction), and the colour recalled correctly (blue, vs "gray" before). Verified no app_demo regression (routing 5/5, recall/web/math hold). Residual glitches (a trivial statement that over-thinks to timeout; "how many shelves" answered as "480 cm") are 1.5B response-quality quirks, not the memory.

Scaling to MANY rallies (10-turn party conversation, 2 corrections). Pushing further surfaced the final design — and two dead ends to avoid:

• _is_factlike gate — only log statements that assert a value/attribute. Logging questions or chit-chat ("explain compound interest") poisons recall: it bled "compound interest"/"1157.63" into "where did I park?". Gate it out.
• Lexical top-K, NOT full-log injection. Injecting the whole recent log on a recall makes the context large and the model recency-bleeds the most recent number (park → "1157.63"). Keep the injected context small (_matches, cap≈4) with ties broken by recency so corrections still win.
• compute_ref — a referential imperative ("recompute the total", "redo it") isn't a question and has no lexical anchor → inject the recent log + a compute instruction.
• Result on the 10-rally party: a mid-conversation 2-fact check is perfect ("chocolate, 10 guests"); the final 4-fact check is 3/4 — both corrections held (theme→dinosaur, guests 8→10, cake→chocolate) and only the oldest fact (turn-1 "Mia") faded. That tail is the fundamental bounded-recency limit: very old facts drop after ~many turns; fixing it needs fact-pinning / summarisation (not done). Verified no app_demo regression (recall 3/3, web+math hold).

6. Known limits (the app-quality boundary — intentionally not chased further)

• Math / long reasoning — the model over-thinks and self-degrades on hard chains (§6.5); pair with a calculator/tool. Short reasoning is fine.
• Groundedness is necessary-not-sufficient — rejects out-of-context / empty / echo answers, but can't catch a wrong in-context span (right page injected, wrong fact quoted). Needs a span-constrained extractor or NLI/LLM judge.
• Retrieval is fuzzy-lexical — paraphrased questions with no shared words can miss; swap in a BGE sentence encoder (runtime/) for semantic matching.
• Latency — ~10–25 s/turn (always thinks); fact-saves instant; web +1–5 s. Free scrapers rate-limit bursts (back off + Wikipedia fallback; use an API key for production).

7. Extension points (if picked up later)

• Semantic retrieval: replace _best/_overlap with a BGE sentence encoder (the runtime/ pipeline already has BGE rerank + a verify/refuse gate — promote it into TieredMemory).
• Tool use: a calculator tool for the math turns; a date-guard for "today/now" web queries.
• Stronger contamination judge: span-constrained decoding or a tiny NLI check for wrong-span answers.
• iPhone: retrieval ports directly (URLSession + SwiftSoup); the model needs the pooler + SP-evict rollout re-implemented on mlx-swift (numerics proven; it's a port).

8. Related docs

• OPERATING.md — full runbook (install, recipe table, runners, tiered stack, contamination guard)
• README.md — portfolio writeup (idea, models, RAG, results)
• RESULTS.md — honest verification numbers
• HANDOFF.md — research handoff (training, deepKL, architecture)