Update SkyIsNotGreen/splinter-memory from the HiveBench Studio

README.md

@@ -74,13 +74,53 @@ milliseconds.
 | Retrieval precision (sentence proxy) | near-chance past the window | ≥70% / ≥85% | **10.7%** - *the encoder ceiling* (§7) |
 | Context sufficiency vs. an equal-size FIFO window | — | ≥80% of turns | **85.1%** of the fact-retrievable turns (4.4:1) |
 | Routing accuracy | — | ≥80% / ≥92% | **100%** |
-| Added per-turn latency | 0 ms | <50 / <30 ms | **≈18 ms** (3.4 ms assembly + ~15 ms scoring) |
+| Added per-turn latency | 0 ms | <50 / <30 ms | **≈18 ms** on a small store (3.4 ms assembly + ~15 ms scoring) - **and it grows with the store**: 22.8 ms at 212 chunks, 156.8 ms at 1,612 (§7.4) |
 | Peak RSS over 500 turns | — | small | **34.7 MB** |
 | Pipeline Efficiency Score | ~30 | ≥65 / ≥80 | **80.0 (GREEN)** live, 73.1 on the 20-conversation evidence run |
 
 Full protocol, per-hypothesis logic chains and falsification conditions:
 §5 of the canonical paper.
 
+### The whole protocol, refreshed on the 35B-A3B
+
+The P1-P11 protocol re-run live against the **35B-A3B ternary** (a
+12-conversation fixture subset), to ask whether the verdict set is a
+property of the architecture or of the generator it was measured on:
+
+| | verdict | the evidence |
+|---|---|---|
+| P1 constant throughput | **PASS** | the throughput does not decay |
+| P2 retrieval precision | **FAIL** | the precision clause (the known encoder ceiling, §7) |
+| P3 context sufficiency | **PASS** | the curated context against an equal-size FIFO window |
+| P4 domain-dependent decay | **PASS** | code m90 1.8 vs prose m90 1.2 - a gap of 0.6, beyond the 0.2 band |
+| P5 targeted masking | SKIP | a training experiment |
+| P6 confidence escalation | **FAIL** | the escalated recall is worse (0.174 vs 0.565) |
+| P7 auditor-human agreement | SKIP | it needs human raters (90.25% measured 2026-08-23) |
+| P8 routing accuracy | **PASS** | 200/200 |
+| P9 densest-duplicate retention | **PASS** | the densest chunk wins 100% of the turns |
+| P10 drift reset | **FAIL** | the reset changes the recall by 0.0 points |
+| P11 comb resurrection | **PASS** | the pressure regime: 100% topic-return recall against 20% without the comb |
+
+**PASS 6/11, 2 skipped - the published verdict set, exactly.** The
+published run (2026-08-23/24, on a 27B, on the full corpus) reported
+P1/P3/P4/P8/P9/P11 PASS, P2/P6/P10 FAIL, P5/P7 SKIP. The same set
+reproduces on a different model and a smaller corpus, which says the
+protocol measures the architecture rather than the generator.
+
+**One of the failures now has a candidate mechanism.** P10 (the drift
+reset) measures a detector whose threshold was calibrated for a different
+encoder (§7): forced on, it fires on 571 of 628 turns, so "the reset has
+no effect" may be measuring a detector that fires on almost everything
+rather than a reset that does not work. The verdict stands as measured;
+the mechanism is now in question, which is a different claim.
+
+**And the one result that does not carry over is P3's margin.** On the
+27B the curated context won 115 turns outright to 26; on the 35B it is a
+tie (§7.3). The *verdict* holds (the curated context is sufficient, and
+at least as good on 88.4% of the turns); the *margin* does not. The
+curation's accuracy advantage is a function of the generator - what
+survives for a strong one is the bounded context and the cost.
+
 ## Use it
 
 Three ways, in increasing order of control:
@@ -140,13 +180,66 @@ generates → the reply is observed back into the store.
    the time, which is enough to drive parameter tuning without a human
    labelling pass.
 
+## The harness that runs it, and the pipeline that manages it
+
+The memory system is a library; the **HiveBench Studio** is the harness
+that runs it, and it is where the management pipeline lives.
+
+### The management pipeline
+
+A long-horizon memory has to be *managed*: every knob that decides what
+the model sees used to live in three different menus, and a wrong setting
+was invisible. The Studio has one place for all of it - the **Global
+harness controls** - and every block of it is a *baseline* that the more
+specific thing beats:
+
+| block | what it decides |
+|---|---|
+| **the encoder** | which small model does the retrieval and the memory management - you choose it, and the next conversation runs on it (no restart) |
+| **Memory & RAG** | the curation baseline: the context window, the stale wall, the dedup and drift thresholds, the comb, the medium drone |
+| **Quality** | the cascade's judge, repair, gate and escalation - the defaults every run starts from |
+| **MCP** | what an external client may read and write of the memory |
+| **Runtime** | the sampling a reply is drawn with, and the context, GPU layers and threads an engine is launched with |
+
+The point is not tidiness. It is that **a setting which is not consulted
+is indistinguishable from a setting which does not work**, and the only
+way to know which one you have is to wire them and then measure.
+
+### The harness itself
+
+| | |
+|---|---|
+| the console | Home, Settings, Global, Cascade, and a "More tools" group (Agent, Engines, Local library, Providers, Hub, Inspector, Training) |
+| the chat | Splinter mode (one curated reply) or Agent mode (the tool loop); the reply says who produced it |
+| the launch panel | llama.cpp or Strata, per engine, from the profiles; stop closes the connection so the *engine* stops; unload gives the VRAM back and retires the orphans |
+| the cascade console | every runner knob in one form, with the Global tab's Quality block as its starting values |
+| the harvest | every turn (the query, the reply, the curation that was used) written to a jsonl, off by default - this is the data the encoder is trained on |
+
+### Answer quality reporting and API escalation
+
+The curation's effect is not only felt, it is reported. Every cascade run
+writes a per-item report (the verdict, the detector findings, the repair
+attempt, the ladder's resolution), a provenance manifest (the model
+sha256, the stack, the git commit, the task-set sha, the decode
+settings) and a generated model card; and the escalation lane harvests
+every ladder-resolved item to `escalation.jsonl` next to the report -
+training-side only, and read by nothing in the runtime.
+
+The escalation ladder itself is the last stage: an answer the local
+verifiers still flag can go to a human gate (verbose on the first
+trigger, compact afterwards) and then to an API tier, with the served
+answer, the cost and the reason recorded. The companion paper
+(*The Hive Cascade*) documents the loop, the gold-free runtime contract
+and the measured verdicts; the benchmark and the run reports are
+published next to this card.
+
 ## The evidence, hypothesis by hypothesis
 
 | | Hypothesis | Verdict |
 |---|---|---|
 | P1 | Constant throughput over 500 turns | **PASS** |
 | P2 | Retrieval precision ≥85% at ≥90% recall | **SPLIT** - recall PASS (90.3%), precision FAIL (the encoder ceiling) |
-| P3 | Context sufficiency ≥80% of turns vs. an equal-size FIFO window | **PASS** (85.1%) |
+| P3 | Context sufficiency ≥80% of turns vs. an equal-size FIFO window | **PASS** (85.1% on the 27B); on the 35B-A3B the answer-level result is a tie (88.4% at-least-as-good, 13 strict wins to 17) - see §7.3 |
 | P4 | Domain-dependent decay curve (code vs. prose separate) | **PASS** (code m90 1.8 vs. prose 1.2) |
 | P5 | Targeted masking | **PASS** |
 | P6 | Confidence escalation | **FAIL** |
@@ -195,11 +288,118 @@ not re-derived.
 | drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
 | drift as a *ranking* signal | **AUC 0.945** - the signal is strong; the threshold is wrong |
 
-The encoder is not the bottleneck. The **operating points** are - and they
-are per-encoder, so they have to travel with the model. The full sweep
-(model families, sizes, dimensions, CPU cost per chunk) is in
-`docs/ENCODER-NOTES.md` in the harness repository, and the evaluation sets
-are published as a dataset next to this card.
+### 7.1 The F1 is the wrong target, and the threshold is not the lever
+
+The obvious conclusion - "move the thresholds to the encoder's optimum" -
+is **falsified** by a deterministic replay of the corpus through the real
+pipeline (50 conversations, 493 measurable turns, the P2 diagnostic's own
+notion of a fact; `experiments/threshold_ab.py`):
+
+| arm | dedup / drift | fact recall | context | verdict |
+|---|---|---|---|---|
+| shipped | 0.92 / 0.6 | **50.6%** | 159 tok | - |
+| the dedup alone | 0.57 / 0.6 | **39.7%** | 90 tok | **falsified** |
+| the drift alone | 0.92 / 0.77 | **50.6%** | 159 tok | **survives** |
+
+**The duplicate test's two errors are not interchangeable.** A duplicate
+test that fires wrongly *deletes* a chunk, and the fact inside it is gone
+irreversibly; a duplicate test that fails to fire only leaves a redundant
+chunk in the context, which costs budget and nothing else. F1 weights
+them equally, so F1 points at the wrong threshold: at 0.57 the encoder
+merges chunks that are *about* the same thing but carry *different* facts,
+and the fact recall falls by a fifth. The shipped 0.92 is conservative
+**for a reason** - its precision is 1.00, so it never deletes a fact - at
+the price of catching only 5% of the redundancy.
+
+The topic-drift test is the opposite case: it only *penalises* the
+off-topic chunks, it never deletes them, so its recalibration is free.
+
+**The model is the lever, not the threshold.** A small encoder tuned on
+the harness's own traffic (17M parameters, one epoch, the dedup task
+only, labelled with the same lexical rule) merges **six times more true
+duplicates** at the *same* precision - and the fact recall goes *up*
+slightly, not down:
+
+| encoder | fact recall | the dedup's recall at 0.92 | the dedup's precision |
+|---|---|---|---|
+| the stock L3-v2 | 50.55% | 0.046 | 1.00 |
+| the tuned encoder | **50.93%** | **0.292** | 1.00 |
+
+The encoder can be improved because it improves the *ranking* of what is a
+duplicate. The threshold cannot be lowered because lowering it moves the
+*boundary*, and the boundary is where the distinct facts get caught.
+
+### 7.4 The curation's cost grows with the conversation it curates
+
+The stability run - 2,000 turns of a growing conversation, the real drone,
+no model (`experiments.stability_500.py`):
+
+| turns | the store | the assembled context | the curation's cost (p50) | the peak RSS |
+|---|---|---|---|---|
+| 250 | 212 chunks | 322 tok | 22.8 ms | 519 MB |
+| 500 | 412 chunks | 320 tok | 37.3 ms | 564 MB |
+| 1,000 | 812 chunks | 335 tok | 89.7 ms | 594 MB |
+| 2,000 | 1,612 chunks | 332 tok | 156.8 ms | 667 MB |
+
+**The context is flat and the cost is not.** The assembled context moves
+by 10 tokens across a 7.6x range of store - the curation's promise holds
+✓✓ - and nothing is OOM-killed over 2,000 turns ✓. But the curation's
+per-turn cost is **linear in the store**: about 7x for 7.6x the chunks,
+because every chunk is scored every turn.
+
+So the memory's trade is explicit, and the published "≈18 ms added per
+turn" is the *small-store* number: the prompt stays bounded (the
+expensive part for the model) while the curation gets more expensive as
+the conversation grows. The two costs trade against each other, and the
+crossover - the point where curating costs more than the tokens it saves
+- is a measurement the system does not yet make.
+
+### 7.3 The curation's advantage is generator-dependent
+
+The paired A/B - the same model, the same turns, the curated context
+against the last-4k-token window of the same history - is the memory
+system's headline measurement (P3). It was measured on a 27B, and it has
+now been measured on the **35B-A3B ternary** (180 measurable turns):
+
+| | the curated arm | the FIFO arm |
+|---|---|---|
+| the answer's fact recall | 81.1% | 81.7% |
+| the context fidelity (the answer's terms sourced from its own context) | **0.241** | 0.208 |
+| the turns where the answer is at least as good | **88.4%** (the bar: 80%) | - |
+| the strictly-better turns | 13 | 17 |
+| the context sufficiency (P3-style) | **97.5%** >= FIFO | - |
+
+**On the 27B the curated arm won 115 turns outright to 26 (4.4:1). On the
+35B it is 13 to 17** - a tie - and the answers are equal (81.1% against
+81.7%). The "neither" bucket collapses too (145 of 175 on the 27B, 34 of
+180 here): the 35B answers from *whatever* context it is given.
+
+**The scoping result: the better the generator is at long context, the
+less the curation has to add.** The FIFO window is the most recent turns
+verbatim, and a model that uses them well does not need a curation to find
+them. This is what Postulate 3 (the separation) implies - the curation is
+a substitute for a *weaker* reader - and it is the first thing the
+measurements had not tested.
+
+**What survives.** The curated context is still sufficient on 97.5% of the
+turns, its fidelity is ahead, and - the part the numbers above do not
+touch - the curation is what keeps the context **bounded**: the FIFO arm
+is 4k tokens of pure recency, and the two arms are equal *at that
+budget*. The honest claim for a strong generator is the bounded context
+and the cost, not the better answers.
+
+### 7.2 What we changed, and what we did not
+
+- The shipped configuration stays: the stock encoder at the shipped
+  thresholds, because that is the configuration the published evidence was
+  produced with.
+- The next fine-tune weights a false merge above a missed merge
+  (precision-weighted at the operating point) instead of optimising F1 -
+  the asymmetry makes F1 the wrong target - and is measured on a held-out
+  split, because the current one was trained and measured on the same
+  traffic (a fit, not a generalisation).
+- The full sweep, the fine-tune runs and the replay are in
+  `docs/ENCODER-NOTES.md` in the harness repository.
 
 ## Limits, honestly
 
@@ -217,6 +417,20 @@ are published as a dataset next to this card.
   isolate conversations per store and filter refusals out of the store.
 - **The decay curve is an engineering heuristic** with cognitive-science
   inspiration, not a validated model of human forgetting.
+- **The large tier is a state, not a model.** The 176B-A6B answers at
+  60-73 tok/s warm and 3 tok/s cold, on the same engine and the same
+  request: a fresh prompt read costs 5 tok/s and a reused prefix 71, and
+  the cold phase is the OS pulling 60 GB of expert weights off the NVMe
+  (the file is 20.8% page-cached on a 30 GB box). Warm and alone it
+  answers the cascade's 28 tasks in 26.9 s an item; co-sited with the
+  small tier, in 229.7 s. A deployment that measures the large tier
+  cold, or with another engine resident, measures the box and not the
+  model. Two levers pay and the rest do not: keep it warm, and run it
+  alone. A whole-file prefetch does not - a 13 GB page cache cannot hold
+  a 60 GB file, and the kernel's `WILLNEED` left it at 0% cached for
+  80 s. The engine's resident tier (`--resident-budget-gib`) does: it
+  keeps what the VRAM cache cannot hold in ordinary RAM, and the storage
+  read on the second request falls from 1.4 GB to 146 MB.
 
 ## Reproduce
 
@@ -225,7 +439,10 @@ are published as a dataset next to this card.
 pytest tests/unit tests/integration -q
 
 # the encoder sweep on your own traffic
-python -m experiments.encoder_suite --encoders l3 l6 bge-small
+python -m experiments.encoder_suite --encoders l3 l6 bge-small e5-small gte-small
+
+# the threshold replay (deterministic: no model, no generation)
+python -m experiments.threshold_ab
 
 # the cascade pilot (local model, deterministic verifiers)
 open http://127.0.0.1:8765/cascade
@@ -234,6 +451,19 @@ open http://127.0.0.1:8765/cascade
 The repository ships the run bundles, the event logs and the label files, so
 a verdict can be traced back to the bytes that produced it.
 
+### The artifacts
+
+| artifact | what it is |
+|---|---|
+| `SkyIsNotGreen/splinter-memory` | this paper, the figures, the license and the citation |
+| `SkyIsNotGreen/splinter-encoder` | the tuned drone encoder, its card, its configuration and its gate results |
+| `SkyIsNotGreen/hive-cascade` | the cascade's benchmark (a 70/30 split) and two full run reports |
+| `SkyIsNotGreen/Scion-35B-A3B` | the ternary 35B-A3B model and its decision head |
+| `SkyIsNotGreen/Scion-FlashNext-176B-A6B` | the ternary 176B-A6B model |
+
+The releases are private while the results are being refreshed; they open
+when the refreshed numbers are in.
+
 ## License and citation
 
 MIT. See `LICENSE` and `CITATION.cff` in this repository.