Point the citation at the Hive monorepo
f8589c0 verified 1 day ago
25.7 kB
---
license: mit
library_name: sentence-transformers
tags:
  - context-curation
  - long-context
  - memory-management
  - conversational-ai
  - retrieval
  - llm
  - research-paper
pipeline_tag: sentence-similarity
---
# Long-Context Memory Curation Management

**Splinter Memory — a managed-decay context curation architecture for long-horizon LLM conversations.**

This card is the Hugging Face edition of the Splinter Memory white paper. It
is written for the people who build LLM applications and want the context
window to *stay useful* as the conversation grows: the memory layer, the
mechanisms, the measured results, and the code that implements them.

The canonical paper lives with the code
([sky-is-green/hive/tree/main/splinter](https://github.com/sky-is-green/hive/tree/main/splinter));
this edition adds the encoder measurement taken on the harness's own traffic
(§7) and the operational notes a practitioner needs.

---

## The problem, in one paragraph

A long conversation is not a long document. Facts that mattered twenty turns
ago fall out of a rolling context window; the model answers from what
survives, not from what happened; and the surviving context is mostly noise
by then. Truncation and recency are cheap and wrong: they throw away
exactly the old fact the user came back for. Keeping everything is worse -
the prompt grows, the KV cache grows, the cost grows, and the model
"loses the middle" while the interesting fact sits somewhere in the middle.

## What this does

Splinter Memory keeps a **managed** memory outside the model and hands the
model a **bounded, relevance-ranked** context every turn:

- a small fleet of **bidirectional encoders** ("drones") scores every stored
  chunk against the current query - cheap, and far better at similarity than
  a generative model of the same size;
- a **membrane** drops what is said twice and notices when the topic
  changes;
- a **retention layer** makes forgetting a *design parameter* - an
  escalating-friction decay, a remembrance pass that rescues facts worth
  keeping, and a "comb" (an SSD tier) that resurrects topics that returned
  after being evicted;
- a **focal layer** assembles a bounded context on an adaptive token budget
  and hands it to any OpenAI-compatible backend (llama.cpp, LM Studio, vLLM,
  a hosted API).

The model never sees the raw history. It sees the part that is worth its
attention, every turn, at a cost that does not grow with the conversation.

**The separation is the point.** The generative model is good at producing
answers and the wrong tool for selecting context; measured live, using the
27B generative model as an encoder proxy cost tens of seconds per turn on
the hot path, while the 17M-parameter drone did the same job in
milliseconds.

## The measured results

| Metric | Naive FIFO (baseline) | Target | Measured |
|---|---|---|---|
| Throughput over 500 turns | decays | flat ±10% | **flat** (14.5 → 15.5 tok/s over 308+ turns), 0 OOM events |
| Retrieval recall (facts the model actually stated) | n/a | ≥75% / ≥90% | **90.3%** live, 93.5% on the isolated-store replay |
| Retrieval precision (sentence proxy) | near-chance past the window | ≥70% / ≥85% | **10.7%** - *the encoder ceiling* (§7) |
| Context sufficiency vs. an equal-size FIFO window | — | ≥80% of turns | **85.1%** of the fact-retrievable turns (4.4:1) |
| Routing accuracy | — | ≥80% / ≥92% | **100%** |
| Added per-turn latency | 0 ms | <50 / <30 ms | **≈18 ms** on a small store (3.4 ms assembly + ~15 ms scoring) - **and it grows with the store**: 22.8 ms at 212 chunks, 156.8 ms at 1,612 (§7.4) |
| Peak RSS over 500 turns | — | small | **34.7 MB** |
| Pipeline Efficiency Score | ~30 | ≥65 / ≥80 | **80.0 (GREEN)** live, 73.1 on the 20-conversation evidence run |

Full protocol, per-hypothesis logic chains and falsification conditions:
§5 of the canonical paper.

### The whole protocol, refreshed on the 35B-A3B

The P1-P11 protocol re-run live against the **35B-A3B ternary** (a
12-conversation fixture subset), to ask whether the verdict set is a
property of the architecture or of the generator it was measured on:

| | verdict | the evidence |
|---|---|---|
| P1 constant throughput | **PASS** | the throughput does not decay |
| P2 retrieval precision | **FAIL** | the precision clause (the known encoder ceiling, §7) |
| P3 context sufficiency | **PASS** | the curated context against an equal-size FIFO window |
| P4 domain-dependent decay | **PASS** | code m90 1.8 vs prose m90 1.2 - a gap of 0.6, beyond the 0.2 band |
| P5 targeted masking | SKIP | a training experiment |
| P6 confidence escalation | **FAIL** | the escalated recall is worse (0.174 vs 0.565) |
| P7 auditor-human agreement | SKIP | it needs human raters (90.25% measured 2026-08-23) |
| P8 routing accuracy | **PASS** | 200/200 |
| P9 densest-duplicate retention | **PASS** | the densest chunk wins 100% of the turns |
| P10 drift reset | **FAIL** | the reset changes the recall by 0.0 points |
| P11 comb resurrection | **PASS** | the pressure regime: 100% topic-return recall against 20% without the comb |

**PASS 6/11, 2 skipped - the published verdict set, exactly.** The
published run (2026-08-23/24, on a 27B, on the full corpus) reported
P1/P3/P4/P8/P9/P11 PASS, P2/P6/P10 FAIL, P5/P7 SKIP. The same set
reproduces on a different model and a smaller corpus, which says the
protocol measures the architecture rather than the generator.

**One of the failures now has a candidate mechanism.** P10 (the drift
reset) measures a detector whose threshold was calibrated for a different
encoder (§7): forced on, it fires on most turns, so "the reset has no
effect" may be measuring a detector that fires far too often rather than
a reset that does not work. (The exact rate was measured on the
pre-correction corpus and is withdrawn with the rest of §7.) The verdict
stands as measured; the mechanism is now in question, which is a
different claim.

**And the one result that does not carry over is P3's margin.** On the
27B the curated context won 115 turns outright to 26; on the 35B it is a
tie (§7.3). The *verdict* holds (the curated context is sufficient, and
at least as good on 88.4% of the turns); the *margin* does not. The
curation's accuracy advantage is a function of the generator - what
survives for a strong one is the bounded context and the cost.

## Use it

Three ways, in increasing order of control:

**1. As a sidecar** - the Studio serves an OpenAI-compatible
`/v1/chat/completions` that curates before it generates. Point your client
at it; nothing else changes.

```bash
git clone https://github.com/sky-is-green/hive
cd hive
python -m pip install -e splinter
python -m harness --port 8765
```

**2. As a library** - the memory system is importable, engine-agnostic and
has no GPU requirement for the encoder tier.

```python
from cortex.splinter import Splinter
from cortex.config import SplinterConfig
s = Splinter(config=SplinterConfig(max_context=8192))
result = s.process_turn("what did we decide about the KV cache?", "my-conversation")
print(result.assembled.content)     # the curated context, bounded
```

**3. As the benchmark** - the P1-P12 protocol, the deterministic
diagnostics and the self-contained run bundles are in the repository, so the
claims can be re-measured on your own hardware and your own backend.

## How it works

| Layer | Function | Core mechanisms |
|---|---|---|
| **Cortex** | Orchestration & routing | task classification, drone dispatch, congestion detection, graceful degradation |
| **Sieve** | Relevance scoring | ultra-small encoder (17M params, 384-dim) for fast similarity; a medium encoder for uncertain cases; confidence via prediction variance |
| **Membrane** | Selective filtering | semantic deduplication (cosine > 0.92, keep the densest), topic-drift detection and reset |
| **Retention** | Memory & decay | remembrance pass, escalating-friction decay, stale-context acceleration, the "comb" (surplus SSD tier, topic-return resurrection) |
| **Focal** | Assembly | adaptive token budget (1k-6k by route tier), confidence-weighted sorting, bounded context construction |

**Per turn:** user input → Cortex classifies it → Sieve scores every stored
chunk → Membrane deduplicates and checks the topic → Retention applies decay
and the remembrance pass → Focal assembles a bounded context → the backend
generates → the reply is observed back into the store.

### The four postulates

1. **Context curation** - for a fixed token budget, a relevance-ranked
   selection from the full history dominates a contiguous window of the same
   size, and the gap widens with conversation length.
2. **Managed decay** - forgetting is a design parameter, not a failure mode;
   escalating friction beats both unbounded growth and uniform forgetting.
3. **Separation** - bidirectional encoders for comprehension, an
   autoregressive decoder for generation, is the better allocation of the
   same compute.
4. **Ground-truth bootstrap** - an LLM auditor asked about *context
   utilization* (not answer correctness) agrees with human labels 90.25% of
   the time, which is enough to drive parameter tuning without a human
   labelling pass.

## The harness that runs it, and the pipeline that manages it

The memory system is a library; the **HiveBench Studio** is the harness
that runs it, and it is where the management pipeline lives.

### The management pipeline

A long-horizon memory has to be *managed*: every knob that decides what
the model sees used to live in three different menus, and a wrong setting
was invisible. The Studio has one place for all of it - the **Global
harness controls** - and every block of it is a *baseline* that the more
specific thing beats:

| block | what it decides |
|---|---|
| **the encoder** | which small model does the retrieval and the memory management - you choose it, and the next conversation runs on it (no restart) |
| **Memory & RAG** | the curation baseline: the context window, the stale wall, the dedup and drift thresholds, the comb, the medium drone |
| **Quality** | the cascade's judge, repair, gate and escalation - the defaults every run starts from |
| **MCP** | what an external client may read and write of the memory |
| **Runtime** | the sampling a reply is drawn with, and the context, GPU layers and threads an engine is launched with |

The point is not tidiness. It is that **a setting which is not consulted
is indistinguishable from a setting which does not work**, and the only
way to know which one you have is to wire them and then measure.

### The harness itself

| | |
|---|---|
| the console | Home, Settings, Global, Cascade, and a "More tools" group (Agent, Engines, Local library, Providers, Hub, Inspector, Training) |
| the chat | Splinter mode (one curated reply) or Agent mode (the tool loop); the reply says who produced it |
| the launch panel | llama.cpp or Strata, per engine, from the profiles; stop closes the connection so the *engine* stops; unload gives the VRAM back and retires the orphans |
| the cascade console | every runner knob in one form, with the Global tab's Quality block as its starting values |
| the harvest | every turn (the query, the reply, the curation that was used) written to a jsonl, off by default - this is the data the encoder is trained on |

### Answer quality reporting and API escalation

The curation's effect is not only felt, it is reported. Every cascade run
writes a per-item report (the verdict, the detector findings, the repair
attempt, the ladder's resolution), a provenance manifest (the model
sha256, the stack, the git commit, the task-set sha, the decode
settings) and a generated model card; and the escalation lane harvests
every ladder-resolved item to `escalation.jsonl` next to the report -
training-side only, and read by nothing in the runtime.

The escalation ladder itself is the last stage: an answer the local
verifiers still flag can go to a human gate (verbose on the first
trigger, compact afterwards) and then to an API tier, with the served
answer, the cost and the reason recorded. The companion paper
(*The Hive Cascade*) documents the loop, the gold-free runtime contract
and the measured verdicts; the benchmark and the run reports are
published next to this card.

## The evidence, hypothesis by hypothesis

| | Hypothesis | Verdict |
|---|---|---|
| P1 | Constant throughput over 500 turns | **PASS** |
| P2 | Retrieval precision ≥85% at ≥90% recall | **SPLIT** - recall PASS (90.3%), precision FAIL (the encoder ceiling) |
| P3 | Context sufficiency ≥80% of turns vs. an equal-size FIFO window | **PASS** (85.1% on the 27B); on the 35B-A3B the answer-level result is a tie (88.4% at-least-as-good, 13 strict wins to 17) - see §7.3 |
| P4 | Domain-dependent decay curve (code vs. prose separate) | **PASS** (code m90 1.8 vs. prose 1.2) |
| P5 | Targeted masking | **PASS** |
| P6 | Confidence escalation | **FAIL** |
| P7 | Auditor-human agreement ≥90% | **PASS** (90.25% on 400 items) |
| P8 | Heuristic routing floor | **PASS** (100%) |
| P9 | Densest-duplicate survives deduplication | **PASS** |
| P10 | Drift reset | **FAIL** |
| P11 | Comb resurrection of returned topics | **PASS** (deterministic and live-validated) |
| P12 | Store-time fact distillation | protocol only |

The failures are kept in the paper with their evidence. A pre-registered
protocol that only reports passes is not evidence.

## 7. The encoder: what we measured on our own traffic

> **Correction (2026-10-21).** The corpus behind this section included an
> ingested knowledge-base attachment: 17 of its 82 files were chunked
> `[block n/m]` documents (5,531 of 6,977 chunks), and one conversation
> was loaded from two state directories. The drift figures below (the
> 0.945 AUC and the "fires on 571 of 628 turns" mechanism) are
> **withdrawn**, and the tuned encoder was rejected by the replay (see
> below). The shipped encoder and the shipped 0.92 stay. The correction
> and the clean measurement are in `AB-RESULTS.md` §2C-septies; the
> retrieval results elsewhere in this card come from different runs and
> are unaffected.

The harness gives its encoder three jobs, and only one of them is a ranking:

| job | question | where |
|---|---|---|
| **rank** | which stored chunk answers this query? | the sieve |
| **dedup** | do these two chunks say the same thing? | the membrane (cosine > 0.92) |
| **drift** | did the conversation change topic? | the membrane (1 - cosine > 0.6) |

The prior encoder comparison measured **rank** only, on the synthetic
fixtures, and found that six encoders (a 22M MiniLM, a 400MB cross-encoder,
a 568M retrieval-specialised encoder and two contrastively tuned variants)
all landed on the same top-K curve - so it concluded the ceiling was
*data-structural* and that the remedy was "stop looking for a better
encoder".

We re-measured all three jobs on the harness's **own** traffic - 81
conversations, 6,975 stored chunks, 4,056 duplicate pairs and 6,533
topic-change cases, labelled lexically so the label is independent of the
encoder being measured (`experiments/encoder_suite.py`). The result
changes the picture, and it is not about the ranking:

**The thresholds that ship with the harness were calibrated for a different
encoder.** The production drone was swapped from `all-MiniLM-L6-v2` to
`paraphrase-MiniLM-L3-v2` on 2026-08-23; the membrane's two thresholds were
not re-derived.

| operating point | measured |
|---|---|
| dedup at the shipped threshold (0.92) | precision 1.00, **recall 0.03** - the duplicate test almost never fires |
| dedup at its own best threshold (~0.56) | F1 **0.75** (precision 0.62, recall 0.95) |
| drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
| drift as a *ranking* signal | **withdrawn** - measured on the pre-correction corpus (79% an ingested attachment), see above |

### 7.1 The F1 is the wrong target, and the threshold is not the lever

The obvious conclusion - "move the thresholds to the encoder's optimum" -
is **falsified** by a deterministic replay of the corpus through the real
pipeline (50 conversations, 493 measurable turns, the P2 diagnostic's own
notion of a fact; `experiments/threshold_ab.py`):

| arm | dedup / drift | fact recall | context | verdict |
|---|---|---|---|---|
| shipped | 0.92 / 0.6 | **50.6%** | 159 tok | - |
| the dedup alone | 0.57 / 0.6 | **39.7%** | 90 tok | **falsified** |
| the drift alone | 0.92 / 0.77 | **50.6%** | 159 tok | **survives** |

**The duplicate test's two errors are not interchangeable.** A duplicate
test that fires wrongly *deletes* a chunk, and the fact inside it is gone
irreversibly; a duplicate test that fails to fire only leaves a redundant
chunk in the context, which costs budget and nothing else. F1 weights
them equally, so F1 points at the wrong threshold: at 0.57 the encoder
merges chunks that are *about* the same thing but carry *different* facts,
and the fact recall falls by a fifth. The shipped 0.92 is conservative
**for a reason** - its precision is 1.00, so it never deletes a fact - at
the price of catching only 5% of the redundancy.

The topic-drift test is the opposite case: it only *penalises* the
off-topic chunks, it never deletes them, so its recalibration is free.

**The model is not the lever either.** A small encoder tuned on the
harness's own traffic (17M parameters, one epoch, the dedup task only)
raises the dedup decision at 0.92 from a recall of 0.046 to 0.986 on the
held-out job test. The replay then prices that decision: at the shipped
0.92 the tuned encoder keeps **32.66%** of the facts against the stock's
**60.51%**, for a context 2.7x smaller. The decision improved; the
curation nearly halved. The shipped encoder and the shipped 0.92 stay.

The reason is structural: a true duplicate merged is a fact deleted from
the context, so improving the dedup decision is improving the deletion.
The threshold and the model are two dials on the same trade-off curve,
and the fact-preservation constraint fixes where on it the system sits.
The clean replay and the three failed fine-tunes are in `AB-RESULTS.md`
§2C-quater through §2C-septies.

### 7.4 The curation's cost grows with the conversation it curates

The stability run - 2,000 turns of a growing conversation, the real drone,
no model (`experiments.stability_500.py`):

| turns | the store | the assembled context | the curation's cost (p50) | the peak RSS |
|---|---|---|---|---|
| 250 | 212 chunks | 322 tok | 22.8 ms | 519 MB |
| 500 | 412 chunks | 320 tok | 37.3 ms | 564 MB |
| 1,000 | 812 chunks | 335 tok | 89.7 ms | 594 MB |
| 2,000 | 1,612 chunks | 332 tok | 156.8 ms | 667 MB |

**The context is flat and the cost is not.** The assembled context moves
by 10 tokens across a 7.6x range of store - the curation's promise holds
✓✓ - and nothing is OOM-killed over 2,000 turns ✓. But the curation's
per-turn cost is **linear in the store**: about 7x for 7.6x the chunks,
because every chunk is scored every turn.

So the memory's trade is explicit, and the published "≈18 ms added per
turn" is the *small-store* number: the prompt stays bounded (the
expensive part for the model) while the curation gets more expensive as
the conversation grows. The two costs trade against each other, and the
crossover - the point where curating costs more than the tokens it saves
- is a measurement the system does not yet make.

### 7.3 The curation's advantage is generator-dependent

The paired A/B - the same model, the same turns, the curated context
against the last-4k-token window of the same history - is the memory
system's headline measurement (P3). It was measured on a 27B, and it has
now been measured on the **35B-A3B ternary** (180 measurable turns):

| | the curated arm | the FIFO arm |
|---|---|---|
| the answer's fact recall | 81.1% | 81.7% |
| the context fidelity (the answer's terms sourced from its own context) | **0.241** | 0.208 |
| the turns where the answer is at least as good | **88.4%** (the bar: 80%) | - |
| the strictly-better turns | 13 | 17 |
| the context sufficiency (P3-style) | **97.5%** >= FIFO | - |

**On the 27B the curated arm won 115 turns outright to 26 (4.4:1). On the
35B it is 13 to 17** - a tie - and the answers are equal (81.1% against
81.7%). The "neither" bucket collapses too (145 of 175 on the 27B, 34 of
180 here): the 35B answers from *whatever* context it is given.

**The scoping result: the better the generator is at long context, the
less the curation has to add.** The FIFO window is the most recent turns
verbatim, and a model that uses them well does not need a curation to find
them. This is what Postulate 3 (the separation) implies - the curation is
a substitute for a *weaker* reader - and it is the first thing the
measurements had not tested.

**What survives.** The curated context is still sufficient on 97.5% of the
turns, its fidelity is ahead, and - the part the numbers above do not
touch - the curation is what keeps the context **bounded**: the FIFO arm
is 4k tokens of pure recency, and the two arms are equal *at that
budget*. The honest claim for a strong generator is the bounded context
and the cost, not the better answers.

### 7.2 What we changed, and what we did not

- The shipped configuration stays: the stock encoder at the shipped
  thresholds, because that is the configuration the published evidence was
  produced with.
- The next fine-tune weights a false merge above a missed merge
  (precision-weighted at the operating point) instead of optimising F1 -
  the asymmetry makes F1 the wrong target - and is measured on a held-out
  split, because the current one was trained and measured on the same
  traffic (a fit, not a generalisation).
- The full sweep, the fine-tune runs and the replay are in
  `docs/ENCODER-NOTES.md` in [`splinter/`](https://github.com/sky-is-green/hive/tree/main/splinter).

## Limits, honestly

- **Precision is the open question.** The context that reaches the model is
  fatter than it needs to be. The recall evidence says the facts the model
  stated are in there; an independent, non-circular precision number is
  still open.
- **The auditor sees the answer.** P7's agreement check mitigates the
  circularity but does not remove it: the auditor is the same model family
  that produced the answer.
- **Synthetic-corpus bias.** The long-horizon corpus is generated. The
  measurement in §7 is on real traffic; the P1-P12 verdicts mostly are not.
- **Cross-conversation contamination and hedge-replies** are harness-level
  failure modes that look like architectural failures if you do not
  isolate conversations per store and filter refusals out of the store.
- **The decay curve is an engineering heuristic** with cognitive-science
  inspiration, not a validated model of human forgetting.
- **The large tier is a state, not a model.** The 176B-A6B answers at
  60-73 tok/s warm and 3 tok/s cold, on the same engine and the same
  request: a fresh prompt read costs 5 tok/s and a reused prefix 71, and
  the cold phase is the OS pulling 60 GB of expert weights off the NVMe
  (the file is 20.8% page-cached on a 30 GB box). Warm and alone it
  answers the cascade's 28 tasks in 26.9 s an item; co-sited with the
  small tier, in 229.7 s. A deployment that measures the large tier
  cold, or with another engine resident, measures the box and not the
  model. Two levers pay and the rest do not: keep it warm, and run it
  alone. A whole-file prefetch does not - a 13 GB page cache cannot hold
  a 60 GB file, and the kernel's `WILLNEED` left it at 0% cached for
  80 s. The engine's resident tier (`--resident-budget-gib`) does: it
  keeps what the VRAM cache cannot hold in ordinary RAM, and the storage
  read on the second request falls from 1.4 GB to 146 MB.

## Reproduce

```bash
# the offline suite (deterministic diagnostics, no model needed)
pytest tests/unit tests/integration -q
# the encoder sweep on your own traffic
python -m experiments.encoder_suite --encoders l3 l6 bge-small e5-small gte-small
# the threshold replay (deterministic: no model, no generation)
python -m experiments.threshold_ab
# the cascade pilot (local model, deterministic verifiers)
open http://127.0.0.1:8765/cascade
```

The repository ships the run bundles, the event logs and the label files, so
a verdict can be traced back to the bytes that produced it.

### The artifacts

| artifact | what it is |
|---|---|
| `SkyIsNotGreen/splinter-memory` | this paper, the figures, the license and the citation |
| `SkyIsNotGreen/splinter-encoder` | the tuned drone encoder, its card, its configuration and its gate results |
| `SkyIsNotGreen/hive-cascade` | the cascade's benchmark (a 70/30 split) and two full run reports |
| `SkyIsNotGreen/Scion-35B-A3B` | the ternary 35B-A3B model and its decision head |
| `SkyIsNotGreen/Scion-FlashNext-176B-A6B` | the ternary 176B-A6B model |

The releases are private while the results are being refreshed; they open
when the refreshed numbers are in.

## License and citation

MIT. See `LICENSE` and `CITATION.cff` in this repository.

```
@misc{splinter-memory-2026,
  title  = {Splinter Memory: A Managed-Decay Context Curation Architecture
            for Long-Horizon LLM Conversations},
  author = {sky-is-green},
  year   = {2026},
  url    = {https://github.com/sky-is-green/hive/tree/main/splinter}
}
```

**Community invitation.** The protocol is written to be challenged: every
prediction has a logic chain, a measurement and a falsification condition.
If your hardware produces a different verdict, that is a result - open an
issue with the run bundle.