README.md

metadata

license: mit
library_name: sentence-transformers
tags:
  - context-curation
  - long-context
  - memory-management
  - conversational-ai
  - retrieval
  - llm
  - research-paper
pipeline_tag: sentence-similarity

Long-Context Memory Curation Management

Splinter Memory — a managed-decay context curation architecture for long-horizon LLM conversations.

This card is the Hugging Face edition of the Splinter Memory white paper. It is written for the people who build LLM applications and want the context window to stay useful as the conversation grows: the memory layer, the mechanisms, the measured results, and the code that implements them.

The canonical paper lives with the code (sky-is-green/hive/tree/main/splinter); this edition adds the encoder measurement taken on the harness's own traffic (§7) and the operational notes a practitioner needs.

---

The problem, in one paragraph

A long conversation is not a long document. Facts that mattered twenty turns ago fall out of a rolling context window; the model answers from what survives, not from what happened; and the surviving context is mostly noise by then. Truncation and recency are cheap and wrong: they throw away exactly the old fact the user came back for. Keeping everything is worse - the prompt grows, the KV cache grows, the cost grows, and the model "loses the middle" while the interesting fact sits somewhere in the middle.

What this does

Splinter Memory keeps a managed memory outside the model and hands the model a bounded, relevance-ranked context every turn:

• a small fleet of bidirectional encoders ("drones") scores every stored chunk against the current query - cheap, and far better at similarity than a generative model of the same size;
• a membrane drops what is said twice and notices when the topic changes;
• a retention layer makes forgetting a design parameter - an escalating-friction decay, a remembrance pass that rescues facts worth keeping, and a "comb" (an SSD tier) that resurrects topics that returned after being evicted;
• a focal layer assembles a bounded context on an adaptive token budget and hands it to any OpenAI-compatible backend (llama.cpp, LM Studio, vLLM, a hosted API).

The model never sees the raw history. It sees the part that is worth its attention, every turn, at a cost that does not grow with the conversation.

The separation is the point. The generative model is good at producing answers and the wrong tool for selecting context; measured live, using the 27B generative model as an encoder proxy cost tens of seconds per turn on the hot path, while the 17M-parameter drone did the same job in milliseconds.

The measured results

Metric	Naive FIFO (baseline)	Target	Measured
Throughput over 500 turns	decays	flat ±10%	flat (14.5 → 15.5 tok/s over 308+ turns), 0 OOM events
Retrieval recall (facts the model actually stated)	n/a	≥75% / ≥90%	90.3% live, 93.5% on the isolated-store replay
Retrieval precision (sentence proxy)	near-chance past the window	≥70% / ≥85%	10.7% - the encoder ceiling (§7)
Context sufficiency vs. an equal-size FIFO window	—	≥80% of turns	85.1% of the fact-retrievable turns (4.4:1)
Routing accuracy	—	≥80% / ≥92%	100%
Added per-turn latency	0 ms	<50 / <30 ms	≈18 ms on a small store (3.4 ms assembly + ~15 ms scoring) - and it grows with the store: 22.8 ms at 212 chunks, 156.8 ms at 1,612 (§7.4)
Peak RSS over 500 turns	—	small	34.7 MB
Pipeline Efficiency Score	~30	≥65 / ≥80	80.0 (GREEN) live, 73.1 on the 20-conversation evidence run

Full protocol, per-hypothesis logic chains and falsification conditions: §5 of the canonical paper.

The whole protocol, refreshed on the 35B-A3B

The P1-P11 protocol re-run live against the 35B-A3B ternary (a 12-conversation fixture subset), to ask whether the verdict set is a property of the architecture or of the generator it was measured on:

	verdict	the evidence
P1 constant throughput	PASS	the throughput does not decay
P2 retrieval precision	FAIL	the precision clause (the known encoder ceiling, §7)
P3 context sufficiency	PASS	the curated context against an equal-size FIFO window
P4 domain-dependent decay	PASS	code m90 1.8 vs prose m90 1.2 - a gap of 0.6, beyond the 0.2 band
P5 targeted masking	SKIP	a training experiment
P6 confidence escalation	FAIL	the escalated recall is worse (0.174 vs 0.565)
P7 auditor-human agreement	SKIP	it needs human raters (90.25% measured 2026-08-23)
P8 routing accuracy	PASS	200/200
P9 densest-duplicate retention	PASS	the densest chunk wins 100% of the turns
P10 drift reset	FAIL	the reset changes the recall by 0.0 points
P11 comb resurrection	PASS	the pressure regime: 100% topic-return recall against 20% without the comb

PASS 6/11, 2 skipped - the published verdict set, exactly. The published run (2026-08-23/24, on a 27B, on the full corpus) reported P1/P3/P4/P8/P9/P11 PASS, P2/P6/P10 FAIL, P5/P7 SKIP. The same set reproduces on a different model and a smaller corpus, which says the protocol measures the architecture rather than the generator.

One of the failures now has a candidate mechanism. P10 (the drift reset) measures a detector whose threshold was calibrated for a different encoder (§7): forced on, it fires on most turns, so "the reset has no effect" may be measuring a detector that fires far too often rather than a reset that does not work. (The exact rate was measured on the pre-correction corpus and is withdrawn with the rest of §7.) The verdict stands as measured; the mechanism is now in question, which is a different claim.

And the one result that does not carry over is P3's margin. On the 27B the curated context won 115 turns outright to 26; on the 35B it is a tie (§7.3). The verdict holds (the curated context is sufficient, and at least as good on 88.4% of the turns); the margin does not. The curation's accuracy advantage is a function of the generator - what survives for a strong one is the bounded context and the cost.

Use it

Three ways, in increasing order of control:

1. As a sidecar - the Studio serves an OpenAI-compatible /v1/chat/completions that curates before it generates. Point your client at it; nothing else changes.

git clone https://github.com/sky-is-green/hive
cd hive
python -m pip install -e splinter
python -m harness --port 8765

2. As a library - the memory system is importable, engine-agnostic and has no GPU requirement for the encoder tier.

from cortex.splinter import Splinter
from cortex.config import SplinterConfig

s = Splinter(config=SplinterConfig(max_context=8192))
result = s.process_turn("what did we decide about the KV cache?", "my-conversation")
print(result.assembled.content)     # the curated context, bounded

3. As the benchmark - the P1-P12 protocol, the deterministic diagnostics and the self-contained run bundles are in the repository, so the claims can be re-measured on your own hardware and your own backend.

How it works

Layer	Function	Core mechanisms
Cortex	Orchestration & routing	task classification, drone dispatch, congestion detection, graceful degradation
Sieve	Relevance scoring	ultra-small encoder (17M params, 384-dim) for fast similarity; a medium encoder for uncertain cases; confidence via prediction variance
Membrane	Selective filtering	semantic deduplication (cosine > 0.92, keep the densest), topic-drift detection and reset
Retention	Memory & decay	remembrance pass, escalating-friction decay, stale-context acceleration, the "comb" (surplus SSD tier, topic-return resurrection)
Focal	Assembly	adaptive token budget (1k-6k by route tier), confidence-weighted sorting, bounded context construction

Per turn: user input → Cortex classifies it → Sieve scores every stored chunk → Membrane deduplicates and checks the topic → Retention applies decay and the remembrance pass → Focal assembles a bounded context → the backend generates → the reply is observed back into the store.

The four postulates

1. Context curation - for a fixed token budget, a relevance-ranked selection from the full history dominates a contiguous window of the same size, and the gap widens with conversation length.
2. Managed decay - forgetting is a design parameter, not a failure mode; escalating friction beats both unbounded growth and uniform forgetting.
3. Separation - bidirectional encoders for comprehension, an autoregressive decoder for generation, is the better allocation of the same compute.
4. Ground-truth bootstrap - an LLM auditor asked about context utilization (not answer correctness) agrees with human labels 90.25% of the time, which is enough to drive parameter tuning without a human labelling pass.

The harness that runs it, and the pipeline that manages it

The memory system is a library; the HiveBench Studio is the harness that runs it, and it is where the management pipeline lives.

The management pipeline

A long-horizon memory has to be managed: every knob that decides what the model sees used to live in three different menus, and a wrong setting was invisible. The Studio has one place for all of it - the Global harness controls - and every block of it is a baseline that the more specific thing beats:

block	what it decides
the encoder	which small model does the retrieval and the memory management - you choose it, and the next conversation runs on it (no restart)
Memory & RAG	the curation baseline: the context window, the stale wall, the dedup and drift thresholds, the comb, the medium drone
Quality	the cascade's judge, repair, gate and escalation - the defaults every run starts from
MCP	what an external client may read and write of the memory
Runtime	the sampling a reply is drawn with, and the context, GPU layers and threads an engine is launched with

The point is not tidiness. It is that a setting which is not consulted is indistinguishable from a setting which does not work, and the only way to know which one you have is to wire them and then measure.

The harness itself

the console	Home, Settings, Global, Cascade, and a "More tools" group (Agent, Engines, Local library, Providers, Hub, Inspector, Training)
the chat	Splinter mode (one curated reply) or Agent mode (the tool loop); the reply says who produced it
the launch panel	llama.cpp or Strata, per engine, from the profiles; stop closes the connection so the engine stops; unload gives the VRAM back and retires the orphans
the cascade console	every runner knob in one form, with the Global tab's Quality block as its starting values
the harvest	every turn (the query, the reply, the curation that was used) written to a jsonl, off by default - this is the data the encoder is trained on

Answer quality reporting and API escalation

The curation's effect is not only felt, it is reported. Every cascade run writes a per-item report (the verdict, the detector findings, the repair attempt, the ladder's resolution), a provenance manifest (the model sha256, the stack, the git commit, the task-set sha, the decode settings) and a generated model card; and the escalation lane harvests every ladder-resolved item to escalation.jsonl next to the report - training-side only, and read by nothing in the runtime.

The escalation ladder itself is the last stage: an answer the local verifiers still flag can go to a human gate (verbose on the first trigger, compact afterwards) and then to an API tier, with the served answer, the cost and the reason recorded. The companion paper (The Hive Cascade) documents the loop, the gold-free runtime contract and the measured verdicts; the benchmark and the run reports are published next to this card.

The evidence, hypothesis by hypothesis

	Hypothesis	Verdict
P1	Constant throughput over 500 turns	PASS
P2	Retrieval precision ≥85% at ≥90% recall	SPLIT - recall PASS (90.3%), precision FAIL (the encoder ceiling)
P3	Context sufficiency ≥80% of turns vs. an equal-size FIFO window	PASS (85.1% on the 27B); on the 35B-A3B the answer-level result is a tie (88.4% at-least-as-good, 13 strict wins to 17) - see §7.3
P4	Domain-dependent decay curve (code vs. prose separate)	PASS (code m90 1.8 vs. prose 1.2)
P5	Targeted masking	PASS
P6	Confidence escalation	FAIL
P7	Auditor-human agreement ≥90%	PASS (90.25% on 400 items)
P8	Heuristic routing floor	PASS (100%)
P9	Densest-duplicate survives deduplication	PASS
P10	Drift reset	FAIL
P11	Comb resurrection of returned topics	PASS (deterministic and live-validated)
P12	Store-time fact distillation	protocol only

The failures are kept in the paper with their evidence. A pre-registered protocol that only reports passes is not evidence.

7. The encoder: what we measured on our own traffic

Correction (2026-10-21). The corpus behind this section included an ingested knowledge-base attachment: 17 of its 82 files were chunked [block n/m] documents (5,531 of 6,977 chunks), and one conversation was loaded from two state directories. The drift figures below (the 0.945 AUC and the "fires on 571 of 628 turns" mechanism) are withdrawn, and the tuned encoder was rejected by the replay (see below). The shipped encoder and the shipped 0.92 stay. The correction and the clean measurement are in AB-RESULTS.md §2C-septies; the retrieval results elsewhere in this card come from different runs and are unaffected.

The harness gives its encoder three jobs, and only one of them is a ranking:

job	question	where
rank	which stored chunk answers this query?	the sieve
dedup	do these two chunks say the same thing?	the membrane (cosine > 0.92)
drift	did the conversation change topic?	the membrane (1 - cosine > 0.6)

The prior encoder comparison measured rank only, on the synthetic fixtures, and found that six encoders (a 22M MiniLM, a 400MB cross-encoder, a 568M retrieval-specialised encoder and two contrastively tuned variants) all landed on the same top-K curve - so it concluded the ceiling was data-structural and that the remedy was "stop looking for a better encoder".

We re-measured all three jobs on the harness's own traffic - 81 conversations, 6,975 stored chunks, 4,056 duplicate pairs and 6,533 topic-change cases, labelled lexically so the label is independent of the encoder being measured (experiments/encoder_suite.py). The result changes the picture, and it is not about the ranking:

The thresholds that ship with the harness were calibrated for a different encoder. The production drone was swapped from all-MiniLM-L6-v2 to paraphrase-MiniLM-L3-v2 on 2026-08-23; the membrane's two thresholds were not re-derived.

operating point	measured
dedup at the shipped threshold (0.92)	precision 1.00, recall 0.03 - the duplicate test almost never fires
dedup at its own best threshold (~0.56)	F1 0.75 (precision 0.62, recall 0.95)
drift at the shipped threshold (0.6)	precision 0.08, recall 0.96 - the topic-change test fires on almost every turn
drift as a ranking signal	withdrawn - measured on the pre-correction corpus (79% an ingested attachment), see above

7.1 The F1 is the wrong target, and the threshold is not the lever

The obvious conclusion - "move the thresholds to the encoder's optimum" - is falsified by a deterministic replay of the corpus through the real pipeline (50 conversations, 493 measurable turns, the P2 diagnostic's own notion of a fact; experiments/threshold_ab.py):

arm	dedup / drift	fact recall	context	verdict
shipped	0.92 / 0.6	50.6%	159 tok	-
the dedup alone	0.57 / 0.6	39.7%	90 tok	falsified
the drift alone	0.92 / 0.77	50.6%	159 tok	survives

The duplicate test's two errors are not interchangeable. A duplicate test that fires wrongly deletes a chunk, and the fact inside it is gone irreversibly; a duplicate test that fails to fire only leaves a redundant chunk in the context, which costs budget and nothing else. F1 weights them equally, so F1 points at the wrong threshold: at 0.57 the encoder merges chunks that are about the same thing but carry different facts, and the fact recall falls by a fifth. The shipped 0.92 is conservative for a reason - its precision is 1.00, so it never deletes a fact - at the price of catching only 5% of the redundancy.

The topic-drift test is the opposite case: it only penalises the off-topic chunks, it never deletes them, so its recalibration is free.

The model is not the lever either. A small encoder tuned on the harness's own traffic (17M parameters, one epoch, the dedup task only) raises the dedup decision at 0.92 from a recall of 0.046 to 0.986 on the held-out job test. The replay then prices that decision: at the shipped 0.92 the tuned encoder keeps 32.66% of the facts against the stock's 60.51%, for a context 2.7x smaller. The decision improved; the curation nearly halved. The shipped encoder and the shipped 0.92 stay.

The reason is structural: a true duplicate merged is a fact deleted from the context, so improving the dedup decision is improving the deletion. The threshold and the model are two dials on the same trade-off curve, and the fact-preservation constraint fixes where on it the system sits. The clean replay and the three failed fine-tunes are in AB-RESULTS.md §2C-quater through §2C-septies.

7.4 The curation's cost grows with the conversation it curates

The stability run - 2,000 turns of a growing conversation, the real drone, no model (experiments.stability_500.py):

turns	the store	the assembled context	the curation's cost (p50)	the peak RSS
250	212 chunks	322 tok	22.8 ms	519 MB
500	412 chunks	320 tok	37.3 ms	564 MB
1,000	812 chunks	335 tok	89.7 ms	594 MB
2,000	1,612 chunks	332 tok	156.8 ms	667 MB

The context is flat and the cost is not. The assembled context moves by 10 tokens across a 7.6x range of store - the curation's promise holds ✓✓ - and nothing is OOM-killed over 2,000 turns ✓. But the curation's per-turn cost is linear in the store: about 7x for 7.6x the chunks, because every chunk is scored every turn.

So the memory's trade is explicit, and the published "≈18 ms added per turn" is the small-store number: the prompt stays bounded (the expensive part for the model) while the curation gets more expensive as the conversation grows. The two costs trade against each other, and the crossover - the point where curating costs more than the tokens it saves

• is a measurement the system does not yet make.

7.3 The curation's advantage is generator-dependent

The paired A/B - the same model, the same turns, the curated context against the last-4k-token window of the same history - is the memory system's headline measurement (P3). It was measured on a 27B, and it has now been measured on the 35B-A3B ternary (180 measurable turns):

	the curated arm	the FIFO arm
the answer's fact recall	81.1%	81.7%
the context fidelity (the answer's terms sourced from its own context)	0.241	0.208
the turns where the answer is at least as good	88.4% (the bar: 80%)	-
the strictly-better turns	13	17
the context sufficiency (P3-style)	97.5% >= FIFO	-

On the 27B the curated arm won 115 turns outright to 26 (4.4:1). On the 35B it is 13 to 17 - a tie - and the answers are equal (81.1% against 81.7%). The "neither" bucket collapses too (145 of 175 on the 27B, 34 of 180 here): the 35B answers from whatever context it is given.

The scoping result: the better the generator is at long context, the less the curation has to add. The FIFO window is the most recent turns verbatim, and a model that uses them well does not need a curation to find them. This is what Postulate 3 (the separation) implies - the curation is a substitute for a weaker reader - and it is the first thing the measurements had not tested.

What survives. The curated context is still sufficient on 97.5% of the turns, its fidelity is ahead, and - the part the numbers above do not touch - the curation is what keeps the context bounded: the FIFO arm is 4k tokens of pure recency, and the two arms are equal at that budget. The honest claim for a strong generator is the bounded context and the cost, not the better answers.

7.2 What we changed, and what we did not

• The shipped configuration stays: the stock encoder at the shipped thresholds, because that is the configuration the published evidence was produced with.
• The next fine-tune weights a false merge above a missed merge (precision-weighted at the operating point) instead of optimising F1 - the asymmetry makes F1 the wrong target - and is measured on a held-out split, because the current one was trained and measured on the same traffic (a fit, not a generalisation).
• The full sweep, the fine-tune runs and the replay are in docs/ENCODER-NOTES.md in splinter/.

Limits, honestly

• Precision is the open question. The context that reaches the model is fatter than it needs to be. The recall evidence says the facts the model stated are in there; an independent, non-circular precision number is still open.
• The auditor sees the answer. P7's agreement check mitigates the circularity but does not remove it: the auditor is the same model family that produced the answer.
• Synthetic-corpus bias. The long-horizon corpus is generated. The measurement in §7 is on real traffic; the P1-P12 verdicts mostly are not.
• Cross-conversation contamination and hedge-replies are harness-level failure modes that look like architectural failures if you do not isolate conversations per store and filter refusals out of the store.
• The decay curve is an engineering heuristic with cognitive-science inspiration, not a validated model of human forgetting.
• The large tier is a state, not a model. The 176B-A6B answers at 60-73 tok/s warm and 3 tok/s cold, on the same engine and the same request: a fresh prompt read costs 5 tok/s and a reused prefix 71, and the cold phase is the OS pulling 60 GB of expert weights off the NVMe (the file is 20.8% page-cached on a 30 GB box). Warm and alone it answers the cascade's 28 tasks in 26.9 s an item; co-sited with the small tier, in 229.7 s. A deployment that measures the large tier cold, or with another engine resident, measures the box and not the model. Two levers pay and the rest do not: keep it warm, and run it alone. A whole-file prefetch does not - a 13 GB page cache cannot hold a 60 GB file, and the kernel's WILLNEED left it at 0% cached for 80 s. The engine's resident tier (--resident-budget-gib) does: it keeps what the VRAM cache cannot hold in ordinary RAM, and the storage read on the second request falls from 1.4 GB to 146 MB.

Reproduce

# the offline suite (deterministic diagnostics, no model needed)
pytest tests/unit tests/integration -q

# the encoder sweep on your own traffic
python -m experiments.encoder_suite --encoders l3 l6 bge-small e5-small gte-small

# the threshold replay (deterministic: no model, no generation)
python -m experiments.threshold_ab

# the cascade pilot (local model, deterministic verifiers)
open http://127.0.0.1:8765/cascade

The repository ships the run bundles, the event logs and the label files, so a verdict can be traced back to the bytes that produced it.

The artifacts

artifact	what it is
SkyIsNotGreen/splinter-memory	this paper, the figures, the license and the citation
SkyIsNotGreen/splinter-encoder	the tuned drone encoder, its card, its configuration and its gate results
SkyIsNotGreen/hive-cascade	the cascade's benchmark (a 70/30 split) and two full run reports
SkyIsNotGreen/Scion-35B-A3B	the ternary 35B-A3B model and its decision head
SkyIsNotGreen/Scion-FlashNext-176B-A6B	the ternary 176B-A6B model

The releases are private while the results are being refreshed; they open when the refreshed numbers are in.

License and citation

MIT. See LICENSE and CITATION.cff in this repository.

@misc{splinter-memory-2026,
  title  = {Splinter Memory: A Managed-Decay Context Curation Architecture
            for Long-Horizon LLM Conversations},
  author = {sky-is-green},
  year   = {2026},
  url    = {https://github.com/sky-is-green/hive/tree/main/splinter}
}

Community invitation. The protocol is written to be challenged: every prediction has a logic chain, a measurement and a falsification condition. If your hardware produces a different verdict, that is a result - open an issue with the run bundle.