SkyIsNotGreen commited on
Commit
14514c2
·
verified ·
1 Parent(s): 34a3f48

Release from the HiveBench Studio (private)

Browse files
Files changed (1) hide show
  1. README.md +21 -20
README.md CHANGED
@@ -33,26 +33,26 @@ this edition adds the encoder measurement taken on the harness's own traffic
33
  A long conversation is not a long document. Facts that mattered twenty turns
34
  ago fall out of a rolling context window; the model answers from what
35
  survives, not from what happened; and the surviving context is mostly noise
36
- by then. Truncation and recency are cheap and wrong. They throw away
37
- exactly the old fact the user came back for. Keeping everything is worse;
38
  the prompt grows, the KV cache grows, the cost grows, and the model
39
- "loses the middle" while the interesting fact *sits somewhere in the middle*.
40
 
41
  ## What this does
42
 
43
  Splinter Memory keeps a **managed** memory outside the model and hands the
44
  model a **bounded, relevance-ranked** context every turn:
45
 
46
- - A small fleet of **bidirectional encoders** ("drones") scores every stored
47
- chunk against the current query; cheap, and far better at similarity than
48
- a generative model of the same size.
49
- - A **membrane** drops what is said twice and notices when the topic
50
- changes.
51
- - A **retention layer** makes forgetting a *design parameter*; an
52
  escalating-friction decay, a remembrance pass that rescues facts worth
53
  keeping, and a "comb" (an SSD tier) that resurrects topics that returned
54
- after being evicted.
55
- - A **focal layer** assembles a bounded context on an adaptive token budget
56
  and hands it to any OpenAI-compatible backend (llama.cpp, LM Studio, vLLM,
57
  a hosted API).
58
 
@@ -60,10 +60,10 @@ The model never sees the raw history. It sees the part that is worth its
60
  attention, every turn, at a cost that does not grow with the conversation.
61
 
62
  **The separation is the point.** The generative model is good at producing
63
- answers and the wrong tool for selecting context; measured live, *using the
64
- 27B generative model as an encoder-proxy cost tens of seconds per turn on
65
  the hot path, while the 17M-parameter drone did the same job in
66
- milliseconds.*
67
 
68
  ## The measured results
69
 
@@ -130,7 +130,7 @@ generates → the reply is observed back into the store.
130
  1. **Context curation** - for a fixed token budget, a relevance-ranked
131
  selection from the full history dominates a contiguous window of the same
132
  size, and the gap widens with conversation length.
133
- 2. **Managed decay** - forgetting is a design parameter, ***not*** a failure mode;
134
  escalating friction beats both unbounded growth and uniform forgetting.
135
  3. **Separation** - bidirectional encoders for comprehension, an
136
  autoregressive decoder for generation, is the better allocation of the
@@ -173,11 +173,11 @@ The harness gives its encoder three jobs, and only one of them is a ranking:
173
  The prior encoder comparison measured **rank** only, on the synthetic
174
  fixtures, and found that six encoders (a 22M MiniLM, a 400MB cross-encoder,
175
  a 568M retrieval-specialised encoder and two contrastively tuned variants)
176
- all landed on the same top-K curve; so it concluded the ceiling was
177
  *data-structural* and that the remedy was "stop looking for a better
178
  encoder".
179
 
180
- We re-measured all three jobs on the harness's **own** traffic; 81
181
  conversations, 6,975 stored chunks, 4,056 duplicate pairs and 6,533
182
  topic-change cases, labelled lexically so the label is independent of the
183
  encoder being measured (`experiments/encoder_suite.py`). The result
@@ -195,7 +195,7 @@ not re-derived.
195
  | drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
196
  | drift as a *ranking* signal | **AUC 0.945** - the signal is strong; the threshold is wrong |
197
 
198
- The encoder is not the bottleneck. The **operating points are**, and they
199
  are per-encoder, so they have to travel with the model. The full sweep
200
  (model families, sizes, dimensions, CPU cost per chunk) is in
201
  `docs/ENCODER-NOTES.md` in the harness repository, and the evaluation sets
@@ -214,7 +214,7 @@ are published as a dataset next to this card.
214
  measurement in §7 is on real traffic; the P1-P12 verdicts mostly are not.
215
  - **Cross-conversation contamination and hedge-replies** are harness-level
216
  failure modes that look like architectural failures if you do not
217
- isolate conversations per-store and filter refusals *out* of the store.
218
  - **The decay curve is an engineering heuristic** with cognitive-science
219
  inspiration, not a validated model of human forgetting.
220
 
@@ -250,4 +250,5 @@ MIT. See `LICENSE` and `CITATION.cff` in this repository.
250
 
251
  **Community invitation.** The protocol is written to be challenged: every
252
  prediction has a logic chain, a measurement and a falsification condition.
253
- If your hardware produces a different verdict, that is a result I am interested in, open a topic/issue with the run bundle.
 
 
33
  A long conversation is not a long document. Facts that mattered twenty turns
34
  ago fall out of a rolling context window; the model answers from what
35
  survives, not from what happened; and the surviving context is mostly noise
36
+ by then. Truncation and recency are cheap and wrong: they throw away
37
+ exactly the old fact the user came back for. Keeping everything is worse -
38
  the prompt grows, the KV cache grows, the cost grows, and the model
39
+ "loses the middle" while the interesting fact sits somewhere in the middle.
40
 
41
  ## What this does
42
 
43
  Splinter Memory keeps a **managed** memory outside the model and hands the
44
  model a **bounded, relevance-ranked** context every turn:
45
 
46
+ - a small fleet of **bidirectional encoders** ("drones") scores every stored
47
+ chunk against the current query - cheap, and far better at similarity than
48
+ a generative model of the same size;
49
+ - a **membrane** drops what is said twice and notices when the topic
50
+ changes;
51
+ - a **retention layer** makes forgetting a *design parameter* - an
52
  escalating-friction decay, a remembrance pass that rescues facts worth
53
  keeping, and a "comb" (an SSD tier) that resurrects topics that returned
54
+ after being evicted;
55
+ - a **focal layer** assembles a bounded context on an adaptive token budget
56
  and hands it to any OpenAI-compatible backend (llama.cpp, LM Studio, vLLM,
57
  a hosted API).
58
 
 
60
  attention, every turn, at a cost that does not grow with the conversation.
61
 
62
  **The separation is the point.** The generative model is good at producing
63
+ answers and the wrong tool for selecting context; measured live, using the
64
+ 27B generative model as an encoder proxy cost tens of seconds per turn on
65
  the hot path, while the 17M-parameter drone did the same job in
66
+ milliseconds.
67
 
68
  ## The measured results
69
 
 
130
  1. **Context curation** - for a fixed token budget, a relevance-ranked
131
  selection from the full history dominates a contiguous window of the same
132
  size, and the gap widens with conversation length.
133
+ 2. **Managed decay** - forgetting is a design parameter, not a failure mode;
134
  escalating friction beats both unbounded growth and uniform forgetting.
135
  3. **Separation** - bidirectional encoders for comprehension, an
136
  autoregressive decoder for generation, is the better allocation of the
 
173
  The prior encoder comparison measured **rank** only, on the synthetic
174
  fixtures, and found that six encoders (a 22M MiniLM, a 400MB cross-encoder,
175
  a 568M retrieval-specialised encoder and two contrastively tuned variants)
176
+ all landed on the same top-K curve - so it concluded the ceiling was
177
  *data-structural* and that the remedy was "stop looking for a better
178
  encoder".
179
 
180
+ We re-measured all three jobs on the harness's **own** traffic - 81
181
  conversations, 6,975 stored chunks, 4,056 duplicate pairs and 6,533
182
  topic-change cases, labelled lexically so the label is independent of the
183
  encoder being measured (`experiments/encoder_suite.py`). The result
 
195
  | drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
196
  | drift as a *ranking* signal | **AUC 0.945** - the signal is strong; the threshold is wrong |
197
 
198
+ The encoder is not the bottleneck. The **operating points** are - and they
199
  are per-encoder, so they have to travel with the model. The full sweep
200
  (model families, sizes, dimensions, CPU cost per chunk) is in
201
  `docs/ENCODER-NOTES.md` in the harness repository, and the evaluation sets
 
214
  measurement in §7 is on real traffic; the P1-P12 verdicts mostly are not.
215
  - **Cross-conversation contamination and hedge-replies** are harness-level
216
  failure modes that look like architectural failures if you do not
217
+ isolate conversations per store and filter refusals out of the store.
218
  - **The decay curve is an engineering heuristic** with cognitive-science
219
  inspiration, not a validated model of human forgetting.
220
 
 
250
 
251
  **Community invitation.** The protocol is written to be challenged: every
252
  prediction has a logic chain, a measurement and a falsification condition.
253
+ If your hardware produces a different verdict, that is a result - open an
254
+ issue with the run bundle.