Sentence Similarity
sentence-transformers
context-curation
long-context
memory-management
conversational-ai
retrieval
llm
research-paper
Instructions to use SkyIsNotGreen/splinter-memory with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use SkyIsNotGreen/splinter-memory with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("SkyIsNotGreen/splinter-memory") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
Release from the HiveBench Studio (private)
Browse files
README.md
CHANGED
|
@@ -33,26 +33,26 @@ this edition adds the encoder measurement taken on the harness's own traffic
|
|
| 33 |
A long conversation is not a long document. Facts that mattered twenty turns
|
| 34 |
ago fall out of a rolling context window; the model answers from what
|
| 35 |
survives, not from what happened; and the surviving context is mostly noise
|
| 36 |
-
by then. Truncation and recency are cheap and wrong
|
| 37 |
-
exactly the old fact the user came back for. Keeping everything is worse
|
| 38 |
the prompt grows, the KV cache grows, the cost grows, and the model
|
| 39 |
-
"loses the middle" while the interesting fact
|
| 40 |
|
| 41 |
## What this does
|
| 42 |
|
| 43 |
Splinter Memory keeps a **managed** memory outside the model and hands the
|
| 44 |
model a **bounded, relevance-ranked** context every turn:
|
| 45 |
|
| 46 |
-
-
|
| 47 |
-
chunk against the current query
|
| 48 |
-
a generative model of the same size
|
| 49 |
-
-
|
| 50 |
-
changes
|
| 51 |
-
-
|
| 52 |
escalating-friction decay, a remembrance pass that rescues facts worth
|
| 53 |
keeping, and a "comb" (an SSD tier) that resurrects topics that returned
|
| 54 |
-
after being evicted
|
| 55 |
-
-
|
| 56 |
and hands it to any OpenAI-compatible backend (llama.cpp, LM Studio, vLLM,
|
| 57 |
a hosted API).
|
| 58 |
|
|
@@ -60,10 +60,10 @@ The model never sees the raw history. It sees the part that is worth its
|
|
| 60 |
attention, every turn, at a cost that does not grow with the conversation.
|
| 61 |
|
| 62 |
**The separation is the point.** The generative model is good at producing
|
| 63 |
-
answers and the wrong tool for selecting context; measured live,
|
| 64 |
-
27B generative model as an encoder
|
| 65 |
the hot path, while the 17M-parameter drone did the same job in
|
| 66 |
-
milliseconds.
|
| 67 |
|
| 68 |
## The measured results
|
| 69 |
|
|
@@ -130,7 +130,7 @@ generates → the reply is observed back into the store.
|
|
| 130 |
1. **Context curation** - for a fixed token budget, a relevance-ranked
|
| 131 |
selection from the full history dominates a contiguous window of the same
|
| 132 |
size, and the gap widens with conversation length.
|
| 133 |
-
2. **Managed decay** - forgetting is a design parameter,
|
| 134 |
escalating friction beats both unbounded growth and uniform forgetting.
|
| 135 |
3. **Separation** - bidirectional encoders for comprehension, an
|
| 136 |
autoregressive decoder for generation, is the better allocation of the
|
|
@@ -173,11 +173,11 @@ The harness gives its encoder three jobs, and only one of them is a ranking:
|
|
| 173 |
The prior encoder comparison measured **rank** only, on the synthetic
|
| 174 |
fixtures, and found that six encoders (a 22M MiniLM, a 400MB cross-encoder,
|
| 175 |
a 568M retrieval-specialised encoder and two contrastively tuned variants)
|
| 176 |
-
all landed on the same top-K curve
|
| 177 |
*data-structural* and that the remedy was "stop looking for a better
|
| 178 |
encoder".
|
| 179 |
|
| 180 |
-
We re-measured all three jobs on the harness's **own** traffic
|
| 181 |
conversations, 6,975 stored chunks, 4,056 duplicate pairs and 6,533
|
| 182 |
topic-change cases, labelled lexically so the label is independent of the
|
| 183 |
encoder being measured (`experiments/encoder_suite.py`). The result
|
|
@@ -195,7 +195,7 @@ not re-derived.
|
|
| 195 |
| drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
|
| 196 |
| drift as a *ranking* signal | **AUC 0.945** - the signal is strong; the threshold is wrong |
|
| 197 |
|
| 198 |
-
The encoder is not the bottleneck. The **operating points
|
| 199 |
are per-encoder, so they have to travel with the model. The full sweep
|
| 200 |
(model families, sizes, dimensions, CPU cost per chunk) is in
|
| 201 |
`docs/ENCODER-NOTES.md` in the harness repository, and the evaluation sets
|
|
@@ -214,7 +214,7 @@ are published as a dataset next to this card.
|
|
| 214 |
measurement in §7 is on real traffic; the P1-P12 verdicts mostly are not.
|
| 215 |
- **Cross-conversation contamination and hedge-replies** are harness-level
|
| 216 |
failure modes that look like architectural failures if you do not
|
| 217 |
-
isolate conversations per
|
| 218 |
- **The decay curve is an engineering heuristic** with cognitive-science
|
| 219 |
inspiration, not a validated model of human forgetting.
|
| 220 |
|
|
@@ -250,4 +250,5 @@ MIT. See `LICENSE` and `CITATION.cff` in this repository.
|
|
| 250 |
|
| 251 |
**Community invitation.** The protocol is written to be challenged: every
|
| 252 |
prediction has a logic chain, a measurement and a falsification condition.
|
| 253 |
-
If your hardware produces a different verdict, that is a result
|
|
|
|
|
|
| 33 |
A long conversation is not a long document. Facts that mattered twenty turns
|
| 34 |
ago fall out of a rolling context window; the model answers from what
|
| 35 |
survives, not from what happened; and the surviving context is mostly noise
|
| 36 |
+
by then. Truncation and recency are cheap and wrong: they throw away
|
| 37 |
+
exactly the old fact the user came back for. Keeping everything is worse -
|
| 38 |
the prompt grows, the KV cache grows, the cost grows, and the model
|
| 39 |
+
"loses the middle" while the interesting fact sits somewhere in the middle.
|
| 40 |
|
| 41 |
## What this does
|
| 42 |
|
| 43 |
Splinter Memory keeps a **managed** memory outside the model and hands the
|
| 44 |
model a **bounded, relevance-ranked** context every turn:
|
| 45 |
|
| 46 |
+
- a small fleet of **bidirectional encoders** ("drones") scores every stored
|
| 47 |
+
chunk against the current query - cheap, and far better at similarity than
|
| 48 |
+
a generative model of the same size;
|
| 49 |
+
- a **membrane** drops what is said twice and notices when the topic
|
| 50 |
+
changes;
|
| 51 |
+
- a **retention layer** makes forgetting a *design parameter* - an
|
| 52 |
escalating-friction decay, a remembrance pass that rescues facts worth
|
| 53 |
keeping, and a "comb" (an SSD tier) that resurrects topics that returned
|
| 54 |
+
after being evicted;
|
| 55 |
+
- a **focal layer** assembles a bounded context on an adaptive token budget
|
| 56 |
and hands it to any OpenAI-compatible backend (llama.cpp, LM Studio, vLLM,
|
| 57 |
a hosted API).
|
| 58 |
|
|
|
|
| 60 |
attention, every turn, at a cost that does not grow with the conversation.
|
| 61 |
|
| 62 |
**The separation is the point.** The generative model is good at producing
|
| 63 |
+
answers and the wrong tool for selecting context; measured live, using the
|
| 64 |
+
27B generative model as an encoder proxy cost tens of seconds per turn on
|
| 65 |
the hot path, while the 17M-parameter drone did the same job in
|
| 66 |
+
milliseconds.
|
| 67 |
|
| 68 |
## The measured results
|
| 69 |
|
|
|
|
| 130 |
1. **Context curation** - for a fixed token budget, a relevance-ranked
|
| 131 |
selection from the full history dominates a contiguous window of the same
|
| 132 |
size, and the gap widens with conversation length.
|
| 133 |
+
2. **Managed decay** - forgetting is a design parameter, not a failure mode;
|
| 134 |
escalating friction beats both unbounded growth and uniform forgetting.
|
| 135 |
3. **Separation** - bidirectional encoders for comprehension, an
|
| 136 |
autoregressive decoder for generation, is the better allocation of the
|
|
|
|
| 173 |
The prior encoder comparison measured **rank** only, on the synthetic
|
| 174 |
fixtures, and found that six encoders (a 22M MiniLM, a 400MB cross-encoder,
|
| 175 |
a 568M retrieval-specialised encoder and two contrastively tuned variants)
|
| 176 |
+
all landed on the same top-K curve - so it concluded the ceiling was
|
| 177 |
*data-structural* and that the remedy was "stop looking for a better
|
| 178 |
encoder".
|
| 179 |
|
| 180 |
+
We re-measured all three jobs on the harness's **own** traffic - 81
|
| 181 |
conversations, 6,975 stored chunks, 4,056 duplicate pairs and 6,533
|
| 182 |
topic-change cases, labelled lexically so the label is independent of the
|
| 183 |
encoder being measured (`experiments/encoder_suite.py`). The result
|
|
|
|
| 195 |
| drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
|
| 196 |
| drift as a *ranking* signal | **AUC 0.945** - the signal is strong; the threshold is wrong |
|
| 197 |
|
| 198 |
+
The encoder is not the bottleneck. The **operating points** are - and they
|
| 199 |
are per-encoder, so they have to travel with the model. The full sweep
|
| 200 |
(model families, sizes, dimensions, CPU cost per chunk) is in
|
| 201 |
`docs/ENCODER-NOTES.md` in the harness repository, and the evaluation sets
|
|
|
|
| 214 |
measurement in §7 is on real traffic; the P1-P12 verdicts mostly are not.
|
| 215 |
- **Cross-conversation contamination and hedge-replies** are harness-level
|
| 216 |
failure modes that look like architectural failures if you do not
|
| 217 |
+
isolate conversations per store and filter refusals out of the store.
|
| 218 |
- **The decay curve is an engineering heuristic** with cognitive-science
|
| 219 |
inspiration, not a validated model of human forgetting.
|
| 220 |
|
|
|
|
| 250 |
|
| 251 |
**Community invitation.** The protocol is written to be challenged: every
|
| 252 |
prediction has a logic chain, a measurement and a falsification condition.
|
| 253 |
+
If your hardware produces a different verdict, that is a result - open an
|
| 254 |
+
issue with the run bundle.
|