Sentence Similarity
sentence-transformers
context-curation
long-context
memory-management
conversational-ai
retrieval
llm
research-paper
Instructions to use SkyIsNotGreen/splinter-memory with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use SkyIsNotGreen/splinter-memory with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("SkyIsNotGreen/splinter-memory") sentences = [ "That is a happy person", "That is a happy dog", "That is a very happy person", "Today is a sunny day" ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [4, 4] - Notebooks
- Google Colab
- Kaggle
Correct the card before publication: withdrawal of the invalidated drift numbers and the rejected threshold recommendation
Browse files
README.md
CHANGED
|
@@ -109,10 +109,12 @@ protocol measures the architecture rather than the generator.
|
|
| 109 |
|
| 110 |
**One of the failures now has a candidate mechanism.** P10 (the drift
|
| 111 |
reset) measures a detector whose threshold was calibrated for a different
|
| 112 |
-
encoder (§7): forced on, it fires on
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
|
|
|
|
|
|
| 116 |
|
| 117 |
**And the one result that does not carry over is P3's margin.** On the
|
| 118 |
27B the curated context won 115 turns outright to 26; on the 35B it is a
|
|
@@ -255,6 +257,17 @@ protocol that only reports passes is not evidence.
|
|
| 255 |
|
| 256 |
## 7. The encoder: what we measured on our own traffic
|
| 257 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 258 |
The harness gives its encoder three jobs, and only one of them is a ranking:
|
| 259 |
|
| 260 |
| job | question | where |
|
|
@@ -286,7 +299,7 @@ not re-derived.
|
|
| 286 |
| dedup at the shipped threshold (0.92) | precision 1.00, **recall 0.03** - the duplicate test almost never fires |
|
| 287 |
| dedup at its own best threshold (~0.56) | F1 **0.75** (precision 0.62, recall 0.95) |
|
| 288 |
| drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
|
| 289 |
-
| drift as a *ranking* signal | **
|
| 290 |
|
| 291 |
### 7.1 The F1 is the wrong target, and the threshold is not the lever
|
| 292 |
|
|
@@ -314,20 +327,20 @@ the price of catching only 5% of the redundancy.
|
|
| 314 |
The topic-drift test is the opposite case: it only *penalises* the
|
| 315 |
off-topic chunks, it never deletes them, so its recalibration is free.
|
| 316 |
|
| 317 |
-
**The model is
|
| 318 |
-
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
| 324 |
-
|
| 325 |
-
|
| 326 |
-
|
| 327 |
-
|
| 328 |
-
|
| 329 |
-
|
| 330 |
-
|
| 331 |
|
| 332 |
### 7.4 The curation's cost grows with the conversation it curates
|
| 333 |
|
|
|
|
| 109 |
|
| 110 |
**One of the failures now has a candidate mechanism.** P10 (the drift
|
| 111 |
reset) measures a detector whose threshold was calibrated for a different
|
| 112 |
+
encoder (§7): forced on, it fires on most turns, so "the reset has no
|
| 113 |
+
effect" may be measuring a detector that fires far too often rather than
|
| 114 |
+
a reset that does not work. (The exact rate was measured on the
|
| 115 |
+
pre-correction corpus and is withdrawn with the rest of §7.) The verdict
|
| 116 |
+
stands as measured; the mechanism is now in question, which is a
|
| 117 |
+
different claim.
|
| 118 |
|
| 119 |
**And the one result that does not carry over is P3's margin.** On the
|
| 120 |
27B the curated context won 115 turns outright to 26; on the 35B it is a
|
|
|
|
| 257 |
|
| 258 |
## 7. The encoder: what we measured on our own traffic
|
| 259 |
|
| 260 |
+
> **Correction (2026-10-21).** The corpus behind this section included an
|
| 261 |
+
> ingested knowledge-base attachment: 17 of its 82 files were chunked
|
| 262 |
+
> `[block n/m]` documents (5,531 of 6,977 chunks), and one conversation
|
| 263 |
+
> was loaded from two state directories. The drift figures below (the
|
| 264 |
+
> 0.945 AUC and the "fires on 571 of 628 turns" mechanism) are
|
| 265 |
+
> **withdrawn**, and the tuned encoder was rejected by the replay (see
|
| 266 |
+
> below). The shipped encoder and the shipped 0.92 stay. The correction
|
| 267 |
+
> and the clean measurement are in `AB-RESULTS.md` §2C-septies; the
|
| 268 |
+
> retrieval results elsewhere in this card come from different runs and
|
| 269 |
+
> are unaffected.
|
| 270 |
+
|
| 271 |
The harness gives its encoder three jobs, and only one of them is a ranking:
|
| 272 |
|
| 273 |
| job | question | where |
|
|
|
|
| 299 |
| dedup at the shipped threshold (0.92) | precision 1.00, **recall 0.03** - the duplicate test almost never fires |
|
| 300 |
| dedup at its own best threshold (~0.56) | F1 **0.75** (precision 0.62, recall 0.95) |
|
| 301 |
| drift at the shipped threshold (0.6) | precision 0.08, **recall 0.96** - the topic-change test fires on almost every turn |
|
| 302 |
+
| drift as a *ranking* signal | **withdrawn** - measured on the pre-correction corpus (79% an ingested attachment), see above |
|
| 303 |
|
| 304 |
### 7.1 The F1 is the wrong target, and the threshold is not the lever
|
| 305 |
|
|
|
|
| 327 |
The topic-drift test is the opposite case: it only *penalises* the
|
| 328 |
off-topic chunks, it never deletes them, so its recalibration is free.
|
| 329 |
|
| 330 |
+
**The model is not the lever either.** A small encoder tuned on the
|
| 331 |
+
harness's own traffic (17M parameters, one epoch, the dedup task only)
|
| 332 |
+
raises the dedup decision at 0.92 from a recall of 0.046 to 0.986 on the
|
| 333 |
+
held-out job test. The replay then prices that decision: at the shipped
|
| 334 |
+
0.92 the tuned encoder keeps **32.66%** of the facts against the stock's
|
| 335 |
+
**60.51%**, for a context 2.7x smaller. The decision improved; the
|
| 336 |
+
curation nearly halved. The shipped encoder and the shipped 0.92 stay.
|
| 337 |
+
|
| 338 |
+
The reason is structural: a true duplicate merged is a fact deleted from
|
| 339 |
+
the context, so improving the dedup decision is improving the deletion.
|
| 340 |
+
The threshold and the model are two dials on the same trade-off curve,
|
| 341 |
+
and the fact-preservation constraint fixes where on it the system sits.
|
| 342 |
+
The clean replay and the three failed fine-tunes are in `AB-RESULTS.md`
|
| 343 |
+
§2C-quater through §2C-septies.
|
| 344 |
|
| 345 |
### 7.4 The curation's cost grows with the conversation it curates
|
| 346 |
|