KIP-Hash v3 β the nano verify-and-dedup word
One 28-bit word per row, two jobs, no neural network at score time:
word = bucket(24 bits, 6 learned scalars x 4-bit equal-mass bins) << 4
| score4(4 bits, 16 quantized quality levels)
fuzzy dedup AND quality verification packed into a single integer you can store in 3.5 bytes, look up in a dict at O(1), and triage without ever re-reading the text. Pure-stdlib inference. The whole artifact β int8 projection (32768 x 10), bin edges, heads, quantizers, eval block β is 606 KB, 10x smaller than v2's 5.9 MB.
KIP-Hash v3 is the hashing layer of KIP, an open $0-cost data pipeline, and the direct upgrade of kip-hash-v2.
What changed from v2 -> v3
| v2 (128-bit signature) | v3 (nano word) | |
|---|---|---|
| signature | 128 bits (16 B/row) | 28-bit word (3.5 B/row) |
| artifact | 5.9 MB | 606 KB |
| dedup geometry | learned sign-bits + Hamming banding (8x16) | **learned quantile-banded scalars: wide equal-mass bins, Β±1-bin probing, per-dim |
| verify | linear heads on 128 bits / projection | battery+projection head; 4-bit score PACKED in the word |
| quality (Spearman vs teacher) | 0.543 (projection head) | 0.719 |
| MAE (0-100) | 11.0 | 8.4 fine / 8.21 from the 4-bit score alone |
| hash-only triage | 0.42 (from 128 bits) | score4: 4 bits, MAE 8.21, O(1) from the word |
| dup pairs sharing a quality level | β | 83% |
| false-merge zone (13 hard not-dups) | 0 merges | 0 merges (and 1/1 true dup caught) |
| fuzzy dup recall (held-out) | β | 95.8% |
| confirmed dup recall @ 0.11% false-merge rate | β | 91.7% |
| throughput (pure stdlib) | 45 docs/s | 224 docs/s |
| teacher batch size | 6 rows/chat | 10 rows/chat + pair chats (staged) |
How one word does two jobs
Features. Every row becomes a sparse L2-normalized log-tf vector
over blake2b-hashed buckets (32768): word unigrams, word bigrams, char
4-grams (CJK uses char uni/bi-grams + 4-grams). One pure-stdlib
function (kip_hash3.features) is imported by the trainer, so
train/infer parity is structural β verified to 1e-16 with zero
bin/level mismatches.
Dedup: fuzzy by construction. Six learned scalar projections z_k = (w_k . x - mu_k) / sigma_k are quantized with equal-mass bin edges learned from the corpus (16 bins each -> a 24-bit bucket). Near-duplicates differ by less than a bin width, so quantization ABSORBS the perturbation instead of betting bit-by-bit (SimHash) or shingle-by-shingle (MinHash). Fewer bucket bits than a classic LSH is a feature, not a bug: buckets are deliberately fuzzy, candidate generation over-generates slightly, and a cheap per-dimension |dz| <= tau confirmation restores precision. Held-out dup pairs: 95.8% are fuzzy-bucket candidates, 91.7% survive confirmation at a 0.11% false-merge rate over 3604 held-out negatives. In the adversarial false-merge zone (14 teacher-adjudicated pairs with real shingle overlap, MinHash sim 0.15-0.46) v3 merges 0 of 13 not-dups and catches the 1 true dup β while plain MinHash-64 at its own best threshold merges 2. The untrained random-projection baseline with the SAME banding pipeline catches 32% of dups; learning the projection is worth 3x the recall at the same bucket budget.
Verify: 4 bits of verdict packed in the low bits. A linear head
over [z, battery] distills the teacher's 0-100 verdict; equal-mass
edges over the predicted distribution quantize it to 16 levels, and
the level index is packed into the low 4 bits of the word. Reading a
stored word triages a row at O(1): decoded-level MAE is 8.21 points β
statistically identical to the full-precision head (8.4) β and 83% of
true duplicate pairs land on the SAME quality level, so dedup
candidates are quality-consistent by construction. The battery is 16
deterministic surface features (length, alnum/punct/upper ratios,
unique-word ratio, 4-gram repetition, sentence stats, script ratios)
computable in pure stdlib at signing time. The projection z alone is
quality-blind (Spearman 0.05) β it is shaped for duplicate geometry β
and the battery supplies the verdict signal; together they reach
0.719 Spearman vs the teacher (v2: 0.543), MAE 8.4.
The teacher: DeepSeek webapp, batched, ephemeral
Same protocol as v2 (full verbatim prompts in
training/teacher_protocol_v3.md): fresh chat -> batched rows +
structured JSON verdict -> chat deleted -> jittered sleep -> next
chat. Every completion rides the webapp's own proof-of-work
(DeepSeekHashV1, solved by DeepSeek's shipped pow.wasm). The v2
corpus (844 row verdicts, 140 dup verdicts, 58 pair adjudications β
40/40 injected-dup recall, self-consistency |Ξq| mean 7.0) is carried
forward unchanged; v3 adds construction-known-truth synthetic pair
supervision (deterministic paraphrase positives, safe-similarity
negatives) generated by the trainer, which scales pair geometry beyond
teacher throughput without fabricating teacher scores.
The SCALED v3 campaign β 1575 rows, 8 domains, 15 languages, 10
rows/chat, 30 triple-injection chats, 60 fresh ambiguous-band pairs,
9 calibration chats, 181 chats total (~21 min) β is fully staged in
scripts/prep_batches_v3.py + scripts/label_run_v3.py and fires the
moment the webapp JWT is refreshed (the previous token expired with
40003 INVALID_TOKEN after 140+ successful chats). v3.1 will retrain
on that corpus with the same script.
Evaluation
Corpus: KIP teacher labels (794 scored rows, 15 languages, 6 domains β wiki-heavy by v2's design) + synthetic known-truth pairs; batch-level 80/20 split (no batch crosses); 157 held-out rows, 96 held-out dup pairs, 3604 held-out negatives (incl. teacher-adjudicated hard negatives); all numbers from the dequantized artifact (the int8 weights ARE the source of truth β an earlier draft's metrics computed on un-dequantized int8 projections and were discarded).
Verify (157 held-out rows; junk n=12)
| verifier | Spearman vs teacher | junk AUC | MAE (0-100) |
|---|---|---|---|
| v2 hash-only head (128 bits) | 0.424 | 0.755 | 11.2 |
| v2 projection head | 0.543 | 0.701 | 11.0 |
| v3 fine (battery + z, at signing) | 0.719 | 0.735 | 8.4 |
| v3 packed score4 (from the word alone) | β | β | 8.21 |
Dedup (held-out pairs)
| engine | exact-bucket recall | fuzzy recall (Β±1 bin) | confirmed recall | false merges |
|---|---|---|---|---|
| v3 learned nano word | 39.6% | 95.8% | 91.7% @ tau op | 0.11% (4/3604) |
| random SimHash, same banding | 4.2% | β | 32.3% | 0% |
| plain MinHash-64 (best F1 thr) | β | β | ~99% F1 (easy band) | 2/13 in the false-merge zone |
| false-merge zone (adjudicated) | β | β | 1/1 dups caught | 0/13 not-dups merged |
Usage
Standalone (this repo, nothing to install):
from kip_hash3 import KipHashV3
vh = KipHashV3("kip-hash-v3.json")
w = vh.sign_text("...row text...")
w["word"] # 28-bit int: 24 bucket bits | 4 score bits
w["score4"] # 0-15 packed quality level
w["quality"] # decoded 0-100 verdict for that level
w["quality_fine"] # full-precision verdict (battery + projection)
w["junk_prob"] # hash-side junk/boilerplate flag
w["word_hex"] # 7 hex chars
vh.fuzzy_keys(w["bins"]) # +-1-bin probe keys
vh.confirm(z_a, z_b) # post-candidate |dz| verification
kept, dups, rep = vh.dedup_stream(texts) # one-pass fuzzy dedup
Command line: python3 kip_hash3.py runs the demo + selftest
(pipeline-scale near-dup merge, different-topic rejection,
same-topic-different-text rejection, packed-score ordering,
determinism, one-pass dedup report).
Inside the KIP pipeline: kip.fullscale.learned_hash gained
load_v3 / nano_dedup / nano_triage β the same (kept, dups,
report) contract as the exact-dedup stage, plus score4-based triage
that reads only the stored word.
Files
| file | what |
|---|---|
kip-hash-v3.json |
the artifact: int8 W (32768x10), bin edges, |
kip_hash3.py |
pure-stdlib inference (features, battery, projection, banding, fuzzy keys, confirm, dedup stream, selftest) |
eval/report.json |
every number in this card, machine-readable |
training/teacher_protocol_v3.md |
the verbatim DeepSeek webapp protocol + prompts + staged scaled run |
training/teacher_labels_v2_carried.jsonl |
844 teacher row verdicts |
training/teacher_dups_v2_carried.jsonl |
per-chat dup verdicts + injection ground truth |
training/teacher_pairs_adjudicated_v2.jsonl |
58 ambiguous-band pair verdicts (train/eval sides never mixed) |
training/batches_manifest_v3.jsonl |
the staged scaled run (metadata only, no corpus text) |
training/train_v3.py |
the deterministic training script |
training/label_run_v3.py |
the resumable chat-per-batch labeling loop |
Limitations (honest)
- v2.0.1 note. While building v3 we found that v2's projection
heads were fit on un-dequantized int8 projections (and its junk
thresholds were selected in a different score space than the module
compares): v2's
quality(from the 128 bits) and all dedup numbers are valid, but its deployedquality_p/junk_proboutputs are mis-scaled until a v2.0.1 head re-fit. v3 is built on the corrected dequantize-first pipeline end to end, and this note ships with v3 so nobody deploys v2's projection heads blind. - Evaluation scale. 157 verify rows, 96 dup pairs, 3604 negatives, 14 adjudicated hard pairs β directional results; the staged scaled campaign (~2400 more labels across 8 domains) is one JWT away and exists precisely to re-measure at scale.
- Teacher noise ceiling. The teacher reproduces within Β±7 points (median 4); the battery head's 0.719 Spearman partly reflects the teacher's own surface-driven scoring style, and may inherit its length/domain biases (battery includes log-length).
- Corpus-scale geometry. Bins, tau and the dup geometry are calibrated on ~1200-1600-char pipeline rows. Short standalone texts need a looser tau (same caveat as v2; the selftest documents it).
- z is quality-blind by design. The projection dims serve dedup; quality rides on the battery head at signing time. Hash-ONLY verdicts are the packed score4 (16 levels), not the fine score.
- Lexical, not semantic. Recall is bounded by n-gram overlap; translation dups are intentionally NOT merged.
- The DeepSeek teacher inherits that model's biases; prompts, labels, verdicts and adjudications all ship for auditability.
Provenance
- Built 2026-11-19 by the KIP pipeline ($0 compute; teacher channel = the user's own DeepSeek webapp session, ephemeral chats, PoW).
- Teacher corpus: KIP shard_v25 β wikipedia dumps, wikidata, wiktionary, eur-lex, openstax, RFCs (permissive sources only).
- Artifact policy: weights + metadata only, no corpus text.
- License: Apache-2.0.
Citations
- Penedo et al., 2024, The FineWeb Datasets β LLM-annotation -> small-classifier distillation for data quality (FineWeb-Edu).
- DeepSeek-AI, 2024/2025, DeepSeek V2/V3 technical reports β the frontier teacher family; webapp annotation protocol in
training/teacher_protocol_v3.md. - Manku, Jain & Das Sarma, 2007, Detecting Near-Duplicates for Web Crawling (WWW 2007) β simhash ancestry; v3 replaces per-bit betting with quantile banding.
- Wang et al., 2016, A Survey on Learning to Hash β supervised hashing vs random projections.
- Gionis et al., 1999, Similarity Search in High Dimensions via Hashing (VLDB) β LSH bucket probing; v3 probes Β±1 quantile bins.
- DataComp-LM (arXiv:2406.11794); Nemotron-CC (arXiv:2412.02595) β model-based filtering context.