KIP-Hash v3 β€” the nano verify-and-dedup word

One 28-bit word per row, two jobs, no neural network at score time:

word = bucket(24 bits, 6 learned scalars x 4-bit equal-mass bins) << 4
     | score4(4 bits, 16 quantized quality levels)

fuzzy dedup AND quality verification packed into a single integer you can store in 3.5 bytes, look up in a dict at O(1), and triage without ever re-reading the text. Pure-stdlib inference. The whole artifact β€” int8 projection (32768 x 10), bin edges, heads, quantizers, eval block β€” is 606 KB, 10x smaller than v2's 5.9 MB.

KIP-Hash v3 is the hashing layer of KIP, an open $0-cost data pipeline, and the direct upgrade of kip-hash-v2.

What changed from v2 -> v3

v2 (128-bit signature) v3 (nano word)
signature 128 bits (16 B/row) 28-bit word (3.5 B/row)
artifact 5.9 MB 606 KB
dedup geometry learned sign-bits + Hamming banding (8x16) **learned quantile-banded scalars: wide equal-mass bins, Β±1-bin probing, per-dim
verify linear heads on 128 bits / projection battery+projection head; 4-bit score PACKED in the word
quality (Spearman vs teacher) 0.543 (projection head) 0.719
MAE (0-100) 11.0 8.4 fine / 8.21 from the 4-bit score alone
hash-only triage 0.42 (from 128 bits) score4: 4 bits, MAE 8.21, O(1) from the word
dup pairs sharing a quality level β€” 83%
false-merge zone (13 hard not-dups) 0 merges 0 merges (and 1/1 true dup caught)
fuzzy dup recall (held-out) β€” 95.8%
confirmed dup recall @ 0.11% false-merge rate β€” 91.7%
throughput (pure stdlib) 45 docs/s 224 docs/s
teacher batch size 6 rows/chat 10 rows/chat + pair chats (staged)

How one word does two jobs

Features. Every row becomes a sparse L2-normalized log-tf vector over blake2b-hashed buckets (32768): word unigrams, word bigrams, char 4-grams (CJK uses char uni/bi-grams + 4-grams). One pure-stdlib function (kip_hash3.features) is imported by the trainer, so train/infer parity is structural β€” verified to 1e-16 with zero bin/level mismatches.

Dedup: fuzzy by construction. Six learned scalar projections z_k = (w_k . x - mu_k) / sigma_k are quantized with equal-mass bin edges learned from the corpus (16 bins each -> a 24-bit bucket). Near-duplicates differ by less than a bin width, so quantization ABSORBS the perturbation instead of betting bit-by-bit (SimHash) or shingle-by-shingle (MinHash). Fewer bucket bits than a classic LSH is a feature, not a bug: buckets are deliberately fuzzy, candidate generation over-generates slightly, and a cheap per-dimension |dz| <= tau confirmation restores precision. Held-out dup pairs: 95.8% are fuzzy-bucket candidates, 91.7% survive confirmation at a 0.11% false-merge rate over 3604 held-out negatives. In the adversarial false-merge zone (14 teacher-adjudicated pairs with real shingle overlap, MinHash sim 0.15-0.46) v3 merges 0 of 13 not-dups and catches the 1 true dup β€” while plain MinHash-64 at its own best threshold merges 2. The untrained random-projection baseline with the SAME banding pipeline catches 32% of dups; learning the projection is worth 3x the recall at the same bucket budget.

Verify: 4 bits of verdict packed in the low bits. A linear head over [z, battery] distills the teacher's 0-100 verdict; equal-mass edges over the predicted distribution quantize it to 16 levels, and the level index is packed into the low 4 bits of the word. Reading a stored word triages a row at O(1): decoded-level MAE is 8.21 points β€” statistically identical to the full-precision head (8.4) β€” and 83% of true duplicate pairs land on the SAME quality level, so dedup candidates are quality-consistent by construction. The battery is 16 deterministic surface features (length, alnum/punct/upper ratios, unique-word ratio, 4-gram repetition, sentence stats, script ratios) computable in pure stdlib at signing time. The projection z alone is quality-blind (Spearman 0.05) β€” it is shaped for duplicate geometry β€” and the battery supplies the verdict signal; together they reach 0.719 Spearman vs the teacher (v2: 0.543), MAE 8.4.

The teacher: DeepSeek webapp, batched, ephemeral

Same protocol as v2 (full verbatim prompts in training/teacher_protocol_v3.md): fresh chat -> batched rows + structured JSON verdict -> chat deleted -> jittered sleep -> next chat. Every completion rides the webapp's own proof-of-work (DeepSeekHashV1, solved by DeepSeek's shipped pow.wasm). The v2 corpus (844 row verdicts, 140 dup verdicts, 58 pair adjudications β€” 40/40 injected-dup recall, self-consistency |Ξ”q| mean 7.0) is carried forward unchanged; v3 adds construction-known-truth synthetic pair supervision (deterministic paraphrase positives, safe-similarity negatives) generated by the trainer, which scales pair geometry beyond teacher throughput without fabricating teacher scores.

The SCALED v3 campaign β€” 1575 rows, 8 domains, 15 languages, 10 rows/chat, 30 triple-injection chats, 60 fresh ambiguous-band pairs, 9 calibration chats, 181 chats total (~21 min) β€” is fully staged in scripts/prep_batches_v3.py + scripts/label_run_v3.py and fires the moment the webapp JWT is refreshed (the previous token expired with 40003 INVALID_TOKEN after 140+ successful chats). v3.1 will retrain on that corpus with the same script.

Evaluation

Corpus: KIP teacher labels (794 scored rows, 15 languages, 6 domains β€” wiki-heavy by v2's design) + synthetic known-truth pairs; batch-level 80/20 split (no batch crosses); 157 held-out rows, 96 held-out dup pairs, 3604 held-out negatives (incl. teacher-adjudicated hard negatives); all numbers from the dequantized artifact (the int8 weights ARE the source of truth β€” an earlier draft's metrics computed on un-dequantized int8 projections and were discarded).

Verify (157 held-out rows; junk n=12)

verifier Spearman vs teacher junk AUC MAE (0-100)
v2 hash-only head (128 bits) 0.424 0.755 11.2
v2 projection head 0.543 0.701 11.0
v3 fine (battery + z, at signing) 0.719 0.735 8.4
v3 packed score4 (from the word alone) β€” β€” 8.21

Dedup (held-out pairs)

engine exact-bucket recall fuzzy recall (Β±1 bin) confirmed recall false merges
v3 learned nano word 39.6% 95.8% 91.7% @ tau op 0.11% (4/3604)
random SimHash, same banding 4.2% β€” 32.3% 0%
plain MinHash-64 (best F1 thr) β€” β€” ~99% F1 (easy band) 2/13 in the false-merge zone
false-merge zone (adjudicated) β€” β€” 1/1 dups caught 0/13 not-dups merged

Usage

Standalone (this repo, nothing to install):

from kip_hash3 import KipHashV3

vh = KipHashV3("kip-hash-v3.json")
w = vh.sign_text("...row text...")
w["word"]          # 28-bit int: 24 bucket bits | 4 score bits
w["score4"]        # 0-15 packed quality level
w["quality"]       # decoded 0-100 verdict for that level
w["quality_fine"]  # full-precision verdict (battery + projection)
w["junk_prob"]     # hash-side junk/boilerplate flag
w["word_hex"]      # 7 hex chars

vh.fuzzy_keys(w["bins"])           # +-1-bin probe keys
vh.confirm(z_a, z_b)               # post-candidate |dz| verification
kept, dups, rep = vh.dedup_stream(texts)   # one-pass fuzzy dedup

Command line: python3 kip_hash3.py runs the demo + selftest (pipeline-scale near-dup merge, different-topic rejection, same-topic-different-text rejection, packed-score ordering, determinism, one-pass dedup report).

Inside the KIP pipeline: kip.fullscale.learned_hash gained load_v3 / nano_dedup / nano_triage β€” the same (kept, dups, report) contract as the exact-dedup stage, plus score4-based triage that reads only the stored word.

Files

file what
kip-hash-v3.json the artifact: int8 W (32768x10), bin edges,
kip_hash3.py pure-stdlib inference (features, battery, projection, banding, fuzzy keys, confirm, dedup stream, selftest)
eval/report.json every number in this card, machine-readable
training/teacher_protocol_v3.md the verbatim DeepSeek webapp protocol + prompts + staged scaled run
training/teacher_labels_v2_carried.jsonl 844 teacher row verdicts
training/teacher_dups_v2_carried.jsonl per-chat dup verdicts + injection ground truth
training/teacher_pairs_adjudicated_v2.jsonl 58 ambiguous-band pair verdicts (train/eval sides never mixed)
training/batches_manifest_v3.jsonl the staged scaled run (metadata only, no corpus text)
training/train_v3.py the deterministic training script
training/label_run_v3.py the resumable chat-per-batch labeling loop

Limitations (honest)

  • v2.0.1 note. While building v3 we found that v2's projection heads were fit on un-dequantized int8 projections (and its junk thresholds were selected in a different score space than the module compares): v2's quality (from the 128 bits) and all dedup numbers are valid, but its deployed quality_p/junk_prob outputs are mis-scaled until a v2.0.1 head re-fit. v3 is built on the corrected dequantize-first pipeline end to end, and this note ships with v3 so nobody deploys v2's projection heads blind.
  • Evaluation scale. 157 verify rows, 96 dup pairs, 3604 negatives, 14 adjudicated hard pairs β€” directional results; the staged scaled campaign (~2400 more labels across 8 domains) is one JWT away and exists precisely to re-measure at scale.
  • Teacher noise ceiling. The teacher reproduces within Β±7 points (median 4); the battery head's 0.719 Spearman partly reflects the teacher's own surface-driven scoring style, and may inherit its length/domain biases (battery includes log-length).
  • Corpus-scale geometry. Bins, tau and the dup geometry are calibrated on ~1200-1600-char pipeline rows. Short standalone texts need a looser tau (same caveat as v2; the selftest documents it).
  • z is quality-blind by design. The projection dims serve dedup; quality rides on the battery head at signing time. Hash-ONLY verdicts are the packed score4 (16 levels), not the fine score.
  • Lexical, not semantic. Recall is bounded by n-gram overlap; translation dups are intentionally NOT merged.
  • The DeepSeek teacher inherits that model's biases; prompts, labels, verdicts and adjudications all ship for auditability.

Provenance

  • Built 2026-11-19 by the KIP pipeline ($0 compute; teacher channel = the user's own DeepSeek webapp session, ephemeral chats, PoW).
  • Teacher corpus: KIP shard_v25 β€” wikipedia dumps, wikidata, wiktionary, eur-lex, openstax, RFCs (permissive sources only).
  • Artifact policy: weights + metadata only, no corpus text.
  • License: Apache-2.0.

Citations

  • Penedo et al., 2024, The FineWeb Datasets β€” LLM-annotation -> small-classifier distillation for data quality (FineWeb-Edu).
  • DeepSeek-AI, 2024/2025, DeepSeek V2/V3 technical reports β€” the frontier teacher family; webapp annotation protocol in training/teacher_protocol_v3.md.
  • Manku, Jain & Das Sarma, 2007, Detecting Near-Duplicates for Web Crawling (WWW 2007) β€” simhash ancestry; v3 replaces per-bit betting with quantile banding.
  • Wang et al., 2016, A Survey on Learning to Hash β€” supervised hashing vs random projections.
  • Gionis et al., 1999, Similarity Search in High Dimensions via Hashing (VLDB) β€” LSH bucket probing; v3 probes Β±1 quantile bins.
  • DataComp-LM (arXiv:2406.11794); Nemotron-CC (arXiv:2412.02595) β€” model-based filtering context.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Papers for IO-AIR/kip-hash-v3