KIP-Hash v2 β a frontier-distilled verify-and-dedup hash
One 128-bit signature, two jobs, no neural network at score time: fuzzy dedup AND quality verification from the same bits, pure-stdlib inference, weights packed as int8 (5.9 MB). KIP-Hash v2 is the hashing layer of KIP, an open $0-cost data pipeline β and the direct upgrade of kip-hash-v1.
The v1 idea β teach a hash to verify rows β was validated with a GLM teacher on 136 rubric labels. v2 rebuilds the whole stack around a frontier teacher (the DeepSeek webapp's default chat model), a pairwise-aware labeling protocol the teacher's verdicts on pairs, not just rows), and a learned projection trained on teacher-adjudicated duplicate pairs and teacher-rejected hard negatives. To our knowledge this is still the only shipped hash whose single signature answers "is this a duplicate?" and "is this worth keeping?" at the same time β and v2 is the first version whose dedup bits are trained on neural-verifier judgments of ambiguous pairs rather than sampled by IDF heuristics.
What changed from v1 β v2
| v1 (GLM) | v2 (DeepSeek webapp) | |
|---|---|---|
| teacher | glm-4-plus, one doc per call | frontier chat model, 6 rows/chat + 6 pairs/chat |
| teacher labels | 136 rows, 0-5 scale | 844 rows, 0-100 scale, + 58 adjudicated pairs |
| languages | 15 | 15 (full spread kept) |
| pair supervision | none | injected pairs (40, recall-tested) + organic (3) + ambiguous-band adjudication (58) |
| dedup bits | IDF-weighted MinHash sampling (heuristic learning) | learned projection W, contrastive on teacher-adjudicated pairs |
| verify from hash | 2-bit quality prefix | linear head on all 128 bits (+ finer head on the projection) |
| junk/boilerplate AUC (new teacher) | 0.689 | 0.755 |
| Spearman vs teacher (cross-teacher for v1) | 0.298 | 0.424 hash-only / 0.543 projection |
| false merges in the ambiguous band (13 hard pairs) | 1 (weighted Jaccard) | 0 |
| artifact | 18.4 MB | 5.9 MB |
How one hash does two jobs
Features. Every row becomes a sparse L2-normalized log-tf vector over blake2b-hashed buckets (32768): word unigrams, word bigrams, char 4-grams (CJK text uses char unigram/bigram + 4-grams). The feature extractor is one pure-stdlib function, imported by the training script, so train/infer parity is structural, not by convention.
Dedup: learned locality bits. The 128 bits are
sign(WΒ·x) where W β R^{32768Γ128} is trained β not sampled:
- positives = near-duplicates the teacher confirmed (paraphrase injections with known ground truth, organic corpus dups, plus the one true dup found in the ambiguous band);
- hard negatives = pairs from the same chat that the teacher scored but did not call duplicates (same language, same domain, shared boilerplate/vocabulary β the exact pairs that fool plain MinHash), plus 44 ambiguous-band pairs adjudicated pair-by-pair (MinHash-128 similarity 0.07-0.69; the teacher says not-dup);
- the objective is a cosine-margin contrastive loss in projection space (positives pulled β₯ 0.70, negatives pushed β€ 0.30), anchored to the random-projection prior for generalization.
Plain MinHash treats every shingle equally; the learned W weights content shingles over boilerplate, so two unrelated pages that share a navigation block stay far in Hamming space while a paraphrased duplicate stays close. Candidate generation is banding (8 bands Γ 16 bits), confirmation is full 128-bit Hamming distance (threshold 7 on pipeline-scale rows).
Verify: distilled verdicts from the same bits. Ridge heads are fit on the frozen quantized model:
quality_bitsβ linear head over the Β±1 bit vector β 0-100 verdict; this is the O(1) triage you can read from a stored hash without the text (v1 compressed this to 2 bits; v2 uses all 128);quality_projβ head over the pre-sign projection β finer verdict;junk_bits/junk_projβ junk/boilerplate flag with stored operating thresholds.
The teacher's quality signal also shaped W itself during training: a straight-through quality head runs alongside the contrastive objective, so the bit planes carry quality structure, not just duplicate geometry.
The teacher: DeepSeek webapp, batched, ephemeral
Protocol (full verbatim prompts in training/teacher_protocol.md):
- fresh chat β 6 corpus rows (same language, same domain; 20 batches
carry seeded paraphrase injections) β structured JSON:
{"rows":[{i, quality 0-100, knowledge 0-100, flag}], "dup_pairs":[[i,j]]}β chat deleted β jittered sleep β next chat. - 9 batches re-labeled in fresh chats for self-consistency: mean |Ξquality| 7.0, median 4.0 (the noise ceiling the student could ever reach).
- Ambiguous-band pairs (real shingle overlap β the false-merge zone) adjudicated pair-by-pair, 6 pairs per chat, chats deleted after.
- Teacher dup-pair recall measured against known ground truth: 40/40 injected pairs found (100%); 3 organic corpus dups discovered on top.
Every completion rides the webapp's own proof-of-work (DeepSeekHashV1
solved by DeepSeek's shipped pow.wasm), so the protocol is
account-bound, rate-limited (jittered 1.5-4.5 s), and leaves no chat
history behind.
Evaluation
Corpus: KIP shard_v25, 794 labeled rows, 15 languages; batch-level 80/20 split (no batch crosses), held-out synthetic paraphrases, teacher-adjudicated eval pairs never seen in training.
Verify (157 held-out rows; junk n=12)
| verifier | Spearman vs teacher | junk AUC | MAE (0-100) |
|---|---|---|---|
| v1 ensemble (GLM-distilled, cross-teacher) | 0.298 | 0.689 | β |
| v2 hash-only head (from 128 stored bits) | 0.424 | 0.755 | 11.2 |
| v2 projection head (at signing time) | 0.543 | 0.701 | 11.0 |
v2 beats v1 on every metric under the new frontier teacher β the honest cross-teacher comparison, since v1 was distilled against GLM.
Dedup β easy band (30 positives / 90 negatives)
| engine | best F1 | P | R |
|---|---|---|---|
| v2 learned bits | 0.983 | 1.00 | 0.967 |
| random-projection SimHash | 0.983 | 1.00 | 0.967 |
| plain MinHash-128 | 1.000 | 1.00 | 1.00 |
Everyone solves the easy band. The discrimination lives below.
Dedup β the false-merge zone (14 teacher-adjudicated pairs, MinHash sim 0.07-0.69)
| engine | not-dups wrongly merged (of 13) | true dups caught (of 1) |
|---|---|---|
| v2 learned bits (threshold 7) | 0 | 1 |
| plain MinHash-128 (its own best threshold) | 2 | 1 |
| IDF-weighted Jaccard (v1 verify stage) | 1 | 1 |
This is the band where boilerplate and same-entity/template overlap manufactures false merges. The learned bits reject every teacher-rejected hard negative while still catching the true duplicate.
Throughput (pure stdlib, one core)
45.0 docs/s signing (features + projection + all four heads + band keys). Slower than v1's 812 docs/s because v2 computes a real 128-dim learned projection per document instead of sampling 128 shingles β the price of trainable bits. A numpy-vectorized bulk path exists for large runs; the stdlib path needs nothing but Python.
Usage
Standalone (this repo, nothing to install):
from kip_hash import KipHashV2
vh = KipHashV2("kip-hash-v2.json")
sig = vh.sign_text("...row text...")
sig["quality"] # 0-100 distilled verdict, read from the 128 bits
sig["quality_p"] # finer verdict from the projection
sig["junk_prob"] # hash-only junk/boilerplate flag
sig["hex"] # 128-bit signature, 32 hex chars
vh.hamming(sig["bits"], sig2["bits"]) # <= 7 -> near-duplicate
kept, dups, report = vh.dedup_stream(texts) # one-pass fuzzy dedup
Command line: python3 kip_hash.py runs a 5-text demo + selftest
(near-dup pair vs different-topic vs same-topic-only, quality ranking,
one-pass dedup report).
Inside the KIP pipeline: the artifact is consumed through
kip.fullscale.learned_hash (v2 hook) or directly via this module;
dedup_stream reuses the survivor policy of the exact-dedup stage, and
quality/junk_prob join the quality spine as features.
Files
| file | what |
|---|---|
kip-hash-v2.json |
the artifact: int8 W (32768Γ128), 4 heads, config, teacher/provenance/eval blocks (5.9 MB) |
kip_hash.py |
pure-stdlib inference (features, projection, heads, hamming dedup, selftest) |
eval/report.json |
every number in this card, machine-readable |
training/teacher_protocol.md |
the verbatim DeepSeek webapp protocol + prompts |
training/teacher_labels_deepseek.jsonl |
844 teacher row verdicts |
training/teacher_dups.jsonl |
per-chat dup verdicts + injection ground truth |
training/teacher_pairs_adjudicated.jsonl |
58 ambiguous-band pair verdicts |
training/batches_manifest.jsonl |
batch metadata (no corpus text) |
training/train_v2.py |
the deterministic training script |
Limitations (honest)
- Evaluation scale. 157 verify rows, 30/90 dedup pairs, 14 adjudicated hard pairs β directional results, re-measure at 100k+ scale before release-gating on them.
- Teacher noise ceiling. The teacher itself reproduces within Β±7 points (median 4); student metrics are bounded by that.
- OOD text. Heads are calibrated on ~1600-char pipeline rows;
short standalone texts saturate (same caveat as v1), and the corpus
contained no
spam-flagged rows, so the junk head knows boilerplate/incoherent/junk but not classic web spam. - Thresholds are corpus-scale. Hamming threshold 7 is selected on pipeline-scale rows; short texts need a looser threshold (~25-30).
- Corpus-shaped bits. W is trained on a wiki-heavy multilingual sample; other domains should retrain with the shipped script and their own teacher labels (the protocol is fully documented).
- Lexical, not semantic. Paraphrase-level recall is bounded by n-gram overlap; this is not an embedding space, and cross-language translation dups are intentionally NOT merged.
- Throughput. 45 docs/s stdlib vs v1's 812 β trainable bits cost compute; use the numpy bulk path for >1M-row runs.
- The DeepSeek teacher inherits that model's biases; prompts, labels, verdicts and adjudications all ship for auditability.
Provenance
- Built 2026-11-18/19 by the KIP pipeline ($0 compute; teacher channel = the user's own DeepSeek webapp session, ephemeral chats).
- Corpus: KIP shard_v25 β wikipedia dumps, wikidata, wiktionary, eur-lex, openstax, RFCs (permissive sources only).
- Artifact policy: weights + metadata only, no corpus text.
- License: Apache-2.0.
Citations
- Penedo et al., 2024, The FineWeb Datasets β LLM-annotation β small-classifier distillation for data quality (FineWeb-Edu).
- DeepSeek-AI, 2024/2025, DeepSeek V2/V3 technical reports β the frontier teacher family; webapp annotation protocol documented in
training/teacher_protocol.md. - Manku, Jain & Das Sarma, 2007, Detecting Near-Duplicates for Web Crawling (WWW 2007) β simhash Hamming-space ancestry.
- Wang et al., 2016, A Survey on Learning to Hash β supervised bits vs random projections.
- Ioffe, 2010, Improved Consistent Sampling, Weighted MinHash and L1 Sketching β v1's sampling rule (superseded here by the learned projection).
- Orgad et al., 2025, LLMs Know More Than They Show; Azaria & Mitchell, 2023, The Internal State of an LLM Knows When It's Lying β probing the teacher's verdicts.
- DataComp-LM (arXiv:2406.11794); Nemotron-CC (arXiv:2412.02595) β model-based filtering context.