KIP-Hash v1 β a verify-and-dedup hash
One hash, two jobs: fuzzy dedup and quality verification, in a single signature, with pure-stdlib inference. KIP-Hash is the hashing layer of KIP, an open $0-cost data pipeline. It pairs corpus-shaped weighted MinHash locality bits (for fuzzy dedup at higher precision than plain MinHash) with a quality prefix distilled from a neural verifier (for triage and filtering without re-reading text). To our knowledge this combination β a learned hash whose signature simultaneously answers "is this a duplicate?" and "is this worth keeping?" β has not been shipped before as a single artifact.
TL;DR
- 128 locality bits: IDF-weighted consistent sampling (Ioffe, ICDM
- over word 5-shingles (char k-grams for CJK/Thai families). Boilerplate is down-weighted before it can manufacture false merges, and candidate pairs are verified with the exact weighted Jaccard.
- 2-bit quality prefix inside the same signature: the verdict of a linear verifier distilled from a large neural verifier (GLM glm-4-plus rubric annotations, FineWeb-Edu pattern). Reading the prefix classifies a stored row junk/low/mid/hi at O(1), without the text.
- 3-member linear-probe ensemble, 2^18 hashed feature buckets (word uni/bi-grams + char trigrams + struct probe bands). No neural network at score time: 812 docs/s signing on one CPU core, pure Python standard library.
- Measured teacher agreement: weak-label verifier AUC 0.809 vs teacher-distilled verifier AUC 0.888 on all teacher-labeled rows; distillation heldout agreement 0.971.
Why: one hash, two jobs
Plain MinHash treats every shingle as equally informative β random permutations over uniform shingles. That is why boilerplate (navigation bars, license footers, repeated collection phrases) manufactures false merges: two unrelated documents sharing a big boilerplate block cross the Jaccard threshold on the strength of text neither reader would call duplicate content. Separately, quality filtering usually runs a whole second stack (heuristic batteries, classifiers) over every row.
KIP-Hash fixes the hash, not just the threshold, and folds the quality verdict into the same bits:
- Corpus-shaped locality bits. Pass 1 counts each shingle's
document frequency; weight
w = log(1 + N/df)(classic IDF β boilerplate that repeats across the corpus is down-weighted, rare content shingles dominate). The signature keeps the L shingles with the smallest keyskey = -log(u)/w, whereuis a deterministic uniform derived from the shingle's own 64-bit hash β consistent weighted sampling (Ioffe 2010), no RNG anywhere. Candidate pairs are verified with the exact weighted Jaccardsum_s min(w_a, w_b) / sum_s max(w_a, w_b), so the weighting that shaped candidate generation also owns the final verdict. - Verifier-shaped quality bits. The first two stored bits are the distilled linear verifier's quality class over Nemotron-CC-style 20 buckets: class 0 = tiers 0-4 (junk), 1 = tiers 5-9 (low), 2 = tiers 10-15 (mid), 3 = tiers 16-19 (HQ). Quality bits never enter the banding keys (a high-quality original and its low-quality scraper copy must still be candidates and merge); they ride along for verification, triage and reporting.
Signature layout: [q1 q0 | m1 .. mL] β quality prefix, then locality
bits. Dedup and verification ride one signature, one pass, no second
model stack.
How the verifier is taught: neural-verifier distillation
KIP cannot run a neural teacher at score time, so the pipeline ships both halves of the distillation contract (the probes lesson: a linear model over the right representation matches the full model β Azaria & Mitchell 2023; Orgad et al. 2024 found truthfulness information concentrated in internal representations that simple probes recover; FineWeb-Edu is exactly this pattern for data quality, teacher annotations β small classifier):
- Teacher:
glm-4-plus(the strongest teacher available in our build environment) scored a deterministic stratified sample of the corpus β 144 documents across 15 languages and 6 domains, in-language rubric 0-5 (knowledge density, self-containedness, formatting, boilerplate), 2000-char excerpts. 136 labels landed; 8 documents are persistently excluded by the teacher API's content filter (recorded in the provenance). Score histogram:{1: 1, 2: 25, 3: 22, 4: 56, 5: 32}. The verbatim rubric ships intraining/teacher_rubric.md, the labels intraining/teacher_labels_glm.jsonl. - Student: sparse logistic regression over the hashed word/bi-gram
- char-trigram + struct-probe feature space, trained on soft teacher labels (confidence becomes example weight); rows without a teacher fall back to intrinsic weak labels at 0.35 weight. Three seeds form the ensemble; mean-prob at score time.
- Feature-space correctness: distilled members are stamped
salt=0, struct=Trueso deploy-time scoring reads exactly the training space (see the artifact verifier stamps).
Head-to-head on every teacher-labeled row (AUC against teacher scores):
| verifier | AUC vs teacher |
|---|---|
| v1: weak labels only (intrinsic heuristics) | 0.809 |
| v2: teacher-distilled ensemble (this artifact) | 0.888 |
Distillation heldout (deterministic 20% split of the labeled mix): agreement AUC 0.971. Calibration (mean teacher score per quality-class band, 1-5 scale): junk 2.43 β mid 2.60 β hi 4.13 β monotone at the ends; the tiny low band (n=2) is noisy.
Evaluation
Corpus: KIP shard_v25 β 3,034 records, 15 languages (en 795, fr 362,
es 301, ru 293, ar 258, de 252, zh 242, ...), 6 domains (facts 2023,
encyclopedic 524, events 288, reference 191, ...).
Fuzzy dedup (threshold 0.8, one pass):
| engine | candidates | false-merge guard | runtime (3034 docs) |
|---|---|---|---|
| plain MinHash (unweighted) | 42 | β | 58.0 s |
| KIP-Hash (learned weights) | 39 | boilerplate crux rejected | 12.9 s |
The analytic boilerplate crux (KIP selftest, reproduced on every run): two unrelated documents share 276 boilerplate 5-shingles β plain Jaccard 0.8214 crosses a 0.8 threshold (false merge, and via transitivity it would cluster unrelated docs); the learned weighted Jaccard is 0.7350 β rejected. The true near-dup pair keeps weighted Jaccard 0.9185 β merged. On the curated shard itself there are no fuzzy duplicates at 0.8 (it is already exact-deduped and versioned); the candidate reduction and the crux are the honest measurable signal at this scale.
Throughput (pure stdlib Python, one CPU core, 300-doc benchmark): 812.6 docs/s signing; the learned one-pass dedup runs 3034 docs in 12.9 s vs 58.0 s for the plain engine (the IDF pass makes banding cheaper, not more expensive).
Usage
Standalone (this repo, nothing to install):
from kip_hash import load_default, sign_text, sig_agreement
ensemble = load_default("kip-hash-v1.json")
corpus = ["...doc 0...", "...doc 1...", "..."]
sig = sign_text("...new doc...", corpus, ensemble=ensemble)
# sig["class_name"] -> junk | low | mid | hi (verify, no text re-read)
# sig["loc"] -> locality bits (fuzzy dedup banding)
# sig["hex"] -> first 16 locality bits as hex
Command line: python3 kip_hash.py runs a 4-text demo (near-dup pair,
different topic, spam) printing classes, tiers and pairwise locality
agreements β near-dup collides (~0.8), different topics do not (0.0).
Note: the quality prefix is calibrated on full pipeline documents;
standalone short or synthetic texts are out-of-distribution (they
saturate) β trust the prefix on in-pipeline data, where it separates
junk/mid/hi across 3,034 real records (eval/report.json).
Inside the KIP pipeline (kip.fullscale):
from kip.fullscale import learned_hash
art = learned_hash.load_artifact("kip-hash-v1.json")
kept, dups, report = learned_hash.fuzzy_dedup(
records, model=art["models"], threshold=0.8)
# report["precision_gain"]["plain_would_merge"] - false merges rejected
# report["quality_prefix"]["histogram"] - hash-only triage
Files
| file | what |
|---|---|
kip-hash-v1.json |
the artifact: hash config + 3-member distilled verifier ensemble + teacher/provenance/eval blocks (18.4 MB) |
kip_hash.py |
standalone stdlib inference (verbatim extraction of the pipeline paths; parity-checked) |
eval/report.json |
agreement, calibration, dedup comparison, throughput |
training/teacher_rubric.md |
the verbatim teacher system prompt |
training/teacher_labels_glm.jsonl |
136 teacher rows ({id, score, label, language, domain, source}) |
Artifact schema: kip.hash.v1 β hash_config (k=5, 128 locality bits,
32 bands Γ 4 rows, threshold 0.8, IDF formula, quality classes,
char-shingle script table), verifier (feature space, feat scale,
ensemble members with salt=0, struct=True stamps), teacher,
provenance, eval.
Limitations (honest)
- Teacher coverage is 136/3034 rows (~4.5%); rows without a teacher inherit the intrinsic weak-label prior at reduced weight (0.35).
- The quality prefix is calibrated on full pipeline documents; on standalone short or synthetic texts it is out-of-distribution and saturates. Also, the weak prior treats very short rows (<30 words) as negative, so short atomic fact rows score lower than their teacher rubric value (2-3) β a known bias to refine with more teacher coverage, and a reason fact rows should not be dropped on the prefix alone.
- The dedup evaluation corpus contains no fuzzy duplicates at 0.8 (curated + exact-deduped); the higher-precision claim rests on the analytic crux plus candidate reduction. At web scale the precision gain should be re-measured per corpus.
- The quality prefix is a 4-class compression of a 20-bucket verdict;
use the artifact's
clf_probwhen finer grading matters. - Locality bits are corpus-shaped: signatures depend on the IDF pass over your corpus. Signatures from different corpora are comparable only via the same weights file.
- KIP-Hash is not a semantic embedding: paraphrase-level dedup recall is bounded by lexical shingle overlap; it is not an all-MiniLM-style similarity space.
- The teacher (glm-4-plus) annotations inherit that model's biases; rubric, labels and raw transcripts ship for auditability.
Provenance
- Built 2026-11-18 by the KIP pipeline (pure stdlib, $0 compute; teacher calls via the z-ai CLI).
- Corpus: KIP
shard_v25.jsonl(wikipedia dumps, wikidata facts, wiktionary, RFCs, eur-lex, openstax β permissive sources only). - Single-epoch regime throughout (
mixtures.EPOCHS_DEFAULT=1). - License: Apache-2.0. The artifact contains model weights and score metadata only β no corpus text.
Citations
- Penedo et al., 2024, The FineWeb Datasets β LLM-annotation β small-classifier distillation for data quality (FineWeb-Edu).
- Orgad et al., 2025, LLMs Know More Than They Show β internal representations + simple probes recover verifier-level truthfulness signal.
- Azaria & Mitchell, 2023, The Internal State of an LLM Knows When It's Lying β linear probes over internal states.
- Ioffe, 2010, Improved Consistent Sampling, Weighted Minhash and L1 Sketching (ICDM 2010, pp. 246-255) β the weighted sampling rule.
- Manku, Jain & Das Sarma, 2007, Detecting Near-Duplicates for Web Crawling (WWW 2007) β simhash Hamming-space dedup ancestry.
- Wang et al., 2016, A Survey on Learning to Hash β supervised/corpus-shaped bits vs random projections.
- DataComp-LM (arXiv:2406.11794) β model-based filtering baseline; Nemotron-CC (arXiv:2412.02595) β ensemble β 20 quality buckets driving the mixture.