KIP-Hash v1 β€” a verify-and-dedup hash

One hash, two jobs: fuzzy dedup and quality verification, in a single signature, with pure-stdlib inference. KIP-Hash is the hashing layer of KIP, an open $0-cost data pipeline. It pairs corpus-shaped weighted MinHash locality bits (for fuzzy dedup at higher precision than plain MinHash) with a quality prefix distilled from a neural verifier (for triage and filtering without re-reading text). To our knowledge this combination β€” a learned hash whose signature simultaneously answers "is this a duplicate?" and "is this worth keeping?" β€” has not been shipped before as a single artifact.

TL;DR

  • 128 locality bits: IDF-weighted consistent sampling (Ioffe, ICDM
    1. over word 5-shingles (char k-grams for CJK/Thai families). Boilerplate is down-weighted before it can manufacture false merges, and candidate pairs are verified with the exact weighted Jaccard.
  • 2-bit quality prefix inside the same signature: the verdict of a linear verifier distilled from a large neural verifier (GLM glm-4-plus rubric annotations, FineWeb-Edu pattern). Reading the prefix classifies a stored row junk/low/mid/hi at O(1), without the text.
  • 3-member linear-probe ensemble, 2^18 hashed feature buckets (word uni/bi-grams + char trigrams + struct probe bands). No neural network at score time: 812 docs/s signing on one CPU core, pure Python standard library.
  • Measured teacher agreement: weak-label verifier AUC 0.809 vs teacher-distilled verifier AUC 0.888 on all teacher-labeled rows; distillation heldout agreement 0.971.

Why: one hash, two jobs

Plain MinHash treats every shingle as equally informative β€” random permutations over uniform shingles. That is why boilerplate (navigation bars, license footers, repeated collection phrases) manufactures false merges: two unrelated documents sharing a big boilerplate block cross the Jaccard threshold on the strength of text neither reader would call duplicate content. Separately, quality filtering usually runs a whole second stack (heuristic batteries, classifiers) over every row.

KIP-Hash fixes the hash, not just the threshold, and folds the quality verdict into the same bits:

  1. Corpus-shaped locality bits. Pass 1 counts each shingle's document frequency; weight w = log(1 + N/df) (classic IDF β€” boilerplate that repeats across the corpus is down-weighted, rare content shingles dominate). The signature keeps the L shingles with the smallest keys key = -log(u)/w, where u is a deterministic uniform derived from the shingle's own 64-bit hash β€” consistent weighted sampling (Ioffe 2010), no RNG anywhere. Candidate pairs are verified with the exact weighted Jaccard sum_s min(w_a, w_b) / sum_s max(w_a, w_b), so the weighting that shaped candidate generation also owns the final verdict.
  2. Verifier-shaped quality bits. The first two stored bits are the distilled linear verifier's quality class over Nemotron-CC-style 20 buckets: class 0 = tiers 0-4 (junk), 1 = tiers 5-9 (low), 2 = tiers 10-15 (mid), 3 = tiers 16-19 (HQ). Quality bits never enter the banding keys (a high-quality original and its low-quality scraper copy must still be candidates and merge); they ride along for verification, triage and reporting.

Signature layout: [q1 q0 | m1 .. mL] β€” quality prefix, then locality bits. Dedup and verification ride one signature, one pass, no second model stack.

How the verifier is taught: neural-verifier distillation

KIP cannot run a neural teacher at score time, so the pipeline ships both halves of the distillation contract (the probes lesson: a linear model over the right representation matches the full model β€” Azaria & Mitchell 2023; Orgad et al. 2024 found truthfulness information concentrated in internal representations that simple probes recover; FineWeb-Edu is exactly this pattern for data quality, teacher annotations β†’ small classifier):

  • Teacher: glm-4-plus (the strongest teacher available in our build environment) scored a deterministic stratified sample of the corpus β€” 144 documents across 15 languages and 6 domains, in-language rubric 0-5 (knowledge density, self-containedness, formatting, boilerplate), 2000-char excerpts. 136 labels landed; 8 documents are persistently excluded by the teacher API's content filter (recorded in the provenance). Score histogram: {1: 1, 2: 25, 3: 22, 4: 56, 5: 32}. The verbatim rubric ships in training/teacher_rubric.md, the labels in training/teacher_labels_glm.jsonl.
  • Student: sparse logistic regression over the hashed word/bi-gram
    • char-trigram + struct-probe feature space, trained on soft teacher labels (confidence becomes example weight); rows without a teacher fall back to intrinsic weak labels at 0.35 weight. Three seeds form the ensemble; mean-prob at score time.
  • Feature-space correctness: distilled members are stamped salt=0, struct=True so deploy-time scoring reads exactly the training space (see the artifact verifier stamps).

Head-to-head on every teacher-labeled row (AUC against teacher scores):

verifier AUC vs teacher
v1: weak labels only (intrinsic heuristics) 0.809
v2: teacher-distilled ensemble (this artifact) 0.888

Distillation heldout (deterministic 20% split of the labeled mix): agreement AUC 0.971. Calibration (mean teacher score per quality-class band, 1-5 scale): junk 2.43 β†’ mid 2.60 β†’ hi 4.13 β€” monotone at the ends; the tiny low band (n=2) is noisy.

Evaluation

Corpus: KIP shard_v25 β€” 3,034 records, 15 languages (en 795, fr 362, es 301, ru 293, ar 258, de 252, zh 242, ...), 6 domains (facts 2023, encyclopedic 524, events 288, reference 191, ...).

Fuzzy dedup (threshold 0.8, one pass):

engine candidates false-merge guard runtime (3034 docs)
plain MinHash (unweighted) 42 β€” 58.0 s
KIP-Hash (learned weights) 39 boilerplate crux rejected 12.9 s

The analytic boilerplate crux (KIP selftest, reproduced on every run): two unrelated documents share 276 boilerplate 5-shingles β€” plain Jaccard 0.8214 crosses a 0.8 threshold (false merge, and via transitivity it would cluster unrelated docs); the learned weighted Jaccard is 0.7350 β†’ rejected. The true near-dup pair keeps weighted Jaccard 0.9185 β†’ merged. On the curated shard itself there are no fuzzy duplicates at 0.8 (it is already exact-deduped and versioned); the candidate reduction and the crux are the honest measurable signal at this scale.

Throughput (pure stdlib Python, one CPU core, 300-doc benchmark): 812.6 docs/s signing; the learned one-pass dedup runs 3034 docs in 12.9 s vs 58.0 s for the plain engine (the IDF pass makes banding cheaper, not more expensive).

Usage

Standalone (this repo, nothing to install):

from kip_hash import load_default, sign_text, sig_agreement

ensemble = load_default("kip-hash-v1.json")
corpus = ["...doc 0...", "...doc 1...", "..."]
sig = sign_text("...new doc...", corpus, ensemble=ensemble)
# sig["class_name"] -> junk | low | mid | hi   (verify, no text re-read)
# sig["loc"]     -> locality bits              (fuzzy dedup banding)
# sig["hex"]     -> first 16 locality bits as hex

Command line: python3 kip_hash.py runs a 4-text demo (near-dup pair, different topic, spam) printing classes, tiers and pairwise locality agreements β€” near-dup collides (~0.8), different topics do not (0.0). Note: the quality prefix is calibrated on full pipeline documents; standalone short or synthetic texts are out-of-distribution (they saturate) β€” trust the prefix on in-pipeline data, where it separates junk/mid/hi across 3,034 real records (eval/report.json).

Inside the KIP pipeline (kip.fullscale):

from kip.fullscale import learned_hash

art = learned_hash.load_artifact("kip-hash-v1.json")
kept, dups, report = learned_hash.fuzzy_dedup(
    records, model=art["models"], threshold=0.8)
# report["precision_gain"]["plain_would_merge"] - false merges rejected
# report["quality_prefix"]["histogram"]         - hash-only triage

Files

file what
kip-hash-v1.json the artifact: hash config + 3-member distilled verifier ensemble + teacher/provenance/eval blocks (18.4 MB)
kip_hash.py standalone stdlib inference (verbatim extraction of the pipeline paths; parity-checked)
eval/report.json agreement, calibration, dedup comparison, throughput
training/teacher_rubric.md the verbatim teacher system prompt
training/teacher_labels_glm.jsonl 136 teacher rows ({id, score, label, language, domain, source})

Artifact schema: kip.hash.v1 β€” hash_config (k=5, 128 locality bits, 32 bands Γ— 4 rows, threshold 0.8, IDF formula, quality classes, char-shingle script table), verifier (feature space, feat scale, ensemble members with salt=0, struct=True stamps), teacher, provenance, eval.

Limitations (honest)

  • Teacher coverage is 136/3034 rows (~4.5%); rows without a teacher inherit the intrinsic weak-label prior at reduced weight (0.35).
  • The quality prefix is calibrated on full pipeline documents; on standalone short or synthetic texts it is out-of-distribution and saturates. Also, the weak prior treats very short rows (<30 words) as negative, so short atomic fact rows score lower than their teacher rubric value (2-3) β€” a known bias to refine with more teacher coverage, and a reason fact rows should not be dropped on the prefix alone.
  • The dedup evaluation corpus contains no fuzzy duplicates at 0.8 (curated + exact-deduped); the higher-precision claim rests on the analytic crux plus candidate reduction. At web scale the precision gain should be re-measured per corpus.
  • The quality prefix is a 4-class compression of a 20-bucket verdict; use the artifact's clf_prob when finer grading matters.
  • Locality bits are corpus-shaped: signatures depend on the IDF pass over your corpus. Signatures from different corpora are comparable only via the same weights file.
  • KIP-Hash is not a semantic embedding: paraphrase-level dedup recall is bounded by lexical shingle overlap; it is not an all-MiniLM-style similarity space.
  • The teacher (glm-4-plus) annotations inherit that model's biases; rubric, labels and raw transcripts ship for auditability.

Provenance

  • Built 2026-11-18 by the KIP pipeline (pure stdlib, $0 compute; teacher calls via the z-ai CLI).
  • Corpus: KIP shard_v25.jsonl (wikipedia dumps, wikidata facts, wiktionary, RFCs, eur-lex, openstax β€” permissive sources only).
  • Single-epoch regime throughout (mixtures.EPOCHS_DEFAULT=1).
  • License: Apache-2.0. The artifact contains model weights and score metadata only β€” no corpus text.

Citations

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Papers for IO-AIR/kip-hash-v1