KIP-Hash v2 β€” a frontier-distilled verify-and-dedup hash

One 128-bit signature, two jobs, no neural network at score time: fuzzy dedup AND quality verification from the same bits, pure-stdlib inference, weights packed as int8 (5.9 MB). KIP-Hash v2 is the hashing layer of KIP, an open $0-cost data pipeline β€” and the direct upgrade of kip-hash-v1.

The v1 idea β€” teach a hash to verify rows β€” was validated with a GLM teacher on 136 rubric labels. v2 rebuilds the whole stack around a frontier teacher (the DeepSeek webapp's default chat model), a pairwise-aware labeling protocol the teacher's verdicts on pairs, not just rows), and a learned projection trained on teacher-adjudicated duplicate pairs and teacher-rejected hard negatives. To our knowledge this is still the only shipped hash whose single signature answers "is this a duplicate?" and "is this worth keeping?" at the same time β€” and v2 is the first version whose dedup bits are trained on neural-verifier judgments of ambiguous pairs rather than sampled by IDF heuristics.

What changed from v1 β†’ v2

v1 (GLM) v2 (DeepSeek webapp)
teacher glm-4-plus, one doc per call frontier chat model, 6 rows/chat + 6 pairs/chat
teacher labels 136 rows, 0-5 scale 844 rows, 0-100 scale, + 58 adjudicated pairs
languages 15 15 (full spread kept)
pair supervision none injected pairs (40, recall-tested) + organic (3) + ambiguous-band adjudication (58)
dedup bits IDF-weighted MinHash sampling (heuristic learning) learned projection W, contrastive on teacher-adjudicated pairs
verify from hash 2-bit quality prefix linear head on all 128 bits (+ finer head on the projection)
junk/boilerplate AUC (new teacher) 0.689 0.755
Spearman vs teacher (cross-teacher for v1) 0.298 0.424 hash-only / 0.543 projection
false merges in the ambiguous band (13 hard pairs) 1 (weighted Jaccard) 0
artifact 18.4 MB 5.9 MB

How one hash does two jobs

Features. Every row becomes a sparse L2-normalized log-tf vector over blake2b-hashed buckets (32768): word unigrams, word bigrams, char 4-grams (CJK text uses char unigram/bigram + 4-grams). The feature extractor is one pure-stdlib function, imported by the training script, so train/infer parity is structural, not by convention.

Dedup: learned locality bits. The 128 bits are sign(WΒ·x) where W ∈ R^{32768Γ—128} is trained β€” not sampled:

  • positives = near-duplicates the teacher confirmed (paraphrase injections with known ground truth, organic corpus dups, plus the one true dup found in the ambiguous band);
  • hard negatives = pairs from the same chat that the teacher scored but did not call duplicates (same language, same domain, shared boilerplate/vocabulary β€” the exact pairs that fool plain MinHash), plus 44 ambiguous-band pairs adjudicated pair-by-pair (MinHash-128 similarity 0.07-0.69; the teacher says not-dup);
  • the objective is a cosine-margin contrastive loss in projection space (positives pulled β‰₯ 0.70, negatives pushed ≀ 0.30), anchored to the random-projection prior for generalization.

Plain MinHash treats every shingle equally; the learned W weights content shingles over boilerplate, so two unrelated pages that share a navigation block stay far in Hamming space while a paraphrased duplicate stays close. Candidate generation is banding (8 bands Γ— 16 bits), confirmation is full 128-bit Hamming distance (threshold 7 on pipeline-scale rows).

Verify: distilled verdicts from the same bits. Ridge heads are fit on the frozen quantized model:

  • quality_bits β€” linear head over the Β±1 bit vector β†’ 0-100 verdict; this is the O(1) triage you can read from a stored hash without the text (v1 compressed this to 2 bits; v2 uses all 128);
  • quality_proj β€” head over the pre-sign projection β†’ finer verdict;
  • junk_bits / junk_proj β€” junk/boilerplate flag with stored operating thresholds.

The teacher's quality signal also shaped W itself during training: a straight-through quality head runs alongside the contrastive objective, so the bit planes carry quality structure, not just duplicate geometry.

The teacher: DeepSeek webapp, batched, ephemeral

Protocol (full verbatim prompts in training/teacher_protocol.md):

  1. fresh chat β†’ 6 corpus rows (same language, same domain; 20 batches carry seeded paraphrase injections) β†’ structured JSON: {"rows":[{i, quality 0-100, knowledge 0-100, flag}], "dup_pairs":[[i,j]]} β†’ chat deleted β†’ jittered sleep β†’ next chat.
  2. 9 batches re-labeled in fresh chats for self-consistency: mean |Ξ”quality| 7.0, median 4.0 (the noise ceiling the student could ever reach).
  3. Ambiguous-band pairs (real shingle overlap β€” the false-merge zone) adjudicated pair-by-pair, 6 pairs per chat, chats deleted after.
  4. Teacher dup-pair recall measured against known ground truth: 40/40 injected pairs found (100%); 3 organic corpus dups discovered on top.

Every completion rides the webapp's own proof-of-work (DeepSeekHashV1 solved by DeepSeek's shipped pow.wasm), so the protocol is account-bound, rate-limited (jittered 1.5-4.5 s), and leaves no chat history behind.

Evaluation

Corpus: KIP shard_v25, 794 labeled rows, 15 languages; batch-level 80/20 split (no batch crosses), held-out synthetic paraphrases, teacher-adjudicated eval pairs never seen in training.

Verify (157 held-out rows; junk n=12)

verifier Spearman vs teacher junk AUC MAE (0-100)
v1 ensemble (GLM-distilled, cross-teacher) 0.298 0.689 β€”
v2 hash-only head (from 128 stored bits) 0.424 0.755 11.2
v2 projection head (at signing time) 0.543 0.701 11.0

v2 beats v1 on every metric under the new frontier teacher β€” the honest cross-teacher comparison, since v1 was distilled against GLM.

Dedup β€” easy band (30 positives / 90 negatives)

engine best F1 P R
v2 learned bits 0.983 1.00 0.967
random-projection SimHash 0.983 1.00 0.967
plain MinHash-128 1.000 1.00 1.00

Everyone solves the easy band. The discrimination lives below.

Dedup β€” the false-merge zone (14 teacher-adjudicated pairs, MinHash sim 0.07-0.69)

engine not-dups wrongly merged (of 13) true dups caught (of 1)
v2 learned bits (threshold 7) 0 1
plain MinHash-128 (its own best threshold) 2 1
IDF-weighted Jaccard (v1 verify stage) 1 1

This is the band where boilerplate and same-entity/template overlap manufactures false merges. The learned bits reject every teacher-rejected hard negative while still catching the true duplicate.

Throughput (pure stdlib, one core)

45.0 docs/s signing (features + projection + all four heads + band keys). Slower than v1's 812 docs/s because v2 computes a real 128-dim learned projection per document instead of sampling 128 shingles β€” the price of trainable bits. A numpy-vectorized bulk path exists for large runs; the stdlib path needs nothing but Python.

Usage

Standalone (this repo, nothing to install):

from kip_hash import KipHashV2

vh = KipHashV2("kip-hash-v2.json")
sig = vh.sign_text("...row text...")
sig["quality"]      # 0-100 distilled verdict, read from the 128 bits
sig["quality_p"]    # finer verdict from the projection
sig["junk_prob"]    # hash-only junk/boilerplate flag
sig["hex"]          # 128-bit signature, 32 hex chars

vh.hamming(sig["bits"], sig2["bits"])          # <= 7 -> near-duplicate
kept, dups, report = vh.dedup_stream(texts)    # one-pass fuzzy dedup

Command line: python3 kip_hash.py runs a 5-text demo + selftest (near-dup pair vs different-topic vs same-topic-only, quality ranking, one-pass dedup report).

Inside the KIP pipeline: the artifact is consumed through kip.fullscale.learned_hash (v2 hook) or directly via this module; dedup_stream reuses the survivor policy of the exact-dedup stage, and quality/junk_prob join the quality spine as features.

Files

file what
kip-hash-v2.json the artifact: int8 W (32768Γ—128), 4 heads, config, teacher/provenance/eval blocks (5.9 MB)
kip_hash.py pure-stdlib inference (features, projection, heads, hamming dedup, selftest)
eval/report.json every number in this card, machine-readable
training/teacher_protocol.md the verbatim DeepSeek webapp protocol + prompts
training/teacher_labels_deepseek.jsonl 844 teacher row verdicts
training/teacher_dups.jsonl per-chat dup verdicts + injection ground truth
training/teacher_pairs_adjudicated.jsonl 58 ambiguous-band pair verdicts
training/batches_manifest.jsonl batch metadata (no corpus text)
training/train_v2.py the deterministic training script

Limitations (honest)

  • Evaluation scale. 157 verify rows, 30/90 dedup pairs, 14 adjudicated hard pairs β€” directional results, re-measure at 100k+ scale before release-gating on them.
  • Teacher noise ceiling. The teacher itself reproduces within Β±7 points (median 4); student metrics are bounded by that.
  • OOD text. Heads are calibrated on ~1600-char pipeline rows; short standalone texts saturate (same caveat as v1), and the corpus contained no spam-flagged rows, so the junk head knows boilerplate/incoherent/junk but not classic web spam.
  • Thresholds are corpus-scale. Hamming threshold 7 is selected on pipeline-scale rows; short texts need a looser threshold (~25-30).
  • Corpus-shaped bits. W is trained on a wiki-heavy multilingual sample; other domains should retrain with the shipped script and their own teacher labels (the protocol is fully documented).
  • Lexical, not semantic. Paraphrase-level recall is bounded by n-gram overlap; this is not an embedding space, and cross-language translation dups are intentionally NOT merged.
  • Throughput. 45 docs/s stdlib vs v1's 812 β€” trainable bits cost compute; use the numpy bulk path for >1M-row runs.
  • The DeepSeek teacher inherits that model's biases; prompts, labels, verdicts and adjudications all ship for auditability.

Provenance

  • Built 2026-11-18/19 by the KIP pipeline ($0 compute; teacher channel = the user's own DeepSeek webapp session, ephemeral chats).
  • Corpus: KIP shard_v25 β€” wikipedia dumps, wikidata, wiktionary, eur-lex, openstax, RFCs (permissive sources only).
  • Artifact policy: weights + metadata only, no corpus text.
  • License: Apache-2.0.

Citations

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Papers for IO-AIR/kip-hash-v2