kip-hash / README.md
IO-AIR's picture
Model card: first production record (W48 dataset build)
081aceb verified
|
Raw History Blame Contribute Delete
5.99 kB
metadata
license: mit
pretty_name: kip-hash
tags:
  - dedup
  - near-duplicate-detection
  - data-curation
  - quality-filter
  - minhash
  - simhash
  - nano
  - distillation

kip-hash

The nano verify-and-dedup word — one 28-bit integer per row, two jobs, pure-stdlib inference. MIT licensed.

word = (bucket << 4) | score4        # 24 learned bucket bits | 4 score bits
  1. DEDUP (fuzzy by design). The 24 bucket bits are six learned scalar projections z_k = (w_k . x - mu_k) / sigma_k of a hashed feature view (word uni/bi-grams + char 4-grams, log-tf, L2), each quantized with equal-mass bin edges learned from the training corpus. Near-duplicates differ by less than a bin width, so quantization absorbs the perturbation instead of betting bit-by-bit (SimHash) or shingle-by-shingle (MinHash): dups collide, and a +-1-bin probe recovers pairs that straddle an edge. A cheap per-dimension |dz| confirmation restores precision without ever re-hashing the text.
  2. VERIFY (packed, not appended). A linear head over the six learned dimensions plus a 16-feature surface battery distills the teacher's 0-100 verdict; the score is quantized to 4 bits (16 equal-mass levels) and packed into the low bits of the same word. Reading a stored word triages a row at O(1) with no text re-read.

3.5 bytes per row (28-bit word), 610 KB artifact (W(32768 x 10) int8 + heads + quantizers), ~220 docs/s signing on one core, pure Python stdlib.

Measured (held-out batches, 946 docs / 157 dup pairs / 9,367 negatives)

All dedup numbers are at the shipped operating point (the F1-optimal tau is baked into the artifact; metrics measured there, not at a looser calibration tau):

metric value
dup confirmed recall 0.8153
pre-guard false-merge rate 0.00149
false merges after lexical guard 0
guard true retention / false rejection 1.0 / 1.0
score4 decoded MAE (0-100) 12.11, from the word alone
junk AUC 0.6631
throughput 210 docs/s, one core, stdlib

Gate drops read straight off the word (gate_eval in the report has junk precision/recall per threshold). The engine's precision contract is owned by the lexical 5-gram guard: confirmed candidates with J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded for teacher adjudication instead of silently merged.

Compact mode (with_compact()): the same learned projection re-banded to 18-bit buckets -> a 22-bit word. Measured on the same eval: 0.9172 fuzzy / 0.8790 confirmed recall, higher candidate rate (4.19%) — the recall-forward mode for <= 1M-row corpora.

Probe geometry knob (config.fuzzy_radii): the trainer sweeps asymmetric per-dim probe radii and ships the F1-optimal spec (measured here: widening to +-2 on the two widest dims raises recall +2.5 pts but leaks pre-guard false merges 26 -> 44 on same-batch hard negatives; post-guard FM is 0 across all specs). Operators under recall pressure can widen fuzzy_radii without code changes.

Engine

  • Collision policy — on merge, the cluster survivor is the row with the highest (score4, quality_fine, length, -arrival); O(1) off the word.
  • Quality gate — rows with score4 < gate never enter the bucket index.
  • legacy=True reproduces the pre-guard mechanics for comparability.
  • CLI selftest: python3 kip_hash.py (demo + contracts + per-dim radii geometry).

Training

Distilled from the teacher roster on 4,839 labeled rows (794 + 1,913

  • 2,132) across 44+ languages and 10 domains:
teacher channel role
DeepSeek-V4.1-Flash-1017 DeepSeek webapp (channel 1) primary; human-rhythm multi-turn batches, quality anchor
kimi-k3.1-pro NVIDIA NIM (channel 2) bulk labeling workhorse, 64-row bundles
MiniMax M3.1 Flash NVIDIA NIM designated fallback

Supervision: teacher-adjudicated dup pairs (injected + organic), same-chat hard negatives, ambiguous-band adjudications, construction-known-truth synthetic paraphrase positives, and safe-similarity synthetic negatives (48,106 train / 6,524 eval pairs). Batch-level 80/20 split (no batch crosses); deterministic trainer (scripts/train_kip_hash.py, SEED=20261119). Weights + metadata only — no corpus text ships in the artifact.

Offline refine experiments (probe radii, language-balanced quality head — measured and rejected, it hurts: Spearman 0.103 vs 0.179) live in research/kip_hash_refine_experiments.json; version lineage in research/kip_hash_lineage/.

Usage

from kip_hash import KipHash
vh = KipHash("kip-hash.json")
w = vh.sign_text("...row text...")
w["word"]          # 28-bit int: 24 bucket bits | 4 score bits
w["score4"]        # 0-15 quality level
w["quality"]       # decoded 0-100 verdict for that level
w["junk_prob"]     # hash-side junk/boilerplate flag
keep, dups, rep = vh.dedup_stream(texts, gate=3)
vhc = vh.with_compact()   # 22-bit word view

Pipeline hooks: kip.fullscale.learned_hash.load_kip_hash / nano_dedup / nano_triage / lexical_veto.

Production record

First production duty (W48 dataset build, 2026-11-20, IO-AIR/kip-dataset): on 12,047 quality-gated rows across 28 languages, kip-hash dropped 1,637 duplicates and 3,298 low-quality rows off the same words, one pass, no text re-reads. A/B against the retired MinHash-LSH stage on the identical 8,749-row post-gate universe: kip-hash's drops are a strict superset minus 10 false merges the lexical guard rejected (only-kip-hash = 0; all 10 MinHash-only merges measure script-aware J5 < 0.25). 67 short-row/template merges were vetoed and resurrected by the pipeline's script-aware lexical policy (kip.fullscale.learned_hash.lexical_veto — pipeline-side; the engine stays frozen, the short-row caveat above is the documented reason).

License

MIT — see LICENSE (and the repo-root LICENSE).