--- license: mit pretty_name: kip-hash tags: - dedup - near-duplicate-detection - data-curation - quality-filter - minhash - simhash - nano - distillation --- # kip-hash **The nano verify-and-dedup word — one 28-bit integer per row, two jobs, pure-stdlib inference. MIT licensed.** word = (bucket << 4) | score4 # 24 learned bucket bits | 4 score bits 1. **DEDUP (fuzzy by design).** The 24 bucket bits are six learned scalar projections z_k = (w_k . x - mu_k) / sigma_k of a hashed feature view (word uni/bi-grams + char 4-grams, log-tf, L2), each quantized with equal-mass bin edges learned from the training corpus. Near-duplicates differ by less than a bin width, so quantization *absorbs* the perturbation instead of betting bit-by-bit (SimHash) or shingle-by-shingle (MinHash): dups collide, and a +-1-bin probe recovers pairs that straddle an edge. A cheap per-dimension |dz| confirmation restores precision without ever re-hashing the text. 2. **VERIFY (packed, not appended).** A linear head over the six learned dimensions plus a 16-feature surface battery distills the teacher's 0-100 verdict; the score is quantized to 4 bits (16 equal-mass levels) and packed into the low bits of the same word. Reading a stored word triages a row at O(1) with no text re-read. 3.5 bytes per row (28-bit word), 610 KB artifact (W(32768 x 10) int8 + heads + quantizers), ~220 docs/s signing on one core, pure Python stdlib. ## Measured (held-out batches, 946 docs / 157 dup pairs / 9,367 negatives) All dedup numbers are at the **shipped operating point** (the F1-optimal tau is baked into the artifact; metrics measured there, not at a looser calibration tau): | metric | value | | --- | --- | | dup confirmed recall | **0.8153** | | pre-guard false-merge rate | 0.00149 | | **false merges after lexical guard** | **0** | | guard true retention / false rejection | 1.0 / 1.0 | | score4 decoded MAE (0-100) | 12.11, from the word alone | | junk AUC | 0.6631 | | throughput | 210 docs/s, one core, stdlib | Gate drops read straight off the word (`gate_eval` in the report has junk precision/recall per threshold). The engine's precision contract is owned by the **lexical 5-gram guard**: confirmed candidates with J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded for teacher adjudication instead of silently merged. **Compact mode** (`with_compact()`): the same learned projection re-banded to 18-bit buckets -> a 22-bit word. Measured on the same eval: 0.9172 fuzzy / 0.8790 confirmed recall, higher candidate rate (4.19%) — the recall-forward mode for <= 1M-row corpora. **Probe geometry knob** (`config.fuzzy_radii`): the trainer sweeps asymmetric per-dim probe radii and ships the F1-optimal spec (measured here: widening to +-2 on the two widest dims raises recall +2.5 pts but leaks pre-guard false merges 26 -> 44 on same-batch hard negatives; post-guard FM is 0 across all specs). Operators under recall pressure can widen `fuzzy_radii` without code changes. ## Engine - **Collision policy** — on merge, the cluster survivor is the row with the highest (score4, quality_fine, length, -arrival); O(1) off the word. - **Quality gate** — rows with score4 < gate never enter the bucket index. - **`legacy=True`** reproduces the pre-guard mechanics for comparability. - CLI selftest: `python3 kip_hash.py` (demo + contracts + per-dim radii geometry). ## Training Distilled from the teacher roster on **4,839 labeled rows** (794 + 1,913 + 2,132) across **44+ languages** and 10 domains: | teacher | channel | role | | --- | --- | --- | | DeepSeek-V4.1-Flash-1017 | DeepSeek webapp (channel 1) | primary; human-rhythm multi-turn batches, quality anchor | | kimi-k3.1-pro | NVIDIA NIM (channel 2) | bulk labeling workhorse, 64-row bundles | | MiniMax M3.1 Flash | NVIDIA NIM | designated fallback | Supervision: teacher-adjudicated dup pairs (injected + organic), same-chat hard negatives, ambiguous-band adjudications, construction-known-truth synthetic paraphrase positives, and safe-similarity synthetic negatives (48,106 train / 6,524 eval pairs). Batch-level 80/20 split (no batch crosses); deterministic trainer (`scripts/train_kip_hash.py`, SEED=20261119). Weights + metadata only — no corpus text ships in the artifact. Offline refine experiments (probe radii, language-balanced quality head — measured and **rejected**, it hurts: Spearman 0.103 vs 0.179) live in `research/kip_hash_refine_experiments.json`; version lineage in `research/kip_hash_lineage/`. ## Usage ```python from kip_hash import KipHash vh = KipHash("kip-hash.json") w = vh.sign_text("...row text...") w["word"] # 28-bit int: 24 bucket bits | 4 score bits w["score4"] # 0-15 quality level w["quality"] # decoded 0-100 verdict for that level w["junk_prob"] # hash-side junk/boilerplate flag keep, dups, rep = vh.dedup_stream(texts, gate=3) vhc = vh.with_compact() # 22-bit word view ``` Pipeline hooks: `kip.fullscale.learned_hash.load_kip_hash` / `nano_dedup` / `nano_triage` / `lexical_veto`. ## Production record First production duty (W48 dataset build, 2026-11-20, [IO-AIR/kip-dataset](https://huggingface.co/datasets/IO-AIR/kip-dataset)): on 12,047 quality-gated rows across 28 languages, kip-hash dropped 1,637 duplicates and 3,298 low-quality rows off the same words, one pass, no text re-reads. A/B against the retired MinHash-LSH stage on the identical 8,749-row post-gate universe: kip-hash's drops are a strict superset minus 10 false merges the lexical guard rejected (only-kip-hash = 0; all 10 MinHash-only merges measure script-aware J5 < 0.25). 67 short-row/template merges were vetoed and resurrected by the pipeline's script-aware lexical policy (`kip.fullscale.learned_hash.lexical_veto` — pipeline-side; the engine stays frozen, the short-row caveat above is the documented reason). ## License MIT — see [LICENSE](LICENSE) (and the repo-root LICENSE).