Download README.md from IO-AIR/kip-hash: direct link, hf CLI and curl.
- Browser
- Download file 5.99 kB
-
https://huggingface.co/IO-AIR/kip-hash/resolve/main/README.md
- Command line
-
hf download hf://IO-AIR/kip-hash/README.md
-
curl -L -o README.md https://huggingface.co/IO-AIR/kip-hash/resolve/main/README.md
license: mit
pretty_name: kip-hash
tags:
- dedup
- near-duplicate-detection
- data-curation
- quality-filter
- minhash
- simhash
- nano
- distillation
kip-hash
The nano verify-and-dedup word — one 28-bit integer per row, two jobs, pure-stdlib inference. MIT licensed.
word = (bucket << 4) | score4 # 24 learned bucket bits | 4 score bits
- DEDUP (fuzzy by design). The 24 bucket bits are six learned scalar projections z_k = (w_k . x - mu_k) / sigma_k of a hashed feature view (word uni/bi-grams + char 4-grams, log-tf, L2), each quantized with equal-mass bin edges learned from the training corpus. Near-duplicates differ by less than a bin width, so quantization absorbs the perturbation instead of betting bit-by-bit (SimHash) or shingle-by-shingle (MinHash): dups collide, and a +-1-bin probe recovers pairs that straddle an edge. A cheap per-dimension |dz| confirmation restores precision without ever re-hashing the text.
- VERIFY (packed, not appended). A linear head over the six learned dimensions plus a 16-feature surface battery distills the teacher's 0-100 verdict; the score is quantized to 4 bits (16 equal-mass levels) and packed into the low bits of the same word. Reading a stored word triages a row at O(1) with no text re-read.
3.5 bytes per row (28-bit word), 610 KB artifact (W(32768 x 10) int8 + heads + quantizers), ~220 docs/s signing on one core, pure Python stdlib.
Measured (held-out batches, 946 docs / 157 dup pairs / 9,367 negatives)
All dedup numbers are at the shipped operating point (the F1-optimal tau is baked into the artifact; metrics measured there, not at a looser calibration tau):
| metric | value |
|---|---|
| dup confirmed recall | 0.8153 |
| pre-guard false-merge rate | 0.00149 |
| false merges after lexical guard | 0 |
| guard true retention / false rejection | 1.0 / 1.0 |
| score4 decoded MAE (0-100) | 12.11, from the word alone |
| junk AUC | 0.6631 |
| throughput | 210 docs/s, one core, stdlib |
Gate drops read straight off the word (gate_eval in the report has
junk precision/recall per threshold). The engine's precision contract is
owned by the lexical 5-gram guard: confirmed candidates with
J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded
for teacher adjudication instead of silently merged.
Compact mode (with_compact()): the same learned projection
re-banded to 18-bit buckets -> a 22-bit word. Measured on the same eval:
0.9172 fuzzy / 0.8790 confirmed recall, higher candidate rate (4.19%) —
the recall-forward mode for <= 1M-row corpora.
Probe geometry knob (config.fuzzy_radii): the trainer sweeps
asymmetric per-dim probe radii and ships the F1-optimal spec (measured
here: widening to +-2 on the two widest dims raises recall +2.5 pts but
leaks pre-guard false merges 26 -> 44 on same-batch hard negatives;
post-guard FM is 0 across all specs). Operators under recall pressure
can widen fuzzy_radii without code changes.
Engine
- Collision policy — on merge, the cluster survivor is the row with the highest (score4, quality_fine, length, -arrival); O(1) off the word.
- Quality gate — rows with score4 < gate never enter the bucket index.
legacy=Truereproduces the pre-guard mechanics for comparability.- CLI selftest:
python3 kip_hash.py(demo + contracts + per-dim radii geometry).
Training
Distilled from the teacher roster on 4,839 labeled rows (794 + 1,913
- 2,132) across 44+ languages and 10 domains:
| teacher | channel | role |
|---|---|---|
| DeepSeek-V4.1-Flash-1017 | DeepSeek webapp (channel 1) | primary; human-rhythm multi-turn batches, quality anchor |
| kimi-k3.1-pro | NVIDIA NIM (channel 2) | bulk labeling workhorse, 64-row bundles |
| MiniMax M3.1 Flash | NVIDIA NIM | designated fallback |
Supervision: teacher-adjudicated dup pairs (injected + organic),
same-chat hard negatives, ambiguous-band adjudications,
construction-known-truth synthetic paraphrase positives, and
safe-similarity synthetic negatives (48,106 train / 6,524 eval pairs).
Batch-level 80/20 split (no batch crosses); deterministic trainer
(scripts/train_kip_hash.py, SEED=20261119). Weights + metadata only —
no corpus text ships in the artifact.
Offline refine experiments (probe radii, language-balanced quality head
— measured and rejected, it hurts: Spearman 0.103 vs 0.179) live in
research/kip_hash_refine_experiments.json; version lineage in
research/kip_hash_lineage/.
Usage
from kip_hash import KipHash
vh = KipHash("kip-hash.json")
w = vh.sign_text("...row text...")
w["word"] # 28-bit int: 24 bucket bits | 4 score bits
w["score4"] # 0-15 quality level
w["quality"] # decoded 0-100 verdict for that level
w["junk_prob"] # hash-side junk/boilerplate flag
keep, dups, rep = vh.dedup_stream(texts, gate=3)
vhc = vh.with_compact() # 22-bit word view
Pipeline hooks: kip.fullscale.learned_hash.load_kip_hash /
nano_dedup / nano_triage / lexical_veto.
Production record
First production duty (W48 dataset build, 2026-11-20,
IO-AIR/kip-dataset):
on 12,047 quality-gated rows across 28 languages, kip-hash dropped
1,637 duplicates and 3,298 low-quality rows off the same words, one
pass, no text re-reads. A/B against the retired MinHash-LSH stage on
the identical 8,749-row post-gate universe: kip-hash's drops are a
strict superset minus 10 false merges the lexical guard rejected
(only-kip-hash = 0; all 10 MinHash-only merges measure script-aware
J5 < 0.25). 67 short-row/template merges were vetoed and resurrected
by the pipeline's script-aware lexical policy
(kip.fullscale.learned_hash.lexical_veto — pipeline-side; the engine
stays frozen, the short-row caveat above is the documented reason).
License
MIT — see LICENSE (and the repo-root LICENSE).