|
Download README.md from IO-AIR/kip-hash: direct link, hf CLI and curl.
- Browser
- Download file 5.99 kB
-
https://huggingface.co/IO-AIR/kip-hash/resolve/main/README.md
- Command line
-
hf download hf://IO-AIR/kip-hash/README.md
-
curl -L -o README.md https://huggingface.co/IO-AIR/kip-hash/resolve/main/README.md
5.99 kB
| license: mit | |
| pretty_name: kip-hash | |
| tags: | |
| - dedup | |
| - near-duplicate-detection | |
| - data-curation | |
| - quality-filter | |
| - minhash | |
| - simhash | |
| - nano | |
| - distillation | |
| # kip-hash | |
| **The nano verify-and-dedup word — one 28-bit integer per row, two jobs, | |
| pure-stdlib inference. MIT licensed.** | |
| word = (bucket << 4) | score4 # 24 learned bucket bits | 4 score bits | |
| 1. **DEDUP (fuzzy by design).** The 24 bucket bits are six learned scalar | |
| projections z_k = (w_k . x - mu_k) / sigma_k of a hashed feature view | |
| (word uni/bi-grams + char 4-grams, log-tf, L2), each quantized with | |
| equal-mass bin edges learned from the training corpus. Near-duplicates | |
| differ by less than a bin width, so quantization *absorbs* the | |
| perturbation instead of betting bit-by-bit (SimHash) or | |
| shingle-by-shingle (MinHash): dups collide, and a +-1-bin probe | |
| recovers pairs that straddle an edge. A cheap per-dimension |dz| | |
| confirmation restores precision without ever re-hashing the text. | |
| 2. **VERIFY (packed, not appended).** A linear head over the six learned | |
| dimensions plus a 16-feature surface battery distills the teacher's | |
| 0-100 verdict; the score is quantized to 4 bits (16 equal-mass levels) | |
| and packed into the low bits of the same word. Reading a stored word | |
| triages a row at O(1) with no text re-read. | |
| 3.5 bytes per row (28-bit word), 610 KB artifact | |
| (W(32768 x 10) int8 + heads + quantizers), ~220 docs/s signing on one | |
| core, pure Python stdlib. | |
| ## Measured (held-out batches, 946 docs / 157 dup pairs / 9,367 negatives) | |
| All dedup numbers are at the **shipped operating point** (the F1-optimal | |
| tau is baked into the artifact; metrics measured there, not at a looser | |
| calibration tau): | |
| | metric | value | | |
| | --- | --- | | |
| | dup confirmed recall | **0.8153** | | |
| | pre-guard false-merge rate | 0.00149 | | |
| | **false merges after lexical guard** | **0** | | |
| | guard true retention / false rejection | 1.0 / 1.0 | | |
| | score4 decoded MAE (0-100) | 12.11, from the word alone | | |
| | junk AUC | 0.6631 | | |
| | throughput | 210 docs/s, one core, stdlib | | |
| Gate drops read straight off the word (`gate_eval` in the report has | |
| junk precision/recall per threshold). The engine's precision contract is | |
| owned by the **lexical 5-gram guard**: confirmed candidates with | |
| J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded | |
| for teacher adjudication instead of silently merged. | |
| **Compact mode** (`with_compact()`): the same learned projection | |
| re-banded to 18-bit buckets -> a 22-bit word. Measured on the same eval: | |
| 0.9172 fuzzy / 0.8790 confirmed recall, higher candidate rate (4.19%) — | |
| the recall-forward mode for <= 1M-row corpora. | |
| **Probe geometry knob** (`config.fuzzy_radii`): the trainer sweeps | |
| asymmetric per-dim probe radii and ships the F1-optimal spec (measured | |
| here: widening to +-2 on the two widest dims raises recall +2.5 pts but | |
| leaks pre-guard false merges 26 -> 44 on same-batch hard negatives; | |
| post-guard FM is 0 across all specs). Operators under recall pressure | |
| can widen `fuzzy_radii` without code changes. | |
| ## Engine | |
| - **Collision policy** — on merge, the cluster survivor is the row with | |
| the highest (score4, quality_fine, length, -arrival); O(1) off the word. | |
| - **Quality gate** — rows with score4 < gate never enter the bucket index. | |
| - **`legacy=True`** reproduces the pre-guard mechanics for comparability. | |
| - CLI selftest: `python3 kip_hash.py` (demo + contracts + per-dim radii | |
| geometry). | |
| ## Training | |
| Distilled from the teacher roster on **4,839 labeled rows** (794 + 1,913 | |
| + 2,132) across **44+ languages** and 10 domains: | |
| | teacher | channel | role | | |
| | --- | --- | --- | | |
| | DeepSeek-V4.1-Flash-1017 | DeepSeek webapp (channel 1) | primary; human-rhythm multi-turn batches, quality anchor | | |
| | kimi-k3.1-pro | NVIDIA NIM (channel 2) | bulk labeling workhorse, 64-row bundles | | |
| | MiniMax M3.1 Flash | NVIDIA NIM | designated fallback | | |
| Supervision: teacher-adjudicated dup pairs (injected + organic), | |
| same-chat hard negatives, ambiguous-band adjudications, | |
| construction-known-truth synthetic paraphrase positives, and | |
| safe-similarity synthetic negatives (48,106 train / 6,524 eval pairs). | |
| Batch-level 80/20 split (no batch crosses); deterministic trainer | |
| (`scripts/train_kip_hash.py`, SEED=20261119). Weights + metadata only — | |
| no corpus text ships in the artifact. | |
| Offline refine experiments (probe radii, language-balanced quality head | |
| — measured and **rejected**, it hurts: Spearman 0.103 vs 0.179) live in | |
| `research/kip_hash_refine_experiments.json`; version lineage in | |
| `research/kip_hash_lineage/`. | |
| ## Usage | |
| ```python | |
| from kip_hash import KipHash | |
| vh = KipHash("kip-hash.json") | |
| w = vh.sign_text("...row text...") | |
| w["word"] # 28-bit int: 24 bucket bits | 4 score bits | |
| w["score4"] # 0-15 quality level | |
| w["quality"] # decoded 0-100 verdict for that level | |
| w["junk_prob"] # hash-side junk/boilerplate flag | |
| keep, dups, rep = vh.dedup_stream(texts, gate=3) | |
| vhc = vh.with_compact() # 22-bit word view | |
| ``` | |
| Pipeline hooks: `kip.fullscale.learned_hash.load_kip_hash` / | |
| `nano_dedup` / `nano_triage` / `lexical_veto`. | |
| ## Production record | |
| First production duty (W48 dataset build, 2026-11-20, | |
| [IO-AIR/kip-dataset](https://huggingface.co/datasets/IO-AIR/kip-dataset)): | |
| on 12,047 quality-gated rows across 28 languages, kip-hash dropped | |
| 1,637 duplicates and 3,298 low-quality rows off the same words, one | |
| pass, no text re-reads. A/B against the retired MinHash-LSH stage on | |
| the identical 8,749-row post-gate universe: kip-hash's drops are a | |
| strict superset minus 10 false merges the lexical guard rejected | |
| (only-kip-hash = 0; all 10 MinHash-only merges measure script-aware | |
| J5 < 0.25). 67 short-row/template merges were vetoed and resurrected | |
| by the pipeline's script-aware lexical policy | |
| (`kip.fullscale.learned_hash.lexical_veto` — pipeline-side; the engine | |
| stays frozen, the short-row caveat above is the documented reason). | |
| ## License | |
| MIT — see [LICENSE](LICENSE) (and the repo-root LICENSE). | |