kip-hash / README.md
IO-AIR's picture
Model card: first production record (W48 dataset build)
081aceb verified
|
Raw History Blame Contribute Delete
5.99 kB
---
license: mit
pretty_name: kip-hash
tags:
- dedup
- near-duplicate-detection
- data-curation
- quality-filter
- minhash
- simhash
- nano
- distillation
---
# kip-hash
**The nano verify-and-dedup word — one 28-bit integer per row, two jobs,
pure-stdlib inference. MIT licensed.**
word = (bucket << 4) | score4 # 24 learned bucket bits | 4 score bits
1. **DEDUP (fuzzy by design).** The 24 bucket bits are six learned scalar
projections z_k = (w_k . x - mu_k) / sigma_k of a hashed feature view
(word uni/bi-grams + char 4-grams, log-tf, L2), each quantized with
equal-mass bin edges learned from the training corpus. Near-duplicates
differ by less than a bin width, so quantization *absorbs* the
perturbation instead of betting bit-by-bit (SimHash) or
shingle-by-shingle (MinHash): dups collide, and a +-1-bin probe
recovers pairs that straddle an edge. A cheap per-dimension |dz|
confirmation restores precision without ever re-hashing the text.
2. **VERIFY (packed, not appended).** A linear head over the six learned
dimensions plus a 16-feature surface battery distills the teacher's
0-100 verdict; the score is quantized to 4 bits (16 equal-mass levels)
and packed into the low bits of the same word. Reading a stored word
triages a row at O(1) with no text re-read.
3.5 bytes per row (28-bit word), 610 KB artifact
(W(32768 x 10) int8 + heads + quantizers), ~220 docs/s signing on one
core, pure Python stdlib.
## Measured (held-out batches, 946 docs / 157 dup pairs / 9,367 negatives)
All dedup numbers are at the **shipped operating point** (the F1-optimal
tau is baked into the artifact; metrics measured there, not at a looser
calibration tau):
| metric | value |
| --- | --- |
| dup confirmed recall | **0.8153** |
| pre-guard false-merge rate | 0.00149 |
| **false merges after lexical guard** | **0** |
| guard true retention / false rejection | 1.0 / 1.0 |
| score4 decoded MAE (0-100) | 12.11, from the word alone |
| junk AUC | 0.6631 |
| throughput | 210 docs/s, one core, stdlib |
Gate drops read straight off the word (`gate_eval` in the report has
junk precision/recall per threshold). The engine's precision contract is
owned by the **lexical 5-gram guard**: confirmed candidates with
J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded
for teacher adjudication instead of silently merged.
**Compact mode** (`with_compact()`): the same learned projection
re-banded to 18-bit buckets -> a 22-bit word. Measured on the same eval:
0.9172 fuzzy / 0.8790 confirmed recall, higher candidate rate (4.19%) —
the recall-forward mode for <= 1M-row corpora.
**Probe geometry knob** (`config.fuzzy_radii`): the trainer sweeps
asymmetric per-dim probe radii and ships the F1-optimal spec (measured
here: widening to +-2 on the two widest dims raises recall +2.5 pts but
leaks pre-guard false merges 26 -> 44 on same-batch hard negatives;
post-guard FM is 0 across all specs). Operators under recall pressure
can widen `fuzzy_radii` without code changes.
## Engine
- **Collision policy** — on merge, the cluster survivor is the row with
the highest (score4, quality_fine, length, -arrival); O(1) off the word.
- **Quality gate** — rows with score4 < gate never enter the bucket index.
- **`legacy=True`** reproduces the pre-guard mechanics for comparability.
- CLI selftest: `python3 kip_hash.py` (demo + contracts + per-dim radii
geometry).
## Training
Distilled from the teacher roster on **4,839 labeled rows** (794 + 1,913
+ 2,132) across **44+ languages** and 10 domains:
| teacher | channel | role |
| --- | --- | --- |
| DeepSeek-V4.1-Flash-1017 | DeepSeek webapp (channel 1) | primary; human-rhythm multi-turn batches, quality anchor |
| kimi-k3.1-pro | NVIDIA NIM (channel 2) | bulk labeling workhorse, 64-row bundles |
| MiniMax M3.1 Flash | NVIDIA NIM | designated fallback |
Supervision: teacher-adjudicated dup pairs (injected + organic),
same-chat hard negatives, ambiguous-band adjudications,
construction-known-truth synthetic paraphrase positives, and
safe-similarity synthetic negatives (48,106 train / 6,524 eval pairs).
Batch-level 80/20 split (no batch crosses); deterministic trainer
(`scripts/train_kip_hash.py`, SEED=20261119). Weights + metadata only —
no corpus text ships in the artifact.
Offline refine experiments (probe radii, language-balanced quality head
— measured and **rejected**, it hurts: Spearman 0.103 vs 0.179) live in
`research/kip_hash_refine_experiments.json`; version lineage in
`research/kip_hash_lineage/`.
## Usage
```python
from kip_hash import KipHash
vh = KipHash("kip-hash.json")
w = vh.sign_text("...row text...")
w["word"] # 28-bit int: 24 bucket bits | 4 score bits
w["score4"] # 0-15 quality level
w["quality"] # decoded 0-100 verdict for that level
w["junk_prob"] # hash-side junk/boilerplate flag
keep, dups, rep = vh.dedup_stream(texts, gate=3)
vhc = vh.with_compact() # 22-bit word view
```
Pipeline hooks: `kip.fullscale.learned_hash.load_kip_hash` /
`nano_dedup` / `nano_triage` / `lexical_veto`.
## Production record
First production duty (W48 dataset build, 2026-11-20,
[IO-AIR/kip-dataset](https://huggingface.co/datasets/IO-AIR/kip-dataset)):
on 12,047 quality-gated rows across 28 languages, kip-hash dropped
1,637 duplicates and 3,298 low-quality rows off the same words, one
pass, no text re-reads. A/B against the retired MinHash-LSH stage on
the identical 8,749-row post-gate universe: kip-hash's drops are a
strict superset minus 10 false merges the lexical guard rejected
(only-kip-hash = 0; all 10 MinHash-only merges measure script-aware
J5 < 0.25). 67 short-row/template merges were vetoed and resurrected
by the pipeline's script-aware lexical policy
(`kip.fullscale.learned_hash.lexical_veto` — pipeline-side; the engine
stays frozen, the short-row caveat above is the documented reason).
## License
MIT — see [LICENSE](LICENSE) (and the repo-root LICENSE).