Spaces:
Running
Running
| title: Does this pack keep what it claims? | |
| emoji: 🔎 | |
| colorFrom: indigo | |
| colorTo: gray | |
| sdk: static | |
| app_file: index.html | |
| pinned: false | |
| license: apache-2.0 | |
| short_description: Check a published quantized model against its own config | |
| # Quantized pack integrity check | |
| Quantization can drop or disable part of a model without failing. The | |
| pack loads, serves, and answers correctly — while a component its card | |
| says it preserved is either absent or present-and-ignored by the | |
| runtime. Nothing errors, and nothing in the logs says so. | |
| Everything runs **in the visitor's browser**. There is no backend and | |
| no token: it reads safetensors headers and `config.json` over ranged | |
| requests, exactly what any visitor could read themselves. A 27 GB pack | |
| costs a few kilobytes to inspect. | |
| ## What it checks | |
| 1. **Does every exclusion entry name a real module?** An entry naming | |
| something the architecture does not have protects nothing. The | |
| common case is a vision tower: Llama and Pixtral call it | |
| `vision_tower`, Qwen calls it `visual`, and the documented default | |
| names the former. | |
| 2. **Did any module of the source model fail to reach the pack?** | |
| `from_pretrained` does not materialise tensors the AutoModel class | |
| has no slot for, so an auxiliary head living inside the checkpoint | |
| never reaches the quantizer and never reaches the artifact. | |
| 3. **Is anything at source precision without being declared?** A | |
| runtime builds those quantized, looks for a packed weight, finds a | |
| plain one, and skips it. | |
| The first two are observations. The third is a prediction, and says so. | |
| ## Two implementations, kept in step mechanically | |
| The rules exist twice: in `../pack_check.py` for the bulk scanner, and | |
| in `pack_check.js` here because a static Space has no Python. That is a | |
| real risk — not that the port is wrong today, but that the two drift | |
| later and only one of them is ever reviewed. | |
| So the agreement is enforced rather than trusted: | |
| ```bash | |
| python gen_convention_golden.py # Python answers the shared cases | |
| node parity.mjs # JavaScript must reproduce them | |
| ``` | |
| `conventions.cases.json` holds inputs only. The expectations are | |
| generated from Python, never typed by hand — a hand-written | |
| expectation records what someone believed on the day. Change a rule on | |
| either side, regenerate, and the port fails until it agrees. Verified | |
| that it can fail: breaking the regex anchoring (`re.match` semantics | |
| to `search`) is caught immediately. | |
| ## Why the rules are per format | |
| | format | field | how an entry matches | | |
| |---|---|---| | |
| | compressed-tensors | `ignore` | exact equality, or a `re:` regex | | |
| | awq | `modules_to_not_convert` | plain substring | | |
| | gptq | `dynamic` | ordered, first match wins; `-:` excludes | | |
| | bitsandbytes | `llm_int8_skip_modules` | exact path component, or a cumulative prefix | | |
| | modelopt | `exclude_modules` | exact, substring, or an fnmatch glob | | |
| | auto-round | `extra_config` | per-module `bits >= 16`, or a block allowlist | | |
| A bare `mtp` covers a whole head under AWQ's substring rule and covers | |
| **nothing** under compressed-tensors' exact-match rule. Every regex | |
| resolves through Python's `re.match`, which anchors at the start: | |
| `re:layers.0` does not match `mtp.layers.0.…`. | |
| These were read out of vLLM's source, not from documentation. A format | |
| that is not on the list reports **not checked** rather than OK — a | |
| checker that cannot tell "healthy" from "not checked" is not a checker. | |
| ## Limits | |
| - A clean result is not a quality measurement. This never loads the | |
| model and never runs it. A pack can pass here and still be a poor | |
| quantization. | |
| - Rules extracted from vLLM `0.23.1rc1.dev552+g4559c43a9` on | |
| 2026-08-11. They are not a stable public API upstream. | |
| - Check 3 tolerates the wrapper-level difference between the names | |
| written into an exclusion list and the names tensors are stored | |
| under, because vLLM resolves that with a per-architecture mapper this | |
| cannot replicate from metadata. It errs toward "declared" — missing | |
| some real cases rather than accusing correct packs. | |
| - Gated and private repos cannot be read. | |
| ## Deploying | |
| ```bash | |
| HF_TOKEN=... ./deploy.sh aleada/pack-integrity-check | |
| ``` | |
| Static Spaces are free; Gradio and Docker Spaces require PRO. A Gradio | |
| version of the same checker lives in [`../space/`](../space/) for if | |
| that changes — it shares `pack_check.py` directly and needs no port. | |