| --- |
| license: apache-2.0 |
| base_model: tiiuae/Falcon-Perception |
| tags: [webgpu, wgsl, int8, browser, object-detection, open-vocabulary, quantized] |
| --- |
| |
| # falcon-perception-webgpu — FPQ4 (q8 + q4-lm) weights |
|
|
| Custom mixed-precision quants of |
| [tiiuae/Falcon-Perception](https://huggingface.co/tiiuae/Falcon-Perception) |
| (0.6B open-vocabulary detection VLM) in the FPQ4 wire format of the |
| [falcon-perception-webgpu-demo](https://huggingface.co/spaces/borkiss/falcon-perception-webgpu-demo) |
| browser runtime — hand-written WGSL kernels, no ONNX Runtime for the language |
| stack, GPU-resident decode loop with speculative decoding for video. |
|
|
| **`falcon-q8-lm4.bin`** (724 MB, single file, everything the engine needs): |
|
|
| - all 113 layer matrices **int8** (group 128) — box quality demands it: any |
| single q4 group's noise flips coordinate bins (q4 box-set IoU vs fp32 is |
| only 0.43; q8 reaches hidden cosine 0.9993); |
| - `lm_head` **int4** — bit-exact argmax vs q8, −34 MB and −half the lm-read |
| traffic per decode token; |
| - coord/size decode heads **f16**, Fourier coord/size encoders f32 — |
| 0/14 bin mismatches vs the ONNX reference; |
| - embeddings f16. |
|
|
| Numbers on an Apple M4 (10-core GPU, ~120 GB/s) in Chrome: |
|
|
| | | this runtime | original Space (fp32 ONNX/WASM) | |
| |---|---|---| |
| | full detection query | **1.2 s** | 7.8–8.7 s | |
| | decode | 91–95 tok/s | ~8.7 tok/s | |
| | download | 724 MB | 2.4 GB | |
| | video keyframes | 2.2–2.5 Hz (≤3 objects) + 30 FPS correlation tracker | — | |
|
|
| Quality vs the fp32 original on identical pixels: 8/8 boxes matched, |
| mean IoU 0.957. The runtime itself is verified bit-exact per layer (75/75) |
| against a numpy dequantized-weights reference, decode token-for-token. |
|
|
| ## FPQ4 container |
|
|
| `'FPQ4'` magic + u32 manifest length + JSON manifest + 256-byte-aligned blobs, |
| absolute offsets — one `fetch`, sliced straight into GPU buffers: |
|
|
| ```js |
| const head = new DataView(await file.slice(0, 8).arrayBuffer()); |
| const mlen = head.getUint32(4, true); |
| const manifest = JSON.parse(new TextDecoder().decode( |
| await file.slice(8, 8 + mlen).arrayBuffer())); |
| // per tensor t: file.slice(t.packed.offset, t.packed.offset + t.packed.length) → writeBuffer |
| ``` |
|
|
| q8 matrix ≈ 1.02 bytes/param (packed + f16 scales + i8 zeros, group 128); |
| q4 ≈ 0.52 bytes/param. Only the mask heads (`anyup`/`segm_head`) stay in ONNX |
| on the demo's segmentation path; detection runs 100% on WGSL. |
|
|
| Try it: **https://huggingface.co/spaces/borkiss/falcon-perception-webgpu-demo** |
| (needs WebGPU with `shader-f16` + `subgroups` — Chrome/Edge 125+). |
|
|
