---
license: apache-2.0
library_name: kernels
---
# OrbitQuant packed GEMV

Bias-free W4A4 integer-surrogate matrix-vector evaluation for 1-8 activation rows. Reads packed nibbles directly, accumulates exact INT32 dot products, and applies the existing OrbitQuant token/row norms and surrogate scales. Output is BF16 or FP16.

The selected YuE2 dispatch uses it only for bias-free row-major packed weights. Larger batches and other layouts retain the original OrbitQuant implementation. The build targets SM120; other architectures require a rebuild and validation.

On RTX 5090, CUDA-graph layer timings improved from 0.1355 to 0.0102 ms (2048x2048) and from 0.4617 to 0.0144 ms (2048x6144), including activation preparation. BF16 reference timings were 0.0083 and 0.0105 ms: these isolated small matrices are still slightly slower than BF16. Whole-model grouped projections are measured separately.

Tests compare exact results without bias to the existing packed native kernel and exercise CUDA graph capture and invalid weight dimensions. Do not infer musical quality or whole-model speed from the microbenchmarks.

Build with kernel-builder-generated project files: `python setup.py build_kernel`. Python ABI3 and manylinux_2_34 checks pass for the supplied binary; manylinux_2_28 does not, due to a GLIBC_2.32 symbol from the Ubuntu build environment.