Quantize and tie q4f16 input embedding
#1
by nico-martin HF Staff - opened
Quantizes the q4f16 input embedding with GatherBlockQuantized and lets its differently shaped initializer share the LM-head packed external byte range. This reduces external data from 2.058 GB to 1.534 GB (decimal) without introducing duplicate tied-weight storage.
Validation:
- shared-range graph is bit-exact to the equivalent duplicate-storage graph for prefill logits, cached logits, and all cache tensors
- ONNX Runtime CPU session creation and Transformers.js WebGPU loading/generation pass
- embedding quantization matched 11/12 varied 32-token greedy generations versus the FP16-embedding graph; the README documents the quantized embedding
- direct WebGPU benchmark improved from 102.16 to 106.05 tokens/s in the local test environment
mlabonne changed pull request status to merged