Running experiment: Expert only ternary preserves routing on Grok-1 block-0
#85
by rmems - opened
Measure on real Grok-1 weights (full block-0), 98% of the block's parameters that experts can be ternarized without router damage:
- Router top-1 & top-2 agreement = 1.000000, residential drift = 0.000000
- What breaks routing is the attention ternarization (~55% top-1)
Full measurement + provenance:
https://github.com/rmems/grok-ozempic/blob/main/reports/grok-1-full-block-forward/results.md