ddasd/gemma4-31b-fp8block-finaltoken-sae-200m

Gemma 4 31B FP8 Block Final-Token SAE

TopK sparse autoencoder trained on vLLM token_embed final-token embeddings from RedHatAI/gemma-4-31B-it-FP8-block.

This is a final-token / embedding-surface SAE, not an internal residual-stream layer SAE.

Training Summary

• Base model: RedHatAI/gemma-4-31B-it-FP8-block
• Activation surface: vLLM token embedding / final-token embeddings
• Architecture: TopK SAE
• Input dimension: 5376
• Dictionary size: 65536
• TopK: 128
• Aux TopK: 2688
• Training tokens: 200,000,000
• Final EMA variance explained: approximately 0.875
• Final dead features: 2693 / 65536

Files

• gemma4_31b_fp8block_finaltoken_sae_final.safetensors: final SAE weights
• gemma4_31b_fp8block_finaltoken_sae_cfg.json: configuration and training metadata

The weights use keys W_enc, W_dec, b_enc, b_dec, and last_fired.