Qwen3.5-4B PLTs
Per-layer PLT checkpoints for Qwen/Qwen3.5-4B trained as same-layer MLP transcoders.
Contents
transcoder_L0.pttranscoder_L2.pt...transcoder_L31.ptlayer_metrics.json
Layer L1 is currently missing because that checkpoint was not preserved in the training artifacts.
Training recipe
- Base model:
Qwen/Qwen3.5-4B - Objective: same-layer MLP input to same-layer MLP output
- Feature width:
16384 - Activation regime:
ReLU + L1with ghost gradients and dead-feature resampling - Training corpus:
20kimage-text subset built from CC3M and CC12M
Intended use
These checkpoints are intended for research on sparse feature discovery, circuit analysis, and PLT-based steering experiments in Qwen3.5-4B.
Notes
Checkpoint quality is not uniform across layers. Later layers generally behaved better than early layers in our experiments.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support