Reason for choosing DSV3-style MTP head?
I am curious of why DeepSeek-V3 style single layer MTP head was selected among other options such as DFlash, DFly etc.
Good question — the short answer is that they're solving different problems, so they weren't really competing options for us.
We adopted the DeepSeek-V3 style MTP module primarily as a training-time objective, not as a speculative decoding drafter. It densifies the training signal per token and encourages the representations to plan ahead, and DS-V3 reported gains on the main model's quality from it — that's why it's in the pretraining stack at all. The fact that it can be repurposed as a draft head at inference is a bonus, not the design goal.
DFlash, DFly and similar drafters are trained after the target model is finished, on top of frozen weights. They're orthogonal to pretraining and can be added to any released checkpoint, including this one — so nothing here forecloses them. Since this is a base model release, we didn't do any inference-side drafter training or benchmarking, and we'd be happy to see the community train those on top of it.
On depth-1 specifically: we followed DS-V3's choice of a single module to keep the extra compute and memory overhead on the pretraining loop small; deeper MTP stacks cost more per step for diminishing returns at this scale.