epic findings on this model

#8
by inikishev - opened

All non expert parameters are identical to base qwen3.8 flash so they were not trained.

Experts have less params - qwen experts are 2560Γ—640 and Edge0 experts are 1792Γ—640. To get the output back to 2560-dimensional space, there are two additional up and down 2560Γ—1792 projections in each layer and they are shared between experts. Presumably the experts were retrained or post trained which would be required to explain the benchmarks in the readme.

The information above was produced by qwen3.8 flash next but it might still be true

Edge0 org

Your reading is correct, and none of it was hidden β€” the card names Qwen3.8-Flash-Next as base_model.

Yes, the non-expert tensors are the base model's. We did not retrain them. It is not that we could not: changing weights is cheap, a few hundred SFT samples would move them. We left them alone because there was no need to.

What we did is the architecture. We replaced the original large experts with a latent MoE, so the experts run in a smaller latent space and each expert carries far fewer parameters. The D/U projections you spotted are new β€” they exist only because of that change, and they are trained along with the experts.

That is the point for us: the expert pool is what has to fit in VRAM and RAM, and shrinking it is what makes a model this size usable on low-resource edge devices.

Edge0 org

Thank you for your epic findings which help clarify our efforts to bring LLM to consumer desktop.

wanglamao changed discussion status to closed

Sign up or log in to comment