Any chance of a smaller coding model in the 30-70b range?
Howdy fine Qwen folks, congratulations on the release!
I'm wondering if you have any plans to release a Qwen 3 coder model in a size range that could be run locally / on home servers?
Something in the 30-70b range would be great, perhaps even a (40b-70b)-A(4-8)b MoE?
30B A3B Coder or A6B would be amazing!
Howdy fine Qwen folks, congratulations on the release!
I'm wondering if you have any plans to release a Qwen 3 coder model in a size range that could be run locally / on home servers?
Something in the 30-70b range would be great, perhaps even a (40b-70b)-A(4-8)b MoE?
This would be perfect!
Something in the 30-70b range would be great, perhaps even a (40b-70b)-A(4-8)b MoE
Let's not forget the enormous context window of this new full-size Coder model (256k tok. native, 1M expanded). I think anything larger than perhaps 50B (or even less?) would have problems fitting into a single A100 or H100 card (Qwen2.5 72B in 8-bit quants definitely could not fit into a single 80G A100 card with its 131K context - 8k tok. was max that fit there, so you had to spill over 30/80 of model layers to the CPU to fit the 131K window, which made inference down by an order of magnitude...).
I also think that using MoE with smaller models is not a good idea (expert model size matters, so you are better of with a dense variant until a single expert in a MoE version can match the size of the dense model).
Let's not forget the enormous context window of this new full-size Coder model .. think anything larger than perhaps 50B (or even less?) would have problems fitting into a single A100
With K/V cache quantisation this is less of an issue, you can now run up very large context windows with llama.cpp with the KV cache set to Q8_0 with practically no quality drop but at a fraction of the overheads of full fp16/32.
You can run up 30b~ models on two RTX3090s (48GB) with a 256k context easily.