Hello @DavidAU ,

can you try to do the MoE versions, please? We could use something between 9B and 27B.

I was surprised about the capability of the small 9B model. While it wasn't capable of creating something as advanced as bigger models and the code it generated from scratch was usually broken, when I gave its own broken code, it was smart enough to fix individual issues. I do believe that it would have been capable to improve the quality iteratively when prompted to do so in small single tasks, just like with fixing the issues. Unfortunately, creating something little bit more ambitious from scratch with the 9B model was not possible in a single prompt.

On the other hand, the 27B model is already a bit too demanding for my hardware and token generation speed is too slow for the model to be useable for me.

The smallest MoE is 35B which is even bigger than the previous generation, but due to its Mixture of Experts architecture, it is still a bit faster than the 27B model for me and there are also some smaller REAP versions like 26B Creative and 24B regular version which are even faster.

I believe these MoE models would nicely fill up the gap between 9B and 27B dense variants in terms of quality, if you tuned them similarly with the Claude datasets.