Eval requests: G4 E4B/MoE ARA vs SOMA
Hey, DontPlanToEnd, lad. I have risen from the dead to nag you with the following models. As you are aware, I have a thing for the tiny E4B multimodal model and comparing, contrasting different ablation methods. They were cooked basically on the same set of markers and score about the same on the PIQA benchmark. However, UGI should shed a brighter light for drawing conclusions.
https://huggingface.co/MuXodious/gemma-4-E4B-it-ARA-heresy
https://huggingface.co/MuXodious/gemma-4-E4B-it-SOMPOA-heresy
Highly optional at this time, as I'm yet to and unsure when to cook its comparate I had it done.:
https://huggingface.co/MuXodious/gemma-4-26B-A4B-it-ARA-heresy
https://huggingface.co/MuXodious/gemma-4-26B-A4B-it-SOMPOA-heresy
MuXodious/gemma-4-E4B-it-ARA-heresy and MuXodious/gemma-4-E4B-it-SOMPOA-heresy are giving me errors like this for some reason:
"""
ValueError: Following weights were not initialized from checkpoint: {'language_model.model.layers.33.self_attn.k_norm.weight', 'language_model.model.layers.31.self_attn.k_norm.weight', 'language_model.model.layers.38.self_attn.k_norm.weight'
...
"""
Added the 26Bs
Thank you for your time and effort, as well as for bringing this issue up with me.
Layers after lang. layer 23 do not have self_attn.k_norm/k_proj/v_norm weighs. It is the case in both versions. I have also checked other Heretic models, and they are a mixed bag. Some do have those weights, while in some, those weights are also missing. This also explains why the model is roughly 100MB shorter than the base. Surprisingly, all works in GGUF, making zero sense to me how as conversion or interfacing should have thrown an error somewhere, and I'm using the SOMPOA version as my daily driver. 🤦♂️
This shouldn't happen and needs to be looked thoroughly after. I cannot do a more in depth analysis at this time. I'll open another request once I get it sorted.
I tried 26B and wow, it's a lot different than the default... I seem to get much more variety in swipes? Will you ever release the 31B model? Thank you tons ❤️🌈