Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
hypothetical 
posted an update 6 days ago

The most discriminating number in this release is the KL column in release-manifest.json, and it flags one repo.

Pulled all 7 manifests, 28 variants.

M and L are the useful comparison because they are the same fixed recipe everywhere: Q4_K_M and Q8_0, no ANNA or RCO search involved. So cross-repo is apples to apples there.

Every L (Q8_0) sits at heldout_kl_mean 0.00062 to 0.00161. Except gemma-4-12B, at 0.03117. That is 19x to 50x the other six, at the same qtype and the same 8.51-8.64 bpw. Its M is 0.14189 against 0.01155-0.04488 for everyone else.

The one structural difference I can see in the manifests: gemma-4-12B is the only repo that retains zero tensors at BF16. Its L is a bare Q8_0:329, where E4B's L is BF16:85 plus Q8_0:260, and every other repo keeps 36 to 85. It is also the only one with output_embedding_mode tied_alias and lm_head_policy skip_tied.

Worth adding that the benchmark you headline cannot see any of this. On IFEval instruction-strict, 4 of the 7 recommended tiers score above their own BF16 reference (Qwen3.5-2B M at +0.84pp, E4B S at +0.48). Largest deviation in either direction across all seven is 1.68pp, and one binomial SE at n=541 is about 1.4pp. Nothing clears 2 sigma. Your own KL column separates the same checkpoints by more than 20x.

These cards are among the best I have read on the Hub, which is the only reason any of this was checkable: base revision pinned, llama.cpp revision pinned, SHA-256 and tensor inventory per file.

So: is the tied output embedding what pushes that whole path through the quantizer on 12B, and would retaining it at BF16 like the other six close the Q8_0 gap?