Requires this PR to run.

This repo contains specialized MoE-quants for Motif-Technologies/Motif-3. The idea being that given the huge size of the FFN tensors compared to the rest of the tensors in the model, it should be possible to achieve a better quality while keeping the overall size of the entire model smaller compared to a similar naive quantization. To that end, the quantization type default is kept in high quality and the FFN UP + FFN GATE tensors are quanted down along with the FFN DOWN tensors.

Quant Size Mixture PPL 1-(Mean PPL(Q)/PPL(base)) KLD
Q8_0 314.95 GiB (8.60 BPW) Q8_0 35.367303 ยฑ 0.451587 +2.8870% 0.010134 ยฑ 0.000061
Q5_K_M 220.07 GiB (6.01 BPW) Q8_0 / Q5_K / Q5_K / Q6_K 35.336996 ยฑ 0.450859 +2.7989% 0.015087 ยฑ 0.000088
Q4_K_M 183.47 GiB (5.01 BPW) Q8_0 / Q4_K / Q4_K / Q5_K 35.635889 ยฑ 0.455428 +3.6684% 0.026133 ยฑ 0.000135
IQ4_XS 143.12 GiB (3.91 BPW) Q8_0 / IQ3_S / IQ3_S / IQ4_XS 36.272353 ยฑ 0.463193 +5.5199% 0.058303 ยฑ 0.000287
IQ3_M 123.79 GiB (3.38 BPW) Q6_K / IQ3_XXS / IQ3_XXS / IQ3_S 38.364498 ยฑ 0.492873 +11.6062% 0.103530 ยฑ 0.000507
IQ3_S 111.84 GiB (3.05 BPW) Q6_K / IQ2_S / IQ2_S / IQ3_S 39.318385 ยฑ 0.502525 +14.3811% 0.157063 ยฑ 0.000752
IQ2_S 101.38 GiB (2.77 BPW) Q6_K / IQ2_XS / IQ2_XS / IQ3_XXS 41.701314 ยฑ 0.536307 +21.3133% 0.222056 ยฑ 0.001044

kld_graph ppl_graph

Downloads last month
1,222
GGUF
Model size
315B params
Architecture
motif3
Hardware compatibility
Log In to add your hardware

2-bit

3-bit

4-bit

5-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for AesSedai/Motif-3-GGUF

Quantized
(10)
this model