Spaces:
Configuration error
READ: Access to our weights
If you want access to our weights for MERGING or something similar. Request access to the weights, and then let me know here.
I'll review your profile and make a decision on allowing it based on your prior work.
For people that want to QUANT the weights, please just request the quant type you want here. We will (eventually) get around to it.
If you want a VLLM quant ask/ping @phaedawg directly.
Be nice, because phaedawg does this on their own free time and can't make any kind of guarantee. They may not do it at all. They have been really busy lately.
NOTE: Because of how many models we have, we're really only going to accept quant requests for recent work. Although, we will decide on a case by case basis to quant older models.
@phaedawg hi guys, thanks for the hard work. I need a AWQ 4 bits (VLLM nightly is compatible with gemma 4 12b) quantize version of the last version of Melody (gemma 12B NSFW) to test it please :) thanks !
I've been messing around with Serenity all three variants and it seems to be doing well. Would love to convert these to MLX. (I'd be more than happy to contribute).
I've been messing around with Serenity all three variants and it seems to be doing well. Would love to convert these to MLX. (I'd be more than happy to contribute).
I can do the MLX quant. Please let me know the specific model you want, along with the bit (4-8).
I did request access to Melody1437-26B-A4B-v2.0 for quanting purposes, but I was hoping to do a range of different i-APEX-style quants to find a good balance between size and precision for my specific setup. That's kind of a niche request to bother people with. Had some very good experience with these configs for standard 26B and the 19B Reap version of the model:
Standard Gemma 4 Apex Mini
blk.0.ffn_gate_inp=Q8_0
blk.0.ffn_gate_up_exps=Q3_K
blk.0.ffn_down_exps=Q3_K
blk.0.ffn_gate=Q5_K
blk.0.ffn_up=Q5_K
blk.0.ffn_down=Q5_K
blk.0.attn_q=Q4_K
blk.0.attn_k=Q4_K
blk.0.attn_v=Q4_K
blk.0.attn_output=Q4_K
blk.1.ffn_gate_inp=Q8_0
blk.1.ffn_gate_up_exps=Q3_K
blk.1.ffn_down_exps=Q3_K
blk.1.ffn_gate=Q5_K
blk.1.ffn_up=Q5_K
blk.1.ffn_down=Q5_K
blk.1.attn_q=Q4_K
blk.1.attn_k=Q4_K
blk.1.attn_v=Q4_K
blk.1.attn_output=Q4_K
blk.2.ffn_gate_inp=Q8_0
blk.2.ffn_gate_up_exps=Q3_K
blk.2.ffn_down_exps=Q3_K
blk.2.ffn_gate=Q5_K
blk.2.ffn_up=Q5_K
blk.2.ffn_down=Q5_K
blk.2.attn_q=Q4_K
blk.2.attn_k=Q4_K
blk.2.attn_v=Q4_K
blk.2.attn_output=Q4_K
blk.3.ffn_gate_inp=Q8_0
blk.3.ffn_gate_up_exps=Q3_K
blk.3.ffn_down_exps=Q3_K
blk.3.ffn_gate=Q4_K
blk.3.ffn_up=Q4_K
blk.3.ffn_down=Q4_K
blk.3.attn_q=Q3_K
blk.3.attn_k=Q3_K
blk.3.attn_v=Q3_K
blk.3.attn_output=Q3_K
blk.4.ffn_gate_inp=Q8_0
blk.4.ffn_gate_up_exps=Q3_K
blk.4.ffn_down_exps=Q3_K
blk.4.ffn_gate=Q4_K
blk.4.ffn_up=Q4_K
blk.4.ffn_down=Q4_K
blk.4.attn_q=Q3_K
blk.4.attn_k=Q3_K
blk.4.attn_v=Q3_K
blk.4.attn_output=Q3_K
blk.5.ffn_gate_inp=Q8_0
blk.5.ffn_gate_up_exps=Q3_K
blk.5.ffn_down_exps=Q3_K
blk.5.ffn_gate=Q4_K
blk.5.ffn_up=Q4_K
blk.5.ffn_down=Q4_K
blk.5.attn_q=Q3_K
blk.5.attn_k=Q3_K
blk.5.attn_output=Q3_K
blk.6.ffn_gate_inp=Q8_0
blk.6.ffn_gate_up_exps=Q3_K
blk.6.ffn_down_exps=Q3_K
blk.6.ffn_gate=Q4_K
blk.6.ffn_up=Q4_K
blk.6.ffn_down=Q4_K
blk.6.attn_q=Q3_K
blk.6.attn_k=Q3_K
blk.6.attn_v=Q3_K
blk.6.attn_output=Q3_K
blk.7.ffn_gate_inp=Q8_0
blk.7.ffn_gate_up_exps=Q3_K
blk.7.ffn_down_exps=Q3_K
blk.7.ffn_gate=Q4_K
blk.7.ffn_up=Q4_K
blk.7.ffn_down=Q4_K
blk.7.attn_q=Q3_K
blk.7.attn_k=Q3_K
blk.7.attn_v=Q3_K
blk.7.attn_output=Q3_K
blk.8.ffn_gate_inp=Q8_0
blk.8.ffn_gate_up_exps=Q3_K
blk.8.ffn_down_exps=Q3_K
blk.8.ffn_gate=Q4_K
blk.8.ffn_up=Q4_K
blk.8.ffn_down=Q4_K
blk.8.attn_q=Q3_K
blk.8.attn_k=Q3_K
blk.8.attn_v=Q3_K
blk.8.attn_output=Q3_K
blk.9.ffn_gate_inp=Q8_0
blk.9.ffn_gate_up_exps=Q3_K
blk.9.ffn_down_exps=Q3_K
blk.9.ffn_gate=Q4_K
blk.9.ffn_up=Q4_K
blk.9.ffn_down=Q4_K
blk.9.attn_q=Q3_K
blk.9.attn_k=Q3_K
blk.9.attn_v=Q3_K
blk.9.attn_output=Q3_K
blk.10.ffn_gate_inp=Q8_0
blk.10.ffn_gate_up_exps=Q3_K
blk.10.ffn_down_exps=Q3_K
blk.10.ffn_gate=Q4_K
blk.10.ffn_up=Q4_K
blk.10.ffn_down=Q4_K
blk.10.attn_q=Q3_K
blk.10.attn_k=Q3_K
blk.10.attn_v=Q3_K
blk.10.attn_output=Q3_K
blk.11.ffn_gate_inp=Q8_0
blk.11.ffn_gate_up_exps=Q3_K
blk.11.ffn_down_exps=Q3_K
blk.11.ffn_gate=Q4_K
blk.11.ffn_up=Q4_K
blk.11.ffn_down=Q4_K
blk.11.attn_q=Q3_K
blk.11.attn_k=Q3_K
blk.11.attn_output=Q3_K
blk.12.ffn_gate_inp=Q8_0
blk.12.ffn_gate_up_exps=Q3_K
blk.12.ffn_down_exps=Q3_K
blk.12.ffn_gate=Q4_K
blk.12.ffn_up=Q4_K
blk.12.ffn_down=Q4_K
blk.12.attn_q=Q3_K
blk.12.attn_k=Q3_K
blk.12.attn_v=Q3_K
blk.12.attn_output=Q3_K
blk.13.ffn_gate_inp=Q8_0
blk.13.ffn_gate_up_exps=Q3_K
blk.13.ffn_down_exps=Q3_K
blk.13.ffn_gate=Q4_K
blk.13.ffn_up=Q4_K
blk.13.ffn_down=Q4_K
blk.13.attn_q=Q3_K
blk.13.attn_k=Q3_K
blk.13.attn_v=Q3_K
blk.13.attn_output=Q3_K
blk.14.ffn_gate_inp=Q8_0
blk.14.ffn_gate_up_exps=Q3_K
blk.14.ffn_down_exps=Q3_K
blk.14.ffn_gate=Q4_K
blk.14.ffn_up=Q4_K
blk.14.ffn_down=Q4_K
blk.14.attn_q=Q3_K
blk.14.attn_k=Q3_K
blk.14.attn_v=Q3_K
blk.14.attn_output=Q3_K
blk.15.ffn_gate_inp=Q8_0
blk.15.ffn_gate_up_exps=Q3_K
blk.15.ffn_down_exps=Q3_K
blk.15.ffn_gate=Q4_K
blk.15.ffn_up=Q4_K
blk.15.ffn_down=Q4_K
blk.15.attn_q=Q3_K
blk.15.attn_k=Q3_K
blk.15.attn_v=Q3_K
blk.15.attn_output=Q3_K
blk.16.ffn_gate_inp=Q8_0
blk.16.ffn_gate_up_exps=Q3_K
blk.16.ffn_down_exps=Q3_K
blk.16.ffn_gate=Q4_K
blk.16.ffn_up=Q4_K
blk.16.ffn_down=Q4_K
blk.16.attn_q=Q3_K
blk.16.attn_k=Q3_K
blk.16.attn_v=Q3_K
blk.16.attn_output=Q3_K
blk.17.ffn_gate_inp=Q8_0
blk.17.ffn_gate_up_exps=Q3_K
blk.17.ffn_down_exps=Q3_K
blk.17.ffn_gate=Q4_K
blk.17.ffn_up=Q4_K
blk.17.ffn_down=Q4_K
blk.17.attn_q=Q3_K
blk.17.attn_k=Q3_K
blk.17.attn_output=Q3_K
blk.18.ffn_gate_inp=Q8_0
blk.18.ffn_gate_up_exps=Q3_K
blk.18.ffn_down_exps=Q3_K
blk.18.ffn_gate=Q4_K
blk.18.ffn_up=Q4_K
blk.18.ffn_down=Q4_K
blk.18.attn_q=Q3_K
blk.18.attn_k=Q3_K
blk.18.attn_v=Q3_K
blk.18.attn_output=Q3_K
blk.19.ffn_gate_inp=Q8_0
blk.19.ffn_gate_up_exps=Q3_K
blk.19.ffn_down_exps=Q3_K
blk.19.ffn_gate=Q4_K
blk.19.ffn_up=Q4_K
blk.19.ffn_down=Q4_K
blk.19.attn_q=Q3_K
blk.19.attn_k=Q3_K
blk.19.attn_v=Q3_K
blk.19.attn_output=Q3_K
blk.20.ffn_gate_inp=Q8_0
blk.20.ffn_gate_up_exps=Q3_K
blk.20.ffn_down_exps=Q3_K
blk.20.ffn_gate=Q4_K
blk.20.ffn_up=Q4_K
blk.20.ffn_down=Q4_K
blk.20.attn_q=Q3_K
blk.20.attn_k=Q3_K
blk.20.attn_v=Q3_K
blk.20.attn_output=Q3_K
blk.21.ffn_gate_inp=Q8_0
blk.21.ffn_gate_up_exps=Q3_K
blk.21.ffn_down_exps=Q3_K
blk.21.ffn_gate=Q4_K
blk.21.ffn_up=Q4_K
blk.21.ffn_down=Q4_K
blk.21.attn_q=Q3_K
blk.21.attn_k=Q3_K
blk.21.attn_v=Q3_K
blk.21.attn_output=Q3_K
blk.22.ffn_gate_inp=Q8_0
blk.22.ffn_gate_up_exps=Q3_K
blk.22.ffn_down_exps=Q3_K
blk.22.ffn_gate=Q4_K
blk.22.ffn_up=Q4_K
blk.22.ffn_down=Q4_K
blk.22.attn_q=Q3_K
blk.22.attn_k=Q3_K
blk.22.attn_v=Q3_K
blk.22.attn_output=Q3_K
blk.23.ffn_gate_inp=Q8_0
blk.23.ffn_gate_up_exps=Q3_K
blk.23.ffn_down_exps=Q3_K
blk.23.ffn_gate=Q4_K
blk.23.ffn_up=Q4_K
blk.23.ffn_down=Q4_K
blk.23.attn_q=Q3_K
blk.23.attn_k=Q3_K
blk.23.attn_output=Q3_K
blk.24.ffn_gate_inp=Q8_0
blk.24.ffn_gate_up_exps=Q3_K
blk.24.ffn_down_exps=Q3_K
blk.24.ffn_gate=Q4_K
blk.24.ffn_up=Q4_K
blk.24.ffn_down=Q4_K
blk.24.attn_q=Q3_K
blk.24.attn_k=Q3_K
blk.24.attn_v=Q3_K
blk.24.attn_output=Q3_K
blk.25.ffn_gate_inp=Q8_0
blk.25.ffn_gate_up_exps=Q3_K
blk.25.ffn_down_exps=Q3_K
blk.25.ffn_gate=Q5_K
blk.25.ffn_up=Q5_K
blk.25.ffn_down=Q5_K
blk.25.attn_q=Q3_K
blk.25.attn_k=Q3_K
blk.25.attn_v=Q3_K
blk.25.attn_output=Q3_K
blk.26.ffn_gate_inp=Q8_0
blk.26.ffn_gate_up_exps=Q3_K
blk.26.ffn_down_exps=Q3_K
blk.26.ffn_gate=Q5_K
blk.26.ffn_up=Q5_K
blk.26.ffn_down=Q5_K
blk.26.attn_q=Q3_K
blk.26.attn_k=Q3_K
blk.26.attn_v=Q3_K
blk.26.attn_output=Q3_K
blk.27.ffn_gate_inp=Q8_0
blk.27.ffn_gate_up_exps=Q3_K
blk.27.ffn_down_exps=Q3_K
blk.27.ffn_gate=Q5_K
blk.27.ffn_up=Q5_K
blk.27.ffn_down=Q5_K
blk.27.attn_q=Q4_K
blk.27.attn_k=Q4_K
blk.27.attn_v=Q4_K
blk.27.attn_output=Q4_K
blk.28.ffn_gate_inp=Q8_0
blk.28.ffn_gate_up_exps=Q3_K
blk.28.ffn_down_exps=Q3_K
blk.28.ffn_gate=Q5_K
blk.28.ffn_up=Q5_K
blk.28.ffn_down=Q5_K
blk.28.attn_q=Q4_K
blk.28.attn_k=Q4_K
blk.28.attn_v=Q4_K
blk.28.attn_output=Q4_K
blk.29.ffn_gate_inp=Q8_0
blk.29.ffn_gate_up_exps=Q3_K
blk.29.ffn_down_exps=Q3_K
blk.29.ffn_gate=Q5_K
blk.29.ffn_up=Q5_K
blk.29.ffn_down=Q5_K
blk.29.attn_q=Q4_K
blk.29.attn_k=Q4_K
blk.29.attn_output=Q4_K
Basic 1-step-Upscale
blk.0.ffn_gate_inp=Q8_0
blk.0.ffn_gate_up_exps=Q4_K
blk.0.ffn_down_exps=Q4_K
blk.0.ffn_gate=Q6_K
blk.0.ffn_up=Q6_K
blk.0.ffn_down=Q6_K
blk.0.attn_q=Q5_K
blk.0.attn_k=Q5_K
blk.0.attn_v=Q5_K
blk.0.attn_output=Q5_K
blk.1.ffn_gate_inp=Q8_0
blk.1.ffn_gate_up_exps=Q4_K
blk.1.ffn_down_exps=Q4_K
blk.1.ffn_gate=Q6_K
blk.1.ffn_up=Q6_K
blk.1.ffn_down=Q6_K
blk.1.attn_q=Q5_K
blk.1.attn_k=Q5_K
blk.1.attn_v=Q5_K
blk.1.attn_output=Q5_K
blk.2.ffn_gate_inp=Q8_0
blk.2.ffn_gate_up_exps=Q4_K
blk.2.ffn_down_exps=Q4_K
blk.2.ffn_gate=Q6_K
blk.2.ffn_up=Q6_K
blk.2.ffn_down=Q6_K
blk.2.attn_q=Q5_K
blk.2.attn_k=Q5_K
blk.2.attn_v=Q5_K
blk.2.attn_output=Q5_K
blk.3.ffn_gate_inp=Q8_0
blk.3.ffn_gate_up_exps=Q4_K
blk.3.ffn_down_exps=Q4_K
blk.3.ffn_gate=Q5_K
blk.3.ffn_up=Q5_K
blk.3.ffn_down=Q5_K
blk.3.attn_q=Q4_K
blk.3.attn_k=Q4_K
blk.3.attn_v=Q4_K
blk.3.attn_output=Q4_K
blk.4.ffn_gate_inp=Q8_0
blk.4.ffn_gate_up_exps=Q4_K
blk.4.ffn_down_exps=Q4_K
blk.4.ffn_gate=Q5_K
blk.4.ffn_up=Q5_K
blk.4.ffn_down=Q5_K
blk.4.attn_q=Q4_K
blk.4.attn_k=Q4_K
blk.4.attn_v=Q4_K
blk.4.attn_output=Q4_K
blk.5.ffn_gate_inp=Q8_0
blk.5.ffn_gate_up_exps=Q4_K
blk.5.ffn_down_exps=Q4_K
blk.5.ffn_gate=Q5_K
blk.5.ffn_up=Q5_K
blk.5.ffn_down=Q5_K
blk.5.attn_q=Q4_K
blk.5.attn_k=Q4_K
blk.5.attn_output=Q4_K
blk.6.ffn_gate_inp=Q8_0
blk.6.ffn_gate_up_exps=Q4_K
blk.6.ffn_down_exps=Q4_K
blk.6.ffn_gate=Q5_K
blk.6.ffn_up=Q5_K
blk.6.ffn_down=Q5_K
blk.6.attn_q=Q4_K
blk.6.attn_k=Q4_K
blk.6.attn_v=Q4_K
blk.6.attn_output=Q4_K
blk.7.ffn_gate_inp=Q8_0
blk.7.ffn_gate_up_exps=Q4_K
blk.7.ffn_down_exps=Q4_K
blk.7.ffn_gate=Q5_K
blk.7.ffn_up=Q5_K
blk.7.ffn_down=Q5_K
blk.7.attn_q=Q4_K
blk.7.attn_k=Q4_K
blk.7.attn_v=Q4_K
blk.7.attn_output=Q4_K
blk.8.ffn_gate_inp=Q8_0
blk.8.ffn_gate_up_exps=Q4_K
blk.8.ffn_down_exps=Q4_K
blk.8.ffn_gate=Q5_K
blk.8.ffn_up=Q5_K
blk.8.ffn_down=Q5_K
blk.8.attn_q=Q4_K
blk.8.attn_k=Q4_K
blk.8.attn_v=Q4_K
blk.8.attn_output=Q4_K
blk.9.ffn_gate_inp=Q8_0
blk.9.ffn_gate_up_exps=Q4_K
blk.9.ffn_down_exps=Q4_K
blk.9.ffn_gate=Q5_K
blk.9.ffn_up=Q5_K
blk.9.ffn_down=Q5_K
blk.9.attn_q=Q4_K
blk.9.attn_k=Q4_K
blk.9.attn_v=Q4_K
blk.9.attn_output=Q4_K
blk.10.ffn_gate_inp=Q8_0
blk.10.ffn_gate_up_exps=Q4_K
blk.10.ffn_down_exps=Q4_K
blk.10.ffn_gate=Q5_K
blk.10.ffn_up=Q5_K
blk.10.ffn_down=Q5_K
blk.10.attn_q=Q4_K
blk.10.attn_k=Q4_K
blk.10.attn_v=Q4_K
blk.10.attn_output=Q4_K
blk.11.ffn_gate_inp=Q8_0
blk.11.ffn_gate_up_exps=Q4_K
blk.11.ffn_down_exps=Q4_K
blk.11.ffn_gate=Q5_K
blk.11.ffn_up=Q5_K
blk.11.ffn_down=Q5_K
blk.11.attn_q=Q4_K
blk.11.attn_k=Q4_K
blk.11.attn_output=Q4_K
blk.12.ffn_gate_inp=Q8_0
blk.12.ffn_gate_up_exps=Q4_K
blk.12.ffn_down_exps=Q4_K
blk.12.ffn_gate=Q5_K
blk.12.ffn_up=Q5_K
blk.12.ffn_down=Q5_K
blk.12.attn_q=Q4_K
blk.12.attn_k=Q4_K
blk.12.attn_v=Q4_K
blk.12.attn_output=Q4_K
blk.13.ffn_gate_inp=Q8_0
blk.13.ffn_gate_up_exps=Q4_K
blk.13.ffn_down_exps=Q4_K
blk.13.ffn_gate=Q5_K
blk.13.ffn_up=Q5_K
blk.13.ffn_down=Q5_K
blk.13.attn_q=Q4_K
blk.13.attn_k=Q4_K
blk.13.attn_v=Q4_K
blk.13.attn_output=Q4_K
blk.14.ffn_gate_inp=Q8_0
blk.14.ffn_gate_up_exps=Q4_K
blk.14.ffn_down_exps=Q4_K
blk.14.ffn_gate=Q5_K
blk.14.ffn_up=Q5_K
blk.14.ffn_down=Q5_K
blk.14.attn_q=Q4_K
blk.14.attn_k=Q4_K
blk.14.attn_v=Q4_K
blk.14.attn_output=Q4_K
blk.15.ffn_gate_inp=Q8_0
blk.15.ffn_gate_up_exps=Q4_K
blk.15.ffn_down_exps=Q4_K
blk.15.ffn_gate=Q5_K
blk.15.ffn_up=Q5_K
blk.15.ffn_down=Q5_K
blk.15.attn_q=Q4_K
blk.15.attn_k=Q4_K
blk.15.attn_v=Q4_K
blk.15.attn_output=Q4_K
blk.16.ffn_gate_inp=Q8_0
blk.16.ffn_gate_up_exps=Q4_K
blk.16.ffn_down_exps=Q4_K
blk.16.ffn_gate=Q5_K
blk.16.ffn_up=Q5_K
blk.16.ffn_down=Q5_K
blk.16.attn_q=Q4_K
blk.16.attn_k=Q4_K
blk.16.attn_v=Q4_K
blk.16.attn_output=Q4_K
blk.17.ffn_gate_inp=Q8_0
blk.17.ffn_gate_up_exps=Q4_K
blk.17.ffn_down_exps=Q4_K
blk.17.ffn_gate=Q5_K
blk.17.ffn_up=Q5_K
blk.17.ffn_down=Q5_K
blk.17.attn_q=Q4_K
blk.17.attn_k=Q4_K
blk.17.attn_output=Q4_K
blk.18.ffn_gate_inp=Q8_0
blk.18.ffn_gate_up_exps=Q4_K
blk.18.ffn_down_exps=Q4_K
blk.18.ffn_gate=Q5_K
blk.18.ffn_up=Q5_K
blk.18.ffn_down=Q5_K
blk.18.attn_q=Q4_K
blk.18.attn_k=Q4_K
blk.18.attn_v=Q4_K
blk.18.attn_output=Q4_K
blk.19.ffn_gate_inp=Q8_0
blk.19.ffn_gate_up_exps=Q4_K
blk.19.ffn_down_exps=Q4_K
blk.19.ffn_gate=Q5_K
blk.19.ffn_up=Q5_K
blk.19.ffn_down=Q5_K
blk.19.attn_q=Q4_K
blk.19.attn_k=Q4_K
blk.19.attn_v=Q4_K
blk.19.attn_output=Q4_K
blk.20.ffn_gate_inp=Q8_0
blk.20.ffn_gate_up_exps=Q4_K
blk.20.ffn_down_exps=Q4_K
blk.20.ffn_gate=Q5_K
blk.20.ffn_up=Q5_K
blk.20.ffn_down=Q5_K
blk.20.attn_q=Q4_K
blk.20.attn_k=Q4_K
blk.20.attn_v=Q4_K
blk.20.attn_output=Q4_K
blk.21.ffn_gate_inp=Q8_0
blk.21.ffn_gate_up_exps=Q4_K
blk.21.ffn_down_exps=Q4_K
blk.21.ffn_gate=Q5_K
blk.21.ffn_up=Q5_K
blk.21.ffn_down=Q5_K
blk.21.attn_q=Q4_K
blk.21.attn_k=Q4_K
blk.21.attn_v=Q4_K
blk.21.attn_output=Q4_K
blk.22.ffn_gate_inp=Q8_0
blk.22.ffn_gate_up_exps=Q4_K
blk.22.ffn_down_exps=Q4_K
blk.22.ffn_gate=Q5_K
blk.22.ffn_up=Q5_K
blk.22.ffn_down=Q5_K
blk.22.attn_q=Q4_K
blk.22.attn_k=Q4_K
blk.22.attn_v=Q4_K
blk.22.attn_output=Q4_K
blk.23.ffn_gate_inp=Q8_0
blk.23.ffn_gate_up_exps=Q4_K
blk.23.ffn_down_exps=Q4_K
blk.23.ffn_gate=Q5_K
blk.23.ffn_up=Q5_K
blk.23.ffn_down=Q5_K
blk.23.attn_q=Q4_K
blk.23.attn_k=Q4_K
blk.23.attn_output=Q4_K
blk.24.ffn_gate_inp=Q8_0
blk.24.ffn_gate_up_exps=Q4_K
blk.24.ffn_down_exps=Q4_K
blk.24.ffn_gate=Q5_K
blk.24.ffn_up=Q5_K
blk.24.ffn_down=Q5_K
blk.24.attn_q=Q4_K
blk.24.attn_k=Q4_K
blk.24.attn_v=Q4_K
blk.24.attn_output=Q4_K
blk.25.ffn_gate_inp=Q8_0
blk.25.ffn_gate_up_exps=Q4_K
blk.25.ffn_down_exps=Q4_K
blk.25.ffn_gate=Q6_K
blk.25.ffn_up=Q6_K
blk.25.ffn_down=Q6_K
blk.25.attn_q=Q4_K
blk.25.attn_k=Q4_K
blk.25.attn_v=Q4_K
blk.25.attn_output=Q4_K
blk.26.ffn_gate_inp=Q8_0
blk.26.ffn_gate_up_exps=Q4_K
blk.26.ffn_down_exps=Q4_K
blk.26.ffn_gate=Q6_K
blk.26.ffn_up=Q6_K
blk.26.ffn_down=Q6_K
blk.26.attn_q=Q4_K
blk.26.attn_k=Q4_K
blk.26.attn_v=Q4_K
blk.26.attn_output=Q4_K
blk.27.ffn_gate_inp=Q8_0
blk.27.ffn_gate_up_exps=Q4_K
blk.27.ffn_down_exps=Q4_K
blk.27.ffn_gate=Q6_K
blk.27.ffn_up=Q6_K
blk.27.ffn_down=Q6_K
blk.27.attn_q=Q5_K
blk.27.attn_k=Q5_K
blk.27.attn_v=Q5_K
blk.27.attn_output=Q5_K
blk.28.ffn_gate_inp=Q8_0
blk.28.ffn_gate_up_exps=Q4_K
blk.28.ffn_down_exps=Q4_K
blk.28.ffn_gate=Q6_K
blk.28.ffn_up=Q6_K
blk.28.ffn_down=Q6_K
blk.28.attn_q=Q5_K
blk.28.attn_k=Q5_K
blk.28.attn_v=Q5_K
blk.28.attn_output=Q5_K
blk.29.ffn_gate_inp=Q8_0
blk.29.ffn_gate_up_exps=Q4_K
blk.29.ffn_down_exps=Q4_K
blk.29.ffn_gate=Q6_K
blk.29.ffn_up=Q6_K
blk.29.ffn_down=Q6_K
blk.29.attn_q=Q5_K
blk.29.attn_k=Q5_K
blk.29.attn_output=Q5_K
We're not going to make apex quants, sorry.
Fair enough. And no way you'll provide me access to the weights for that singular purpose, just so we're clear on that too? :)
Fair enough. And no way you'll provide me access to the weights for that singular purpose, just so we're clear on that too? :)
I'll give you access to ONE of our weights (any of them). Pick, apply and let me know which one.
I'll give you access to ONE of our weights (any of them). Pick, apply and let me know which one.
Cheers. I've already got a request pending on ReadyArt/Melody1437-26B-A4B-v2.0, so how about I stick with that choice? I've reapplied it.
Also, do you or your team have any interest at all in seeing a range of APEX quants of the model? From my perspective, MoE-aware mixed-precision is a major upgrade, and I'm surprised it's catching on so slowly. I can't say I've got a factory set up here, but I can make the effort if there is an interest.
I'll give you access to ONE of our weights (any of them). Pick, apply and let me know which one.
Cheers. I've already got a request pending on ReadyArt/Melody1437-26B-A4B-v2.0, so how about I stick with that choice? I've reapplied it.
have fun!
have fun!
Thanks again! :)
Hey guys, any chance we can get a Melody1437-26B-A4B-v2.0 6.0bpw H16 elx3 quant at some point?
Hey guys, any chance we can get a Melody1437-26B-A4B-v2.0 6.0bpw H16 elx3 quant at some point?
Why do you want EXL3 over GGUF with tensor parallelism?
Hey guys, any chance we can get a Melody1437-26B-A4B-v2.0 6.0bpw H16 elx3 quant at some point?
Why do you want EXL3 over GGUF with tensor parallelism?
I use koboldcpp for gguf and it doesn't allow parallel generation for multiple swipes, plus I feel like exl3 runs better on my set up. If you guys can't get around to it I understand and I appreciate the work you do.
I see a 3x performance increase on Llamacpp with sm = tensor over exl3.
@FrenzyBiscuit I set up a repo with the 4 basic APEX quant types of Melody. Any objections to me making it accessible?
I've been messing around with Serenity all three variants and it seems to be doing well. Would love to convert these to MLX. (I'd be more than happy to contribute).
I can do the MLX quant. Please let me know the specific model you want, along with the bit (4-8).
Sweet!, Serenity 12B, 26B, 31B v1.1 - MLX 8bit preferred. I have been testing them at each of the varieties. I appreciate you. Thanks.
I can do one of them. Which would you like?
I doubt you're using all three models ;)
I can do one of them. Which would you like?
I doubt you're using all three models ;)
haha, I've been testing with them all, on my M3 Ultra and my 128gb M4Max... but if I had to pick one that I'm most interested in... oof... 26B or 12B..... let's go 12B...
I can do one of them. Which would you like?
I doubt you're using all three models ;)
haha, I've been testing with them all, on my M3 Ultra and my 128gb M4Max... but if I had to pick one that I'm most interested in... oof... 26B or 12B..... let's go 12B...
Oof. You like 12B over the 26B??
:S
I can do one of them. Which would you like?
I doubt you're using all three models ;)
haha, I've been testing with them all, on my M3 Ultra and my 128gb M4Max... but if I had to pick one that I'm most interested in... oof... 26B or 12B..... let's go 12B...
Oof. You like 12B over the 26B??
:S
Well I already have 26B MLX models that I'm pretty happy with, but haven't found a 12B that I'm happy with. And I can run the 26B well on my strix halo box, but for a model to just keep loaded on my local Mac while I'm doing other work... 12B fits that bill.
I can do one of them. Which would you like?
I doubt you're using all three models ;)
haha, I've been testing with them all, on my M3 Ultra and my 128gb M4Max... but if I had to pick one that I'm most interested in... oof... 26B or 12B..... let's go 12B...
Oof. You like 12B over the 26B??
:S
Well I already have 26B MLX models that I'm pretty happy with, but haven't found a 12B that I'm happy with. And I can run the 26B well on my strix halo box, but for a model to just keep loaded on my local Mac while I'm doing other work... 12B fits that bill.
Okay. I made the ReadyArt/Serenity-12B public.
Enjoy.
I can do one of them. Which would you like?
I doubt you're using all three models ;)
haha, I've been testing with them all, on my M3 Ultra and my 128gb M4Max... but if I had to pick one that I'm most interested in... oof... 26B or 12B..... let's go 12B...
Oof. You like 12B over the 26B??
:S
Well I already have 26B MLX models that I'm pretty happy with, but haven't found a 12B that I'm happy with. And I can run the 26B well on my strix halo box, but for a model to just keep loaded on my local Mac while I'm doing other work... 12B fits that bill.
Okay. I made the ReadyArt/Serenity-12B public.
Enjoy.
oh wow, thanks man, going to cgo convert it now :) appreciate it.