I misunderstood the question. Honest answer is I'm not 100% sure, but as an educated guess I'd say it's due to the combination of types actually used (see Q3_K vs Q4_K for example) and the dequant cost of each. As a rule of thumb, Q4_K is the fastest to dequant, followed by Q6_K, Q2_K, Q5_K and Q3_K is the slowest.
Ed Addario PRO
eaddario
AI & ML interests
Finding ways to optimize LLMs' inference performance in resource-constrained environments (e.g. commodity hardware, desktops, laptops, mobiles, edge devices, etc.)
Recent Activity
updated a model 1 day ago
eaddario/IQ2_NL-IQ3_NL-Test-Models updated a model 1 day ago
eaddario/JQ2_K-JQ6_K-Test-Models repliedto their post 2 days ago
Experimental global target bits‑per‑weight quantization of IFM/K2-Horizon-7B and IFM/K2-Horizon-MoVA-36B-A4B.
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
https://huggingface.co/eaddario/K2-Horizon-7B-GGUF
https://huggingface.co/eaddario/K2-Horizon-MoVA-36B-A4B-GGUFOrganizations
replied to their post 2 days ago
replied to their post 6 days ago
Hi @Twu31 , that's certainly a possibility and no doubts it plays a part, although I suspect it has to do more with the architecture itself (Dense vs MoE with Mixture-of-Values attention).
posted an update 17 days ago
Post
5,213
Experimental global target bits‑per‑weight quantization of IFM/K2-Horizon-7B and IFM/K2-Horizon-MoVA-36B-A4B.
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/K2-Horizon-7B-GGUF
eaddario/K2-Horizon-MoVA-36B-A4B-GGUF
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/K2-Horizon-7B-GGUF
eaddario/K2-Horizon-MoVA-36B-A4B-GGUF
- 5 replies
posted an update about 2 months ago
Post
1,357
Experimental global target bits‑per‑weight quantization of openbmb/MiniCPM5-1B and openbmb/MiniCPM5-2B.
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/MiniCPM5-1B-GGUF
eaddario/MiniCPM5-2B-GGUF
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/MiniCPM5-1B-GGUF
eaddario/MiniCPM5-2B-GGUF
posted an update about 2 months ago
Post
189
Experimental global target bits‑per‑weight quantization of XHToken/Spark-X2.5-1.7B and XHToken/Spark-X2.5-4B.
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/Spark-X2.5-1.7B-GGUF
eaddario/Spark-X2.5-4B-GGUF
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/Spark-X2.5-1.7B-GGUF
eaddario/Spark-X2.5-4B-GGUF
posted an update 2 months ago
Post
2,822
Experimental global target bits‑per‑weight quantization of Qwen/Qwen3.8-27B, including MTP tensors in imatrix file.
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
- imatrix includes data for MTP/NextN layers.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/Qwen3.8-27B-GGUF
Unlike standard llama.cpp quantization that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach automatically optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
- imatrix includes data for MTP/NextN layers.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/Qwen3.8-27B-GGUF
posted an update 6 months ago
Post
401
Experimental global target bits‑per‑weight quantization of ibm-granite/granite-4.1-8b and ibm-granite/granite-4.1-30b.
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/granite-4.1-8b-GGUF
eaddario/granite-4.1-30b-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the model's card.
eaddario/granite-4.1-8b-GGUF
eaddario/granite-4.1-30b-GGUF
posted an update 6 months ago
Post
3,382
Experimental global target bits‑per‑weight quantization of Qwen/Qwen3.6-27B and Qwen/Qwen3.6-35B-A3B.
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the models' cards.
eaddario/Qwen3.6-27B-GGUF
eaddario/Qwen3.6-35B-A3B-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, GPQA, MMLU, etc.) and methodology in the models' cards.
eaddario/Qwen3.6-27B-GGUF
eaddario/Qwen3.6-35B-A3B-GGUF
posted an update 7 months ago
Post
291
Experimental global target bits‑per‑weight quantization of google/gemma-4-E2B-it, google/gemma-4-E4B-it and google/gemma-4-26B-A4B-it
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/gemma-4-E2B-it-GGUF
eaddario/gemma-4-E4B-it-GGUF
eaddario/gemma-4-26B-A4B-it-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/gemma-4-E2B-it-GGUF
eaddario/gemma-4-E4B-it-GGUF
eaddario/gemma-4-26B-A4B-it-GGUF
replied to their post 7 months ago
Thank you @Green-Sky ! I'm planning to have a go at the Gemma 4s over the weekend and I'll take your dataset for a spin
replied to their post 7 months ago
On this occasion, no difference in size is expected.
I'm benchmarking quality instead of size, and to facilitate apples-to-apples comparisons, models IQ1_M, IQ2_M, Q3_K, Q4_K, Q5_K, Q6_K and Q8_0 were quantized at the same bits-per-weight (bpw) of naive models, and Q4_K-B and Q4_K-U were matched to the ones produced by Bartwoski and Unsloth respectively.
The file sizes are the same, but the quality is better.
You're welcome to the enhanced versions of llama-imatrix and llama-quantize if you require a particular size. If this is not practical, let me know which ones you'd need, and I'll be happy to upload.
posted an update 7 months ago
Post
234
eaddario/imatrix-calibration datasets updated to include Southeast Asian languages (Burmese, Filipino, Indonesian, Thai & Vietnamese).
posted an update 7 months ago
Post
193
Experimental global target bits‑per‑weight quantization of Qwen/Qwen3.5-4B and Qwen/Qwen3.5-9B
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Qwen3.5-4B-GGUF
eaddario/Qwen3.5-9B-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Qwen3.5-4B-GGUF
eaddario/Qwen3.5-9B-GGUF
- 4 replies
posted an update 9 months ago
Post
3,131
Experimental global target bits‑per‑weight quantization of mistralai/Ministral-3-14B-Instruct-2512 and mistralai/Ministral-3-14B-Reasoning-2512
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Ministral-3-14B-Instruct-2512-GGUF
eaddario/Ministral-3-14B-Reasoning-2512-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Ministral-3-14B-Instruct-2512-GGUF
eaddario/Ministral-3-14B-Reasoning-2512-GGUF
posted an update 10 months ago
Post
1,833
Experimental global target bits‑per‑weight quantization of allenai/Olmo-3-7B-Instruct and allenai/Olmo-3-7B-Think
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Olmo-3-7B-Instruct-GGUF
eaddario/Olmo-3-7B-Think-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Olmo-3-7B-Instruct-GGUF
eaddario/Olmo-3-7B-Think-GGUF
posted an update 10 months ago
Post
2,141
Experimental global target bits‑per‑weight quantization of ServiceNow-AI/Apriel-1.6-15b-Thinker and zai-org/GLM-4.6V-Flash
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Apriel-1.6-15b-Thinker-GGUF
eaddario/GLM-4.6V-Flash-GGUF
Unlike standard llama.cpp quantizations that rely on fixed type heuristics (e.g., Q4_K_M), the Target BPW approach optimizes per-tensor precision where it matters the most, and produces high quality models that meet a precise global file size target.
Key Advantages:
- VRAM Maximization: Can generate high quality models sized exactly to fit hardware constraints (e.g., fitting the model into exactly 24GB VRAM).
- Data-Driven Precision: Quantization mix is determined by actual weight error sensitivity rather than hardcoded rules, often yielding better PPL/KLD size trade-offs.
Full benchmarks (PPL, KLD, ARC, MMLU, etc.) and methodology in the models' cards
eaddario/Apriel-1.6-15b-Thinker-GGUF
eaddario/GLM-4.6V-Flash-GGUF
reacted to hesamation's post with ❤️ about 1 year ago
Post
15,126
a senior engineer at google just dropped a 400-page free book on docs for review: agentic design patterns.
the table of contents looks like everything you need to know about agents + code:
> advanced prompt techniques
> multi-agent patterns
> tool use and MCP
> you name it
read it here: https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/edit?tab=t.0#heading=h.pxcur8v2qagu
you can also pre-order on Amazon (published by Springer) and the royalties goes to Save the Children: https://www.amazon.com/Agentic-Design-Patterns-Hands-Intelligent/dp/3032014018/
the table of contents looks like everything you need to know about agents + code:
> advanced prompt techniques
> multi-agent patterns
> tool use and MCP
> you name it
read it here: https://docs.google.com/document/d/1rsaK53T3Lg5KoGwvf8ukOUvbELRtH-V0LnOIFDxBryE/edit?tab=t.0#heading=h.pxcur8v2qagu
you can also pre-order on Amazon (published by Springer) and the royalties goes to Save the Children: https://www.amazon.com/Agentic-Design-Patterns-Hands-Intelligent/dp/3032014018/
reacted to AdinaY's post with 👀 over 1 year ago
Post
2,758
Skywork UniPic 🔥a unified autoregressive multimodal model for image understanding, generation, & editing, by Skywork 天工
Skywork/skywork-unipic-6888c0789cdb82457b2acf32
✨ 1.5 B - MIT License
✨ Runs on RTX 4090
✨ Truly unified architecture
Skywork/skywork-unipic-6888c0789cdb82457b2acf32
✨ 1.5 B - MIT License
✨ Runs on RTX 4090
✨ Truly unified architecture
posted an update over 1 year ago
Post
560
Layer-wise and Pruned versions of mistralai/Devstral-Small-2505 and mistralai/Mistral-Small-3.2-24B-Instruct-2506
- Tesor-wise:
eaddario/Devstral-Small-2505-GGUF
eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF
- Pruned:
eaddario/Devstral-Small-2505-pruned-GGUF
eaddario/Mistral-Small-3.2-24B-Instruct-2506-pruned-GGUF
Summary in the models' cards and test results in the ./scores directory. Questions/feedback is always welcomed.
- Tesor-wise:
eaddario/Devstral-Small-2505-GGUF
eaddario/Mistral-Small-3.2-24B-Instruct-2506-GGUF
- Pruned:
eaddario/Devstral-Small-2505-pruned-GGUF
eaddario/Mistral-Small-3.2-24B-Instruct-2506-pruned-GGUF
Summary in the models' cards and test results in the ./scores directory. Questions/feedback is always welcomed.
reacted to AdinaY's post with 🔥 over 1 year ago
Post
1,395
Seed-X 🔥 a suite of multilingual translation models released by ByteDance.
ByteDance-Seed/seed-x-6878753f2858bc17afa78543
✨ instruction/reinforcement learning/reward model
✨ Supports 28 languages, bidirectional translation
✨ Optimized for deployment & inference: 7B with mistral architecture
✨ Excels across domains: science, law, finance, literature & more
ByteDance-Seed/seed-x-6878753f2858bc17afa78543
✨ instruction/reinforcement learning/reward model
✨ Supports 28 languages, bidirectional translation
✨ Optimized for deployment & inference: 7B with mistral architecture
✨ Excels across domains: science, law, finance, literature & more