File size: 12,148 Bytes
221ca5c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 | # Final 7-Model Comparison — Agent Task Performance on Mac Mini M4 16GB
**Date:** 2026-04-07 | **Hardware:** Mac Mini M4 16GB (Dyson)
---
## Results At a Glance
```
Model Size Speed Agent Fit Verdict
(GB) (tok/s) Score 16GB?
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Qwopus3.5-27B Q3 11.0 OOM N/A ❌ Too big — needs 24GB+
Qwen3.5-9B Q4 5.6 10 4.5/6 ✅ Most reliable agent
Gemma4 E4B 4bit 5.0 35 4.5/6 ✅ Fastest viable agent
Bonsai-8B 1bit 1.15 49 1.0/6 ✅ Single-turn only
LFM2.5-Nova 1.2B 0.70 118 0.0/6 ✅ Context too small (4K)
FunctionGemma 270M 0.28 197 0.0/6 ✅ Repeats infinitely
Qwopus3.5-27B Q3 11.0 OOM N/A ❌ OOM at any context
```
## Detailed Comparison
```
┌─────────────────────┬────────┬────────┬───────┬───────┬────────┬───────┬──────┐
│ Model │Params │Disk │Memory │Gen │Agent │Multi │Proxy │
│ │(active)│(GB) │(GB) │tok/s │Score │-step? │Need? │
├─────────────────────┼────────┼────────┼───────┼───────┼────────┼───────┼──────┤
│ Qwopus3.5-27B v3 Q3 │ 27B │ 11.0 │ 14+ │ OOM │ N/A │ ? │ No │
│ Qwen3.5-9B Q4_K_XL │ 9B │ 5.6 │ 6.5 │ 10 │ 4.5/6 │ ✅ │ No │
│ Gemma4 E4B 4bit │ 4B MoE │ 5.0 │ 5.4 │ 35 │ 4.5/6 │ ✅ │ Yes │
│ Bonsai-8B Q1_0 │ 8B │ 1.15 │ 1.5 │ 49 │ 1.0/6 │ ❌ │ No │
│ LFM2.5-Nova 1.2B Q4 │ 1.2B │ 0.70 │ 0.8 │ 118 │ 0.0/6 │ ❌ │ No │
│ FunctionGemma 270M │ 270M │ 0.28 │ 0.3 │ 197 │ 0.0/6 │ ❌ │ No │
└─────────────────────┴────────┴────────┴───────┴───────┴────────┴───────┴──────┘
```
## Speed vs Capability Cliff
```
Agent Score (6 = perfect)
5 │ ★ Qwen (10 tok/s) ★ Gemma4 (35 tok/s)
│
4 │
│
3 │
│
2 │
│
1 │ ★ Bonsai (49 tok/s)
│
0 │ ✕ Qwopus ★ LFM2.5 ★ FuncGemma
│ (OOM) (118 tok/s) (197 tok/s)
└──┬──────────┬──────────┬──────────┬──────────┬──────────┬───
0 10 35 50 118 197
Generation Speed (tok/s)
★ = tested ✕ = OOM
THE CLIFF: There is a hard capability cliff between 4B active params
and below. Models under 4B active params cannot do multi-step agent
tasks regardless of how fast they are.
```
## Task Breakdown (6 tests)
```
┌────┬─────────────────┬────┬──────────┬──────────┬──────────┬──────────┬──────────┐
│ # │ Task │Diff│ Qwen 9B │Gemma4 E4B│Bonsai 8B │ LFM 1.2B │FuncG 270M│
├────┼─────────────────┼────┼──────────┼──────────┼──────────┼──────────┼──────────┤
│ 1 │ Wikipedia info │ E │ ⚠️ 1T │ ✅ 4T │ ⚠️ 1T │ ❌ OOC │ ❌ Loop │
│ 2 │ DDG search │ M │ ⚠️ 6T │ ✅ 6T │ ❌ 16T │ ❌ OOC │ ❌ Loop │
│ 3 │ HN top story │ E │ ✅ 2T │ ✅ 2T │ ⚠️ 1T │ ❌ 0T │ ❌ Loop │
│ 4 │ Cat vision (FP) │ M │ ✅ 2T │ ✅ 3T │ ❌ 1T │ ❌ 0T │ ❌ Loop │
│ 5 │ Form filling │ M │ ✅ 6T │ ❌ 1T │ ❌ 1T │ ❌ 0T │ ❌ Loop │
│ 6 │ reCAPTCHA │ H │ ⚠️ 6T │ ⚠️ 13T │ ❌ 1T │ ❌ 0T │ ❌ Loop │
├────┼─────────────────┼────┼──────────┼──────────┼──────────┼──────────┼──────────┤
│ │ Score │ │ 4.5/6 │ 4.5/6 │ 1.0/6 │ 0.0/6 │ 0.0/6 │
│ │ Total time │ │ 834s │ 340s │ 103s │ ~5s │ ~5s │
│ │ Total tools │ │ 23 │ 29 │ 21 │ 0 │ 0 │
└────┴─────────────────┴────┴──────────┴──────────┴──────────┴──────────┴──────────┘
T = tool calls | E = Easy | M = Medium | H = Hard
OOC = Out of Context | Loop = infinite repetition
```
## Why Small Models Fail at Agent Tasks
```
┌─────────────────────────────────────────────────────────────────────┐
│ THE AGENT CAPABILITY LADDER │
│ │
│ Level 1: FORMAT A TOOL CALL │
│ └─ Can generate {"name": "func", "arguments": {...}} │
│ └─ ALL models pass this (even 270M at 197 tok/s) │
│ └─ This is what BFCL benchmarks measure │
│ │
│ Level 2: UNDERSTAND TOOL RESULTS │
│ └─ Read tool output and decide next action │
│ └─ Requires: context understanding, error handling │
│ └─ Bonsai-8B fails here (makes 1 call then stops) │
│ │
│ Level 3: CHAIN MULTIPLE TOOLS │
│ └─ Navigate → type → click → extract → report │
│ └─ Requires: planning, sequential reasoning, 5K+ context │
│ └─ Minimum ~4B active params (Gemma4 E4B) │
│ │
│ Level 4: HANDLE ERRORS AND ADAPT │
│ └─ Tool fails → try different approach → recover │
│ └─ Requires: robust reasoning, error patterns │
│ └─ Qwen 9B reliable, Gemma4 E4B partial │
│ │
│ Level 5: COMPLEX MULTI-FIELD INTERACTION │
│ └─ Fill forms, interact with dynamic UIs │
│ └─ Requires: deep context, field mapping, DOM understanding │
│ └─ Only Qwen 9B succeeds (form filling) │
│ └─ Likely needs 27B+ for consistent success │
│ │
│ BFCL = Level 1 only. Our tests = Levels 1-5. │
│ That's why Bonsai scores 73% on BFCL but 1/6 on our tests. │
└─────────────────────────────────────────────────────────────────────┘
```
## Memory Map on 16GB
```
Available: 16 GB
Qwopus 27B Q3:
████████████████████████████████████████████████████████████████ 14+ GB → ❌ OOM
▓▓▓▓ OS (3GB)
Qwen 9B + Falcon:
████████████████████████████ 6.5 GB model
██████ 1.5 GB Falcon
██ 0.8 GB GUA+Browser
▓▓▓▓ OS (3GB)
░░░░░ 4.2 GB free ✅
Gemma4 E4B + Falcon:
██████████████████████ 5.4 GB model
██████ 1.5 GB Falcon
██ 0.8 GB GUA+Browser+Proxy
▓▓▓▓ OS (3GB)
░░░░░░░ 5.2 GB free ✅ (most headroom)
Bonsai 8B + Qwen 9B (dual!):
█████ 1.5 GB Bonsai
████████████████████████████ 6.5 GB Qwen
██████ 1.5 GB Falcon
██ 0.8 GB GUA+Browser
▓▓▓▓ OS (3GB)
░░ 2.7 GB free ✅ (tight but fits!)
```
## Final Verdict
```
┌────────────────────────────────────────────────────────────────────┐
│ │
│ 🏆 BEST OVERALL: Gemma4 E4B (with proxy fixes) │
│ • 4.5/6 score, 35 tok/s, 5.4 GB │
│ • Best speed-to-capability ratio │
│ • Wins DDG search, Wikipedia, HN, vision tasks │
│ • Needs proxy (7 fixes) but works reliably │
│ │
│ 🥈 MOST RELIABLE: Qwen3.5-9B │
│ • 4.5/6 score, 10 tok/s, 6.5 GB │
│ • ONLY model that fills forms successfully │
│ • No proxy needed, native tool calling │
│ • Slower but handles edge cases better │
│ │
│ 🥉 HONORABLE: Bonsai-8B │
│ • 1.0/6 but only 1.15 GB — fits alongside any other model │
│ • Could serve as fast first-call router │
│ │
│ ❌ DON'T USE FOR AGENTS: │
│ • LFM2.5-Nova (4K context too small) │
│ • FunctionGemma (loops infinitely) │
│ • Qwopus-27B (doesn't fit 16GB) │
│ │
│ MINIMUM FOR MULTI-STEP AGENTS: ~4B active parameters │
│ BFCL SCORE ≠ AGENT CAPABILITY │
│ │
└────────────────────────────────────────────────────────────────────┘
```
---
*7 models tested on identical GUA_Blazor agent loop with Falcon Perception v2.*
*All tests: navigate, search, extract, vision detect, form fill, captcha solve.*
*Mac Mini M4 16GB, April 2026.*
|