File size: 10,911 Bytes
221ca5c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 | # Three-Model Comparison: Qwen3.5-9B vs Gemma4 E4B vs Bonsai-8B
**Date:** 2026-04-07 | **Hardware:** Mac Mini M4 16GB | **Vision:** Falcon Perception v2
---
## Test Results
```
┌────┬───────────────────┬────────┬─────────────────┬─────────────────┬─────────────────┐
│ │ │ │ Qwen3.5-9B │ Gemma4 E4B │ Bonsai-8B │
│ # │ Task │ Diff. │ llama.cpp │ mlx_vlm+proxy │ prism llama.cpp │
├────┼───────────────────┼────────┼─────────────────┼─────────────────┼─────────────────┤
│ 1 │ Wikipedia extract │ Easy │ 51s 1T ⚠️ │ 30s 4T ✅ │ 33s 1T ⚠️ │
│ 2 │ DDG search │ Medium │ 180s 6T ⚠️ │ 60s 6T ✅ │ 53s 16T ❌ │
│ 3 │ HN top story │ Easy │ 32s 2T ✅ │ 30s 2T ✅ │ 4s 1T ⚠️ │
│ 4 │ Cat vision (FP) │ Medium │ 34s 2T ✅ │ 40s 3T ✅ │ 5s 1T ❌ │
│ 5 │ Form filling │ Medium │ 237s 6T ✅ │ 60s 1T ❌ │ 4s 1T ❌ │
│ 6 │ reCAPTCHA │ Hard │ 300s 6T ⚠️ │ 120s 13T ⚠️ │ 4s 1T ❌ │
├────┼───────────────────┼────────┼─────────────────┼─────────────────┼─────────────────┤
│ │ PASS / PARTIAL │ │ 3✅ 2⚠️ 1❌ │ 4✅ 1⚠️ 1❌ │ 0✅ 2⚠️ 4❌ │
│ │ Total time │ │ 834s │ 340s │ 103s │
│ │ Total tool calls │ │ 23 │ 29 │ 21 │
└────┴───────────────────┴────────┴─────────────────┴─────────────────┴─────────────────┘
T = tool calls | ✅ = pass | ⚠️ = partial | ❌ = fail
```
## Speed Comparison
```
Generation speed (tok/s):
Bonsai-8B ████████████████████████████████████████████████ 48.8
Gemma4 E4B ██████████████████████████████████ 35.0
Qwen3.5-9B ██████████ 10.0
Total suite time:
Bonsai-8B ██████ 103s (1.7 min)
Gemma4 E4B █████████████████████ 340s (5.7 min)
Qwen3.5-9B █████████████████████████████████████████████████████ 834s (13.9 min)
```
## Model Specs
```
┌──────────────────┬──────────────┬──────────────┬──────────────┐
│ │ Qwen3.5-9B │ Gemma4 E4B │ Bonsai-8B │
├──────────────────┼──────────────┼──────────────┼──────────────┤
│ Architecture │ Dense │ MoE (4B act) │ Dense 1-bit │
│ Total params │ 9B │ 9B │ 8B │
│ Active params │ 9B │ 4B │ 8B │
│ Quantization │ Q4_K_XL │ 4-bit MLX │ Q1_0_g128 │
│ Disk size │ 5.6 GB │ ~5 GB │ 1.15 GB │
│ Memory (peak) │ ~6.5 GB │ ~5.4 GB │ ~1.5 GB │
│ Generation tok/s │ ~10 │ ~35 │ ~49 │
│ Base model │ Qwen3.5 │ Gemma4 │ Qwen3 │
│ Vision │ ✅ mmproj │ ✅ native │ ❌ text only │
│ Tool calling │ ✅ native │ ✅ patched │ ✅ native │
│ Proxy needed │ ❌ │ ✅ │ ❌ │
│ Server │ llama.cpp │ mlx_vlm │ PrismML fork │
└──────────────────┴──────────────┴──────────────┴──────────────┘
```
## Task-by-Task Analysis
### T1: Wikipedia Extract (Easy)
```
Qwen: 1 tool (scrape_url), got some info ⚠️ Partial
Gemma: 4 tools (navigate+extract), all 3 answers ✅ Pass
Bonsai: 1 tool (navigate), stopped too early ⚠️ Partial
```
### T2: DuckDuckGo Search (Medium)
```
Qwen: 6 tools, hit 180s timeout ⚠️ Partial (slow)
Gemma: 6 tools (go+type+click+scroll+extract) ✅ Pass
Bonsai: 16 tools but 21 idle turns, confused ❌ Fail
```
### T3: HN Top Story (Easy)
```
Qwen: 2 tools, clean stop_loop ✅ Pass
Gemma: 2 tools, found title ✅ Pass
Bonsai: 1 tool, 4s but didn't extract properly ⚠️ Partial
```
### T4: Cat Vision with Falcon (Medium)
```
Qwen: 2 tools incl vision_detect ✅ Pass
Gemma: 3 tools, 2x vision_detect, found 6 cats ✅ Pass
Bonsai: 1 tool, never called vision_detect ❌ Fail
```
### T5: Form Filling (Medium)
```
Qwen: 6 tools (go+input+click x3), stop_loop ✅ Pass — ONLY WINNER
Gemma: 1 tool, stuck thinking about form fields ❌ Fail
Bonsai: 1 tool, only navigated ❌ Fail
```
### T6: reCAPTCHA (Hard)
```
Qwen: 6 tools, 3 vision, 1 batch click, timeout ⚠️ Partial
Gemma: 13 tools, 9 vision, 5 batch clicks, timeout ⚠️ Partial (most active)
Bonsai: 1 tool, only navigated ❌ Fail
```
## Scoring Matrix
```
┌───────────────────┬───────┬───────┬───────┐
│ Task │ Qwen │ Gemma │Bonsai │
├───────────────────┼───────┼───────┼───────┤
│ T1 Wikipedia │ 0.5 │ 1.0 │ 0.5 │
│ T2 DDG Search │ 0.5 │ 1.0 │ 0.0 │
│ T3 HN Story │ 1.0 │ 1.0 │ 0.5 │
│ T4 Cat Vision │ 1.0 │ 1.0 │ 0.0 │
│ T5 Form Fill │ 1.0 │ 0.0 │ 0.0 │
│ T6 reCAPTCHA │ 0.5 │ 0.5 │ 0.0 │
├───────────────────┼───────┼───────┼───────┤
│ TOTAL │ 4.5 │ 4.5 │ 1.0 │
│ Speed factor │ 1x │ 3.5x │ 4.9x │
│ Score/minute │ 0.32 │ 0.79 │ 0.58 │
└───────────────────┴───────┴───────┴───────┘
Score/minute = total_score / suite_time_minutes
```
## Memory Budget on 16GB
```
┌──────────────────────┬──────────┬──────────┬──────────┐
│ Component │ +Qwen │ +Gemma4 │ +Bonsai │
├──────────────────────┼──────────┼──────────┼──────────┤
│ LLM model │ 6.5 GB │ 5.4 GB │ 1.5 GB │
│ Falcon Perception │ 1.5 GB │ 1.5 GB │ 1.5 GB │
│ GUA_Blazor + Browser │ 0.8 GB │ 0.8 GB │ 0.8 GB │
│ Proxy │ — │ 0.1 GB │ — │
│ OS + system │ 3.0 GB │ 3.0 GB │ 3.0 GB │
├──────────────────────┼──────────┼──────────┼──────────┤
│ TOTAL │ 11.8 GB │ 10.8 GB │ 6.8 GB │
│ HEADROOM │ 4.2 GB │ 5.2 GB │ 9.2 GB ★ │
└──────────────────────┴──────────┴──────────┴──────────┘
Bonsai leaves 9.2 GB free — enough to run ANOTHER model simultaneously!
```
## Verdict
```
┌────────────────────────────────────────────────────────────────┐
│ │
│ 🏆 BEST OVERALL: Gemma4 E4B │
│ 4.5 score, 3.5x speed, best score/minute (0.79) │
│ Wins: Wikipedia, DDG search, HN, Cat vision │
│ │
│ 🥈 MOST RELIABLE: Qwen3.5-9B │
│ 4.5 score, slowest but ONLY model that fills forms │
│ Wins: Form filling (exclusive), HN, Cat vision │
│ │
│ 🥉 FASTEST BUT WEAKEST: Bonsai-8B │
│ 1.0 score, 4.9x speed, but can't do multi-step tasks │
│ Only 1.15 GB — could pair with another model │
│ │
│ IDEAL SETUP: │
│ Bonsai-8B (1.5 GB) + Qwen3.5-9B (6.5 GB) = 8 GB │
│ Route simple→Bonsai, complex→Qwen │
│ Both fit on 16GB simultaneously! │
│ │
│ OR: Gemma4 E4B alone (best score/minute, most versatile) │
│ │
└────────────────────────────────────────────────────────────────┘
```
---
*All tests run on Mac Mini M4 16GB with identical GUA_Blazor agent loop,*
*Falcon Perception v2 vision backend, and same prompts.*
|