| <!doctype html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Automaticity Benchmark: MiniCPM5 v7 Fine-Tune</title> |
| <link rel="stylesheet" href="./elephant-hand-artifact.css"> |
| <style> |
| .rank-table td:nth-child(n+3), .rank-table th:nth-child(n+3) { text-align: right; } |
| .model-name { font-weight: 900; } |
| .bar { display: grid; gap: 6px; min-width: 120px; } |
| .bar span { font-weight: 800; } |
| .bar i { display: block; height: 9px; border: 2px solid var(--surface-border); background: var(--accent); box-shadow: var(--shadow-hard-small); } |
| .bar.good i { background: var(--good); } |
| .bar.warn i { background: var(--accent-yellow); } |
| .mono { font-family: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace; font-size: 0.92em; } |
| .compact-list { margin: 0; padding-left: 18px; } |
| .compact-list li { margin: 6px 0; } |
| .two-up { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 16px; } |
| </style> |
| </head> |
| <body> |
| <main class="page"> |
| <section class="hero"> |
| <p class="eyebrow">Automaticity Benchmark</p> |
| <h1>MiniCPM got faster, but the v7 LoRA overcalls no-op prompts.</h1> |
| <p class="lede"> |
| On the same 92-row automaticity benchmark, the incumbent FunctionGemma v7 Q8 remains the leader at 82/92 exact. |
| MiniCPM5 base is surprisingly strong at 78/92 exact; the MiniCPM5 v7 LoRA variants fall behind because no-op recall drops sharply. |
| </p> |
| <div class="metric-grid"> |
| <div class="metric"> |
| <span class="metric-label">Current Leader</span> |
| <span class="metric-value">89.1%</span> |
| <p>FunctionGemma_AUTOMATICITY_V7_Q8, 82/92 exact.</p> |
| </div> |
| <div class="metric"> |
| <span class="metric-label">Best MiniCPM Row</span> |
| <span class="metric-value">84.8%</span> |
| <p>MiniCPM5_Base, 78/92 exact before fine-tuning.</p> |
| </div> |
| <div class="metric"> |
| <span class="metric-label">Training Time</span> |
| <span class="metric-value">4:54</span> |
| <p>Wall-clock for MiniCPM5 base + LoRA training + Q4 export. Trainer loop was 249.9 seconds; Q8 export-only was about 25 seconds.</p> |
| </div> |
| <div class="metric"> |
| <span class="metric-label">Next Dataset</span> |
| <span class="metric-value">V8</span> |
| <p>Materialized with 1,070 train rows and 565 no-op rows for overcall hardening.</p> |
| </div> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <h2>Comparison</h2> |
| <div class="overflow"> |
| <table class="rank-table"> |
| <thead> |
| <tr> |
| <th>Run Label</th> |
| <th>Kind</th> |
| <th>Exact</th> |
| <th>Tool Name</th> |
| <th>Arguments</th> |
| <th>No-op Recall</th> |
| <th>p50</th> |
| <th>p95</th> |
| <th>Failures</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr> |
| <td><span class="model-name">FunctionGemma_AUTOMATICITY_V7_Q8</span></td> |
| <td><span class="badge good">Leader</span> baked GGUF Q8</td> |
| <td><div class="bar good"><span>82/92 路 89.1%</span><i style="width:89.1%"></i></div></td> |
| <td>96.7%</td> |
| <td>90.2%</td> |
| <td>94.7%</td> |
| <td>180 ms</td> |
| <td>568 ms</td> |
| <td>7 wrong args, 3 wrong tool</td> |
| </tr> |
| <tr> |
| <td><span class="model-name">MiniCPM5_Base</span></td> |
| <td><span class="badge good">Best MiniCPM</span> HF base</td> |
| <td><div class="bar good"><span>78/92 路 84.8%</span><i style="width:84.8%"></i></div></td> |
| <td>92.4%</td> |
| <td>87.0%</td> |
| <td>86.8%</td> |
| <td>701 ms</td> |
| <td>2,070 ms</td> |
| <td>7 wrong args, 7 wrong tool</td> |
| </tr> |
| <tr> |
| <td><span class="model-name">MiniCPM5_Base + AUTOMATICITY_V7_LORA_ADAPTER</span></td> |
| <td><span class="badge warn">LoRA Hat</span> unmerged HF adapter</td> |
| <td><div class="bar warn"><span>59/92 路 64.1%</span><i style="width:64.1%"></i></div></td> |
| <td>67.4%</td> |
| <td>75.0%</td> |
| <td>21.1%</td> |
| <td>316 ms</td> |
| <td>478 ms</td> |
| <td>3 wrong args, 30 wrong tool</td> |
| </tr> |
| <tr> |
| <td><span class="model-name">MiniCPM5_AUTOMATICITY_V7_Q8</span></td> |
| <td><span class="badge warn">Merged</span> baked GGUF Q8</td> |
| <td><div class="bar warn"><span>60/92 路 65.2%</span><i style="width:65.2%"></i></div></td> |
| <td>69.6%</td> |
| <td>75.0%</td> |
| <td>26.3%</td> |
| <td>151 ms</td> |
| <td>248 ms</td> |
| <td>4 wrong args, 28 wrong tool</td> |
| </tr> |
| <tr> |
| <td><span class="model-name">MiniCPM5_AUTOMATICITY_V7_Q4</span></td> |
| <td><span class="badge bad">Merged</span> baked GGUF Q4</td> |
| <td><div class="bar"><span>54/92 路 58.7%</span><i style="width:58.7%"></i></div></td> |
| <td>64.1%</td> |
| <td>67.4%</td> |
| <td>13.2%</td> |
| <td>141 ms</td> |
| <td>226 ms</td> |
| <td>5 wrong args, 33 wrong tool</td> |
| </tr> |
| </tbody> |
| </table> |
| </div> |
| </section> |
|
|
| <section class="section two-up"> |
| <div class="card"> |
| <h3>Interpretation</h3> |
| <p> |
| MiniCPM5 base is the better MiniCPM target today. The v7 LoRA learned to emit MiniCPM XML or compact XML fragments and became much faster in GGUF form, but it lost the base model's restraint on hypothetical, negated, deferred, and partial prompts. |
| </p> |
| </div> |
| <div class="card"> |
| <h3>Training Target</h3> |
| <p> |
| MiniCPM fine-tuning is using MiniCPM XML tool calls, not JSON. The parser accepts full XML and the compact fragments the exported model often emits. SGLang's native MiniCPM path should stay aligned with this XML convention. |
| </p> |
| </div> |
| <div class="card"> |
| <h3>Why The Earlier 52.5% Number Differed</h3> |
| <p> |
| The 52.5% MiniCPM result came from the older 120-row FunctionGemma spine benchmark. This report uses the newer 92-row automaticity-hard benchmark, so those percentages are not the same population. |
| </p> |
| </div> |
| <div class="card"> |
| <h3>Next Action</h3> |
| <p> |
| Train `AUTOMATICITY_V8` before promoting MiniCPM. The v8 dataset has added no-op-heavy contrastive rows and should be judged against the same 92 frozen rows with FunctionGemma v7 Q8 included as the leader row. |
| </p> |
| </div> |
| </section> |
|
|
| <section class="section"> |
| <h2>Artifacts</h2> |
| <div class="callout"> |
| <ul class="compact-list"> |
| <li><span class="mono">/home/turnercore/automaticity-training-v8/automaticity-train-v8.jsonl</span> 路 1,070 rows, 565 no-op rows.</li> |
| <li><span class="mono">/home/turnercore/automaticity-benchmark-v1/automaticity-hard-v1.jsonl</span> 路 frozen 92-row benchmark.</li> |
| <li><span class="mono">/tmp/ai-gateway/training/adapters/minicpm5-automaticity-v1</span> 路 unmerged LoRA adapter tested as the LoRA hat row.</li> |
| <li><span class="mono">/tmp/ai-gateway/training/gguf/minicpm5-automaticity-v1-q8/q8_0_gguf/MiniCPM5-1B.Q8_0.gguf</span> 路 MiniCPM5_AUTOMATICITY_V7_Q8.</li> |
| <li><span class="mono">/tmp/ai-gateway/training/gguf/minicpm5-automaticity-v1-q4/q4_k_m_gguf/MiniCPM5-1B.Q4_K_M.gguf</span> 路 MiniCPM5_AUTOMATICITY_V7_Q4.</li> |
| </ul> |
| </div> |
| </section> |
| </main> |
| </body> |
| </html> |
|
|
