| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - transit |
| - kiosk |
| - benchmark |
| - metrollm-bench |
| - apple-silicon |
| - mac-bench |
| --- |
| |
| # MetroLLM-Bench Mac Probe |
|
|
| Slim distribution of the [MetroLLM-Bench](https://github.com/continker/metrollm-bench) |
| hardware-envelope tools for Apple Silicon. Pulls GGUF weights from |
| `continker/Qwen3.5-{2B,4B,9B}-metro-v23`, runs `llama-server` locally on |
| Metal, executes a 15-case stratified MARTA probe (and optional sustained-load |
| thermal curve), and emits JSON telemetry — decode tok/s, TTFT, peak RAM, |
| Tier-1 deterministic accuracy + Tier-2 LLM-judge composite. |
|
|
| This repo exists so corporate or ephemeral Macs can `git clone` the bench |
| without VPN access to the private project repo. |
|
|
| ## Prerequisites (one-time) |
|
|
| ```bash |
| brew install uv llama.cpp |
| export ANTHROPIC_API_KEY=sk-ant-... # for the Tier-2 LLM judge |
| ``` |
|
|
| If brew is unavailable: `uv` has a `curl -LsSf https://astral.sh/uv/install.sh | sh` |
| fallback; `llama.cpp` ships official Apple Silicon release binaries on GitHub. |
|
|
| ## Run a 15-case probe (~10-30 min depending on Mac) |
|
|
| ```bash |
| git clone https://huggingface.co/continker/metrollm-bench-mac /tmp/mac-bench |
| cd /tmp/mac-bench |
| uv sync |
| bash scripts/mac_bench/run_probe.sh 2b |
| cat results/mac_bench/*-2b-probe/telemetry.json |
| ``` |
|
|
| Output captures: |
| - `tier1_composite`, `metrollm_composite` — bench scores (deterministic + judge) |
| - `decode_tok_s_median` / `_p10` / `_p90` — single-stream Metal decode throughput |
| - `ttft_ms_median` — first-token latency end-to-end (HTTP + decode) |
| - `peak_rss_gb` — max RSS of `llama-server` during decode |
| - `runner_wallclock_s` — total wall time |
|
|
| ## Run a sustained-load thermal curve (fanless Macs only, ~45 min) |
|
|
| ```bash |
| bash scripts/mac_bench/run_thermal.sh 2b --duration 45m |
| ``` |
|
|
| Replays MARTA cases on a loop while `thermal_sampler.py` records tok/s + RSS |
| every 30 s. Captures cold → sustained → throttle behaviour. Output: |
| `results/mac_bench/<chip>-<ram>gb-2b-thermal/thermal_curve.{csv,json}`. |
|
|
| ## Cleanup |
|
|
| ```bash |
| rm -rf /tmp/mac-bench |
| ``` |
|
|
| (Optionally `brew uninstall uv llama.cpp` if you don't keep them around.) |
|
|
| ## Per-Mac context-size requirements |
|
|
| `llama.cpp` allocates the full KV cache upfront at server start. Defaults |
| already cover the measured p99 conversation length per model: |
|
|
| | Size | Default ctx | KV memory | Total RAM | |
| |---|---:|---:|---:| |
| | 2B | 32 768 | 1.21 GB | ~4 GB | |
| | 4B | 16 384 | 2.42 GB | ~6.5 GB | |
| | 9B | 16 384 | 2.42 GB | ~9.2 GB (tight on 16 GB Macs) | |
|
|
| Override with `--ctx N` (e.g. `bash scripts/mac_bench/run_probe.sh 9b --ctx 8192`). |
|
|
| ## License & attribution |
|
|
| Apache 2.0. See parent project for full citation. |
|
|
