Add bench.sh + make bench for measured tok/s

The README's hardware table called itself estimates ("Numbers above are
estimates" in Known limitations). This wires up Ollama's authoritative
eval_count / eval_duration metadata into a repeatable benchmark so
anyone can replace estimates with measurements for their own machine.

The script:
- warms up first (model-load cost shouldn't pollute the first sample)
- mixes short / medium / long output prompts (single shape would lock
the average onto one decode pattern)
- reads timings from Ollama's response, not a client stopwatch (no
network or jq overhead in the number)

Reference data point added inline to the hardware section: Ryzen AI
Max+ 395 / Radeon 8060S iGPU at Q3_K_S clocks ~4.5 tok/s — sits
between the CPU-only 1-3 tok/s row and the 24 GB discrete card row,
which matches expectation for an iGPU with unified memory.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

CHANGELOG.md

@@ -25,6 +25,14 @@ and documentation**, not the underlying base model.
   user turn. Reproducible via `make smoke` before/after.
 
 ### Added
+- `scripts/bench.sh` + `make bench`: tok/s benchmark using Ollama's
+  `eval_count` / `eval_duration` response metadata (avoids client-side
+  stopwatch noise). Mixes short / medium / long prompts so the average
+  doesn't lock onto a single output shape, and warms up first so the
+  model-load cost doesn't pollute the first sample. Reference data:
+  Ryzen AI Max+ 395 / Radeon 8060S iGPU clocks ~4.5 tok/s at Q3_K_S.
+  README hardware section now points readers at `make bench` to measure
+  their own rather than trusting the estimated table.
 - `scripts/smoke_test.sh`: token-leakage guard. The previous round-trip
   check passed on any non-empty response — including the broken EOS-bleed
   output that motivated the stop-token fix above. The smoke test now

Makefile

@@ -25,7 +25,7 @@ MODEL ?= $(TAG)
 
 PRECISION ?= F16
 
-.PHONY: help build smoke check hooks mmproj clean
+.PHONY: help build smoke bench check hooks mmproj clean
 
 help:  ## Show this help.
 	@awk 'BEGIN {FS = ":.*##"; printf "Targets:\n"} /^[a-zA-Z_-]+:.*?##/ { printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2 }' $(MAKEFILE_LIST)
@@ -42,6 +42,9 @@ build:  ## Download GGUF (if needed) and run 'ollama create'.
 smoke:  ## Verify the model is reachable and round-trips.
 	MODEL=$(MODEL) ./scripts/smoke_test.sh
 
+bench:  ## Measure tok/s using Ollama's eval timing (3 prompts).
+	MODEL=$(MODEL) ./scripts/bench.sh
+
 mmproj:  ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
 	./scripts/fetch_mmproj.sh $(PRECISION)
 

README.md

@@ -107,6 +107,7 @@ The 27B is **dense**: every parameter participates in every forward pass. It's s
 | `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python |
 | `scripts/build.sh` | One-shot helper: pulls a GGUF and runs `ollama create` for you |
 | `scripts/smoke_test.sh` | Verifies an Ollama daemon + model, runs a round-trip, and asserts no chat-template tokens leak into the response |
+| `scripts/bench.sh` | Measures real tok/s using Ollama's `eval_count` / `eval_duration` metadata over a 3-prompt mix (run `make bench`) |
 | `scripts/fetch_mmproj.sh` | Pulls the vision projector for llama.cpp (Ollama vision is broken upstream — see [Vision](#vision)) |
 | `scripts/check.sh` | Local lint: `bash -n`, `pyflakes`, `py_compile`, footgun-grep |
 | `scripts/install-hooks.sh` | Installs `check.sh` as a git pre-commit hook |
@@ -257,6 +258,15 @@ The dense 27B is the easier of the two Janus models to deploy.
 | Mac Studio M2/M3 32 GB+ unified | Works, ~15-25 tok/s |
 | 32 GB unified-memory laptops (Mac M-series, Ryzen AI Max+, etc.) | Borderline at Q4. Drop to Q3_K_S (~12 GB) and trim `num_ctx` for headroom. |
 
+Most numbers in this table are estimates from comparable models; the
+gradient is right but the absolute values will move ±20% with prompt
+shape, KV cache type, and parallel-request count. Measure your own
+machine with `make bench` (3-prompt mix, reports tok/s from Ollama's
+`eval_count` / `eval_duration` so it's not stopwatch-noisy). Reference
+data point: a Ryzen AI Max+ 395 / Radeon 8060S iGPU at Q3_K_S clocks
+**~4.5 tok/s**, sitting between CPU-only and a 24 GB discrete card as
+expected.
+
 ## Chat template
 
 Standard Qwen 3.x ChatML with `<|im_start|>` / `<|im_end|>` role markers,

scripts/bench.sh

@@ -0,0 +1,98 @@
+#!/usr/bin/env bash
+# Janus-27B — tok/s benchmark via Ollama.
+#
+# Reads timing from Ollama's /api/chat response metadata (eval_count and
+# eval_duration are authoritative — no client-side stopwatch noise) and
+# averages over a handful of prompts that vary in output length so the
+# number generalises a bit beyond a single shape.
+#
+# Usage:
+#   ./scripts/bench.sh                       # uses MODEL=janus-27b
+#   MODEL=janus-27b NPROMPTS=5 ./scripts/bench.sh
+#   HOST=http://localhost:11434 ./scripts/bench.sh
+#
+# Requires: curl, jq, a running Ollama daemon with the model created.
+set -euo pipefail
+
+MODEL="${MODEL:-janus-27b}"
+HOST="${HOST:-http://localhost:11434}"
+
+red()   { printf "\033[31m%s\033[0m\n" "$*"; }
+green() { printf "\033[32m%s\033[0m\n" "$*"; }
+blue()  { printf "\033[34m%s\033[0m\n" "$*"; }
+
+for dep in curl jq; do
+    if ! command -v "$dep" >/dev/null 2>&1; then
+        red "[!] missing dependency: $dep"; exit 1
+    fi
+done
+
+if ! curl -fsS "${HOST}/api/tags" >/dev/null; then
+    red "[!] Ollama not reachable at ${HOST}"
+    exit 1
+fi
+if ! curl -fsS "${HOST}/api/tags" | jq -e --arg m "${MODEL}" '.models[] | select(.name | startswith($m))' >/dev/null; then
+    red "[!] Model '${MODEL}' not found. Build it first: ./scripts/build.sh"
+    exit 1
+fi
+
+# Mix of short / medium / long output lengths — single shape would skew
+# the average toward whatever the model decides to do for that prompt.
+PROMPTS=(
+    "Reply with only the word OK."
+    "Explain the time complexity of mergesort in one short paragraph."
+    "Write a 120-word explanation of what a Bloom filter is and when to use it."
+)
+
+blue "[*] host:  ${HOST}"
+blue "[*] model: ${MODEL}"
+blue "[*] prompts: ${#PROMPTS[@]}"
+echo
+
+# Warmup — first call pays the model-load cost; we don't want that in
+# the average. Result is discarded.
+blue "[*] warmup..."
+curl -fsS "${HOST}/api/chat" \
+    -H 'Content-Type: application/json' \
+    -d "$(jq -n --arg m "${MODEL}" '{
+        model: $m,
+        messages: [{role:"user", content:"warmup"}],
+        stream: false
+    }')" >/dev/null
+
+TOTAL_TOKENS=0
+TOTAL_NS=0
+
+printf "%-4s  %8s  %12s  %8s\n" "#" "tokens" "eval_ms" "tok/s"
+printf "%-4s  %8s  %12s  %8s\n" "----" "--------" "------------" "--------"
+
+for i in "${!PROMPTS[@]}"; do
+    prompt="${PROMPTS[$i]}"
+    resp="$(curl -fsS "${HOST}/api/chat" \
+        -H 'Content-Type: application/json' \
+        -d "$(jq -n --arg m "${MODEL}" --arg p "$prompt" '{
+            model: $m,
+            messages: [{role:"user", content:$p}],
+            stream: false
+        }')")"
+
+    eval_count="$(jq -r '.eval_count // 0' <<<"$resp")"
+    eval_ns="$(jq -r '.eval_duration // 0' <<<"$resp")"
+
+    if [[ "$eval_count" -eq 0 || "$eval_ns" -eq 0 ]]; then
+        red "[!] prompt $((i+1)) returned no timing data"
+        echo "$resp" | jq -r '.message.content // .' | head -3
+        exit 1
+    fi
+
+    eval_ms=$(( eval_ns / 1000000 ))
+    toks_per_s="$(jq -n --argjson c "$eval_count" --argjson n "$eval_ns" '($c / ($n / 1000000000)) | . * 100 | floor / 100')"
+    printf "%-4s  %8s  %12s  %8s\n" "$((i+1))" "$eval_count" "$eval_ms" "$toks_per_s"
+
+    TOTAL_TOKENS=$(( TOTAL_TOKENS + eval_count ))
+    TOTAL_NS=$(( TOTAL_NS + eval_ns ))
+done
+
+echo
+avg="$(jq -n --argjson c "$TOTAL_TOKENS" --argjson n "$TOTAL_NS" '($c / ($n / 1000000000)) | . * 100 | floor / 100')"
+green "[+] aggregate: ${TOTAL_TOKENS} tokens / $(( TOTAL_NS / 1000000 )) ms = ${avg} tok/s"