DSV4-Flash homunculus: scale-model instrument + qualification (median 1.4% prediction error across 10 configs)

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+h0-skeleton-f16.gguf filter=lfs diff=lfs merge=lfs -text
+h1-topology-f16.gguf filter=lfs diff=lfs merge=lfs -text
+h1-topology-mxfp4.gguf filter=lfs diff=lfs merge=lfs -text
+homunculus-poster.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,210 @@
+---
+license: mit
+tags:
+  - gguf
+  - llama.cpp
+  - benchmark
+  - infrastructure
+  - deepseek
+  - not-a-language-model
+library_name: gguf
+---
+
+# DSV4-Flash Homunculus
+
+**A scale model of DeepSeek-V4-Flash, for measuring inference rigs.**
+
+> ## ⚠️ The weights are random. This is not a language model.
+>
+> It has DeepSeek-V4-Flash's architecture and op graph, but every tensor is random and it was
+> never trained or distilled from anything. It will happily load and generate at 300 tokens/s,
+> and every one of those tokens will be garbage. **Do not download this expecting a small
+> DeepSeek-V4.** It is a test article — a piece of measuring equipment shaped like a model.
+>
+> The one thing it does carry from the real model is **the tokenizer**, verbatim (129,280
+> tokens) — see [License and provenance](#license-and-provenance).
+
+---
+
+## What it's for
+
+Tuning a large model across multiple machines is slow because every data point costs a model
+load. Changing one flag on a 145 GiB two-node setup costs five minutes before you learn
+anything, so a sweep of ten configurations is an hour of mostly waiting.
+
+The homunculus is the same graph with the width taken out: same operators, same expert count,
+same routing, same cross-device split — 606× less weight. A configuration that takes **316
+seconds** to measure on the real model takes **2.9 seconds** on this one.
+
+The question is whether the toy's answers transfer. On a two-node Strix Halo pair they do,
+with a stated and measured domain of validity.
+
+![qualification results](homunculus-poster.png)
+
+## The result
+
+Across 10 configurations measured on both models, predicted decode throughput for
+DeepSeek-V4-Flash landed within a **median 1.4%** of measured, **9 of 10 inside 5%**, with **no
+fitted parameters**:
+
+```
+predicted_tg = 1000 / (read_budget_gb / gbs * 1000 + n_layer_real * f_toy)
+```
+
+- `read_budget_gb` — bytes read per token, computed from the **real** model's tensor table.
+  Free: no GPU, no load, it is just arithmetic over the GGUF header.
+- `gbs` — the rig's memory bandwidth, measured once.
+- `n_layer_real` — the real model's layer count.
+- `f_toy` — per-layer overhead in ms, **measured on the homunculus in about three seconds**.
+
+| configuration | f toy | f real | predicted | measured | error |
+|---|---|---|---|---|---|
+| two-node, default | 240 µs | 238 µs | 16.28 | 16.29 | −0.0% |
+| expert parallel OFF | 252 µs | 253 µs | 16.14 | 16.13 | +0.1% |
+| two-node, `-ts 1/1` | 239 µs | 239 µs | 16.29 | 16.27 | +0.1% |
+| batch 4096 / 2048 | 239 µs | 239 µs | 16.29 | 16.27 | +0.1% |
+| `mmap` | 239 µs | 241 µs | 16.29 | 16.25 | +0.2% |
+| `mmap` + `mlock` | 239 µs | 218 µs | 16.29 | 16.52 | −1.4% |
+| direct I/O | 240 µs | 216 µs | 16.28 | 16.54 | −1.5% |
+| `--no-mmap` | 240 µs | 216 µs | 16.28 | 16.54 | −1.6% |
+| `mlock` | 240 µs | 216 µs | 16.28 | 16.54 | −1.6% |
+| expert parallel ON | 601 µs | 810 µs | 13.00 | 11.64 | **+11.7%** |
+
+Decode t/s, two nodes over 100GbE, `llama-bench` pp512/tg128, 3 reps.
+
+## Why it works
+
+A token costs `bytes / bandwidth + n_layer × f`.
+
+The first term is **bandwidth** and scales with the weights, so a toy cannot see it — but you
+do not need to measure it, because it falls out of the tensor table.
+
+The second is **latency per layer**: scheduler dispatch, the cross-device hop,
+`ggml_backend_synchronize`. That is a property of *the rig and the graph*, not of how many bytes
+the tensors happen to hold. So it should survive being scaled down — and it does:
+
+> **Fence per layer: 253 µs on the 144.3 GiB model, 252 µs on the 244 MiB homunculus.**
+> Same rig, same build. 0.4% apart.
+
+That agreement holds across regimes that could hardly be more different: the per-layer residual
+is 17.6% of the real model's token time and 92.8% of the toy's.
+
+## Where it fails
+
+Both failures are on the bandwidth side, which is the useful kind — you can tell in advance
+which questions not to ask it.
+
+**1. Overheads that are themselves proportional to bytes.** Expert parallelism is the worked
+example and the only prediction outside 5%. Its overhead is a cross-device *exchange of expert
+tensors*, so it scales with bytes moved; the toy's experts are 606× smaller, its exchange is
+nearly free, and it under-charges the fence (601 vs 810 µs/layer). It still gets the sign right
+— real EP/serial 0.721, toy 0.438, both say EP loses — so it would have correctly rejected
+expert parallelism without a single 145 GiB load. Trust the ranking, not the magnitude.
+
+**2. Anything measured in bytes per token.** Load mode moves real prefill by 8.0% and the toy by
+2.4% with no consistent ordering. At 244 MiB the weights are resident under every mode. Do not
+use this to size a quantization win or a load-mode change.
+
+## The models
+
+| | h0-skeleton | h1-topology | DeepSeek-V4-Flash-0731 |
+|---|---|---|---|
+| purpose | does it load at all | the qualified instrument | the real thing |
+| size | 177 MiB (f16) | 735 MiB f16 / **244 MiB MXFP4** | 144.3 GiB |
+| tensors | 150 | 364 | 1328 |
+| layers | 5 | 12 | 43 |
+| embedding dim | 256 | 256 | 4096 |
+| attention heads | 32 | 32 | 64 |
+| **KV heads (MLA)** | **1** | **1** | **1** |
+| **experts** | 32 | **256** | **256** |
+| **experts used** | **6** | **6** | **6** |
+| **shared experts** | **1** | **1** | **1** |
+| expert FFN | 128 | 128 | 2048 |
+| **hyper-connections** | **4** | **4** | **4** |
+| **hash layers** | **3** | **3** | **3** |
+| indexer heads | 4 | 4 | 64 |
+
+Bold rows are **preserved exactly** — they determine which operators run and how the graph
+branches. The unbolded rows are width, and width is what gets scaled away.
+
+**Use `h1-topology-mxfp4.gguf`** for measurement: it is the file every number above was produced
+with, and MXFP4 experts match how the real model is quantized. `h0-skeleton` exists to answer
+"does a generated `deepseek4` GGUF load and generate at all" and is not qualified for anything
+else.
+
+## Reproducing
+
+Requires a llama.cpp build with `deepseek4` support.
+
+```bash
+# one configuration, ~3 seconds
+python qualify.py run \
+  --model h1-topology-mxfp4.gguf \
+  --profile m6 --bench /path/to/llama-bench \
+  --rpc <peer>:50052 --straddle \
+  --roofline-gbs <your measured GB/s> \
+  --only rpc_split_even --out toy.json
+```
+
+`qualify.py` derives `f` per configuration and prints it. Feed that into the formula above with
+your real model's `read_budget_gb` (which `qualify.py` also computes, from the tensor table
+alone) to get a prediction.
+
+`gen_homunculus.py` builds these files. It has presets `h0`, `h1`, `h2`, and the parameters are
+plain constants — pointing it at a different architecture is the intended way to make a
+homunculus for something other than DSV4.
+
+```bash
+PYTHONPATH=/path/to/llama.cpp/gguf-py python gen_homunculus.py --preset h1 --out h1.gguf
+```
+
+### Two caveats for anyone reproducing
+
+- **`f` is normalised per layer, and h1 is 12 layers, not 43.** The predictor multiplies `f_toy`
+  by the *real* model's layer count. The toy does not preserve depth and does not need to.
+- **Expert-parallel configurations must run against a build that actually has the TP code.**
+  With a build that doesn't, the environment variables are silently inert and you measure a
+  plain layer split while believing you measured EP.
+
+## Files
+
+| file | what |
+|---|---|
+| `h1-topology-mxfp4.gguf` | the qualified instrument — use this one |
+| `h1-topology-f16.gguf` | same topology, unquantized |
+| `h0-skeleton-f16.gguf` | minimal load-and-generate test article |
+| `gen_homunculus.py` | builds homunculi from an architecture spec |
+| `qualify.py` | the measurement harness; runs the config sweep and derives `f` |
+| `assets/dsv4_tokenizer.json.gz` | the DSV4 tokenizer — `gen_homunculus.py` requires it |
+| `results/*.json` | raw `llama-bench` output behind every number on this page |
+| `homunculus-poster.png` | the qualification summary above |
+
+## Measurement conditions
+
+Two Strix Halo boxes (128 GB unified each), 100GbE, llama.cpp with two-node RPC. Roofline
+209.25 GB/s measured per node. Both nodes pinned to `dpm=high`. `llama-bench` pp512/tg128,
+3 repetitions. The real model is `DeepSeek-V4-Flash-0731`, MXFP4 routed experts with Q8_0 dense
+tensors, 144.3 GiB.
+
+Numbers here are specific to that rig. The *method* is not — the claim is about which terms
+survive scaling, and that argument is hardware-independent even though the constants are not.
+
+## License and provenance
+
+`gen_homunculus.py`, `qualify.py`, and the results are MIT, matching llama.cpp's `gguf-py`
+which the generator builds on. **All tensor data in the GGUFs is randomly generated** and carries
+nothing from DeepSeek-V4-Flash.
+
+Two things *are* taken from the real model, and neither is a weight:
+
+- **The tokenizer, verbatim** — 129,280 tokens, embedded in every GGUF here and shipped as
+  `assets/dsv4_tokenizer.json.gz` because the generator cannot run without it. It is
+  DeepSeek's, redistributed from DeepSeek-V4-Flash, and it is why `h0-skeleton` is 177 MiB
+  despite having only 5 layers: `token_embd` dominates a model this small.
+- **The config values** — layer counts, expert counts, and the hyper-connection / indexer
+  parameters, i.e. the numbers in the comparison table above.
+
+The tokenizer is present so that a homunculus tokenizes identically to the real model, which
+keeps prompt lengths and therefore batch shapes honest. Nothing about the measurement needs
+the tokenizer to be *this* tokenizer — any 129k-vocab tokenizer would give the same timings —
+so if its redistribution is inconvenient for you, substitute your own and regenerate.

assets/dsv4_tokenizer.json.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22a17c579c95db443cae029790254a47af4033c98711d1a39fa84d2d45d11e15
+size 1729594

gen_homunculus.py

@@ -0,0 +1,332 @@
+#!/usr/bin/env python3
+"""
+gen_homunculus.py -- emit a scale-model DeepSeek-V4 ("homunculus") GGUF.
+
+A homunculus is a toy DSV4: small enough to iterate on in minutes, but structurally
+complete -- every pathway the real model has (hyper-connections, MLA + sparse indexer,
+the 0/4/128 compressor pattern, hash-routed layers, 256-way MoE, the nextn/MTP block)
+is present so that parallelization findings transfer.
+
+Ground truth for the real model was read from the GGUF headers on m6 (2026-08-03):
+  main model  m6:/mnt/weights/models/dsv4-flash/dsv4-0731-MXFP4.gguf   block_count 43, no nextn
+  MTP draft   m6:/mnt/weights/models/dsv4-mtp/...-MXFP4_MOE.gguf       block_count 44, nextn 1
+See REAL below, and the akb parent card 2adc38bb65.
+
+Usage:
+  python3 gen_homunculus.py --preset h0 --out /path/to/homunculus-h0-f16.gguf
+  ../llama.cpp/build/bin/llama-quantize <f16.gguf> <mxfp4.gguf> MXFP4_MOE
+"""
+
+import argparse, gzip, json, os, sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "llama.cpp", "gguf-py"))
+import numpy as np
+import gguf
+
+ASSETS = os.path.join(os.path.dirname(os.path.abspath(__file__)), "assets")
+
+# --------------------------------------------------------------------------------------
+# Measured real-DSV4 configuration. Do not edit without re-reading the GGUF header.
+# --------------------------------------------------------------------------------------
+REAL = dict(
+    n_layer=43, n_layer_nextn=0,          # main file: 43 blocks, MTP ships separately
+    n_embd=4096, n_head=64, n_head_kv=1,
+    n_embd_head=512,                       # attention.key_length == value_length (MLA latent)
+    rope_dim=64, rope_freq_base=10000.0,
+    q_lora_rank=1024, o_group_count=8, o_lora_rank=1024,
+    n_expert=256, n_expert_used=6, n_expert_shared=1, n_ff_exp=2048,
+    expert_weights_scale=1.5, expert_weights_norm=True,
+    hc_mult=4, sinkhorn_iters=20, hc_eps=1e-6,
+    indexer_n_head=64, indexer_head_size=128, indexer_top_k=512,
+    hash_layer_count=3, sliding_window=128,
+    compress_rope_freq_base=160000.0, swiglu_clamp=10.0,
+    rms_eps=1e-6, n_ctx=1048576, n_vocab=129280,
+)
+
+# compress_ratios for the real 43-block trunk: two dense layers, then 4/128 alternating.
+# 4   -> compressor (coff=2) PLUS the full sparse-indexer stack
+# 128 -> compressor only (coff=1)
+# 0   -> no compressor at all
+REAL_RATIOS_43 = [0, 0] + [4 if i % 2 == 0 else 128 for i in range(2, 43)]
+assert len(REAL_RATIOS_43) == 43 and REAL_RATIOS_43[42] == 4
+
+
+def ratios_for(n_trunk, n_nextn):
+    """Real ratio pattern truncated/extended to n_trunk, with ratio 0 for nextn blocks."""
+    if n_trunk <= len(REAL_RATIOS_43):
+        r = REAL_RATIOS_43[:n_trunk]
+    else:
+        r = REAL_RATIOS_43 + [4 if i % 2 == 0 else 128 for i in range(43, n_trunk)]
+    # keep at least one of each variant so no pathway is lost to truncation
+    if n_trunk >= 4:
+        assert 0 in r and 4 in r and 128 in r, f"truncation lost a compressor variant: {r}"
+    return r + [0] * n_nextn
+
+
+# --------------------------------------------------------------------------------------
+# Presets. Every preset preserves the DIMENSIONLESS invariants exactly:
+#   hc_mult, n_expert/n_expert_used, n_expert_shared, hash_layer_count, the ratio pattern,
+#   n_ff_exp/n_embd, n_head*n_embd_head/n_embd, sinkhorn_iters, gating func.
+# Only absolute widths and depth scale down.
+# --------------------------------------------------------------------------------------
+def scaled(n_embd, n_layer, n_expert, *, n_nextn=0, n_vocab=None, name=""):
+    """Build a config by scaling widths off n_embd while holding real ratios."""
+    s = REAL["n_embd"] // n_embd                      # width divisor
+    assert REAL["n_embd"] % n_embd == 0, "n_embd must divide 4096 to keep ratios exact"
+    # HARD CONSTRAINT, measured: llama_kv_cache::build_input_k_rot derives the Hadamard
+    # rotation size with `nrot=64; do nrot*=2 while (n_embd_head_k_all % nrot == 0); nrot/=2`.
+    # That only resolves back to the head dim when the head dim is a multiple of 64. Anything
+    # smaller yields a 64x64 rotation against an N-dim head and aborts in ggml_reshape_2d.
+    n_embd_head = max(64, (REAL["n_embd_head"] // s) // 64 * 64)
+    # hold n_head*n_embd_head / n_embd == 8 (the real MLA expansion)
+    n_head = (8 * n_embd) // n_embd_head
+    rope_dim = max(4, (REAL["rope_dim"] * n_embd_head) // REAL["n_embd_head"])
+    rope_dim -= rope_dim % 2
+    cfg = dict(REAL)
+    cfg.update(
+        name=name,
+        n_embd=n_embd, n_layer=n_layer, n_layer_nextn=n_nextn,
+        n_head=n_head, n_embd_head=n_embd_head, rope_dim=rope_dim,
+        q_lora_rank=max(16, REAL["q_lora_rank"] // s),
+        o_lora_rank=max(16, REAL["o_lora_rank"] // s),
+        n_ff_exp=max(16, REAL["n_ff_exp"] // s),
+        n_expert=n_expert,
+        # indexer_head_size DOES NOT SCALE. For DSV4 the lightning-indexer KV cache always has
+        # attn_rot_k forced on, so it is subject to the same power-of-two Hadamard resolution as
+        # above; the real 128 is kept verbatim. 64 also works; below that it aborts.
+        indexer_head_size=REAL["indexer_head_size"],
+        indexer_n_head=max(4, REAL["indexer_n_head"] // s),
+        indexer_top_k=64,
+        n_ctx=8192,
+        n_vocab=n_vocab or REAL["n_vocab"],
+    )
+    # o_group_count must divide n_head*n_embd_head
+    while (n_head * n_embd_head) % cfg["o_group_count"] != 0:
+        cfg["o_group_count"] //= 2
+    return cfg
+
+
+PRESETS = {
+    # H0 -- skeleton. Does it wake up? Minimum dims, every pathway present.
+    "h0": lambda: scaled(n_embd=256, n_layer=5, n_expert=32, name="h0-skeleton"),
+    # H1 -- topology-true. Real expert count and ratios, ~1-2 GiB.
+    "h1": lambda: scaled(n_embd=256, n_layer=12, n_expert=256, name="h1-topology"),
+    # H2 -- latency-true. The REAL trunk depth (43), no nextn: the deployed main GGUF
+    # has block_count 43 and ships MTP as a separate draft file. Narrow.
+    "h2": lambda: scaled(n_embd=256, n_layer=43, n_expert=256, name="h2-latency"),
+}
+
+
+# --------------------------------------------------------------------------------------
+def load_tokenizer():
+    path = os.path.join(ASSETS, "dsv4_tokenizer.json.gz")
+    with gzip.open(path, "rt", encoding="utf-8") as f:
+        return json.load(f)
+
+
+def write_tokenizer(w, tok, n_vocab):
+    """Replay the real DSV4 tokenizer KVs. n_vocab must match the real vocab (129280);
+    truncating would desync merges from tokens, so we do not support it."""
+    real_n = len(tok["tokenizer.ggml.tokens"]["v"]["v"])
+    if n_vocab != real_n:
+        raise SystemExit(f"n_vocab {n_vocab} != real tokenizer size {real_n}; truncation not supported")
+    for k, ent in tok.items():
+        t, v = ent["t"], ent["v"]
+        if t == 9:  # array
+            w.add_array(k, v["v"])
+        elif t == 8:
+            w.add_string(k, v)
+        elif t == 7:
+            w.add_bool(k, v)
+        elif t == 4:
+            w.add_uint32(k, v)
+        elif t == 6:
+            w.add_float32(k, v)
+        else:
+            w.add_key_value(k, v, gguf.GGUFValueType(t))
+
+
+def add_kv(w, c, ratios):
+    n_all = c["n_layer"] + c["n_layer_nextn"]
+    w.add_context_length(c["n_ctx"])
+    w.add_embedding_length(c["n_embd"])
+    w.add_block_count(n_all)
+    w.add_head_count(c["n_head"])
+    w.add_head_count_kv(c["n_head_kv"])
+    w.add_key_length(c["n_embd_head"])
+    w.add_value_length(c["n_embd_head"])
+    w.add_rope_dimension_count(c["rope_dim"])
+    w.add_rope_freq_base(c["rope_freq_base"])
+    w.add_layer_norm_rms_eps(c["rms_eps"])
+    w.add_q_lora_rank(c["q_lora_rank"])
+    w.add_sliding_window(c["sliding_window"])
+    w.add_expert_count(c["n_expert"])
+    w.add_expert_used_count(c["n_expert_used"])
+    w.add_expert_shared_count(c["n_expert_shared"])
+    w.add_expert_feed_forward_length(c["n_ff_exp"])
+    w.add_expert_weights_scale(c["expert_weights_scale"])
+    w.add_expert_weights_norm(c["expert_weights_norm"])
+    w.add_expert_gating_func(gguf.ExpertGatingFuncType.SQRTSOFTPLUS)
+    w.add_swiglu_clamp_exp([c["swiglu_clamp"]] * n_all)
+    w.add_swiglu_clamp_shexp([c["swiglu_clamp"]] * n_all)
+    w.add_indexer_head_count(c["indexer_n_head"])
+    w.add_indexer_key_length(c["indexer_head_size"])
+    w.add_indexer_top_k(c["indexer_top_k"])
+    w.add_attention_output_group_count(c["o_group_count"])
+    w.add_attention_output_lora_rank(c["o_lora_rank"])
+    w.add_attention_compress_ratios(ratios)
+    w.add_attention_compress_rope_freq_base(c["compress_rope_freq_base"])
+    w.add_hyper_connection_count(c["hc_mult"])
+    w.add_hyper_connection_sinkhorn_iterations(c["sinkhorn_iters"])
+    w.add_hyper_connection_epsilon(c["hc_eps"])
+    w.add_hash_layer_count(c["hash_layer_count"])
+    if c["n_layer_nextn"]:
+        w.add_nextn_predict_layers(c["n_layer_nextn"])
+
+
+def build(cfg, out_path, seed=0, ratios=None):
+    rng = np.random.default_rng(seed)
+    c = cfg
+    n_all = c["n_layer"] + c["n_layer_nextn"]
+    ratios = ratios if ratios is not None else ratios_for(c["n_layer"], c["n_layer_nextn"])
+    hc_dim = c["hc_mult"] * c["n_embd"]
+    hc_mix = (2 + c["hc_mult"]) * c["hc_mult"]
+    n_qk = c["n_head"] * c["n_embd_head"]
+
+    w = gguf.GGUFWriter(out_path, "deepseek4")
+    w.add_name(f"homunculus-{c['name']}")
+    w.add_type(gguf.GGUFType.MODEL)
+    add_kv(w, c, ratios)
+    write_tokenizer(w, load_tokenizer(), c["n_vocab"])
+
+    def rnd(*ne, dtype=np.float16, std=0.02):
+        """ne is in ggml order (ne0, ne1, ...); numpy wants it reversed."""
+        return (rng.standard_normal(tuple(reversed(ne))) * std).astype(dtype)
+
+    def ones(*ne):
+        return np.ones(tuple(reversed(ne)), dtype=np.float32)
+
+    w.add_tensor("token_embd.weight", rnd(c["n_embd"], c["n_vocab"]))
+    w.add_tensor("output_norm.weight", ones(c["n_embd"]))
+    w.add_tensor("output.weight", rnd(c["n_embd"], c["n_vocab"]))
+    w.add_tensor("output_hc_fn.weight", rnd(hc_dim, c["hc_mult"], dtype=np.float32))
+    w.add_tensor("output_hc_base.weight", ones(c["hc_mult"]))
+    w.add_tensor("output_hc_scale.weight", ones(1))
+
+    for i in range(n_all):
+        p = f"blk.{i}."
+        w.add_tensor(p + "attn_norm.weight", ones(c["n_embd"]))
+        w.add_tensor(p + "attn_sinks.weight", np.zeros(c["n_head"], dtype=np.float32))
+        w.add_tensor(p + "attn_q_a.weight", rnd(c["n_embd"], c["q_lora_rank"]))
+        w.add_tensor(p + "attn_q_a_norm.weight", ones(c["q_lora_rank"]))
+        w.add_tensor(p + "attn_q_b.weight", rnd(c["q_lora_rank"], n_qk))
+        w.add_tensor(p + "attn_kv.weight", rnd(c["n_embd"], c["n_embd_head"]))
+        w.add_tensor(p + "attn_kv_a_norm.weight", ones(c["n_embd_head"]))
+        w.add_tensor(p + "attn_output_a.weight",
+                     rnd(n_qk // c["o_group_count"], c["o_lora_rank"] * c["o_group_count"]))
+        w.add_tensor(p + "attn_output_b.weight",
+                     rnd(c["o_group_count"] * c["o_lora_rank"], c["n_embd"]))
+
+        for tag in ("attn", "ffn"):
+            w.add_tensor(p + f"hc_{tag}_fn.weight", rnd(hc_dim, hc_mix, dtype=np.float32))
+            w.add_tensor(p + f"hc_{tag}_base.weight", np.zeros(hc_mix, dtype=np.float32))
+            w.add_tensor(p + f"hc_{tag}_scale.weight", ones(3))
+
+        ratio = ratios[i]
+        if ratio != 0:
+            coff = 2 if ratio == 4 else 1
+            cd = coff * c["n_embd_head"]
+            w.add_tensor(p + "attn_compressor_kv.weight", rnd(c["n_embd"], cd))
+            w.add_tensor(p + "attn_compressor_gate.weight", rnd(c["n_embd"], cd))
+            w.add_tensor(p + "attn_compressor_ape.weight", rnd(cd, ratio, dtype=np.float32))
+            w.add_tensor(p + "attn_compressor_norm.weight", ones(c["n_embd_head"]))
+            if ratio == 4:
+                ihs = c["indexer_head_size"]
+                w.add_tensor(p + "indexer.proj.weight", rnd(c["n_embd"], c["indexer_n_head"]))
+                w.add_tensor(p + "indexer.attn_q_b.weight",
+                             rnd(c["q_lora_rank"], c["indexer_n_head"] * ihs))
+                w.add_tensor(p + "indexer_compressor_kv.weight", rnd(c["n_embd"], 2 * ihs))
+                w.add_tensor(p + "indexer_compressor_gate.weight", rnd(c["n_embd"], 2 * ihs))
+                w.add_tensor(p + "indexer_compressor_ape.weight", rnd(2 * ihs, ratio, dtype=np.float32))
+                w.add_tensor(p + "indexer_compressor_norm.weight", ones(ihs))
+
+        w.add_tensor(p + "ffn_gate_inp.weight", rnd(c["n_embd"], c["n_expert"]))
+        if i < c["hash_layer_count"]:
+            # deterministic token-id -> expert-id table, I32, never quantized
+            tid2eid = rng.integers(0, c["n_expert"],
+                                   size=(c["n_vocab"], c["n_expert_used"]), dtype=np.int32)
+            w.add_tensor(p + "ffn_gate_tid2eid.weight", tid2eid)
+        else:
+            w.add_tensor(p + "exp_probs_b.bias", np.zeros(c["n_expert"], dtype=np.float32))
+        w.add_tensor(p + "ffn_norm.weight", ones(c["n_embd"]))
+
+        ne, nf, nx = c["n_embd"], c["n_ff_exp"], c["n_expert"]
+        w.add_tensor(p + "ffn_gate_exps.weight", rnd(ne, nf, nx))
+        w.add_tensor(p + "ffn_down_exps.weight", rnd(nf, ne, nx))
+        w.add_tensor(p + "ffn_up_exps.weight", rnd(ne, nf, nx))
+
+        nsh = nf * c["n_expert_shared"]
+        w.add_tensor(p + "ffn_gate_shexp.weight", rnd(ne, nsh))
+        w.add_tensor(p + "ffn_down_shexp.weight", rnd(nsh, ne))
+        w.add_tensor(p + "ffn_up_shexp.weight", rnd(ne, nsh))
+
+        if i >= c["n_layer"]:
+            w.add_tensor(p + "nextn.eh_proj.weight", rnd(2 * c["n_embd"], c["n_embd"]))
+            w.add_tensor(p + "nextn.enorm.weight", ones(c["n_embd"]))
+            w.add_tensor(p + "nextn.hnorm.weight", ones(c["n_embd"]))
+
+    w.write_header_to_file()
+    w.write_kv_data_to_file()
+    w.write_tensors_to_file()
+    w.close()
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--preset", required=True, choices=sorted(PRESETS))
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--seed", type=int, default=0)
+    # bisect / override knobs -- for isolating which structural pathway breaks a build
+    ap.add_argument("--ratios", help="'dense' (all 0), 'hca' (all 128), 'csa' (all 4), or a comma list")
+    ap.add_argument("--rope-dim", type=int)
+    ap.add_argument("--head-dim", type=int)
+    ap.add_argument("--n-layer", type=int)
+    ap.add_argument("--nextn", type=int, default=0,
+                    help="append N nextn/MTP blocks (default 0 -- spec decode is out of scope; "
+                         "the real main GGUF has none)")
+    a = ap.parse_args()
+
+    cfg = PRESETS[a.preset]()
+    if a.n_layer:
+        cfg["n_layer"] = a.n_layer
+    cfg["n_layer_nextn"] = a.nextn
+    if a.head_dim:
+        cfg["n_embd_head"] = a.head_dim
+        cfg["n_head"] = (8 * cfg["n_embd"]) // a.head_dim
+        while (cfg["n_head"] * a.head_dim) % cfg["o_group_count"] != 0:
+            cfg["o_group_count"] //= 2
+    if a.rope_dim:
+        cfg["rope_dim"] = a.rope_dim
+
+    n_all = cfg["n_layer"] + cfg["n_layer_nextn"]
+    if a.ratios:
+        fixed = {"dense": 0, "hca": 128, "csa": 4}.get(a.ratios)
+        if fixed is not None:
+            ratios = [fixed] * cfg["n_layer"] + [0] * cfg["n_layer_nextn"]
+        else:
+            ratios = [int(x) for x in a.ratios.split(",")]
+            assert len(ratios) == n_all, f"--ratios needs {n_all} entries"
+    else:
+        ratios = ratios_for(cfg["n_layer"], cfg["n_layer_nextn"])
+    print(f"preset {a.preset}: n_embd={cfg['n_embd']} n_head={cfg['n_head']} "
+          f"head_dim={cfg['n_embd_head']} rope_dim={cfg['rope_dim']} "
+          f"n_layer={cfg['n_layer']}+{cfg['n_layer_nextn']}nextn "
+          f"experts={cfg['n_expert']}/{cfg['n_expert_used']} n_ff_exp={cfg['n_ff_exp']} "
+          f"hc_mult={cfg['hc_mult']} o_groups={cfg['o_group_count']}")
+    print(f"  ratios: {ratios}")
+    build(cfg, a.out, a.seed, ratios)
+    print(f"  wrote {a.out} ({os.path.getsize(a.out)/2**20:.1f} MiB)")
+
+
+if __name__ == "__main__":
+    main()

h0-skeleton-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56973c8573d810c69c77535953e94d92f438a8ba3f7255f99ab0a3c66cb3b9ac
+size 185779328

h1-topology-f16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c20c2ef57dcb4e2ab6b5fdb32ea27b4d3747897db17df432767b835660d4faae
+size 770349760

h1-topology-mxfp4.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4c00f22850c7324d0395400b7dd7a88b0e587ed76c35f8ac0094c2aae57b8a24
+size 255728448

qualify.py

@@ -0,0 +1,633 @@
+#!/usr/bin/env python3
+"""
+qualify.py -- the homunculus qualification harness.
+
+Runs a fixed set of parallelization configurations against ONE model via llama-bench, and
+records throughput per configuration. Run it on a homunculus, run it on real DSV4, then
+`compare` the two result files.
+
+THE CRITERION IS A RELATIONSHIP, NOT AN ABSOLUTE. A toy will never reproduce DSV4's t/s and
+trying to fit that would just be chasing the bandwidth term, which is analytic anyway. What
+qualifies a homunculus is that its per-configuration CHANGES track the real model's changes:
+same ordering, and ideally a stable scale factor between the two sets of speedups, so a toy
+delta PREDICTS a real delta.
+
+Everything here is stdlib-only (no numpy) so it runs unchanged on m5/m6 system python.
+
+  # measure
+  python3 qualify.py run --model h1-mxfp4.gguf --profile bigboy --out toy-h1.json
+  python3 qualify.py run --model dsv4-0731-MXFP4.gguf --profile m6 --out real.json
+
+  # qualify
+  python3 qualify.py compare --real real.json --toy toy-h1.json [--toy toy-h2.json ...]
+"""
+
+import argparse, json, math, os, re, shutil, statistics, subprocess, sys, time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+DEFAULT_BENCH = os.path.join(HERE, "..", "llama.cpp", "build", "bin", "llama-bench")
+
+# --------------------------------------------------------------------------------------
+# Configurations. The NAMES are the contract -- a config name means the same thing on the
+# toy and on the real model, which is what makes the deltas comparable.
+#
+# `needs` gates a config on host capability:
+#   gpu     -- at least one offload device
+#   dev2    -- at least two devices to split across
+#   rpc     -- an --rpc endpoint was supplied
+# --------------------------------------------------------------------------------------
+CONFIGS = {
+    # --- baselines -------------------------------------------------------------------
+    "cpu_only":        dict(args=["-ngl", "0"], needs=[]),
+    "gpu_all":         dict(args=["-ngl", "999"], needs=["gpu"]),
+
+    # --- expert placement: the axis that produced the real 2.58 -> 4.0 t/s jump -------
+    "experts_cpu_ot":  dict(args=["-ngl", "999", "-ot", "exps=CPU"], needs=["gpu"]),
+    "experts_cpu_all": dict(args=["-ngl", "999", "-ncmoe", "999"], needs=["gpu"]),
+    "experts_cpu_half":dict(args=["-ngl", "999", "-ncmoe", "{half_layers}"], needs=["gpu"]),
+
+    # --- single-node multi-device split ----------------------------------------------
+    "dev_split_even":  dict(args=["-ngl", "999", "-dev", "{dev0},{dev1}", "-ts", "1/1"],
+                            needs=["dev2"]),
+    "dev_split_skew":  dict(args=["-ngl", "999", "-dev", "{dev0},{dev1}", "-ts", "3/1"],
+                            needs=["dev2"]),
+
+    # --- load path: the mmap-vs-resident axis that dominates single-node m5 -----------
+    # Full --load-mode sweep. -lm none is what the meld serve actually uses (it passes the
+    # deprecated --no-mmap). mmap+mlock beat every other config on BOTH metrics in the first
+    # real run (+1.5 pct tg, +7.6 pct pp vs default), so the serve's choice is worth re-testing.
+    # -lm dio is recorded as a dead end in the DSV4 memory; kept so the toy can be checked
+    # against a KNOWN-BAD config, which is as useful as checking it against a known-good one.
+    "load_none":       dict(args=["-ngl", "999", "-lm", "none"], needs=["gpu"]),
+    "load_mmap":       dict(args=["-ngl", "999", "-lm", "mmap"], needs=["gpu"]),
+    "load_mlock_only": dict(args=["-ngl", "999", "-lm", "mlock"], needs=["gpu"]),
+    "load_mlock":      dict(args=["-ngl", "999", "-lm", "mmap+mlock"], needs=["gpu"]),
+    "load_dio":        dict(args=["-ngl", "999", "-lm", "dio"], needs=["gpu"]),
+
+    # --- batch shape: the -ub 256 -b 256 axis from the Nemotron two-node run ----------
+    "batch_small":     dict(args=["-ngl", "999", "-b", "256", "-ub", "256"], needs=["gpu"]),
+    "batch_large":     dict(args=["-ngl", "999", "-b", "2048", "-ub", "512"], needs=["gpu"]),
+
+    # --- two-node RPC layer split -----------------------------------------------------
+    # This is the CURRENT PRODUCTION SHAPE and the bar every other strategy has to clear.
+    # Measured 2026-08-03 on real DSV4: 15.14 t/s serial layer-split; EP-only lost at 13.85.
+    "rpc_split_even":  dict(args=["-ngl", "999", "--rpc", "{rpc}", "-ts", "1/1"], needs=["rpc"]),
+    "rpc_split_local": dict(args=["-ngl", "999", "--rpc", "{rpc}", "-ts", "3/1"], needs=["rpc"]),
+    "rpc_split_remote":dict(args=["-ngl", "999", "--rpc", "{rpc}", "-ts", "1/3"], needs=["rpc"]),
+
+    # --- tensor parallel --------------------------------------------------------------
+    # -ts 0,1 NOT 1,0: model.devices puts RPC0 FIRST even though --list-devices prints
+    # Vulkan0 first. That gotcha cost session 0cc8d352 twenty minutes; do not re-derive it.
+    # TP wins were pure graph-node ORDERING, which is exactly the fence term a toy measures:
+    #   Cydonia-24B  serial-order TP 10.83 -> join-on-peer 16.49 -> +uid graph cache 17.21 t/s
+    # --- expert parallelism: the axis with a KNOWN real answer -------------------------
+    # Real DSV4 measured serial 15.14 vs EP 13.85 t/s, i.e. EP = 0.915x. EP LOSES. If the toy
+    # reproduces that ordering the instrument is qualified for the fence/topology axis; if the
+    # toy says EP wins, it is not. Both are memory-safe on the real model (EP measured m5 68.4 /
+    # m6 77.6 GiB) so this is the comparison the sweep was missing.
+    # REQUIRES build-ep: build-ds is branch ds-merged and contains NO TP code, so LLAMA_TP_EP
+    # would be an inert env var and this would silently measure plain -ts 0/1 instead.
+    # guard_exempt: -ts 0/1 nominally puts the whole model on one device, but EP then moves half
+    # the routed experts to the peer, so the naive -ts share guard would wrongly reject it.
+    "ep_serial":       dict(args=["-ngl", "999", "--rpc", "{rpc}", "-sm", "layer", "-ts", "1/1"],
+                            needs=["rpc"]),
+    "ep_split":        dict(args=["-ngl", "999", "--rpc", "{rpc}", "-sm", "layer", "-ts", "0/1"],
+                            env={"LLAMA_TP": "1", "LLAMA_TP_EP": "1",
+                                 "LLAMA_TP_JOIN": "cpu", "GGML_RPC_PUSH": "1"},
+                            guard_exempt=True, needs=["rpc"]),
+
+    "tp_join_peer":    dict(args=["-ngl", "999", "--rpc", "{rpc}", "-ts", "0/1"],
+                            env={"LLAMA_TP": "1"}, needs=["rpc", "tp"]),
+    # LLAMA_TP_JOIN=cpu reruns the measured-NEGATIVE CPU-join experiment (16.98 vs 17.21).
+    # Kept because reproducing a known loss is how we validate the instrument.
+    "tp_join_cpu":     dict(args=["-ngl", "999", "--rpc", "{rpc}", "-ts", "0/1"],
+                            env={"LLAMA_TP": "1", "LLAMA_TP_JOIN": "cpu"}, needs=["rpc", "tp"]),
+}
+
+# Host profiles. `devs` are llama-bench -dev names; discover them with --list-devices.
+#   caps  -- which config classes this host can run
+#   env   -- applied to every config on this host
+#
+# bigboy_cpu exists because of a MEASURED bigboy limitation (2026-08-03): any DSV4-arch model
+# aborts the moment the CUDA backend is merely PRESENT. The lightning indexer's Hadamard op
+# (ggml_cuda_op_fwht) fails inside ggml_cuda_kernel_can_use_pdl on compute capability 8.6 --
+# PDL is a Hopper-era feature. -ngl 0 is NOT enough, the op still gets scheduled to CUDA; the
+# device has to be hidden outright. So bigboy generates homunculi, it does not measure them.
+PROFILES = {
+    "bigboy":     dict(caps=["gpu"], devs=["CUDA0"], env={}),
+    "bigboy_cpu": dict(caps=[], devs=[], env={"CUDA_VISIBLE_DEVICES": ""}),
+    "m5":         dict(caps=["gpu", "dev2", "tp"], devs=["Vulkan0", "Vulkan1"], env={}),
+    "m6":         dict(caps=["gpu", "dev2", "tp"], devs=["Vulkan0", "Vulkan1"], env={}),
+    "cpu":        dict(caps=[], devs=[], env={}),
+}
+
+
+# --------------------------------------------------------------------------------------
+# GGUF introspection + the analytic bandwidth term.
+#
+# The q8dense result (2026-08-03) proved the bandwidth term is predictable from the tensor
+# table to within 0.1 percent: read budget 11.594 -> 10.71 GB/token predicted +8.2 percent,
+# measured +8.1 percent. So we COMPUTE it here rather than measuring it, and spend measurement
+# only on the fence term, which is the part a toy can actually reproduce.
+#
+# Pure stdlib -- this has to run on m5/m6 system python, which has no numpy (so no gguf-py).
+# --------------------------------------------------------------------------------------
+QUANT_SIZES = {  # ggml type -> (block_size, bytes_per_block)
+    0: (1, 4), 1: (1, 2), 2: (32, 18), 3: (32, 20), 6: (32, 22), 7: (32, 24),
+    8: (32, 34), 9: (32, 36), 10: (256, 84), 11: (256, 110), 12: (256, 144),
+    13: (256, 176), 14: (256, 210), 15: (256, 292), 24: (1, 1), 25: (1, 2),
+    26: (1, 4), 27: (1, 8), 28: (1, 8), 30: (1, 2), 39: (32, 17),
+}
+_FMT = {0: ("<B", 1), 1: ("<b", 1), 2: ("<H", 2), 3: ("<h", 2), 4: ("<I", 4), 5: ("<i", 4),
+        6: ("<f", 4), 7: ("<?", 1), 10: ("<Q", 8), 11: ("<q", 8), 12: ("<d", 8)}
+
+
+def gguf_meta(path):
+    """Parse a GGUF header: selected KVs + the full tensor table. No numpy."""
+    import struct
+    with open(path, "rb") as f:
+        def raw(n):
+            return f.read(n)
+
+        def scalar(t):
+            fmt, n = _FMT[t]
+            return struct.unpack(fmt, raw(n))[0]
+
+        def string():
+            return raw(scalar(10)).decode("utf-8", "replace")
+
+        def value(t):
+            if t == 8:
+                return string()
+            if t == 9:
+                et, n = scalar(4), scalar(10)
+                return [value(et) for _ in range(n)]
+            return scalar(t)
+
+        assert raw(4) == b"GGUF", "not a GGUF file"
+        scalar(4)
+        n_tensors, n_kv = scalar(10), scalar(10)
+        kv = {}
+        for _ in range(n_kv):
+            k = string()
+            t = scalar(4)
+            v = value(t)
+            if not k.startswith("tokenizer."):   # skip the bulky vocab arrays
+                kv[k] = v
+        tensors = []
+        for _ in range(n_tensors):
+            name = string()
+            ne = [scalar(10) for _ in range(scalar(4))]
+            ttype = scalar(4)
+            scalar(10)  # offset
+            tensors.append((name, ne, ttype))
+    return dict(kv=kv, tensors=tensors)
+
+
+def tensor_bytes(ne, ttype):
+    n = 1
+    for d in ne:
+        n *= d
+    blk, size = QUANT_SIZES.get(ttype, (1, 4))
+    return n // blk * size if n % blk == 0 else (n + blk - 1) // blk * size
+
+
+def read_budget_gb(meta):
+    """Bytes read per decoded token, in GB.
+
+    Accounting matches the q8dense analysis exactly:
+      - token_embd is a ROW GATHER and does NOT count
+      - ffn_gate_tid2eid is likewise indexed by token id -- does not count
+      - routed experts count only n_expert_used / n_expert of their bytes
+      - everything else (including output.weight) is read every token
+    """
+    kv = meta["kv"]
+    arch = kv.get("general.architecture", "")
+    n_exp = kv.get(f"{arch}.expert_count", 0) or 0
+    n_used = kv.get(f"{arch}.expert_used_count", 0) or 0
+    frac = (n_used / n_exp) if n_exp else 1.0
+    total = 0
+    for name, ne, ttype in meta["tensors"]:
+        if name.startswith("token_embd") or "tid2eid" in name:
+            continue
+        b = tensor_bytes(ne, ttype)
+        if "_exps." in name:
+            b *= frac
+        total += b
+    return total / 1e9
+
+
+def n_layer_of(meta):
+    kv = meta["kv"]
+    arch = kv.get("general.architecture", "")
+    return kv.get(f"{arch}.block_count", 0) or 0
+
+
+def fence_cost_ms(tg, n_layer, budget_gb, roofline_gbs):
+    """Per-layer fence cost: the scale-invariant term a homunculus can actually reproduce.
+
+        f = (measured ms/token  -  analytic read time)  /  n_layer
+
+    Returns None when it cannot be derived. A NEGATIVE f means the roofline is wrong for this
+    host (or the run was not bandwidth-bound), not that fences are free -- report it, do not
+    clamp it.
+
+    READ THE NAME CAREFULLY. This is a RESIDUAL: everything per layer that is not weight reads.
+    On a two-node split that residual really is dominated by fences (RTTs, graph serialize,
+    submit latency), which is the quantity we claim transfers from toy to real. On a SINGLE-NODE
+    config there are no cross-node fences at all, so f there is compute + kernel launch, and it
+    is NOT comparable to a two-node f. Only compare f within the same topology class.
+    """
+    if not tg or not n_layer or not budget_gb or not roofline_gbs:
+        return None
+    measured_ms = 1000.0 / tg
+    reads_ms = budget_gb / roofline_gbs * 1000.0
+    return (measured_ms - reads_ms) / n_layer
+
+
+def substitute(args, subs):
+    out = []
+    for a in args:
+        for k, v in subs.items():
+            a = a.replace("{" + k + "}", str(v))
+        out.append(a)
+    return out
+
+
+def straddle(args, rpc):
+    """Force a config onto BOTH nodes.
+
+    Real DSV4 is 145.6 GiB and does not fit one 112 GiB device, so any single-node config
+    (gpu_all, batch_*, load_*, experts_*) is TOY-ONLY and can never appear in a toy-vs-real
+    ranking. But nothing about those axes REQUIRES one node - batch shape and load mode are
+    orthogonal to the split. Adding the two-node base makes the whole sweep comparable
+    instead of throwing twelve of thirteen configs away.
+
+    Configs that already name --rpc keep their own split. Configs that drive a
+    single-host multi-device split (-dev) are left alone: they are a different axis.
+    """
+    if "--rpc" in args or "-dev" in args:
+        return args
+    return ["--rpc", rpc, "-ts", "1/1"] + args
+
+
+_TS_RE = re.compile(r"^\s*(\d+)\s*[/,]\s*(\d+)\s*$")
+
+
+def device_share_gib(args, model_gib):
+    """Largest per-device share this config implies, in GiB.
+
+    -ts 3/1 on a 145.6 GiB model puts 109.2 GiB on ONE device against ~112 GiB of usable
+    GTT. That is the condition that wedged m5 on 2026-08-04 and cost a physical power
+    cycle. Toy-safe does not imply real-safe and the harness could not previously tell.
+    """
+    ts = None
+    for i, x in enumerate(args):
+        if x == "-ts" and i + 1 < len(args):
+            ts = args[i + 1]
+    if ts is None:
+        return model_gib          # unsplit: the whole model lands on one device
+    m = _TS_RE.match(ts)
+    if not m:
+        return model_gib
+    a, b = int(m.group(1)), int(m.group(2))
+    if a + b == 0:
+        return model_gib
+    return model_gib * max(a, b) / (a + b)
+
+
+def runnable(cfg, caps, have_rpc):
+    for need in cfg["needs"]:
+        if need == "rpc":
+            if not have_rpc:
+                return False
+        elif need not in caps:
+            return False
+    return True
+
+
+def bench_one(bench, model, args, n_prompt, n_gen, reps, timeout, env=None):
+    cmd = [bench, "-m", model, "-p", str(n_prompt), "-n", str(n_gen),
+           "-r", str(reps), "-o", "json"] + args
+    runenv = dict(os.environ)
+    runenv.update(env or {})
+    t0 = time.time()
+    p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=runenv)
+    wall = time.time() - t0
+    if p.returncode != 0:
+        return dict(ok=False, error=(p.stderr or p.stdout)[-800:], wall=wall, cmd=cmd)
+    try:
+        recs = json.loads(p.stdout)
+    except json.JSONDecodeError:
+        return dict(ok=False, error="unparseable llama-bench json: " + p.stdout[-400:],
+                    wall=wall, cmd=cmd)
+    out = dict(ok=True, wall=wall, cmd=cmd, pp=None, tg=None)
+    for r in recs:
+        ts = r.get("avg_ts")
+        if r.get("n_prompt", 0) > 0 and r.get("n_gen", 0) == 0:
+            out["pp"] = ts
+        elif r.get("n_gen", 0) > 0:
+            out["tg"] = ts
+    return out
+
+
+def cmd_run(a):
+    bench = a.bench or DEFAULT_BENCH
+    if not os.path.exists(bench):
+        sys.exit(f"llama-bench not found at {bench} (build it, or pass --bench)")
+    prof = PROFILES[a.profile]
+    caps = list(prof["caps"])
+    devs = prof["devs"]
+    subs = dict(
+        dev0=devs[0] if devs else "",
+        dev1=devs[1] if len(devs) > 1 else "",
+        rpc=a.rpc or "",
+        half_layers=a.half_layers,
+    )
+
+    # analytic side: compute the read budget from the tensor table rather than measuring it
+    meta = gguf_meta(a.model)
+    n_layer = n_layer_of(meta)
+    budget = a.read_budget_gb or read_budget_gb(meta)
+    print(f"model: {os.path.basename(a.model)}  arch={meta['kv'].get('general.architecture')} "
+          f"n_layer={n_layer}")
+    print(f"analytic read budget: {budget:.4f} GB/token"
+          + ("" if a.read_budget_gb else " (computed from the tensor table)"))
+    if a.roofline_gbs:
+        print(f"roofline {a.roofline_gbs} GB/s -> {budget/a.roofline_gbs*1000:.2f} ms/token of reads; "
+              f"fence cost f will be derived per config")
+    else:
+        print("no --roofline-gbs given: fence cost f cannot be derived, only raw t/s recorded")
+
+    only = set(a.only.split(",")) if a.only else None
+
+    # total model size on disk, for the per-device memory guard
+    try:
+        model_gib = os.path.getsize(a.model) / (1024 ** 3)
+    except OSError:
+        model_gib = 0.0
+    if a.straddle and not a.rpc:
+        sys.exit("--straddle needs --rpc: it works by adding the two-node base to every config")
+    if a.straddle:
+        print(f"straddle: single-node configs get --rpc {a.rpc} -ts 1/1 so they are "
+              f"comparable against a model that cannot fit one device")
+    if a.max_device_gib:
+        print(f"memory guard: skipping any config whose largest per-device share exceeds "
+              f"{a.max_device_gib} GiB (model is {model_gib:.1f} GiB)")
+
+    results, skipped = {}, {}
+    for name, cfg in CONFIGS.items():
+        if only and name not in only:
+            continue
+        if not runnable(cfg, caps, bool(a.rpc)):
+            skipped[name] = "host lacks " + ",".join(cfg["needs"])
+            continue
+        args = substitute(cfg["args"], subs)
+        if a.straddle:
+            args = straddle(args, a.rpc)
+        if a.max_device_gib and model_gib and not cfg.get("guard_exempt"):
+            share = device_share_gib(args, model_gib)
+            if share > a.max_device_gib:
+                skipped[name] = (f"per-device share {share:.1f} GiB > "
+                                 f"--max-device-gib {a.max_device_gib}")
+                continue
+        env = dict(prof.get("env") or {})
+        env.update(cfg.get("env") or {})
+        print(f"  {name:18s} ...", end="", flush=True)
+        try:
+            r = bench_one(bench, a.model, args, a.n_prompt, a.n_gen, a.reps, a.timeout, env)
+        except subprocess.TimeoutExpired:
+            r = dict(ok=False, error=f"timeout after {a.timeout}s", cmd=None)
+        r["env"] = env
+        r["fence_ms_per_layer"] = fence_cost_ms(r.get("tg"), n_layer, budget, a.roofline_gbs)
+        results[name] = r
+        if r["ok"]:
+            f = r["fence_ms_per_layer"]
+            print(f" pp={r.get('pp')} tg={r.get('tg')}"
+                  + (f" f={f*1000:.0f}us/layer" if f is not None else ""))
+        else:
+            print(f" FAILED: {r['error'][:120]}")
+
+    doc = dict(
+        label=a.label or os.path.basename(a.model),
+        model=os.path.abspath(a.model),
+        profile=a.profile, rpc=a.rpc,
+        n_prompt=a.n_prompt, n_gen=a.n_gen, reps=a.reps,
+        host=os.uname().nodename,
+        arch=meta["kv"].get("general.architecture"),
+        n_layer=n_layer,
+        read_budget_gb=budget,
+        roofline_gbs=a.roofline_gbs,
+        results=results, skipped=skipped,
+    )
+    with open(a.out, "w") as f:
+        json.dump(doc, f, indent=2)
+    print(f"\nwrote {a.out}  ({sum(1 for r in results.values() if r['ok'])} ok, "
+          f"{sum(1 for r in results.values() if not r['ok'])} failed, {len(skipped)} skipped)")
+    if skipped:
+        for k, v in skipped.items():
+            print(f"  SKIPPED {k}: {v}")
+
+
+# --------------------------------------------------------------------------------------
+# statistics -- pure python, no numpy (must run on m5/m6 system python)
+# --------------------------------------------------------------------------------------
+def ranks(xs):
+    order = sorted(range(len(xs)), key=lambda i: xs[i])
+    r = [0.0] * len(xs)
+    i = 0
+    while i < len(order):
+        j = i
+        while j + 1 < len(order) and xs[order[j + 1]] == xs[order[i]]:
+            j += 1
+        avg = (i + j) / 2.0 + 1.0
+        for k in range(i, j + 1):
+            r[order[k]] = avg
+        i = j + 1
+    return r
+
+
+def pearson(xs, ys):
+    n = len(xs)
+    if n < 2:
+        return None
+    mx, my = statistics.fmean(xs), statistics.fmean(ys)
+    num = sum((x - mx) * (y - my) for x, y in zip(xs, ys))
+    dx = math.sqrt(sum((x - mx) ** 2 for x in xs))
+    dy = math.sqrt(sum((y - my) ** 2 for y in ys))
+    if dx == 0 or dy == 0:
+        return None
+    return num / (dx * dy)
+
+
+def spearman(xs, ys):
+    return pearson(ranks(xs), ranks(ys))
+
+
+def linfit(xs, ys):
+    """least-squares slope/intercept of ys ~ a*xs + b"""
+    n = len(xs)
+    if n < 2:
+        return None, None
+    mx, my = statistics.fmean(xs), statistics.fmean(ys)
+    den = sum((x - mx) ** 2 for x in xs)
+    if den == 0:
+        return None, None
+    a = sum((x - mx) * (y - my) for x, y in zip(xs, ys)) / den
+    return a, my - a * mx
+
+
+def series(doc, metric):
+    return {k: v[metric] for k, v in doc["results"].items()
+            if v.get("ok") and v.get(metric) is not None}
+
+
+def cmd_compare(a):
+    real = json.load(open(a.real))
+    toys = [json.load(open(p)) for p in a.toy]
+
+    for metric in ("tg", "pp"):
+        rs = series(real, metric)
+        if not rs:
+            continue
+        print(f"\n{'='*78}\nmetric: {metric}   real = {real['label']} on {real['host']}")
+        print(f"{'='*78}")
+        for toy in toys:
+            ts = series(toy, metric)
+            common = [k for k in CONFIGS if k in rs and k in ts]
+            if len(common) < 3:
+                print(f"\n{toy['label']}: only {len(common)} shared configs -- not enough to qualify")
+                continue
+
+            # baseline-relative speedups make the comparison scale-free
+            base = a.baseline if a.baseline in common else common[0]
+            rsp = [rs[k] / rs[base] for k in common]
+            tsp = [ts[k] / ts[base] for k in common]
+            lr = [math.log(v) for v in rsp]
+            lt = [math.log(v) for v in tsp]
+
+            sp = spearman([rs[k] for k in common], [ts[k] for k in common])
+            pe = pearson(lt, lr)
+            slope, _ = linfit(lt, lr)
+
+            print(f"\n{toy['label']}  ({toy['host']}, {len(common)} shared configs, baseline={base})")
+            print(f"  rank correlation (ordering) : {fmt(sp)}")
+            print(f"  log-speedup correlation     : {fmt(pe)}   <- is there a RELATIONSHIP")
+            print(f"  log-speedup slope           : {fmt(slope)}   <- 1.0 = toy deltas are real deltas")
+
+            # per-config table + inversions against the real ordering
+            real_order = sorted(common, key=lambda k: -rs[k])
+            toy_order = sorted(common, key=lambda k: -ts[k])
+            rf = {k: real["results"][k].get("fence_ms_per_layer") for k in common}
+            tf = {k: toy["results"][k].get("fence_ms_per_layer") for k in common}
+            print(f"  {'config':<18} {'real t/s':>10} {'toy t/s':>10} {'real x':>8} {'toy x':>8}"
+                  f" {'real f':>9} {'toy f':>9}")
+            for k in real_order:
+                fr = f"{rf[k]*1000:.0f}us" if rf.get(k) is not None else "-"
+                ft = f"{tf[k]*1000:.0f}us" if tf.get(k) is not None else "-"
+                print(f"  {k:<18} {rs[k]:>10.2f} {ts[k]:>10.2f} "
+                      f"{rs[k]/rs[base]:>8.2f} {ts[k]/ts[base]:>8.2f} {fr:>9} {ft:>9}")
+
+            # The fence cost is the quantity we actually claim transfers, so score it directly.
+            fk = [k for k in common if rf.get(k) is not None and tf.get(k) is not None]
+            if len(fk) >= 3:
+                fsp = spearman([rf[k] for k in fk], [tf[k] for k in fk])
+                fsl, _ = linfit([tf[k] for k in fk], [rf[k] for k in fk])
+                print(f"  fence-cost rank correlation : {fmt(fsp)}  over {len(fk)} configs")
+                print(f"  fence-cost slope (real/toy) : {fmt(fsl)}   <- 1.0 = the toy measures "
+                      f"the real fence directly")
+
+            inv = [(real_order[i], real_order[j])
+                   for i in range(len(real_order)) for j in range(i + 1, len(real_order))
+                   if ts[real_order[i]] < ts[real_order[j]]]
+            if inv:
+                print(f"  INVERSIONS ({len(inv)}) -- each one localises a broken invariant:")
+                for x, y in inv[:12]:
+                    print(f"    real says {x} > {y}; toy says the opposite")
+            else:
+                print("  no inversions: the toy reproduces the real ordering exactly")
+
+            missing = sorted(set(rs) - set(ts))
+            if missing:
+                print(f"  NOT COMPARED (real has, toy lacks): {', '.join(missing)}")
+
+
+def cmd_budget(a):
+    """Predict the bandwidth term without running anything.
+
+    This is the q8dense workflow: requantize, compute both budgets, and you already know the
+    decode gain before you benchmark. That prediction landed within 0.1 percent on the real
+    model (predicted +8.2, measured +8.1).
+    """
+    rows = []
+    for path in a.model:
+        meta = gguf_meta(path)
+        gb = read_budget_gb(meta)
+        rows.append((os.path.basename(path), n_layer_of(meta), gb,
+                     gb / a.roofline_gbs * 1000.0))
+    w = max(len(r[0]) for r in rows)
+    print(f"{'model':<{w}} {'layers':>7} {'GB/token':>10} {'ms reads':>9} {'roof t/s':>9}")
+    for name, nl, gb, ms in rows:
+        print(f"{name:<{w}} {nl:>7} {gb:>10.4f} {ms:>9.2f} {1000.0/ms:>9.2f}")
+    if len(rows) == 2:
+        a0, a1 = rows[0][2], rows[1][2]
+        print(f"\nread budget {a0:.4f} -> {a1:.4f} GB/token"
+              f"  =  predicted decode gain {(a0/a1 - 1)*100:+.2f}%")
+        print("(decode-only; prefill is compute-bound and will move far less)")
+
+
+def fmt(v):
+    return "n/a" if v is None else f"{v:+.3f}"
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    r = sub.add_parser("run", help="benchmark one model across the configuration set")
+    r.add_argument("--model", required=True)
+    r.add_argument("--profile", required=True, choices=sorted(PROFILES))
+    r.add_argument("--out", required=True)
+    r.add_argument("--label")
+    r.add_argument("--bench", help=f"path to llama-bench (default {DEFAULT_BENCH})")
+    r.add_argument("--rpc", help="rpc endpoint host:port; enables the rpc_* configs")
+    r.add_argument("--only", help="comma-separated config names to run")
+    r.add_argument("--straddle", action="store_true",
+                   help="add the two-node base (--rpc, -ts 1/1) to single-node configs "
+                        "so they are comparable on a model that needs both nodes")
+    r.add_argument("--max-device-gib", type=float,
+                   help="skip configs whose largest per-device share exceeds this "
+                        "(guards the -ts 3/1 wedge on big models; ~108 for a 112 GiB device)")
+    r.add_argument("--n-prompt", type=int, default=512)
+    r.add_argument("--n-gen", type=int, default=128)
+    r.add_argument("--reps", type=int, default=3)
+    r.add_argument("--half-layers", type=int, default=22,
+                   help="value for -ncmoe in experts_cpu_half (half the block count)")
+    r.add_argument("--timeout", type=int, default=1800)
+    r.add_argument("--read-budget-gb", type=float,
+                   help="override the computed read budget (GB/token)")
+    r.add_argument("--roofline-gbs", type=float,
+                   help="host memory bandwidth in GB/s; required to derive the fence cost f. "
+                        "The DSV4 two-node analysis used 209.25")
+    r.set_defaults(func=cmd_run)
+
+    b = sub.add_parser("budget", help="compute the analytic read budget of a GGUF, no benchmarking")
+    b.add_argument("--model", required=True, action="append",
+                   help="repeatable; pass two to see the delta between quantizations")
+    b.add_argument("--roofline-gbs", type=float, default=209.25)
+    b.set_defaults(func=cmd_budget)
+
+    c = sub.add_parser("compare", help="qualify one or more toys against a real-model run")
+    c.add_argument("--real", required=True)
+    c.add_argument("--toy", required=True, action="append")
+    c.add_argument("--baseline", default="gpu_all",
+                   help="config used as the 1.0 reference for speedups")
+    c.set_defaults(func=cmd_compare)
+
+    a = ap.parse_args()
+    a.func(a)
+
+
+if __name__ == "__main__":
+    main()

results/real-dsv4.json

@@ -0,0 +1,195 @@
+{
+  "label": "dsv4-0731-q8dense.gguf",
+  "model": "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+  "profile": "m6",
+  "rpc": "10.10.10.1:50052",
+  "n_prompt": 512,
+  "n_gen": 128,
+  "reps": 2,
+  "host": "m6",
+  "arch": "deepseek4",
+  "n_layer": 43,
+  "read_budget_gb": 10.693818268,
+  "roofline_gbs": 209.0,
+  "results": {
+    "gpu_all": {
+      "ok": true,
+      "wall": 267.69734954833984,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999"
+      ],
+      "pp": 219.225,
+      "tg": 16.292123,
+      "env": {},
+      "fence_ms_per_layer": 0.23750605430712518
+    },
+    "load_mmap": {
+      "ok": true,
+      "wall": 296.21423840522766,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap"
+      ],
+      "pp": 217.077537,
+      "tg": 16.247997,
+      "env": {},
+      "fence_ms_per_layer": 0.24138263291319575
+    },
+    "load_mlock": {
+      "ok": true,
+      "wall": 310.26967692375183,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap+mlock"
+      ],
+      "pp": 233.925787,
+      "tg": 16.522971,
+      "env": {},
+      "fence_ms_per_layer": 0.21756299228459522
+    },
+    "batch_small": {
+      "ok": false,
+      "error": "ERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x11a)[0x73bb01ef599a]\n/home/dad/llama.cpp-ds/build-ds/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x3b8)[0x73bb01efc718]\n/home/dad/llama.cpp-ds/build-ds/bin/libllama.so.0(llama_decode+0xf)[0x73bb01efe39f]\n/home/dad/llama.cpp-ds/build-ds/bin/libllama-bench-impl.so(+0x1bd61)[0x73bb0305fd61]\n/home/dad/llama.cpp-ds/build-ds/bin/libllama-bench-impl.so(_Z11llama_benchiPPc+0xe1e)[0x73bb0306e64e]\n/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x73bb0282a1ca]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x73bb0282a28b]\n/home/dad/llama.cpp-ds/build-ds/bin/llama-bench(+0x12a5)[0x5eac31aa52a5]\nterminate called after throwing an instance of 'vk::DeviceLostError'\n  what():  vk::Queue::submit: ErrorDeviceLost\n",
+      "wall": 298.254257440567,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-b",
+        "256",
+        "-ub",
+        "256"
+      ],
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "batch_large": {
+      "ok": true,
+      "wall": 313.57823181152344,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-b",
+        "2048",
+        "-ub",
+        "512"
+      ],
+      "pp": 217.886656,
+      "tg": 16.272132,
+      "env": {},
+      "fence_ms_per_layer": 0.2392597083660938
+    },
+    "rpc_split_even": {
+      "ok": true,
+      "wall": 330.1122715473175,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1"
+      ],
+      "pp": 217.329445,
+      "tg": 16.27393,
+      "env": {},
+      "fence_ms_per_layer": 0.2391018075663653
+    }
+  },
+  "skipped": {
+    "rpc_split_local": "per-device share 108.3 GiB > --max-device-gib 108.0",
+    "rpc_split_remote": "per-device share 108.3 GiB > --max-device-gib 108.0",
+    "tp_join_peer": "per-device share 144.3 GiB > --max-device-gib 108.0",
+    "tp_join_cpu": "per-device share 144.3 GiB > --max-device-gib 108.0"
+  }
+}

results/real-ep.json

@@ -0,0 +1,80 @@
+{
+  "label": "real-ep",
+  "model": "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+  "profile": "m6",
+  "rpc": "10.10.10.1:50052",
+  "n_prompt": 512,
+  "n_gen": 128,
+  "reps": 3,
+  "host": "m6",
+  "arch": "deepseek4",
+  "n_layer": 43,
+  "read_budget_gb": 10.693818268,
+  "roofline_gbs": 209.25,
+  "results": {
+    "ep_serial": {
+      "ok": true,
+      "wall": 488.03170013427734,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ep/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-sm",
+        "layer",
+        "-ts",
+        "1/1"
+      ],
+      "pp": 207.340335,
+      "tg": 16.130248,
+      "env": {},
+      "fence_ms_per_layer": 0.25325263706681345
+    },
+    "ep_split": {
+      "ok": true,
+      "wall": 491.44082021713257,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ep/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-sm",
+        "layer",
+        "-ts",
+        "0/1"
+      ],
+      "pp": 135.456388,
+      "tg": 11.637701,
+      "env": {
+        "LLAMA_TP": "1",
+        "LLAMA_TP_EP": "1",
+        "LLAMA_TP_JOIN": "cpu",
+        "GGML_RPC_PUSH": "1"
+      },
+      "fence_ms_per_layer": 0.8098176911308856
+    }
+  },
+  "skipped": {}
+}

results/real-loadmodes.json

@@ -0,0 +1,162 @@
+{
+  "label": "dsv4-0731-q8dense.gguf",
+  "model": "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+  "profile": "m6",
+  "rpc": "10.10.10.1:50052",
+  "n_prompt": 512,
+  "n_gen": 128,
+  "reps": 3,
+  "host": "m6",
+  "arch": "deepseek4",
+  "n_layer": 43,
+  "read_budget_gb": 10.693818268,
+  "roofline_gbs": 209.0,
+  "results": {
+    "load_none": {
+      "ok": true,
+      "wall": 243.08253479003906,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "none"
+      ],
+      "pp": 235.564841,
+      "tg": 16.53925,
+      "env": {},
+      "fence_ms_per_layer": 0.21617765566267108
+    },
+    "load_mmap": {
+      "ok": true,
+      "wall": 280.40933322906494,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap"
+      ],
+      "pp": 219.095126,
+      "tg": 16.328015,
+      "env": {},
+      "fence_ms_per_layer": 0.23436830579279866
+    },
+    "load_mlock_only": {
+      "ok": true,
+      "wall": 248.1985890865326,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mlock"
+      ],
+      "pp": 235.030111,
+      "tg": 16.539419,
+      "env": {},
+      "fence_ms_per_layer": 0.2161632881301883
+    },
+    "load_mlock": {
+      "ok": true,
+      "wall": 294.4846339225769,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap+mlock"
+      ],
+      "pp": 234.170396,
+      "tg": 16.527009,
+      "env": {},
+      "fence_ms_per_layer": 0.21721910552501877
+    },
+    "load_dio": {
+      "ok": true,
+      "wall": 225.16271948814392,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dsv4-flash/dsv4-0731-q8dense.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "dio"
+      ],
+      "pp": 235.83676,
+      "tg": 16.535795,
+      "env": {},
+      "fence_ms_per_layer": 0.2164714468096389
+    }
+  },
+  "skipped": {}
+}

results/toy-ep.json

@@ -0,0 +1,80 @@
+{
+  "label": "toy-ep",
+  "model": "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+  "profile": "m6",
+  "rpc": "10.10.10.1:50052",
+  "n_prompt": 512,
+  "n_gen": 128,
+  "reps": 3,
+  "host": "m6",
+  "arch": "deepseek4",
+  "n_layer": 12,
+  "read_budget_gb": 0.049312852,
+  "roofline_gbs": 209.25,
+  "results": {
+    "ep_serial": {
+      "ok": true,
+      "wall": 5.138993501663208,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ep/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-sm",
+        "layer",
+        "-ts",
+        "1/1"
+      ],
+      "pp": 15341.239918,
+      "tg": 306.631127,
+      "env": {},
+      "fence_ms_per_layer": 0.2521318956054091
+    },
+    "ep_split": {
+      "ok": true,
+      "wall": 4.609577178955078,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ep/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-sm",
+        "layer",
+        "-ts",
+        "0/1"
+      ],
+      "pp": 5997.841429,
+      "tg": 134.253405,
+      "env": {
+        "LLAMA_TP": "1",
+        "LLAMA_TP_EP": "1",
+        "LLAMA_TP_JOIN": "cpu",
+        "GGML_RPC_PUSH": "1"
+      },
+      "fence_ms_per_layer": 0.6010779906036318
+    }
+  },
+  "skipped": {}
+}

results/toy-h1-straddle.json

@@ -0,0 +1,470 @@
+{
+  "label": "h1-topology-mxfp4.gguf",
+  "model": "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+  "profile": "m6",
+  "rpc": "10.10.10.1:50052",
+  "n_prompt": 512,
+  "n_gen": 128,
+  "reps": 2,
+  "host": "m6",
+  "arch": "deepseek4",
+  "n_layer": 12,
+  "read_budget_gb": 0.049312852,
+  "roofline_gbs": null,
+  "results": {
+    "cpu_only": {
+      "ok": true,
+      "wall": 1.1713976860046387,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "0"
+      ],
+      "pp": 3570.430355,
+      "tg": 505.44811,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "gpu_all": {
+      "ok": true,
+      "wall": 2.617534875869751,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999"
+      ],
+      "pp": 15582.895451,
+      "tg": 321.452023,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "experts_cpu_ot": {
+      "ok": true,
+      "wall": 3.5321099758148193,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-ot",
+        "exps=CPU"
+      ],
+      "pp": 3907.97141,
+      "tg": 146.2889,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "experts_cpu_all": {
+      "ok": true,
+      "wall": 4.073307514190674,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-ncmoe",
+        "999"
+      ],
+      "pp": 3777.039085,
+      "tg": 148.639188,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "experts_cpu_half": {
+      "ok": true,
+      "wall": 3.53596568107605,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-ncmoe",
+        "22"
+      ],
+      "pp": 3845.900727,
+      "tg": 147.245819,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "dev_split_even": {
+      "ok": false,
+      "error": "ggml_vulkan: Found 1 Vulkan devices:\nggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: dot2 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: KHR_coopmat\nerror: invalid device: Vulkan1\nerror: invalid parameter for argument: -dev\n",
+      "wall": 0.010591983795166016,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "-dev",
+        "Vulkan0,Vulkan1",
+        "-ts",
+        "1/1"
+      ],
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "dev_split_skew": {
+      "ok": false,
+      "error": "ggml_vulkan: Found 1 Vulkan devices:\nggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: dot2 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: KHR_coopmat\nerror: invalid device: Vulkan1\nerror: invalid parameter for argument: -dev\n",
+      "wall": 0.010843038558959961,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "-dev",
+        "Vulkan0,Vulkan1",
+        "-ts",
+        "3/1"
+      ],
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "load_mmap": {
+      "ok": true,
+      "wall": 2.610311508178711,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap"
+      ],
+      "pp": 15446.844927,
+      "tg": 321.834936,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "load_mlock": {
+      "ok": true,
+      "wall": 2.610318660736084,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap+mlock"
+      ],
+      "pp": 15565.496584,
+      "tg": 321.876554,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "batch_small": {
+      "ok": true,
+      "wall": 2.623216152191162,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-b",
+        "256",
+        "-ub",
+        "256"
+      ],
+      "pp": 12934.798133,
+      "tg": 321.866921,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "batch_large": {
+      "ok": true,
+      "wall": 2.612264394760132,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-b",
+        "2048",
+        "-ub",
+        "512"
+      ],
+      "pp": 15505.585415,
+      "tg": 322.102516,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "rpc_split_even": {
+      "ok": true,
+      "wall": 2.607177495956421,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1"
+      ],
+      "pp": 15514.418971,
+      "tg": 321.84373,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "rpc_split_local": {
+      "ok": true,
+      "wall": 3.292717218399048,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "3/1"
+      ],
+      "pp": 15716.61865,
+      "tg": 302.911374,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "rpc_split_remote": {
+      "ok": true,
+      "wall": 1.8777427673339844,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/3"
+      ],
+      "pp": 16355.534594,
+      "tg": 332.586155,
+      "env": {},
+      "fence_ms_per_layer": null
+    },
+    "tp_join_peer": {
+      "ok": true,
+      "wall": 0.9517486095428467,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "0/1"
+      ],
+      "pp": 34327.399852,
+      "tg": 403.44726,
+      "env": {
+        "LLAMA_TP": "1"
+      },
+      "fence_ms_per_layer": null
+    },
+    "tp_join_cpu": {
+      "ok": true,
+      "wall": 0.9493029117584229,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "2",
+        "-o",
+        "json",
+        "-ngl",
+        "999",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "0/1"
+      ],
+      "pp": 34328.079727,
+      "tg": 403.314183,
+      "env": {
+        "LLAMA_TP": "1",
+        "LLAMA_TP_JOIN": "cpu"
+      },
+      "fence_ms_per_layer": null
+    }
+  },
+  "skipped": {}
+}

results/toy-loadmodes.json

@@ -0,0 +1,162 @@
+{
+  "label": "h1-topology-mxfp4.gguf",
+  "model": "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+  "profile": "m6",
+  "rpc": "10.10.10.1:50052",
+  "n_prompt": 512,
+  "n_gen": 128,
+  "reps": 3,
+  "host": "m6",
+  "arch": "deepseek4",
+  "n_layer": 12,
+  "read_budget_gb": 0.049312852,
+  "roofline_gbs": 209.0,
+  "results": {
+    "load_none": {
+      "ok": true,
+      "wall": 3.1830012798309326,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "none"
+      ],
+      "pp": 15826.95342,
+      "tg": 320.659779,
+      "env": {},
+      "fence_ms_per_layer": 0.24021862021984272
+    },
+    "load_mmap": {
+      "ok": true,
+      "wall": 3.049060821533203,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap"
+      ],
+      "pp": 15831.616962,
+      "tg": 321.29154,
+      "env": {},
+      "fence_ms_per_layer": 0.2397076121162177
+    },
+    "load_mlock_only": {
+      "ok": true,
+      "wall": 3.054117441177368,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mlock"
+      ],
+      "pp": 15780.65358,
+      "tg": 320.533263,
+      "env": {},
+      "fence_ms_per_layer": 0.24032119642141203
+    },
+    "load_mlock": {
+      "ok": true,
+      "wall": 3.041682720184326,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "mmap+mlock"
+      ],
+      "pp": 15785.781199,
+      "tg": 321.214943,
+      "env": {},
+      "fence_ms_per_layer": 0.23976946151517323
+    },
+    "load_dio": {
+      "ok": true,
+      "wall": 3.102926731109619,
+      "cmd": [
+        "/home/dad/llama.cpp-ds/build-ds/bin/llama-bench",
+        "-m",
+        "/mnt/weights/models/dev/h1-topology-mxfp4.gguf",
+        "-p",
+        "512",
+        "-n",
+        "128",
+        "-r",
+        "3",
+        "-o",
+        "json",
+        "--rpc",
+        "10.10.10.1:50052",
+        "-ts",
+        "1/1",
+        "-ngl",
+        "999",
+        "-lm",
+        "dio"
+      ],
+      "pp": 15783.352468,
+      "tg": 321.35924,
+      "env": {},
+      "fence_ms_per_layer": 0.2396529712795368
+    }
+  },
+  "skipped": {}
+}