README.md

---
license: gpl-3.0
language:
  - en
library_name: llama.cpp
pretty_name: OldIron
tags:
  - llama.cpp
  - gguf
  - cpu-inference
  - inference-engine
  - quantization
  - avx
  - avx1
  - sandy-bridge
  - hp-z820
  - xeon
  - cuda
pipeline_tag: text-generation
---

# OldIron

An inference engine plus **one named weight file**, built for hardware that current engines have
written off. OldIron is a product line: each variant names the *lowest* machine it must serve, and nothing in it
is allowed to be slower than that floor needs. The weights are third-party and are not shipped here — see
*The weights* below.

| variant | lowest supported CPU | lowest supported GPU | status |
|---|---|---|---|
| **OldIron-AVX** | Xeon E5-2665 (Sandy Bridge-EP, **AVX1**: no AVX2/FMA/F16C/BMI2) | RTX 3060 8 GB (**sm_86**) | built, staged, **measured on two HP Z820s**; CPU kernel work finished (+13.5% at fixed format, +6.4% net) |
| OldIron-Core2 (planned) | SSE4.1/SSSE3 (2007) | none | not started |

## Results, plainly

**One measured step, and no more.** The AVX1 kernel work shipped and it is worth **+13.5% at fixed format and
+6.4% net** (7.45 → 7.86-7.93 t/s). Every other CPU-lane lever was measured and rejected. The "~3x" projection in
the first version of this README **did not survive the engine tests and is retracted below** — the large multiple
on this hardware is on the GPU lane, not in the matvec.

All numbers measured on an HP Z820 (2× Xeon E5-2665, 32 GB DDR3, RTX 5060 Ti) with the 35B-A3B weight file named in
each run, coherent output throughout, `nice -n 19`, box verified idle between arms.

### CPU lane — shipped state

| what | measured | note |
|---|---|---|
| stock engine, IQ3_S 3.66 bpw | **7.45 ± 0.04 t/s** | `-ngl 0 --no-op-offload -t 24`, the best prior config |
| stock engine, Q4_0 4.50 bpw | 6.99 ± 0.47 t/s | the wider format is *slower* on stock kernels (23% more bytes) |
| **shipped W3 AVX1 kernel, Q4_0** | **7.86 ± 0.08 t/s** (bench) · 7.93 ± 0.02 (harness) | **+13.5% vs stock Q4_0, +6.4% net over stock IQ3_S**; coherent |
| prefill pp128 / pp512 / pp4096 | 29.70 / 29.9 / 28.0 t/s | 8× the context costs 6% per token |
| W1 GEMV kernel alone | 4.4 GMAC/s/core from DRAM; 8.9 / 18.1 L1-resident (1 / 2 rows) | exact-integer vs the scalar reference, AVX1-clean |
| gcc vs clang 18 vs icx 2026 | gcc best; clang −9%, icx −15.5% | compiler lever closed |

The trade the kernel buys: the Q4_0 weight file is ~19 GiB against 15.7 GiB for IQ3_S. It still fits a 32 GB box,
but it is a size regression in exchange for the speed.

### Retractions — what this README claimed, and what the tests returned

| claim in the first version | measured result |
|---|---|
| "projection when W3 lands ~20-24 t/s, about 3x today" | **retracted.** W3 landed and bought 13.5% at fixed format. The engine moves ~28 GB/s where the pure kernel reaches 44 GB/s, so closing that gap entirely is **~1.35× end-to-end** on a 77% matvec share — not 3×. The kernel path is close to exhausted |
| "NUMA interleave, one flag, +59%" | **0% in the engine** — 3.13 vs 3.13 t/s, two interleaved rounds, on a copy materialised under the interleave policy. `numastat` also shows the page cache already split 50/50 across nodes. The +59% was a 64 MB streaming microbenchmark; the engine's matvec is not purely bandwidth-bound, so it did not transfer |
| "11.96 GB/s = 16% of the 74.9 GB/s ceiling, so 3-5× headroom" | the 74.9 GB/s figure is a **sequential** streaming measurement that this access pattern never reaches. Like-for-like the engine is at 28 GB/s against the kernel's 44 GB/s — 1.6×, worth ~1.35× end-to-end |
| lm_head Q6_K → Q4_0 (417 MB/token on the old path) | **0%** — 7.83 vs 7.93 t/s, inside noise |

### Also measured and rejected (detail in [`kernels/RESULTS.md`](kernels/RESULTS.md))

- **hoisted-correction GEMM: −40% in situ** (7.93 → 4.67 t/s) despite +113% on the isolated matrix — the engine
  makes ~1.9M small `vec_dot` calls per token, so the per-call precompute is never amortised. Reverted.
- **4-row GEMM batching:** inseparable from the hoist that broke it; not shipped.
- **speculative decoding:** 7.3 → 6.8 t/s (**−7%**) with a 0.8B draft on this thinking-heavy output.
- **NUMA chunk granularity** (dropping the `ggml_is_numa()` term): prefill flat, **decode −7%**. Reverted.
- **K-quant tiled GEMM instead of Q4_0 + `vec_dot`:** the tiled path is **44% slower at prefill** on AVX1
  (19.00/15.74 t/s vs 29.9/28.0), because it compiles its generic path without AVX2/VNNI. Q4_0 + the W3 kernel
  wins on decode *and* prefill.
- **CONCAT "10% of prefill":** a profiler artifact — it was the previous node's barrier wait. Parallelising the
  copy changed nothing (pp512 29.97 → 29.92).
- **Prefill ceiling:** pp512 is ~2.5 TMAC at 29.9 t/s ≈ 145 GMAC/s against the measured 170 GMAC/s int8 floor,
  i.e. **85-90%**, not the 52% an earlier active-parameter estimate assumed.

### Where the multiple actually is: the GPU lane

The CPU lane is at its ISA ceiling. The 16 GB card is not. Best long-context split on the 35B weights, measured at
a real 32k context with a 140 KB prompt (not `llama-bench`'s short context):

| config | prompt | generation | peak VRAM |
|---|---|---|---|
| **`-ngl 99 -ncmoe 10 -t 24 -c 32768`, f16 KV** | **832.9 t/s** | **52.5 t/s** | 15339 MiB |
| same + KV q8_0 | 839.2 | 48.3 (−8%) | 15339 MiB |
| `-ncmoe 12 -t 24` | 731.7 | 51.6 | 14465 MiB (1.4 GB spare) |
| CPU only (fallback) | 29.9 | 7.86 | — |

**6.7× the CPU lane's decode and 29× its prefill.** Only 508 MiB spare at `-ncmoe 10`, so 32k is the ceiling with
this split; `-ncmoe 12` costs 2% and holds 1.4 GB of headroom. KV quantisation is a net loss here — the hybrid
DeltaNet KV is ~12 KB/token, so q8 mostly adds dequant work.

## What OldIron-AVX is

One binary, one weight file, three lanes:

| lane | how it runs | target
|---|---|---|
| A — CPU | `-ngl 0` | the AVX1 Xeon pair, no GPU at all |
| B — GPU | `-ngl 99` | RTX 5060 Ti 16 GB (sm_120) |
| C — 8 GB budget | `-ngl 99 -ncmoe N` | RTX 3060 8 GB (sm_86): attention on the card, routed experts on the CPU |

### Engine provenance

- source: llama.cpp `2744e48d1` (build 816, 0.7.0-dev), `qwen35moe` architecture support present
- CPU backend: **one variant**, `-march=sandybridge -mno-avx2 -mno-fma -mno-f16c -mno-bmi2` + `-msse4.2 -mavx`
  (ggml prints that as `-msse4.2;-mavx` / `GGML_SSE42;GGML_AVX` — on Sandy Bridge that *is* AVX1, not a lower floor)
- `-march=sandybridge` also sets `-mtune=sandybridge`, so the code is scheduled for the Z820 core even though it is
  built on a Zen4 host
- CUDA fatbin: **`sm_86` + `sm_120`**, 145 kernels each (RTX 3060 is the supported floor, 5060 Ti the test card)
- `GGML_LLAMAFILE=ON`, `GGML_CPU_REPACK=ON`, `GGML_OPENMP=ON`
- `RUNPATH=$ORIGIN/../lib` on binaries, `$ORIGIN` on libs — the tree is relocatable

### ISA gate (measured, not asserted)

`scripts/isa-audit.sh` on the shipped `libggml-cpu.so`:

```
AVX2-only mnemonics : 0        FMA      : 0
int-vector on %ymm  : 0        F16C     : 0
AVX-512 (%zmm)      : 0        AVX1 %ymm ops : 5335
VERDICT: AVX1-clean and using 256-bit width
```

**The audit pattern matters.** A naive grep for "AVX2 opcodes" reports false positives: `vpmulld`, `vpblendvb`,
`vpminsd`/`vpmaxsd` are SSE4.1 instructions whose **VEX.128 form is legal AVX1**, and unanchored patterns like
`v(p[a-z]*|ps[a-z]*)` match the `vps` inside `vdivps`/`vpermilps`. The sound tests are (a) genuinely AVX2-only
mnemonics (`vpermd`, `vpermq`, `vpbroadcast[dwq]`, `vpsllvd`, `vpgather*`, `vpblendd`) and (b) **integer-vector ops
on `%ymm`** — AVX1 has no 256-bit integer arithmetic at all. `vmovdqa/vmovdqu` on `%ymm` are AVX1-legitimate.

### The weights: third-party, and not shipped

`Qwen3.6-35B-A3B` in GGUF — 35B total / 3B active, arch `qwen35moe` (256 experts, 8 routed + 1 shared, 40 layers,
3:1 Gated DeltaNet : full attention, 262k context, 248320 vocab, MTP-trained). Three files appear in these
measurements: the stock `Qwen3.6-35B-A3B-IQ3_M.gguf` (**16.9 GB, 3.52 bits per weight**), an `IQ3_S` mix
(3.66 bpw, 15.73 GiB) used for the lane matrix, and a `Q4_0` re-quant of the same model (4.50 bpw, ~19 GiB)
written only to A/B the AVX1 kernel.

**These weights are not ours, and this project does not include them.** They are third-party GGUF files of a public
model, downloaded and checksum-verified in transit; the `Q4_0` copy is a local re-quant of the same model and is a
test artefact, not a product. What is published here is engine source, kernel source, scripts and measurements — no
weight file is redistributed, and no weight file is claimed as part of the deliverable.

Why this shape of model: decode speed is set by *active* parameters and RAM is the scarce resource, so a 35B/3B
MoE at 3.5-4.5 bits fits a 32 GB box with ~15 GB to spare for a long-context KV cache (20 KB/token → 32k ctx
≈ 0.67 GB), where a 27B dense at a higher quant would be ~7× slower for less quality headroom.

## Measured: HP Z820 (2× E5-2665, 16 cores, RTX 5060 Ti)

`Qwen3.6-35B-A3B-IQ3_S` mix, 3.66 bpw, 15.73 GiB, 34.66B params — coherent on every lane
("The capital of France is **Paris**.").

| lane | config | decode | prefill | peak VRAM | verdict |
|---|---|---|---|---|---|
| A — CPU only | `-ngl 0 -nopo 1 -t 16` | 7.25 ± 0.10 t/s | 28.8 t/s (pp128) | — | works, kernel-bound |
| A — CPU only | `-ngl 0 -nopo 1 -t 24` | **7.56 ± 0.03 t/s** | 28.5 t/s | — | best CPU arm on stock kernels |
| A — CPU, shipped kernel | Q4_0 + W3, `-t 24` | **7.86 ± 0.08 t/s** | 29.70 t/s (pp128) | — | what actually ships |
| B — 5060 Ti maximised | `-ngl 99 -ncmoe 4` | **73.3 ± 1.0 t/s** | **763.7 t/s** (pp512) | 14797 MiB | best short-context GPU arm |
| B — | `-ngl 99 -ncmoe 8` | 50.8 t/s | — | 13345 MiB | |
| C — 3060 8 GB, tight | `-ncmoe 24` | 29.7 t/s | — | 7739 MiB | fits, ~450 MiB margin |
| **C — 3060 8 GB, safe** | **`-ncmoe 32`** | **23.8 t/s** | 63.9 t/s | **4905 MiB** | **recommended: 3 GB headroom for context** |
| C — | `-ncmoe 40` | 19.1 t/s | 52.7 t/s | 1869 MiB | |
| — | `-ngl 99` alone | **OOM** | — | — | 15898 MiB allocation fails |

**"GPU full" does not exist for this weight set on a 16 GB card.** 15.73 GiB of weights + KV + context will not fit
15.8 GiB of VRAM, so the GPU lane is *always* hybrid here: `-ncmoe 4` on a 16 GB card, `-ncmoe 32` on an 8 GB card.
For long context the split that matters is in *Results, plainly* above (`-ncmoe 10`, 52.5 t/s at 32k).

CPU lane effective throughput: ~1.54 GB/token (3B active + the 248k-vocab lm_head) ÷ 132 ms ≈ **11.7 GB/s.** That
is against the **44 GB/s** the pure kernel reaches on this access pattern, not against the box's 74.9 GB/s
sequential streaming figure — see the retractions above for why the difference matters.

### Earlier proxy run (kept for comparison)

`gpt-oss-20b Q4_K_M`, 20.9B total / 3.6B active, 10.81 GiB, measured before the 35B weights finished copying:

| lane | decode | prefill | note |
|---|---|---|---|
| A CPU only, `-t 16` | **8.03 ± 0.02 t/s** | **23.9 t/s** (pp512, `--no-op-offload`) | coherent ("…is Paris") |
| B GPU full (`-ngl 99`) | **142.7 ± 2.4 t/s** | 297 t/s | coherent |
| C `-ncmoe 16` | 19.7 t/s | — | peak VRAM 4605 MiB |
| C `-ncmoe 24/32` | 13.4–13.9 t/s | — | peak VRAM 1373 MiB |

### Two measurement traps, both found here

**1. `-ngl 0` is not CPU-only when a CUDA backend is present.** Op-offload defaults to ON and moves the batched
matmuls to the GPU. The contaminated arms reported **242–341 t/s of "CPU prefill"** — physically impossible on this
box (it implies ~3 TMAC/s against a ~0.3 TMAC/s ceiling). With `--no-op-offload` the same arm reads **23.9 ± 0.9
t/s**. Decode was never affected, only prefill. Any CPU-lane prefill number without `--no-op-offload` is a GPU
number wearing a CPU label.

**2. `-fitt` is not a VRAM cap** (see below).

### AVX1 floor vs `-march=native`: the floor costs nothing

Same source (2744e48d1), same model, same flags, built both ways on the Z820 itself:

| build | decode (tg48/128) | prefill (pp512) |
|---|---|---|
| OldIron-AVX floor (`-march=sandybridge`, cross-built on Zen4) | 8.55 / 9.03 t/s | 23.9 ± 0.9 t/s |
| native (`-march=native`, built on the E5-2665) | 8.74 / 8.83 t/s | ~25 t/s |

Parity within run-to-run spread. Repeated on the 35B weights: floor **7.56 ± 0.03** (t=24) / 7.25 ± 0.10 (t=16)
vs native **7.63 ± 0.05** t/s — ~1% apart. One detached harness pass measured the two back to back:
floor **7.38 ± 0.02** vs native **7.43 ± 0.05**: **0.7%**, which is the run-to-run spread itself. Three independent passes of the same arms now agree within ~3-6%, so the floor is confirmed free. The two targets differ only in `-maes` (AES-NI), which no inference kernel
uses, so the cross-built floor is not leaving AVX1 performance on the table and OldIron-AVX can be built anywhere.

(An A/B against the fleet's older `ik_llama-sb-avx1` floor build could not be run: its `llama-cli` exits
`Permission denied` even after `chmod +x`, so that binary is unusable as an arm — unresolved, and not needed while
the same-commit native comparison exists.)

CPU lane effective weight throughput on that run: 3.6B active × 4.85 bits ≈ 2.18 GB/token ÷ 125 ms ≈
**17.5 GB/s.** Measured at 8 / 16 threads the engine moves 24 / **28 GB/s**, against 44 GB/s for the pure kernel on
a sequential 64 MB matrix — within 1.6×, which is the honest size of the kernel headroom.

### Open item: `-fitt` is not a safe VRAM cap

`-fitt 7400` produced a **peak of 8515 MiB** and `-fitt 6400` a peak of **9447 MiB** — both above an 8 GB card's
budget. The fit heuristic and `-ncmoe` fight each other, and the peak includes the CUDA context. Lane C must be
pinned with an explicit `-ncmoe` and verified by sampling `nvidia-smi` *during* the run (sampling before/after
reads 7 MiB and proves nothing).


## One binary, both Z820s (the compatibility claim, measured)

Same OldIron-AVX binary, CPU lane, `-ngl 0 -nopo 1`:

| host | CPU | quant | decode | prefill | effective weight throughput |
|---|---|---|---|---|---|
| Z820 #1 | 2× E5-2665 @ 2.4 GHz | IQ3_M, 3.66 bpw | 7.45 ± 0.04 t/s (t=24) | 28.5 t/s | **11.96 GB/s** |
| Z820 #2 | 2× E5-2687W @ 3.1 GHz | Q4_K_M, 4.85 bpw | 5.66 ± 0.02 t/s (t=16) | 40.7 t/s | **12.04 GB/s** |

Normalised for quant, the two boxes move **within 0.7% of the same bytes per second** — decode is DRAM-bound, so the
29% clock advantage buys nothing there, exactly as the memory model predicts. Prefill is compute-bound and *does*
scale with clock (1.43×). Coherent on both ("The capital of France is **Paris**.").

That is the compatibility measurement, not an argument: one AVX1 binary covers both boxes with no per-host build, and
the only thing that changed the token rate was the weight file.

## Operational notes (learned the hard way on this build)

- **Run the harness detached, in tmux** — the wifi link to the Z820s drops ssh sessions mid-run. tmux is installed
  without root on both Z820s (`scripts/install-tmux-noroot.sh`: `apt-get download` + `dpkg -x` into `~/tmux-local`,
  wrapper at `~/bin/tmux`); the build host has it from apt.
- **Under tmux the CLI gets a pty and enters its interactive REPL** after the first turn, hanging until the timeout
  instead of exiting. Every non-interactive CLI arm needs `< /dev/null`. `llama-cli` also writes a **0-byte log**
  under a tmux pty; capture CLI output over a plain ssh pipe (`llama-bench` is fine under tmux).
- **Never point `cuobjdump` at a symlink and its target together**: given `libggml-cuda.so libggml-cuda.so.0
  libggml-cuda.so.0.27.0` it prints a bogus mixed summary (`sm_50…sm_101`, 1–2 each) that looks like arch
  contamination. On one `readlink -f` path it reports the truth: 145 `sm_120` + 145 `sm_86`.
- **Coherence capture belongs on non-interactive ssh**, not the tmux arm: under the pty the answer text is not
  captured even with stdin closed, so the gate is run over plain ssh where it has produced the correct answer
  repeatedly.
- **Confirm the box is idle between arms** — a stale `llama-bench` survived an ssh timeout and competed with three
  subsequent measurements for 23 minutes (one produced a nonsense 1.31 t/s).
- **CMake does not rebuild on a compile-definition change** — flipping `GGML_CPU_REPACK` rewrote `flags.make` and
  left every object in place, producing a fake "repack off" result. Delete the target's `*.o` and touch the sources.
- **Never quote a rate from a profiled run** — `OLDIRON_OPPROF=1` costs ~1.6x (19.0 s of op time for 96 tokens vs
  12.3 s), so a profiler run and a clean run are not comparable.

## Acceptance gates (a fast wrong answer is a fail)

1. **ISA** — `scripts/isa-audit.sh` on the shipped libs: 0 above the floor, `%ymm` present.
2. **Fatbin coverage** — `cuobjdump --list-elf` shows `sm_86` and `sm_120`.
3. **Coherence** — fixed prompts through each lane, text compared to a known-good engine. This fleet has shipped a
   floored build that answered `The capital of France is!!!!!!!!` at 65 t/s; speed is measured last, on purpose.
4. **Speed** — `llama-bench`, thread sweep on the CPU lane (expect an optimum, not the largest count).
5. **8 GB class** — lane C must peak under ~7.8 GB sampled during the run.

## What is in this repository

Source, scripts and measurements. Not included, deliberately:

- `src/` - the engine is an unmodified llama.cpp worktree at commit `2744e48d1`; clone it from upstream rather than
  vendoring it here.
- weights - `Qwen3.6-35B-A3B` and `gpt-oss-20b` GGUFs are third-party downloads (HF), not redistributed.
- build output, staged binaries, kernel test binaries.

```
kernels/   the AVX1 quantised GEMV/GEMM work (source only) and RESULTS.md, the waypoint log
scripts/   build (AVX1 floor + sm_86;120), ISA audit, staging, weight transfer, acceptance matrix
results/   the measured runs
```

## License

GPL-3.0-or-later. See `LICENSE`; the copyright notice and the third-party terms (llama.cpp is MIT and stays MIT)
are in `NOTICE`.

## Layout

```
/home/bob/OldIron/
  src/            llama.cpp worktree @ 2744e48d1
  build/          the built engine (bin/ + lib/)
  stage/          patched-RUNPATH copy for shipping
  scripts/
    build-oldiron-avx.sh    build (AVX1 floor + sm_86;120)
    isa-audit.sh            the ISA gate (see note above about the pattern)
    stage-to-target.sh      relocatable copy + load check
    fetch-weights.sh        parallel-stream weight transfer (link is wifi/WAN here)
    acceptance-hp-z820.sh    the full matrix on the Z820
```
On a target machine the engine sits at `<root>/engine/{bin,lib}` with the weight file wherever you keep it — nothing
under this repository's tree.