decoder-bundle-minimization/REPORT.md

# B6 — minimizing the shipped paq8px decoder bundle

**Scope.** Decoder-side only: no compression runs, no changes to the archive, nothing touched under
`V1-verify-queue/`, `publish67/` or the flagship run directory, no publication or HF API calls.
All work under `/root/hutter67/swarm67/B6-bundle/`.

## Headline

| | bytes | sha256 | notes |
|---|---:|---|---|
| P5/published paq8px blob-chain bundle | 766,308 | `debed42c9ec425e5461dd886148814e55c71c52ea43b2675d80b563030074dce` | b.xz 760,856 + v.xz 4,948 + d 236 + 268 framing |
| **B6 winner (`dyn_Os`)** | **204,244** | `7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a` | same launcher/inverse, new payload |
| saving | **562,064** | | one-line: **the winner ships 562,064 bytes smaller than the 766,308 B baseline.** |

## SHIP THIS

| | |
|---|---|
| artefact | `/root/hutter67/swarm67/B6-bundle/out/bundle_dyn_Os/decompressor.zip` |
| bytes / sha256 | **204,244** / `7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a` |
| contents | `b.xz` 198,792 (XZ of the 476,864 B `dyn_Os` ELF, sha256 `94cc29db…`) + `v.xz` 4,948 (unchanged P5 member) + `d` 409 raw / 236 deflated (unchanged P5 launcher) + 268 B ZIP framing |
| fallback if the verifier's `-9` decode hits anything unexpected | `/root/hutter67/swarm67/B6-bundle/out/bundle_dyn_O3/decompressor.zip` — 443,156 B, sha256 `411bd72a8deb791e…`, P1's exact `-O3 -flto` recipe with only `-static` removed, also small-chain proven (3/3) |
| evidence table for every variant | §4, with per-binary proofs in §6a/§6b and machine-readable form in `bundle67.json` |

**562,064 B** off the counted total, from a rebuild + repack of the payload only. On the assets basis
the entry is `A_blob + 204,244` (was `A_blob + 766,308`); with the flagship blob archive projecting to
~16.1 MB the entry lands at ≈16.29 MB, versus 17,969,555 B for the published v4 baseline. The launcher `d`
and the blob inverse `v.xz` are byte-identical to the published P5 package, so the decode chain a
verifier runs is unchanged; only the paq8px executable it unpacks is smaller.

## 1. The decoder chain that has to be shipped

Flagship production chain (`P1-paq8px-cal/fullrun_blob.sh`, `-9`):

```
archive.paq8px216  --paq8px -d-->  blob (60,634,501 B)  --blob inverse-->  enwik8 (100,000,000 B)
```

P5 accounted that whole chain as a 766,308 B ZIP holding exactly three members
(`paq8px-blob/package.json`): `b.xz` 760,856 (XZ of the 1,895,080 B static paq8px binary),
`v.xz` 4,948 (the dedicated cmix-store blob inverse), `d` 409 raw / 236 deflated (Python launcher),
plus 268 B of ZIP framing. **B6 changes only `b.xz`.**

Byte-accounting validation: `work/make_bundle.py --verify-baseline` rebuilds the P5 ZIP from its
members and reproduces **766,308 B / sha256 `debed42c9ec425e5461dd886148814e55c71c52ea43b2675d80b563030074dce` byte-for-byte**. Stated plainly, because
it is the whole basis of the claim: **because the packer reproduces P5's ZIP byte-for-byte, the
−562,064 B is an apples-to-apples accounting change (same three members, same launcher, same inverse,
same 268 B of framing) and not a change of accounting basis.** The only member whose bytes differ is
the payload `b.xz`.

## 2. Lever 1 — rebuild the binary smaller (this is where the bytes were)

The shipped binary is a `-O3 -flto -march=native -static` build: 1,895,080 B raw. Static linking of
libc/libstdc++ plus `-O3` inlining is most of it. Rebuilding the **pinned source**
(`paq8px-29237fb44cb1995690e3eb72c6c3b1e4aede5791`, the same commit P1/P5 used) with size flags and
dynamic linking, **changing no source line**, gives:

| build | raw B | XZ payload B | bundle B |
|---|---:|---:|---:|
| P1 shipped (`-O3 -flto -march=native -static`), P5's own tuned XZ | 1,895,080 | 760,856 | 766,308 (the baseline) |
| `-O3 -flto=2` dynamic (identical codegen flags, only the link mode changes) | 1,012,192 | 437,704 | 443,156 |
| `-O2 -flto=2` dynamic | 647,648 | 264,480 | 269,932 |
| clang++-17 `-Oz` dynamic | 523,208 | 211,248 | 216,700 |
| **`-Os -flto=2` dynamic + gc-sections + strip** | **476,864** | **198,792** | **204,244** |
| `-Os -flto=2` dynamic + hard flags (`-fno-plt -fno-stack-protector …`) | 473,552 | 201,004 | 206,456 (smaller raw, *worse* packed) |
| `-Os` dynamic, no LTO | 509,632 | 204,672 | 210,124 |
| `-Os` static (self-contained fallback) | 1,277,800 | 480,876 | 486,328 |

Exact commands are in §5. `-flto=1` and `-flto=2` produced a byte-identical binary here
(sha256 `94cc29db…`), so that knob is a no-op. Every payload column below uses the same swept-optimal
XZ parameters as the winner, so variants are comparable at bundle level.

Cost of the size build: in the single-run measurements of §6a the `-Os` decode is **12.25 s** on the
32 KiB binary slice where the original binary takes **10.42 s** (≈ +18 %, within session noise on a
box that was simultaneously running a 4.5 GB encode; `dyn_O3` measured 9.38 s). This affects *decode
time only* — it is not a scoring term — but it is the honest trade: the smallest bundle decodes
somewhat slower than the `-O3` binary, and the fallback `dyn_O3` variant does not have that penalty.

**`-march=native` was deliberately NOT used** even though P1's build had it: it would bake AVX-512
instructions from this EPYC into the shipped decoder and could crash on a verifier's CPU. The
`-Os` build targets baseline x86-64. This is a portability gain, not just a size choice.

## 3. Lever 2 — pack the payload with the strongest *shippable* codec

All measured on the winner binary (`dyn_Os`, 476,864 B). "Shippable" means the decoder is either
already present on the host (Python's `lzma`) or small enough to pay for itself:

| codec | bytes | note |
|---|---:|---|
| xz + BCJ/x86, LZMA2 dict512K lc3 lp0 pb0 mf=bt2 nice96 (SHIPPED b.xz) | 198,792 | decoder is free on the host (launcher already uses Python lzma); winner |
| zpaq zc56 stream (libzpaq level 5, 64 MiB block) | 187,197 | smallest raw payload, but +16,476 B shipped libzpaq extractor = 203,673 B effective; loses |
| UPX 4.2.4 --best --lzma (upx -d restores byte-identical) | 202,768 | self-extracting; no host decoder, no xz payload; loses by 3,912 B |
| 7z PPMd mem=512m o=32 -mx=9 | 204,453 | needs a 7z binary on the host |
| brotli -q 11 -w 24 | 213,027 | needs brotli on the host |
| zstd --ultra -22 --long=27 | 222,742 | needs zstd on the host |
| bzip2 -9 | 248,621 | host-decodable via Python bz2, but far larger than xz |

XZ + x86/BCJ wins: it needs **zero extra shipped bytes** (the launcher already uses Python's `lzma`),
while zpaq's 11,595 B head start is eaten by the 16,476 B libzpaq extractor it would require
(187,197 + 16,476 = 203,673 > 198,792), and UPX's self-extracting stub lands at 202,768.
Swept per-parameter: `dict 1–64 MiB`, `lc 0–4`, `lp 0/1`, `pb 0–4`, `nice 128/192/273`,
`mf ∈ {bt2,bt3,bt4}`, BCJ `start` offsets 0…0x7fffffff — best is `dict=512 KiB, lc=3, lp=0, pb=0, mf=bt2, nice=96`,
whole-file BCJ. A `.text`/`.rodata` split with BCJ on code only saves a further 452 B but needs a
reassembly step in the launcher (≈ +60 B of launcher, +90 B of ZIP framing), so it was not shipped.

## 4. Every bundle variant built (size, sha256, decode round-trip)

Bundle = `b.xz` + `v.xz` + `d` + 268 B ZIP framing, all built with the same deterministic packer.
"binary" = the candidate executable decoded a real project archive byte-exactly (§6a);
"bundle" = the *packed bundle* ran the complete chain (`launcher → payload → paq8px -d → inverse`) with
the expected sha256 (§6b). Non-candidates were not decoded; `dyn_Os_flto1` is byte-identical to
`dyn_Os` (sha256 `94cc29db…`) so it inherits that proof by identity.

| label | raw binary B | payload (`b.xz`) B | bundle B | bundle sha256 | decode round-trip | what it is |
|---|---:|---:|---:|---|---|---|
| `dyn_Os` | 476,864 | 198,792 | **204,244** | `7a3899220efe888b…` | binary: PASS / bundle: PASS | same source, -Os -flto=2, dynamic, gc-sections, stripped |
| `dyn_Os_flto1` | 476,864 | 198,792 | **204,244** | `7a3899220efe888b…` | binary: PASS / bundle: not run | -Os -flto=1 dynamic |
| `dyn_Os_hard` | 473,552 | 201,004 | **206,456** | `0fe90be2b6518b8e…` | binary: PASS / bundle: not run | -Os -flto=2 dynamic + hard size flags (no-plt/stack-protector off) |
| `dyn_Os_noflto` | 509,632 | 204,672 | **210,124** | `0cd663827adf0895…` | binary: not run / bundle: not run | -Os dynamic, no LTO |
| `dyn_Oz_clang` | 523,208 | 211,248 | **216,700** | `47300c83eea0212b…` | binary: not run / bundle: not run | clang++-17 -Oz dynamic |
| `dyn_O2` | 647,648 | 264,480 | **269,932** | `37c10e004f267e3e…` | binary: not run / bundle: not run | -O2 -flto=2 dynamic |
| `dyn_O3` | 1,012,192 | 437,704 | **443,156** | `411bd72a8deb791e…` | binary: PASS / bundle: PASS | P1 flags verbatim (-O3 -flto) except dynamic link instead of -static |
| `static_Os` | 1,277,800 | 480,876 | **486,328** | `5f2c887528df97a9…` | binary: PASS / bundle: PASS | -Os static (self-contained: no host libstdc++/libc needed) |
| `orig_static_p1` | 1,895,080 | 761,640 | **767,092** | `df55287589ac25f0…` | binary: PASS / bundle: not run | P1's shipped static binary, unmodified (baseline payload) |

The last row re-packs the *baseline* binary with the winner's XZ parameters (761,640 B); P5's own
tuned parameters do 784 B better on that large binary (760,856 B → the 766,308 B published baseline),
which is why the baseline comparison uses 766,308 and not 767,092. On the ~477 KB winner the swept
parameters above are the better ones.

## 5. Winner and exact commands

Winner: **`dyn_Os`**, bundle **204,244 B**, sha256
`7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a`, payload `b.xz` 198,792 B sha256
`35d1f4ff1886a9d6a9714cede09cb7c0d74ce00ca3e3b652d48ab311fc9f6960`.

```sh
# zlib objects (9 C files)
gcc -c -DNDEBUG -I<src>/zlib -Os -fno-fast-math -ffp-contract=off -fexceptions -include unistd.h <zlib>.c -o <obj>.o
# paq8px link (single g++ invocation, sequential TUs)
g++ -DNDEBUG -I<src>/zlib -fno-rtti -std=gnu++17 -ffp-contract=off -fno-fast-math \
    -Os -flto=2 -ffunction-sections -fdata-sections -Wl,--gc-sections \
    -fno-unwind-tables -fno-asynchronous-unwind-tables -Wl,-s \
    <zlib-objects> src/file/*.cpp src/filter/*.cpp src/model/*.cpp src/text/*.cpp src/lstm/*.cpp src/*.cpp \
    -o paq8px
strip --strip-all paq8px

python3 work/make_bundle.py --binary out/build/dyn_Os --out out/bundle_dyn_Os --label dyn_Os
# == inside make_bundle.py:
#   b.xz = lzma.compress(binary, FORMAT_XZ, CHECK_NONE,
#                        filters=[FILTER_X86, FILTER_LZMA2(dict_size=1<<19, lc=3, lp=0, pb=0,
#                                 mode=MODE_NORMAL, mf=MF_BT2, nice_len=96, depth=0)])
#   v.xz, d  = byte-identical copies of the P5 paq8px-blob members
#   zip      = STORED b.xz, STORED v.xz, DEFLATED(9) d with mode 0755 and DOS-epoch timestamps
```

Rebuild the same bundle from scratch (all in `work/`):
```sh
./build_variant.sh dyn_Os dyn_Os          # → out/build/dyn_Os  (476,864 B)
python3 make_bundle.py --binary ../out/build/dyn_Os --out ../out/bundle_dyn_Os --label dyn_Os
bash e2e_bundle.sh dyn_Os ../out/bundle_dyn_Os   # end-to-end proof, writes test/e2e_dyn_Os.json
```

## 6. Proof — what was actually decoded

### 6a. Binary-level proof (real project archives, `test/decode_proof.tsv`)

| binary | archive | out bytes | out sha256 | match expected | rss kB | wall |
|---|---|---:|---|---|---:|---|
| orig | slice32k.paq8px216 | 32768 | 252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8 | YES | 561508 | 0:10.42 |
| dyn_O3 | slice32k.paq8px216 | 32768 | 252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8 | YES | 564120 | 0:09.38 |
| dyn_Os | slice32k.paq8px216 | 32768 | 252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8 | YES | 563812 | 0:12.25 |
| static_Os | slice32k.paq8px216 | 32768 | 252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8 | YES | 562876 | 0:13.42 |
| dyn_Os_hard | slice32k.paq8px216 | 32768 | 252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8 | YES | 563776 | 0:12.15 |
| orig | small.paq8px216 | 53 | c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430 | YES | 347900 | 0:00.25 |
| dyn_O3 | small.paq8px216 | 53 | c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430 | YES | 350696 | 0:00.22 |
| dyn_Os | small.paq8px216 | 53 | c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430 | YES | 350280 | 0:00.27 |
| static_Os | small.paq8px216 | 53 | c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430 | YES | 347376 | 0:00.29 |
| dyn_Os_hard | small.paq8px216 | 53 | c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430 | YES | 350380 | 0:00.33 |

`slice32k.paq8px216` is a **fresh `-1` archive of the first 32,768 bytes of the project's real
`store67/blob67.bin`**, encoded with the original binary (block type `default` = generic/binary path,
12,962 B archive). `small.paq8px216` is P1's calibration fixture (text path). Every candidate binary
reproduces the original binary's output exactly — including the two binaries that were *not* expected
to win (`dyn_Os_hard`, `static_Os`).

### 6b. Full-chain proof through the packed bundle (`test/e2e_dyn_Os.json`)

Fresh directory, `decompressor.zip` unpacked exactly as a verifier would, then the shipped launcher
was run on real archives:

| case | exit | out bytes | out sha256 | match | rss kB | wall | what |
|---|---:|---:|---|---:|---:|---:|---|
| chain_p5fixture | 0 | 17 | `4ab9cb28f180436abbd3a768…` | YES | 21732 | 0:00.12 | P5-certified 41 B paq8px -0L archive of blob35.bin -> full chain -> 17 B original |
| chain_blob35_m1 | 0 | 17 | `4ab9cb28f180436abbd3a768…` | YES | 532868 | 0:00.93 | fresh -1 archive of blob35.bin -> full chain -> 17 B original |
| paq8px_stage_slice32k | 0 | 32768 | `252c561ddcdefb261e605f8e…` | YES | 563896 | 0:12.46 | real 32 KiB slice of blob67.bin, -1 archive -> shipped paq8px stage only |

Case 1 is the P5-certified fixture (`blob-paq.paq`, a genuine `paq8px -0L` archive of the real
35-byte blob `blob35.bin` whose chain output P5 recorded as sha256 `4ab9cb28…`) — **the shipped
bundle produces that same 17-byte output**. Case 2 is a *fresh* `-1` archive of the same blob, so the
chain is also proven through the CM model path at level 1. Case 3 decodes the 32 KiB real-blob-slice
archive with the payload extracted from `b.xz` (extracted payload sha256
`94cc29dbfb2dd1e9ac2ae96fd6699789935cc16585b145ed02ca83361e9764ce` = the built `dyn_Os` binary, i.e. the XZ round-trip is
byte-exact), exercising the generic/binary model path.

### 6c. Inverse stage on a real 2 MiB blob, and independent `unzip` re-verification

The bundle's `v` member (extracted from `v.xz`, sha256 `60015c60…`) was run on P5's real
1,666,504 B blob fixture
(`R1-store-recovery/b2case/blob2m.bin`): exit 0, 2,097,152 B out, sha256
`9dd214e64458c0c36f75a6100aed1a90a7b76ff9dfbb85dc032dea17a1963f50` — identical to P5's certified
`unblob2m.out`, in 0.13 s at 6,780 kB RSS. So both stages of the shipped chain are exercised on real
project data: the payload stage in §6a/§6b, the inverse stage here.

The shipped `out/bundle_dyn_Os/decompressor.zip` (204,244 B, sha256
`7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a`) was unpacked with the system
`unzip` in a fresh directory (`unzip -t`: "No errors detected"), and the launcher was run on both
archives: two runs, exit 0 each, 17-byte outputs, both sha256
`4ab9cb28f180436abbd3a76866c13eb3dc0797888908947f298eec7fb546bcd6` (expected).

### 6d. Flagship `-9` acceptance test

**By director decision (after V1's timing correction) B6 did not run a full-archive decode.**

Decoding the 60,634,501 B blob at `-9` costs roughly as long as the encode (~7 h at ~4.3 GB RSS; P1's own receipt for the 1 MB blob arm at `-9` is 7:36 min / 4,042,460 kB, which scales linearly), so a B6 run would have duplicated the verifier's certification decode hour for hour. The waiter B6 had armed under `locked_run.sh 14400` (full67 + a poc slot, MemAvailable >= 2,500,000 kB, archive-quiescence check) was cancelled before it ever acquired `full67`; the lock was never taken by B6. **The definitive full-archive proof is therefore V1's single certification decode, running exactly the shipped 204,244 B artifact** (`7a389922…`, components in §5) against `runs/full_blob_m9/blob_full.paq8px216`, expecting blob 60,634,501 B / `9c869f14…` and then `enwik8` 100,000,000 B / `2b49720e…`. What this report contributes to that decode: the artifact receipt, the measured performance envelope (level-1 decode peak RSS 563,896 kB; the `-Os` payload runs ~10-20 % slower than the `-O3` binary in §6a's single-run timings, so size the certificate window at ~7-9 h and ~4.3 GB), and the fact that both stages were already exercised end-to-end on real project archives below.

### 6e. What is *not* yet proven

The `-9` level itself has not been exercised by B6 — every B6 measurement is at `-0L`/`-1` (≤ 564 MB RSS) because a `-9` decode needs ~4.3 GB and hours, outside the light lane and duplicative of V1's certification decode. The residual risk is therefore exactly this: codegen behaviour at `-9`'s larger model tables. It is bounded by construction — identical source commit, `-ffp-contract=off -fno-fast-math` preserved, no `-march=native`, and `Shared::mem = 65536 << level` means the level changes only table *sizes*, not which models run, so the level-1 proof exercises the same model code — but it is retired only by the verifier's single `-9` certification decode of this artifact, and by nothing B6 did. If that decode fails, the fallback is `dyn_O3` (443,156 B, §7, also proven on the small chain, packaged and waiting at `out/bundle_dyn_O3/decompressor.zip`), whose codegen is P1's exact `-O3 -flto` recipe with only `-static` removed.

```sh
cd <unpacked bundle> && ./d /path/blob_full.paq8px216 /path/restored_blob.bin   # the inverse stage
# paq8px stage alone (no inverse), to compare against the blob hash directly:
python3 -c "import lzma,pathlib;p=pathlib.Path('.');(p/'b').write_bytes(lzma.open(p/'b.xz').read())"
./b -d /path/blob_full.paq8px216 /path/restored_blob.bin
sha256sum /path/restored_blob.bin   # must equal 9c869f141bc91a9b164e2cd4faecfea5bb648bd996c8c0f5e865b4e4010be425
```

Residual risk is bounded by construction: same source commit, same `-ffp-contract=off -fno-fast-math`
semantics, no `-march=native`, and `Shared::mem = 65536 << level` means the level changes only model
*table sizes*, not which models run — the level-1 proof therefore exercises the same model code as
`-9`. The remaining uncertainty is codegen at larger tables, which the single `-9` decode settles.

## 7. Fallback ladder and lower-risk alternatives

If the verifier's `-9` decode of the winner hits anything unexpected, use the next rung — each rung is
packaged, hashed and small-chain proven, so no rework is needed:

1. **`dyn_O3`** — `out/bundle_dyn_O3/decompressor.zip`, **443,156 B**, sha256
   `411bd72a8deb791e304f7d2ee56b86d35c6990166d90515b6d8f3de4be83b570` (proven 3/3 at bundle level).
2. **`static_Os`** — `out/bundle_static_Os/decompressor.zip`, 486,328 B (proven 3/3; no host DSOs).
3. **P5's original** — `P5-bundles/paq8px-blob/decompressor.zip`, 766,308 B, sha256
   `debed42c…` (the published baseline, unchanged on disk).

Rungs 1–3 cost 238,912 / 282,084 / 562,064 B more than the winner respectively.

## 7b. Lower-risk alternatives (if a dynamic payload is unacceptable)

* **`dyn_O3`** (bundle 443,156 B): P1's exact `-O3 -flto=2` flags, only
  `-static` removed. Same optimization level as the proven binary, so the codegen risk is the
  smallest of any candidate; costs 238,912 B more.
* **`static_Os`** (bundle 486,328 B): fully static, needs no host
  libstdc++/libc at all. Costs 282,084 B more
  than the winner but is self-contained.
* The winner and `dyn_O3`/`static_Os` are dynamic/static respectively: `dyn_Os` requires
  `ld-linux-x86-64.so.2`, `libc.so.6`, `libstdc++.so.6` (+ transitive `libm`/`libgcc_s`) on the
  decode host, exactly like the published 16,306 B baseline ELF and like cmix/paq8l/paq9a in P5's
  table. `static_Os` removes that requirement entirely.

## 8. Boundaries

* No source line was modified; no transform/model was removed; nothing about what the decoder outputs
  changed. Only build flags, link mode, and payload packing.
* Not attempted (and reported as such): pruning encoder-only code or the LSTM (would change the
  decoder's capability set and needs its own behavioural proof), and `-march=native` (portability).
* Measured-but-rejected levers, with numbers, are in §2/§3 so nothing is silently dropped.
* Light-lane discipline held: every build ≤ 285,260 kB peak RSS, every decode test ≤ 564,120 kB;
  no heavy encode, no lock taken, no pattern-kill, nothing written outside `swarm67/B6-bundle/`.
* The full-archive `-9` decode was planned, armed under `locked_run.sh`, and then **cancelled before it
  ever acquired `full67`** once V1's timing correction (~7 h, not ~23 min) showed it would duplicate
  the verifier's certification decode. That decode, on exactly this 204,244 B artifact, is the
  definitive full-archive proof; B6's evidence is the small-chain/real-archive work in §6.