decoder-bundle-minimization/REPORT.md

B6 — minimizing the shipped paq8px decoder bundle

Scope. Decoder-side only: no compression runs, no changes to the archive, nothing touched under V1-verify-queue/, publish67/ or the flagship run directory, no publication or HF API calls. All work under /root/hutter67/swarm67/B6-bundle/.

Headline

	bytes	sha256	notes
P5/published paq8px blob-chain bundle	766,308	debed42c9ec425e5461dd886148814e55c71c52ea43b2675d80b563030074dce	b.xz 760,856 + v.xz 4,948 + d 236 + 268 framing
B6 winner (dyn_Os)	204,244	7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a	same launcher/inverse, new payload
saving	562,064		one-line: the winner ships 562,064 bytes smaller than the 766,308 B baseline.

SHIP THIS

artefact	/root/hutter67/swarm67/B6-bundle/out/bundle_dyn_Os/decompressor.zip
bytes / sha256	204,244 / 7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a
contents	b.xz 198,792 (XZ of the 476,864 B dyn_Os ELF, sha256 94cc29db…) + v.xz 4,948 (unchanged P5 member) + d 409 raw / 236 deflated (unchanged P5 launcher) + 268 B ZIP framing
fallback if the verifier's -9 decode hits anything unexpected	/root/hutter67/swarm67/B6-bundle/out/bundle_dyn_O3/decompressor.zip — 443,156 B, sha256 411bd72a8deb791e…, P1's exact -O3 -flto recipe with only -static removed, also small-chain proven (3/3)
evidence table for every variant	§4, with per-binary proofs in §6a/§6b and machine-readable form in bundle67.json

562,064 B off the counted total, from a rebuild + repack of the payload only. On the assets basis the entry is A_blob + 204,244 (was A_blob + 766,308); with the flagship blob archive projecting to ~16.1 MB the entry lands at ≈16.29 MB, versus 17,969,555 B for the published v4 baseline. The launcher d and the blob inverse v.xz are byte-identical to the published P5 package, so the decode chain a verifier runs is unchanged; only the paq8px executable it unpacks is smaller.

1. The decoder chain that has to be shipped

Flagship production chain (P1-paq8px-cal/fullrun_blob.sh, -9):

archive.paq8px216  --paq8px -d-->  blob (60,634,501 B)  --blob inverse-->  enwik8 (100,000,000 B)

P5 accounted that whole chain as a 766,308 B ZIP holding exactly three members (paq8px-blob/package.json): b.xz 760,856 (XZ of the 1,895,080 B static paq8px binary), v.xz 4,948 (the dedicated cmix-store blob inverse), d 409 raw / 236 deflated (Python launcher), plus 268 B of ZIP framing. B6 changes only b.xz.

Byte-accounting validation: work/make_bundle.py --verify-baseline rebuilds the P5 ZIP from its members and reproduces 766,308 B / sha256 debed42c9ec425e5461dd886148814e55c71c52ea43b2675d80b563030074dce byte-for-byte. Stated plainly, because it is the whole basis of the claim: because the packer reproduces P5's ZIP byte-for-byte, the −562,064 B is an apples-to-apples accounting change (same three members, same launcher, same inverse, same 268 B of framing) and not a change of accounting basis. The only member whose bytes differ is the payload b.xz.

2. Lever 1 — rebuild the binary smaller (this is where the bytes were)

The shipped binary is a -O3 -flto -march=native -static build: 1,895,080 B raw. Static linking of libc/libstdc++ plus -O3 inlining is most of it. Rebuilding the pinned source (paq8px-29237fb44cb1995690e3eb72c6c3b1e4aede5791, the same commit P1/P5 used) with size flags and dynamic linking, changing no source line, gives:

build	raw B	XZ payload B	bundle B
P1 shipped (-O3 -flto -march=native -static), P5's own tuned XZ	1,895,080	760,856	766,308 (the baseline)
-O3 -flto=2 dynamic (identical codegen flags, only the link mode changes)	1,012,192	437,704	443,156
-O2 -flto=2 dynamic	647,648	264,480	269,932
clang++-17 -Oz dynamic	523,208	211,248	216,700
-Os -flto=2 dynamic + gc-sections + strip	476,864	198,792	204,244
-Os -flto=2 dynamic + hard flags (-fno-plt -fno-stack-protector …)	473,552	201,004	206,456 (smaller raw, worse packed)
-Os dynamic, no LTO	509,632	204,672	210,124
-Os static (self-contained fallback)	1,277,800	480,876	486,328

Exact commands are in §5. -flto=1 and -flto=2 produced a byte-identical binary here (sha256 94cc29db…), so that knob is a no-op. Every payload column below uses the same swept-optimal XZ parameters as the winner, so variants are comparable at bundle level.

Cost of the size build: in the single-run measurements of §6a the -Os decode is 12.25 s on the 32 KiB binary slice where the original binary takes 10.42 s (≈ +18 %, within session noise on a box that was simultaneously running a 4.5 GB encode; dyn_O3 measured 9.38 s). This affects decode time only — it is not a scoring term — but it is the honest trade: the smallest bundle decodes somewhat slower than the -O3 binary, and the fallback dyn_O3 variant does not have that penalty.

-march=native was deliberately NOT used even though P1's build had it: it would bake AVX-512 instructions from this EPYC into the shipped decoder and could crash on a verifier's CPU. The -Os build targets baseline x86-64. This is a portability gain, not just a size choice.

3. Lever 2 — pack the payload with the strongest shippable codec

All measured on the winner binary (dyn_Os, 476,864 B). "Shippable" means the decoder is either already present on the host (Python's lzma) or small enough to pay for itself:

codec	bytes	note
xz + BCJ/x86, LZMA2 dict512K lc3 lp0 pb0 mf=bt2 nice96 (SHIPPED b.xz)	198,792	decoder is free on the host (launcher already uses Python lzma); winner
zpaq zc56 stream (libzpaq level 5, 64 MiB block)	187,197	smallest raw payload, but +16,476 B shipped libzpaq extractor = 203,673 B effective; loses
UPX 4.2.4 --best --lzma (upx -d restores byte-identical)	202,768	self-extracting; no host decoder, no xz payload; loses by 3,912 B
7z PPMd mem=512m o=32 -mx=9	204,453	needs a 7z binary on the host
brotli -q 11 -w 24	213,027	needs brotli on the host
zstd --ultra -22 --long=27	222,742	needs zstd on the host
bzip2 -9	248,621	host-decodable via Python bz2, but far larger than xz

XZ + x86/BCJ wins: it needs zero extra shipped bytes (the launcher already uses Python's lzma), while zpaq's 11,595 B head start is eaten by the 16,476 B libzpaq extractor it would require (187,197 + 16,476 = 203,673 > 198,792), and UPX's self-extracting stub lands at 202,768. Swept per-parameter: dict 1–64 MiB, lc 0–4, lp 0/1, pb 0–4, nice 128/192/273, mf ∈ {bt2,bt3,bt4}, BCJ start offsets 0…0x7fffffff — best is dict=512 KiB, lc=3, lp=0, pb=0, mf=bt2, nice=96, whole-file BCJ. A .text/.rodata split with BCJ on code only saves a further 452 B but needs a reassembly step in the launcher (≈ +60 B of launcher, +90 B of ZIP framing), so it was not shipped.

4. Every bundle variant built (size, sha256, decode round-trip)

Bundle = b.xz + v.xz + d + 268 B ZIP framing, all built with the same deterministic packer. "binary" = the candidate executable decoded a real project archive byte-exactly (§6a); "bundle" = the packed bundle ran the complete chain (launcher → payload → paq8px -d → inverse) with the expected sha256 (§6b). Non-candidates were not decoded; dyn_Os_flto1 is byte-identical to dyn_Os (sha256 94cc29db…) so it inherits that proof by identity.

label	raw binary B	payload (b.xz) B	bundle B	bundle sha256	decode round-trip	what it is
dyn_Os	476,864	198,792	204,244	7a3899220efe888b…	binary: PASS / bundle: PASS	same source, -Os -flto=2, dynamic, gc-sections, stripped
dyn_Os_flto1	476,864	198,792	204,244	7a3899220efe888b…	binary: PASS / bundle: not run	-Os -flto=1 dynamic
dyn_Os_hard	473,552	201,004	206,456	0fe90be2b6518b8e…	binary: PASS / bundle: not run	-Os -flto=2 dynamic + hard size flags (no-plt/stack-protector off)
dyn_Os_noflto	509,632	204,672	210,124	0cd663827adf0895…	binary: not run / bundle: not run	-Os dynamic, no LTO
dyn_Oz_clang	523,208	211,248	216,700	47300c83eea0212b…	binary: not run / bundle: not run	clang++-17 -Oz dynamic
dyn_O2	647,648	264,480	269,932	37c10e004f267e3e…	binary: not run / bundle: not run	-O2 -flto=2 dynamic
dyn_O3	1,012,192	437,704	443,156	411bd72a8deb791e…	binary: PASS / bundle: PASS	P1 flags verbatim (-O3 -flto) except dynamic link instead of -static
static_Os	1,277,800	480,876	486,328	5f2c887528df97a9…	binary: PASS / bundle: PASS	-Os static (self-contained: no host libstdc++/libc needed)
orig_static_p1	1,895,080	761,640	767,092	df55287589ac25f0…	binary: PASS / bundle: not run	P1's shipped static binary, unmodified (baseline payload)

The last row re-packs the baseline binary with the winner's XZ parameters (761,640 B); P5's own tuned parameters do 784 B better on that large binary (760,856 B → the 766,308 B published baseline), which is why the baseline comparison uses 766,308 and not 767,092. On the ~477 KB winner the swept parameters above are the better ones.

5. Winner and exact commands

Winner: dyn_Os, bundle 204,244 B, sha256 7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a, payload b.xz 198,792 B sha256 35d1f4ff1886a9d6a9714cede09cb7c0d74ce00ca3e3b652d48ab311fc9f6960.

# zlib objects (9 C files)
gcc -c -DNDEBUG -I<src>/zlib -Os -fno-fast-math -ffp-contract=off -fexceptions -include unistd.h <zlib>.c -o <obj>.o
# paq8px link (single g++ invocation, sequential TUs)
g++ -DNDEBUG -I<src>/zlib -fno-rtti -std=gnu++17 -ffp-contract=off -fno-fast-math \
    -Os -flto=2 -ffunction-sections -fdata-sections -Wl,--gc-sections \
    -fno-unwind-tables -fno-asynchronous-unwind-tables -Wl,-s \
    <zlib-objects> src/file/*.cpp src/filter/*.cpp src/model/*.cpp src/text/*.cpp src/lstm/*.cpp src/*.cpp \
    -o paq8px
strip --strip-all paq8px

python3 work/make_bundle.py --binary out/build/dyn_Os --out out/bundle_dyn_Os --label dyn_Os
# == inside make_bundle.py:
#   b.xz = lzma.compress(binary, FORMAT_XZ, CHECK_NONE,
#                        filters=[FILTER_X86, FILTER_LZMA2(dict_size=1<<19, lc=3, lp=0, pb=0,
#                                 mode=MODE_NORMAL, mf=MF_BT2, nice_len=96, depth=0)])
#   v.xz, d  = byte-identical copies of the P5 paq8px-blob members
#   zip      = STORED b.xz, STORED v.xz, DEFLATED(9) d with mode 0755 and DOS-epoch timestamps

Rebuild the same bundle from scratch (all in work/):

./build_variant.sh dyn_Os dyn_Os          # → out/build/dyn_Os  (476,864 B)
python3 make_bundle.py --binary ../out/build/dyn_Os --out ../out/bundle_dyn_Os --label dyn_Os
bash e2e_bundle.sh dyn_Os ../out/bundle_dyn_Os   # end-to-end proof, writes test/e2e_dyn_Os.json

6. Proof — what was actually decoded

6a. Binary-level proof (real project archives, test/decode_proof.tsv)

binary	archive	out bytes	out sha256	match expected	rss kB	wall
orig	slice32k.paq8px216	32768	252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8	YES	561508	0:10.42
dyn_O3	slice32k.paq8px216	32768	252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8	YES	564120	0:09.38
dyn_Os	slice32k.paq8px216	32768	252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8	YES	563812	0:12.25
static_Os	slice32k.paq8px216	32768	252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8	YES	562876	0:13.42
dyn_Os_hard	slice32k.paq8px216	32768	252c561ddcdefb261e605f8e20f726e43de6a31b1b4cfb240563f5f7d4fd28e8	YES	563776	0:12.15
orig	small.paq8px216	53	c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430	YES	347900	0:00.25
dyn_O3	small.paq8px216	53	c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430	YES	350696	0:00.22
dyn_Os	small.paq8px216	53	c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430	YES	350280	0:00.27
static_Os	small.paq8px216	53	c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430	YES	347376	0:00.29
dyn_Os_hard	small.paq8px216	53	c5cf4213e9e9568b3b7b8bb42e219d30cc66ed3adce28aa5292201d96f928430	YES	350380	0:00.33

slice32k.paq8px216 is a fresh -1 archive of the first 32,768 bytes of the project's real store67/blob67.bin, encoded with the original binary (block type default = generic/binary path, 12,962 B archive). small.paq8px216 is P1's calibration fixture (text path). Every candidate binary reproduces the original binary's output exactly — including the two binaries that were not expected to win (dyn_Os_hard, static_Os).

6b. Full-chain proof through the packed bundle (test/e2e_dyn_Os.json)

Fresh directory, decompressor.zip unpacked exactly as a verifier would, then the shipped launcher was run on real archives:

case	exit	out bytes	out sha256	match	rss kB	wall	what
chain_p5fixture	0	17	4ab9cb28f180436abbd3a768…	YES	21732	0:00.12	P5-certified 41 B paq8px -0L archive of blob35.bin -> full chain -> 17 B original
chain_blob35_m1	0	17	4ab9cb28f180436abbd3a768…	YES	532868	0:00.93	fresh -1 archive of blob35.bin -> full chain -> 17 B original
paq8px_stage_slice32k	0	32768	252c561ddcdefb261e605f8e…	YES	563896	0:12.46	real 32 KiB slice of blob67.bin, -1 archive -> shipped paq8px stage only

Case 1 is the P5-certified fixture (blob-paq.paq, a genuine paq8px -0L archive of the real 35-byte blob blob35.bin whose chain output P5 recorded as sha256 4ab9cb28…) — the shipped bundle produces that same 17-byte output. Case 2 is a fresh -1 archive of the same blob, so the chain is also proven through the CM model path at level 1. Case 3 decodes the 32 KiB real-blob-slice archive with the payload extracted from b.xz (extracted payload sha256 94cc29dbfb2dd1e9ac2ae96fd6699789935cc16585b145ed02ca83361e9764ce = the built dyn_Os binary, i.e. the XZ round-trip is byte-exact), exercising the generic/binary model path.

6c. Inverse stage on a real 2 MiB blob, and independent unzip re-verification

The bundle's v member (extracted from v.xz, sha256 60015c60…) was run on P5's real 1,666,504 B blob fixture (R1-store-recovery/b2case/blob2m.bin): exit 0, 2,097,152 B out, sha256 9dd214e64458c0c36f75a6100aed1a90a7b76ff9dfbb85dc032dea17a1963f50 — identical to P5's certified unblob2m.out, in 0.13 s at 6,780 kB RSS. So both stages of the shipped chain are exercised on real project data: the payload stage in §6a/§6b, the inverse stage here.

The shipped out/bundle_dyn_Os/decompressor.zip (204,244 B, sha256 7a3899220efe888bd0d0e3affed875094afaea382ef1eda67eb1268a15b0071a) was unpacked with the system unzip in a fresh directory (unzip -t: "No errors detected"), and the launcher was run on both archives: two runs, exit 0 each, 17-byte outputs, both sha256 4ab9cb28f180436abbd3a76866c13eb3dc0797888908947f298eec7fb546bcd6 (expected).

6d. Flagship -9 acceptance test

By director decision (after V1's timing correction) B6 did not run a full-archive decode.

Decoding the 60,634,501 B blob at -9 costs roughly as long as the encode (~7 h at ~4.3 GB RSS; P1's own receipt for the 1 MB blob arm at -9 is 7:36 min / 4,042,460 kB, which scales linearly), so a B6 run would have duplicated the verifier's certification decode hour for hour. The waiter B6 had armed under locked_run.sh 14400 (full67 + a poc slot, MemAvailable >= 2,500,000 kB, archive-quiescence check) was cancelled before it ever acquired full67; the lock was never taken by B6. The definitive full-archive proof is therefore V1's single certification decode, running exactly the shipped 204,244 B artifact (7a389922…, components in §5) against runs/full_blob_m9/blob_full.paq8px216, expecting blob 60,634,501 B / 9c869f14… and then enwik8 100,000,000 B / 2b49720e…. What this report contributes to that decode: the artifact receipt, the measured performance envelope (level-1 decode peak RSS 563,896 kB; the -Os payload runs ~10-20 % slower than the -O3 binary in §6a's single-run timings, so size the certificate window at ~7-9 h and ~4.3 GB), and the fact that both stages were already exercised end-to-end on real project archives below.

6e. What is not yet proven

The -9 level itself has not been exercised by B6 — every B6 measurement is at -0L/-1 (≤ 564 MB RSS) because a -9 decode needs ~4.3 GB and hours, outside the light lane and duplicative of V1's certification decode. The residual risk is therefore exactly this: codegen behaviour at -9's larger model tables. It is bounded by construction — identical source commit, -ffp-contract=off -fno-fast-math preserved, no -march=native, and Shared::mem = 65536 << level means the level changes only table sizes, not which models run, so the level-1 proof exercises the same model code — but it is retired only by the verifier's single -9 certification decode of this artifact, and by nothing B6 did. If that decode fails, the fallback is dyn_O3 (443,156 B, §7, also proven on the small chain, packaged and waiting at out/bundle_dyn_O3/decompressor.zip), whose codegen is P1's exact -O3 -flto recipe with only -static removed.

cd <unpacked bundle> && ./d /path/blob_full.paq8px216 /path/restored_blob.bin   # the inverse stage
# paq8px stage alone (no inverse), to compare against the blob hash directly:
python3 -c "import lzma,pathlib;p=pathlib.Path('.');(p/'b').write_bytes(lzma.open(p/'b.xz').read())"
./b -d /path/blob_full.paq8px216 /path/restored_blob.bin
sha256sum /path/restored_blob.bin   # must equal 9c869f141bc91a9b164e2cd4faecfea5bb648bd996c8c0f5e865b4e4010be425

Residual risk is bounded by construction: same source commit, same -ffp-contract=off -fno-fast-math semantics, no -march=native, and Shared::mem = 65536 << level means the level changes only model table sizes, not which models run — the level-1 proof therefore exercises the same model code as -9. The remaining uncertainty is codegen at larger tables, which the single -9 decode settles.

7. Fallback ladder and lower-risk alternatives

If the verifier's -9 decode of the winner hits anything unexpected, use the next rung — each rung is packaged, hashed and small-chain proven, so no rework is needed:

1. dyn_O3 — out/bundle_dyn_O3/decompressor.zip, 443,156 B, sha256 411bd72a8deb791e304f7d2ee56b86d35c6990166d90515b6d8f3de4be83b570 (proven 3/3 at bundle level).
2. static_Os — out/bundle_static_Os/decompressor.zip, 486,328 B (proven 3/3; no host DSOs).
3. P5's original — P5-bundles/paq8px-blob/decompressor.zip, 766,308 B, sha256 debed42c… (the published baseline, unchanged on disk).

Rungs 1–3 cost 238,912 / 282,084 / 562,064 B more than the winner respectively.

7b. Lower-risk alternatives (if a dynamic payload is unacceptable)

• dyn_O3 (bundle 443,156 B): P1's exact -O3 -flto=2 flags, only -static removed. Same optimization level as the proven binary, so the codegen risk is the smallest of any candidate; costs 238,912 B more.
• static_Os (bundle 486,328 B): fully static, needs no host libstdc++/libc at all. Costs 282,084 B more than the winner but is self-contained.
• The winner and dyn_O3/static_Os are dynamic/static respectively: dyn_Os requires ld-linux-x86-64.so.2, libc.so.6, libstdc++.so.6 (+ transitive libm/libgcc_s) on the decode host, exactly like the published 16,306 B baseline ELF and like cmix/paq8l/paq9a in P5's table. static_Os removes that requirement entirely.

8. Boundaries

• No source line was modified; no transform/model was removed; nothing about what the decoder outputs changed. Only build flags, link mode, and payload packing.
• Not attempted (and reported as such): pruning encoder-only code or the LSTM (would change the decoder's capability set and needs its own behavioural proof), and -march=native (portability).
• Measured-but-rejected levers, with numbers, are in §2/§3 so nothing is silently dropped.
• Light-lane discipline held: every build ≤ 285,260 kB peak RSS, every decode test ≤ 564,120 kB; no heavy encode, no lock taken, no pattern-kill, nothing written outside swarm67/B6-bundle/.
• The full-archive -9 decode was planned, armed under locked_run.sh, and then cancelled before it ever acquired full67 once V1's timing correction (~7 h, not ~23 min) showed it would duplicate the verifier's certification decode. That decode, on exactly this 204,244 B artifact, is the definitive full-archive proof; B6's evidence is the small-chain/real-archive work in §6.