davanstrien/gists/sys1-unpadded-flash - Storage Bucket

sys1: unpadded (packed) forward for mixed-length batches

Prototype diff and benchmark results for alvarobartt/sys1.

With flash attention, a batch of mixed-length sequences is padded to the longest row. Flash attention skips the padding, but the linear, norm and MLP layers still run on it. This prototype packs the tokens once and runs the encoder and the Laya decision head on the packed [tokens, hidden] tensor.

Files

• unpadded-flash.diff: the change, against sys1 main at 9f0b5d7 (about 290 lines in src/models/modernbert.rs and src/models/laya.rs). SYS1_PACKED=0 or 1 switches the path off or on; it defaults to on with flash attention.
• build_and_bench.sh: builds the patched binary on the GPU node and runs the four configurations. The /bucket/... paths are where the diff, the client and the results were mounted for the run.
• client_bench.py: the HTTP client (64 concurrent requests, optional length sorting).
• results/: one JSON per configuration.

Setup

• A10G, --dtype bf16 --attention flash-attn-2, model convaiinnovations/laya
• 2,048 pages from the WebOrganizer/FormatAnnotations-Llama-3.1-405B-FP8 test split, truncated to 512 tokens (mean 372 tokens, 44% at the cap)
• 1 question (5-level score) and 3 questions (score, yes/no, 24-way choice) per page

Results

Path	Request order	1 question (docs/s)	3 questions (decisions/s)
padded (main)	arrival	36.2	37.1
padded (main)	sorted by length	51.3	51.4
packed (this diff)	arrival	67.4	70.0
packed (this diff)	sorted by length	63.5	66.6

Top-1 on the 24-way choice is identical in all four runs (0.1655 agreement with the dataset's labels). On CPU, padded and packed gave identical probabilities for a small mixed-length batch.

Limits

One run per configuration. Prototype quality: no unit tests, flash-attn-3 and the other Laya variants are untested, and the non-flash fallback is a per-sequence loop written only to check parity.