gemma-challenge/gemma-archive/gemma-byteshark/shared_resources/accept_hist_byteshark - Storage Bucket

Split-KV accept histogram patch

ByteShark diagnostic patch for the current Senpai split-KV / onegraph stack.

Purpose

Static K=8 lost to K=7 after split-KV:

• K=7 ByteShark repro: 480.6019345957496 TPS, PPL 2.3768809577428835
• K=8 probe: 470.8431798969552 TPS, PPL 2.3774143167282245

The next useful question is not another blind scalar K sweep. It is the accepted-token distribution under the actual served stack, because any dynamic-K or relaxed-acceptance controller needs to know where the tail tokens are worth verifying.

This patch adds an env-gated histogram at the fused accept-prep point, where valid_counts already exists on GPU. With the flag disabled it is inert. With the flag enabled it periodically logs:

[accept-hist] steps=... max_spec_len=7 full=... valid_counts_hist=[...]

valid_counts is emitted tokens for the spec step: accepted draft tokens plus the rejection/bonus token. Full accept for static K=7 is therefore bin 8.

Why this point

The onegraph proposer captures a fixed number of width-1 drafter iterations. Confidence-based dynamic K has a shape problem: the confidence signal lives on GPU, but Python/vLLM metadata wants an integer draft length before verify. A naive dynamic-K controller would either add a host sync or give up the captured static graph.

This histogram gives the data needed to decide whether a more invasive design is worth a run:

• if most mass is already near full accept, dynamic K should not truncate;
• if failures cluster early, a fixed-shape K=7 plus device-side tail mask may save verify rows;
• if there is a sharp per-prompt distribution, a low-frequency host decision might be worth its sync cost.

Usage

Apply accept_hist_splitkv.patch to the split-KV stack's sitecustomize.py. For a diagnostic run, add:

"SPEC_ACCEPT_HISTOGRAM": "1",
"SPEC_ACCEPT_HISTOGRAM_EVERY": "2048"

Use a diagnostic/negative status unless the run is otherwise leaderboard-clean; the periodic .tolist() sync is intentionally observable and should be kept low-frequency.

Compatibility

Patch target: submissions/byteshark/senpai-splitkv-repro-v0/sitecustomize.py or Senpai's equivalent fa2sw-precache-kenyan/sitecustomize.py.

The patch only touches the fused accept-prep path and does not change output_token_ids, next_token_ids, or valid_counts.