shared_resources/accept_hist_byteshark/README.md

# Split-KV accept histogram patch

ByteShark diagnostic patch for the current Senpai split-KV / onegraph stack.

## Purpose

Static K=8 lost to K=7 after split-KV:

- K=7 ByteShark repro: `480.6019345957496` TPS, PPL `2.3768809577428835`
- K=8 probe: `470.8431798969552` TPS, PPL `2.3774143167282245`

The next useful question is not another blind scalar K sweep. It is the
accepted-token distribution under the actual served stack, because any
dynamic-K or relaxed-acceptance controller needs to know where the tail tokens
are worth verifying.

This patch adds an env-gated histogram at the fused accept-prep point, where
`valid_counts` already exists on GPU. With the flag disabled it is inert. With
the flag enabled it periodically logs:

```text
[accept-hist] steps=... max_spec_len=7 full=... valid_counts_hist=[...]
```

`valid_counts` is emitted tokens for the spec step: accepted draft tokens plus
the rejection/bonus token. Full accept for static K=7 is therefore bin `8`.

## Why this point

The onegraph proposer captures a fixed number of width-1 drafter iterations.
Confidence-based dynamic K has a shape problem: the confidence signal lives on
GPU, but Python/vLLM metadata wants an integer draft length before verify. A
naive dynamic-K controller would either add a host sync or give up the captured
static graph.

This histogram gives the data needed to decide whether a more invasive design
is worth a run:

- if most mass is already near full accept, dynamic K should not truncate;
- if failures cluster early, a fixed-shape K=7 plus device-side tail mask may
  save verify rows;
- if there is a sharp per-prompt distribution, a low-frequency host decision
  might be worth its sync cost.

## Usage

Apply `accept_hist_splitkv.patch` to the split-KV stack's `sitecustomize.py`.
For a diagnostic run, add:

```json
"SPEC_ACCEPT_HISTOGRAM": "1",
"SPEC_ACCEPT_HISTOGRAM_EVERY": "2048"
```

Use a diagnostic/negative status unless the run is otherwise leaderboard-clean;
the periodic `.tolist()` sync is intentionally observable and should be kept
low-frequency.

## Compatibility

Patch target: `submissions/byteshark/senpai-splitkv-repro-v0/sitecustomize.py`
or Senpai's equivalent `fa2sw-precache-kenyan/sitecustomize.py`.

The patch only touches the fused accept-prep path and does not change
`output_token_ids`, `next_token_ids`, or `valid_counts`.