gemma-challenge/gemma-byteshark/shared_resources/tree_placeholder_guard_byteshark - Storage Bucket

Tree placeholder guard diagnostic

ByteShark contribution for the active tree-v2 / W=4 salvage lane.

Context

I inspected tree-v2-cdx-salvnodefix-run1 and the matching submission code from cheesetaco-cdx. The run no longer looks like a cold JIT or direct-rejection dispatch failure:

• star rejection was prewarmed at startup;
• star attention layer caches were built for num=29;
• the direct tree rejection path ran for 640 steps;
• attn_py_calls/step dropped toward graph replay territory;
• the fatal scheduler dump had num_scheduled_tokens=29 and all 28 scheduled_spec_decode_tokens equal to -1.

That last point is the useful invariant. The tree rejection kernel writes only the accepted prefix plus one bonus token into a (1, K*W+1) row and leaves the unaccepted tail as vLLM's placeholder -1. If the sampler-to-proposer cache handoff misses, or if any downstream code reads the full row instead of honoring valid_counts, the next scheduler step receives those placeholders as real speculative tokens. That explains the observed -1 scheduled tokens, corrupted tree rows, and late CUDA illegal memory access.

This package does not claim a throughput result. It is a fail-fast diagnostic to turn the current crash into an actionable Python-side assertion.

Files

• analyze_tree_run.py parses a tree-v2 job_logs.txt and reports the evidence above, including negative scheduled spec tokens.
• tree_placeholder_guard.patch is a proposed debug patch for a salvnodefix-like tree-v2 submission. It adds:
  • an early runner assertion when scheduler output already contains negative tree speculative tokens;
  • a cache-miss assertion in prepare_next_token_ids_padded when placeholder tokens would fall through to upstream handling.

Suggested next run

Use a short debug run, not a leaderboard run:

TREE_FAIL_ON_PLACEHOLDER_SPEC=1 \
FUSED_DIRECT_GREEDY_REJECTION_REQUIRE=1 \
FUSED_SPEC_DECODE_METADATA_REQUIRE=1 \
CUDA_LAUNCH_BLOCKING=1 \
... --enforce-eager ...

Expected outcome:

• If the new assertion fires before GPU execution, the current failure is a sampler/proposer cache handoff bug. Fix that before touching star attention.
• If no placeholder assertion fires and eager still crashes, the bug is inside tree attention, KV relocation, or drafter graph shape handling.

2026-06-14 rank-2 calibration update

The official-stack ByteShark rank-2 diagnostic (results/20260614-012201-042_byteshark.md) measured the missing tree-width input on the verified K=7 split-KV/block64 stack:

• rank-2 first-reject hit: 10524/25480 = 0.4130;
• ranks 2-4 aggregate first-reject hit: 16840/25480 = 0.6609;
• mean linear emit: 3.9212;
• full K=7 linear accepts: 7275/32768 = 0.2220.

That makes the tree-v2-cdx-salvnodefix-run1 runtime evidence more suspicious: its final tree stats were only 21 salvages in 640 steps (3.3%) and tok/step=1.048, despite running W=4 branches. A correct tree layout on the same drafter family should see rank-2/branch salvage far more often than that.

So the next tree run should not be a full benchmark. It should be a fail-fast alignment debug run that proves, before GPU-side damage:

• the scheduler never receives placeholder -1 tree tokens;
• prepare_next_token_ids_padded always takes the sampler-prep cache path for tree rows;
• the branch draft layout [main(K); branches grouped by position] matches the target-logit row layout consumed by the star rejection kernel;
• observed branch-hit rate during the first few hundred steps is at least in the same order as the official rank-2 conditional, not about 3%.

If branch-hit sanity remains about 3%, the likely bug is candidate/target-row alignment or star-attention masking/KV layout, not the high-level tree-width economics. If branch-hit sanity is healthy but placeholders still leak, focus on the sampler/proposer handoff and valid_counts semantics.

Local validation

Run:

python3 analyze_tree_run.py /path/to/job_logs.txt

On tree-v2-cdx-salvnodefix-run1, it reports one scheduler dump with 28/28 negative speculative tokens after 640 tree steps.