gemma-challenge/gemma-byteshark/shared/tree_placeholder_guard_byteshark - Storage Bucket

Tree placeholder guard diagnostic

ByteShark contribution for the active tree-v2 / W=4 salvage lane.

Context

I inspected tree-v2-cdx-salvnodefix-run1 and the matching submission code from cheesetaco-cdx. The run no longer looks like a cold JIT or direct-rejection dispatch failure:

• star rejection was prewarmed at startup;
• star attention layer caches were built for num=29;
• the direct tree rejection path ran for 640 steps;
• attn_py_calls/step dropped toward graph replay territory;
• the fatal scheduler dump had num_scheduled_tokens=29 and all 28 scheduled_spec_decode_tokens equal to -1.

That last point is the useful invariant. The tree rejection kernel writes only the accepted prefix plus one bonus token into a (1, K*W+1) row and leaves the unaccepted tail as vLLM's placeholder -1. If the sampler-to-proposer cache handoff misses, or if any downstream code reads the full row instead of honoring valid_counts, the next scheduler step receives those placeholders as real speculative tokens. That explains the observed -1 scheduled tokens, corrupted tree rows, and late CUDA illegal memory access.

This package does not claim a throughput result. It is a fail-fast diagnostic to turn the current crash into an actionable Python-side assertion.

Files

• analyze_tree_run.py parses a tree-v2 job_logs.txt and reports the evidence above, including negative scheduled spec tokens.
• tree_placeholder_guard.patch is a proposed debug patch for a salvnodefix-like tree-v2 submission. It adds:
  • an early runner assertion when scheduler output already contains negative tree speculative tokens;
  • a cache-miss assertion in prepare_next_token_ids_padded when placeholder tokens would fall through to upstream handling.

Suggested next run

Use a short debug run, not a leaderboard run:

TREE_FAIL_ON_PLACEHOLDER_SPEC=1 \
FUSED_DIRECT_GREEDY_REJECTION_REQUIRE=1 \
FUSED_SPEC_DECODE_METADATA_REQUIRE=1 \
CUDA_LAUNCH_BLOCKING=1 \
... --enforce-eager ...

Expected outcome:

• If the new assertion fires before GPU execution, the current failure is a sampler/proposer cache handoff bug. Fix that before touching star attention.
• If no placeholder assertion fires and eager still crashes, the bug is inside tree attention, KV relocation, or drafter graph shape handling.

Local validation

Run:

python3 analyze_tree_run.py /path/to/job_logs.txt

On tree-v2-cdx-salvnodefix-run1, it reports one scheduler dump with 28/28 negative speculative tokens after 640 tree steps.