Spaces:
Sleeping
Sleeping
Commit History
28 days ago
-
Verify early-exit EOS before terminating output
Verified signature5419a23
-
Fix early exit inference loop to eliminate redundant computation
Florian valadeKey fixes: - _draft_single_token now always returns a token (never None) - When no early exit head is confident, continues to lm_head instead of returning None and triggering a redundant full model pass - Extracts 'bonus token' from verification pass when all drafts accepted - Same fixes applied to generate_streaming method This eliminates the double computation bug where layers were processed twice when no head was confident, and adds the bonus token optimization that extracts an extra token from each verification pass. Adds comprehensive tests in tests/test_inference_loop.pya781577