Clarify CUDA graph capture sizing
#1
by oliverholworthy - opened
Separate the model's validated 32,768-token request limit from the deployment-specific batched-token and CUDA graph capture budgets. Use 4,096 as a conservative starting point, explain why the two tuning values should match, and call out the startup-time and memory tradeoff of larger captures.
ybabakhin changed pull request status to merged