Article
Continuous batching from first principles
- +1
ror, ArthurZ, mcpotato
• • 411Shawn Wei Chew
SleepyWoodpecker
·
AI & ML interests
None yet
Recent Activity
upvoted an article 26 days ago
Continuous batching from first principles upvoted an article 26 days ago
Unlocking asynchronicity in continuous batching commentedon an article 26 days ago
Unlocking asynchronicity in continuous batchingOrganizations
None yet
upvoted 2
articles 26 days ago
Article
Unlocking asynchronicity in continuous batching
- +1
ror, pcuenq, ariG23498
• • 61 commented on Unlocking asynchronicity in continuous batching 26 days ago
This was a really enjoyable read! Just have a slight clarification to make:
Based on the execution timeline that you gave, it seems like carry over from batch 0 is not done for batch 1, but rather, is delayed until batch 2. Did I read this correctly? And if so, is this a common practice for inference engines?