Spaces:
Running
Running
File size: 25,130 Bytes
0c6c82c 20fb354 44745f2 20fb354 0c6c82c 20fb354 44745f2 0c6c82c 44745f2 0c6c82c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 | <!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>InferScale-Sim</title>
<meta name="description" content="Interactive LLM serving simulator and SLO-aware capacity planner." />
<link rel="preconnect" href="https://cdn.jsdelivr.net" crossorigin>
<link rel="stylesheet" href="styles.css" />
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.5.1/dist/chart.umd.min.js"></script>
</head>
<body>
<header class="topbar" id="topbar">
<div class="brand-wrap">
<div>
<div class="brand">InferScale-Sim</div>
<div class="subtitle">interactive serving-systems research prototype</div>
</div>
</div>
<div class="runtime-pill" id="runtimePill"><span class="dot"></span><span id="runtimeText">Loading Python runtime...</span></div>
</header>
<main class="shell">
<section class="intro">
<div class="intro-copy">
<div class="eyebrow">InferScale-Sim / v0.3.0</div>
<h1>Explore serving policies, cache reuse, and P/D disaggregation without provisioning a GPU.</h1>
<p>Generate workloads, compare schedulers, model KV pressure and prefix reuse, separate prefill from decode, and search the SLO-constrained design space. The simulator is Python running locally in a Pyodide Web Worker.</p>
</div>
<dl class="project-facts">
<div><dt>Backend</dt><dd>None</dd></div>
<div><dt>Runtime</dt><dd>Python / WASM</dd></div>
<div><dt>Network data</dt><dd>None after load</dd></div>
<div><dt>Profiles</dt><dd>Analytical reference</dd></div>
</dl>
</section>
<div class="reference-note"><strong>Reference-profile mode.</strong> Absolute L4, A10G, and A100 latency values are analytical predictions, not measured GPU benchmarks. Queueing, scheduling, cache, transfer, and SLO behavior is simulated live.</div>
<nav class="tabs" aria-label="InferScale sections">
<button class="tab active" data-tab="lab">Serving Lab</button>
<button class="tab" data-tab="arena">Scheduler Arena</button>
<button class="tab" data-tab="planner">Capacity Planner</button>
<button class="tab" data-tab="modern">Modern Serving</button>
<button class="tab" data-tab="design">Design Explorer</button>
<button class="tab" data-tab="method">Methodology</button>
</nav>
<section id="lab" class="tab-panel active">
<div class="workspace">
<aside class="panel controls-panel">
<div class="panel-title-row"><h2>Experiment</h2><span class="tag">Deterministic seed</span></div>
<div class="field-grid two">
<label>Model<select id="model"><option>Qwen2.5-3B</option><option>Llama-3.1-8B</option><option>Mistral-7B-v0.3</option></select></label>
<label>Weight precision<select id="quantization"><option value="fp16">FP16</option><option value="int8" selected>INT8 scenario</option><option value="int4">INT4 scenario</option></select></label>
</div>
<div class="field-grid two">
<label>Topology<select id="topology"><option value="colocated">Colocated</option><option value="disaggregated_pd">P/D disaggregated</option></select></label>
<label id="colocatedAcceleratorLabel">Accelerator<select id="accelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label>
</div>
<div class="field-grid two">
<label>Scheduler<select id="scheduler"><option value="continuous_fcfs">Continuous - FCFS</option><option value="continuous_sjf">Continuous - SJF</option><option value="continuous_slo">Continuous - SLO-aware</option><option value="chunked_slo">Chunked prefill - SLO-aware</option><option value="static_fcfs">Static batching - FCFS</option></select></label>
<label>Prefix reuse<select id="prefixCache"><option value="off">Off</option><option value="on">On</option></select></label>
</div>
<div id="prefixControls" class="subcontrols hidden">
<div class="section-kicker">Shared-prefix scenario</div>
<div class="field-grid two">
<label>Shared prefix<input id="sharedPrefix" type="number" min="0" step="32" value="256" /><span class="unit">tokens</span></label>
<label>Reuse fraction<input id="prefixReuse" type="number" min="0" max="1" step="0.05" value="0.60" /><span class="unit">fraction</span></label>
</div>
</div>
<div id="pdControls" class="subcontrols hidden">
<div class="section-kicker">P/D topology</div>
<div class="field-grid two">
<label>Prefill accelerator<select id="prefillAccelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label>
<label>Decode accelerator<select id="decodeAccelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label>
<label>Prefill workers<input id="prefillWorkers" type="number" min="1" max="8" step="1" value="1" /></label>
<label>Decode workers<input id="decodeWorkers" type="number" min="1" max="8" step="1" value="1" /></label>
<label>Interconnect<input id="interconnect" type="number" min="1" step="1" value="50" /><span class="unit">GB/s</span></label>
<label>Transfer base<input id="transferBase" type="number" min="0" step="0.05" value="0.20" /><span class="unit">ms</span></label>
</div>
</div>
<hr />
<div class="section-kicker">Workload</div>
<div class="field-grid two">
<label>Arrival process<select id="arrival"><option value="poisson">Poisson</option><option value="constant">Constant</option><option value="bursty">Bursty</option></select></label>
<label>Request rate<input id="rate" type="number" min="0.1" step="0.1" value="4" /><span class="unit">req/s</span></label>
<label>Duration<input id="duration" type="number" min="2" step="1" value="30" /><span class="unit">sim s</span></label>
<label>Seed<input id="seed" type="number" step="1" value="7" /></label>
</div>
<div id="burstControls" class="field-grid two hidden">
<label>Burst multiplier<input id="burstMultiplier" type="number" min="1" step="0.25" value="3" /><span class="unit">x</span></label>
<label>Burst period<input id="burstPeriod" type="number" min="0.5" step="0.5" value="10" /><span class="unit">s</span></label>
</div>
<div class="field-grid two">
<label>Prompt mean<input id="promptMean" type="number" min="16" value="512" /><span class="unit">tokens</span></label>
<label>Prompt CV<input id="promptCv" type="number" min="0" max="2" step="0.05" value="0.50" /></label>
<label>Output mean<input id="outputMean" type="number" min="1" value="64" /><span class="unit">tokens</span></label>
<label>Output CV<input id="outputCv" type="number" min="0" max="2" step="0.05" value="0.60" /></label>
</div>
<hr />
<div class="section-kicker">Serving controls</div>
<div class="field-grid two">
<label>Max batch size<input id="maxBatch" type="number" min="1" max="128" value="16" /></label>
<label>Max batch tokens<input id="maxBatchTokens" type="number" min="128" step="128" value="8192" /></label>
<label>Prefill chunk<input id="chunkSize" type="number" min="64" step="64" value="512" /><span class="unit">tokens</span></label>
<label>KV block<input id="kvBlock" type="number" min="1" value="16" /><span class="unit">tokens</span></label>
</div>
<hr />
<div class="section-kicker">SLO</div>
<div class="field-grid two">
<label>TTFT limit<input id="sloTtft" type="number" min="1" value="500" /><span class="unit">ms</span></label>
<label>E2E limit<input id="sloE2e" type="number" min="100" value="5000" /><span class="unit">ms</span></label>
</div>
<button id="runBtn" class="primary" disabled>Run simulation</button>
<div class="button-row"><button id="copyResultBtn" class="secondary" disabled>Copy JSON</button><button id="exportBtn" class="secondary" disabled>Download JSON</button></div>
</aside>
<div class="results-column">
<section class="panel result-panel">
<div class="panel-title-row"><h2>Run summary</h2><span id="runState" class="tag neutral">Waiting</span></div>
<div id="emptyState" class="empty-state"><h3>Configure a workload and run it</h3><p>The Python simulator executes in a background Web Worker and returns request-level virtual timestamps.</p></div>
<div id="resultContent" class="hidden">
<div class="metric-grid">
<div class="metric"><span>p95 TTFT</span><strong id="mTtft">N/A</strong></div>
<div class="metric"><span>p95 E2E</span><strong id="mE2e">N/A</strong></div>
<div class="metric"><span>Goodput</span><strong id="mGoodput">N/A</strong></div>
<div class="metric"><span>SLO attainment</span><strong id="mSlo">N/A</strong></div>
<div class="metric"><span>Throughput</span><strong id="mReq">N/A</strong></div>
<div class="metric"><span>Peak KV</span><strong id="mKv">N/A</strong></div>
</div>
<section class="diagnostic-card" id="diagnosticCard">
<div class="diagnostic-head"><span>Simulator diagnosis</span><strong id="mBottleneck">N/A</strong></div>
<p id="mDiagnosis">Run a simulation to generate a bottleneck explanation.</p>
<p class="diagnostic-action" id="mRecommendation"></p>
<div class="evidence-row" id="mEvidence"></div>
</section>
<div class="chart-grid">
<div class="chart-card" data-chart-card data-chart-name="latency-percentiles">
<div class="chart-head"><div class="chart-title">Latency percentiles</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body"><canvas id="latencyChart"></canvas></div>
</div>
<div class="chart-card" data-chart-card data-chart-name="queue-decode-kv-timeline">
<div class="chart-head"><div class="chart-title">Queue, decode, and KV timeline</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body"><canvas id="timelineChart"></canvas></div>
</div>
</div>
<div class="chart-card full" data-chart-card data-chart-name="ttft-vs-prompt-length">
<div class="chart-head"><div class="chart-title">Request TTFT vs prompt length</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body"><canvas id="scatterChart"></canvas></div>
</div>
<div id="warnings" class="warnings hidden"></div>
</div>
</section>
</div>
</div>
</section>
<section id="arena" class="tab-panel">
<div class="panel wide-panel">
<div class="panel-title-row arena-title-row">
<div><div class="section-kicker">Same workload - same seed</div><h2>Scheduler Arena</h2><p class="muted">Run every colocated v0.3 scheduler against the current Serving Lab workload and rank by SLO attainment, then goodput.</p></div>
<button id="arenaBtn" class="primary compact" disabled>Compare schedulers</button>
</div>
<div id="arenaEmpty" class="empty-state small"><h3>No comparison yet</h3><p>Your Serving Lab workload and model controls are reused automatically.</p></div>
<div id="arenaContent" class="hidden">
<div class="chart-card full" data-chart-card data-chart-name="scheduler-throughput-comparison">
<div class="chart-head"><div class="chart-title">Useful vs raw request throughput</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body large"><canvas id="arenaChart"></canvas></div>
</div>
<div class="table-toolbar"><span>Scheduler comparison</span><div><button id="arenaCopyBtn" class="mini-button" disabled>Copy table</button><button id="arenaCsvBtn" class="mini-button" disabled>Download CSV</button></div></div>
<div class="table-wrap"><table><thead><tr><th>Scheduler</th><th>Goodput</th><th>SLO attainment</th><th>p95 TTFT</th><th>p95 E2E</th><th>KV peak</th><th>Unfinished</th><th>Diagnosis</th></tr></thead><tbody id="arenaRows"></tbody></table></div>
</div>
</div>
</section>
<section id="planner" class="tab-panel">
<div class="workspace planner-grid">
<aside class="panel controls-panel">
<div class="panel-title-row"><h2>Capacity search</h2><span class="tag">Robust binary search</span></div>
<p class="muted">Find the highest offered load where every repetition meets the SLO target and fully drains.</p>
<label>Required SLO attainment<input id="targetSlo" type="number" min="0.5" max="1" step="0.001" value="0.99" /></label>
<div class="field-grid two">
<label>Minimum rate<input id="minRate" type="number" min="0.05" step="0.1" value="0.25" /><span class="unit">req/s</span></label>
<label>Maximum rate<input id="maxRate" type="number" min="0.1" step="1" value="20" /><span class="unit">req/s</span></label>
<label>Search iterations<input id="searchIter" type="number" min="2" max="12" value="7" /></label>
<label>Repetitions / rate<input id="repetitions" type="number" min="1" max="5" value="2" /></label>
</div>
<label>Safety headroom<input id="headroom" type="number" min="0" max="0.8" step="0.05" value="0.20" /><span class="unit">fraction</span></label>
<button id="capacityBtn" class="primary" disabled>Find sustainable capacity</button>
</aside>
<section class="panel result-panel">
<div class="panel-title-row"><h2>Planner result</h2><span id="plannerState" class="tag neutral">Waiting</span></div>
<div id="plannerEmpty" class="empty-state"><h3>No search yet</h3><p>The planner repeatedly runs the simulator at different offered loads.</p></div>
<div id="plannerContent" class="hidden">
<div class="metric-grid four">
<div class="metric emphasis"><span>Estimated capacity</span><strong id="pCapacity">N/A</strong></div>
<div class="metric"><span>Recommended load</span><strong id="pRecommended">N/A</strong></div>
<div class="metric"><span>Safety headroom</span><strong id="pHeadroom">N/A</strong></div>
<div class="metric"><span>Status</span><strong id="pStatus">N/A</strong></div>
</div>
<div class="planner-note" id="plannerCriterion">A rate passes only if every repetition meets the target and drains all generated requests.</div>
<div class="chart-card full" data-chart-card data-chart-name="capacity-slo-curve">
<div class="chart-head"><div class="chart-title">SLO attainment across searched rates</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body large"><canvas id="capacityChart"></canvas></div>
</div>
<div class="table-toolbar"><span>Search trace</span><div><button id="capacityCopyBtn" class="mini-button" disabled>Copy table</button><button id="capacityCsvBtn" class="mini-button" disabled>Download CSV</button></div></div>
<div class="table-wrap"><table><thead><tr><th>Rate</th><th>Pass</th><th>Mean SLO</th><th>Worst repetition</th><th>Target</th><th>SLO range</th><th>Goodput</th><th>p95 TTFT</th><th>p95 E2E</th></tr></thead><tbody id="capacityRows"></tbody></table></div>
</div>
</section>
</div>
</section>
<section id="modern" class="tab-panel">
<div class="panel wide-panel">
<div class="panel-title-row arena-title-row">
<div><div class="section-kicker">P/D disaggregation + prefix reuse</div><h2>Modern Serving Lab</h2><p class="muted">Compare colocated and prefill/decode-disaggregated serving, each with and without the current shared-prefix reuse scenario. P/D worker counts and interconnect settings come from Serving Lab.</p></div>
<button id="topologyBtn" class="primary compact" disabled>Compare 4 scenarios</button>
</div>
<div id="topologyEmpty" class="empty-state small"><h3>No topology study yet</h3><p>Set shared-prefix and P/D parameters in Serving Lab, then run this controlled comparison.</p></div>
<div id="topologyContent" class="hidden">
<div class="chart-card full" data-chart-card data-chart-name="topology-goodput-ttft">
<div class="chart-head"><div class="chart-title">Goodput vs p95 TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body large"><canvas id="topologyChart"></canvas></div>
</div>
<div class="table-toolbar"><span>Topology and cache comparison</span><div><button id="topologyCopyBtn" class="mini-button" disabled>Copy table</button><button id="topologyCsvBtn" class="mini-button" disabled>Download CSV</button></div></div>
<div class="table-wrap"><table><thead><tr><th>Scenario</th><th>GPU instances</th><th>Goodput</th><th>Goodput / GPU</th><th>SLO attainment</th><th>p95 TTFT</th><th>p95 E2E</th><th>p95 KV transfer</th><th>Cache hit</th><th>Prefill saved</th><th>Diagnosis</th></tr></thead><tbody id="topologyRows"></tbody></table></div>
</div>
</div>
</section>
<section id="design" class="tab-panel">
<div class="panel wide-panel">
<div class="panel-title-row arena-title-row">
<div><div class="section-kicker">Bounded what-if sweep</div><h2>Design Explorer</h2><p class="muted">Sweep 10 colocated candidates plus six P/D/cache variants, then identify the non-dominated goodput/TTFT frontier. This is an interactive design study, not an exhaustive optimizer.</p></div>
<div class="action-stack"><label class="checkline"><input id="includePd" type="checkbox" checked /> Include P/D candidates</label><button id="designBtn" class="primary compact" disabled>Explore design space</button></div>
</div>
<div id="designEmpty" class="empty-state small"><h3>No design sweep yet</h3><p>The current workload, model, SLO, and P/D parameters are reused from Serving Lab.</p></div>
<div id="designContent" class="hidden">
<div class="metric-grid four"><div class="metric emphasis"><span>Candidates</span><strong id="dCandidates">N/A</strong></div><div class="metric"><span>Performance Pareto</span><strong id="dPareto">N/A</strong></div><div class="metric"><span>Efficiency Pareto</span><strong id="dEfficiencyPareto">N/A</strong></div><div class="metric"><span>Profile</span><strong class="metric-small">Analytical</strong></div></div>
<div class="chart-grid">
<div class="chart-card" data-chart-card data-chart-name="design-performance-pareto-frontier">
<div class="chart-head"><div class="chart-title">Raw goodput vs p95 TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body large"><canvas id="designChart"></canvas></div>
</div>
<div class="chart-card" data-chart-card data-chart-name="design-efficiency-pareto-frontier">
<div class="chart-head"><div class="chart-title">Goodput per accelerator vs p95 TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
<div class="chart-body large"><canvas id="designEfficiencyChart"></canvas></div>
</div>
</div>
<div class="table-toolbar"><span>Candidate configurations</span><div><button id="designCopyBtn" class="mini-button" disabled>Copy table</button><button id="designCsvBtn" class="mini-button" disabled>Download CSV</button></div></div>
<div class="table-wrap"><table><thead><tr><th>Candidate</th><th>Perf Pareto</th><th>Efficiency Pareto</th><th>SLO pass</th><th>GPU instances</th><th>Goodput</th><th>Goodput / GPU</th><th>p95 TTFT</th><th>p95 E2E</th><th>KV peak</th><th>Diagnosis</th></tr></thead><tbody id="designRows"></tbody></table></div>
</div>
</div>
</section>
<section id="method" class="tab-panel">
<div class="method-grid">
<article class="panel prose"><div class="section-kicker">Simulation core</div><h2>What is actually simulated?</h2><p>Requests are generated from deterministic workload distributions and advanced through virtual time. Colocated runs model admission, prefill, paged KV allocation, dynamic decode batches, and completion. P/D runs use separate prefill and decode worker pools plus an explicit serialized KV-transfer link.</p><div class="formula">request -> queue -> prefill -> [KV transfer] -> decode -> completion</div></article>
<article class="panel prose"><div class="section-kicker">Prefix reuse</div><h2>Cache without pretending to implement a radix tree</h2><p>v0.3 models a single shared prompt prefix with configurable length and reuse fraction. Cache hits avoid redundant prefill work and share one persistent KV allocation. It is deliberately a controlled what-if abstraction, not a claim to reproduce SGLang's full RadixAttention policy.</p></article>
<article class="panel prose"><div class="section-kicker">P/D disaggregation</div><h2>Role-specific resources and transfer cost</h2><p>Prefill and decode have separate accelerator profiles and worker counts. Prompt KV state crosses a modeled interconnect before decode admission. The simulator reports role utilization, transfer latency, and transfer pressure so the benefit of isolation can be weighed against data-movement overhead.</p></article>
<article class="panel prose"><div class="section-kicker">Design search</div><h2>Pareto, not one magic configuration</h2><p>The Design Explorer reports both a raw-performance frontier and a resource-normalized frontier using goodput per accelerator. That prevents a multi-GPU P/D layout from looking unconditionally better merely because it uses more simulated hardware.</p></article>
<article class="panel prose wide-method"><div class="section-kicker">Research lineage</div><h2>Why these features are in v0.3</h2><p>Vidur established the value of simulation for avoiding expensive deployment sweeps. Recent systems have pushed toward heterogeneous and disaggregated serving, communication-aware modeling, runtime optimizations, and SLA-dependent design-space exploration. InferScale-Sim remains intentionally smaller and inspectable.</p><div class="paper-grid"><div><strong>Vidur / 2024</strong><span>Predictive profiling, workload-aware serving simulation, configuration search.</span></div><div><strong>TokenSim / 2025</strong><span>Extensible scheduling and memory-management simulation.</span></div><div><strong>Revati / 2026</strong><span>GPU-free time-warp emulation of serving control logic.</span></div><div><strong>LLMServingSim 2.0 / 2026</strong><span>Heterogeneous and disaggregated infrastructure, memory and communication.</span></div><div><strong>Frontier / May 2026</strong><span>P/D disaggregation, runtime optimizations, stateful workloads, Pareto exploration.</span></div><div><strong>HeteroPanacea / Aug 2026</strong><span>Heterogeneous stage specialization motivates resource-aware P/D comparison.</span></div><div><strong>SGLang / RadixAttention</strong><span>Automatic shared-prefix KV reuse motivates the controlled cache scenario.</span></div></div></article>
</div>
</section>
</main>
<div id="toast" class="toast" role="status"></div>
<script src="app.js"></script>
</body>
</html>
|