Spaces:
Running
Running
Commit ·
20fb354
1
Parent(s): 0c6c82c
Release InferScale-Sim v0.2.0
Browse files- CHANGELOG.md +21 -0
- README.md +12 -7
- app.js +60 -189
- docs/methodology.md +16 -7
- docs/research.md +2 -2
- docs/validation.md +6 -2
- index.html +41 -187
- py/inferscale/__init__.py +1 -1
- py/inferscale/api.py +1 -1
- py/inferscale/diagnostics.py +81 -0
- py/inferscale/latency.py +1 -1
- py/inferscale/metrics.py +10 -1
- py/inferscale/models.py +1 -0
- py/inferscale/optimizer.py +55 -29
- py/inferscale/simulator.py +5 -1
- pyproject.toml +1 -1
- scripts/release_check.py +22 -7
- src/inferscale/__init__.py +1 -1
- src/inferscale/api.py +1 -1
- src/inferscale/diagnostics.py +81 -0
- src/inferscale/latency.py +1 -1
- src/inferscale/metrics.py +10 -1
- src/inferscale/models.py +1 -0
- src/inferscale/optimizer.py +55 -29
- src/inferscale/simulator.py +5 -1
- styles.css +12 -79
- tests/test_diagnostics.py +32 -0
- tests/test_optimizer.py +11 -2
- worker.mjs +1 -0
CHANGELOG.md
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Changelog
|
| 2 |
+
|
| 3 |
+
## v0.2.0
|
| 4 |
+
|
| 5 |
+
- auto-hide runtime header after Pyodide becomes ready; hover/focus at the top edge reveals it
|
| 6 |
+
- redesigned the interface around a flatter, quieter engineering-dashboard visual system
|
| 7 |
+
- replaced decorative/non-ASCII ranking glyphs with explicit text labels
|
| 8 |
+
- added chart expansion with Escape-to-close support
|
| 9 |
+
- added copy-to-clipboard and CSV export for Scheduler Arena and Capacity Planner tables
|
| 10 |
+
- added JSON copy alongside download for individual simulations
|
| 11 |
+
- exposed burst multiplier and burst-period controls
|
| 12 |
+
- added component TTFT/E2E SLO-attainment accounting and failure breakdowns
|
| 13 |
+
- added heuristic simulator-side bottleneck diagnosis with evidence and next-step guidance
|
| 14 |
+
- made capacity feasibility conservative across repetitions: every repetition must meet the target and fully drain
|
| 15 |
+
- added minimum/mean/maximum repetition evidence to capacity traces
|
| 16 |
+
- changed the capacity plot to a true linear request-rate x-axis and added worst-repetition evidence
|
| 17 |
+
- strengthened release checks for browser-module synchronization, package version, provenance, and ASCII-safe UI labels
|
| 18 |
+
|
| 19 |
+
## v0.1.0
|
| 20 |
+
|
| 21 |
+
Initial browser-native Python release with workload generation, static/continuous scheduling, chunked prefill, KV-cache accounting, analytical latency profiles, Scheduler Arena, and SLO capacity search.
|
README.md
CHANGED
|
@@ -10,7 +10,7 @@ license: mit
|
|
| 10 |
short_description: Interactive LLM serving simulator and SLO planner
|
| 11 |
---
|
| 12 |
|
| 13 |
-
# InferScale-Sim v0.
|
| 14 |
|
| 15 |
**Interactive LLM serving simulator and SLO-aware capacity planner — written in Python, executed entirely in the browser.**
|
| 16 |
|
|
@@ -21,7 +21,7 @@ InferScale-Sim explores a practical systems question:
|
|
| 21 |
The public Hugging Face Space uses **no server CPU, no GPU, no API key, and no paid inference provider**. Hugging Face serves static files; Pyodide executes the same Python simulator used by the local test suite inside a Web Worker on the visitor's ordinary CPU.
|
| 22 |
|
| 23 |
> [!IMPORTANT]
|
| 24 |
-
> v0.
|
| 25 |
|
| 26 |
## Why simulation?
|
| 27 |
|
|
@@ -31,7 +31,7 @@ The area has continued moving quickly. InferScale-Sim references the broader lin
|
|
| 31 |
|
| 32 |
InferScale-Sim is intentionally narrower: a compact, inspectable Python implementation focused on workload dynamics, scheduling, KV-cache behavior, SLOs, and interactive configuration search.
|
| 33 |
|
| 34 |
-
## v0.
|
| 35 |
|
| 36 |
- deterministic **constant, Poisson, and bursty** arrival processes
|
| 37 |
- log-normal prompt/output-length distributions
|
|
@@ -47,8 +47,13 @@ InferScale-Sim is intentionally narrower: a compact, inspectable Python implemen
|
|
| 47 |
- throughput, output-token throughput, **goodput**, and SLO attainment
|
| 48 |
- live queue / decode / KV-cache timeline
|
| 49 |
- scheduler arena using an identical deterministic workload
|
| 50 |
-
- binary **capacity search**
|
| 51 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 52 |
- completely client-side Hugging Face deployment via **Pyodide Web Worker**
|
| 53 |
|
| 54 |
## Architecture
|
|
@@ -185,9 +190,9 @@ See [`docs/research.md`](docs/research.md) and [`docs/methodology.md`](docs/meth
|
|
| 185 |
|
| 186 |
## Roadmap
|
| 187 |
|
| 188 |
-
**v0.
|
| 189 |
|
| 190 |
-
Potential v0.
|
| 191 |
|
| 192 |
- empirical calibration profile import
|
| 193 |
- prefill/decode disaggregation + KV transfer
|
|
|
|
| 10 |
short_description: Interactive LLM serving simulator and SLO planner
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# InferScale-Sim v0.2.0
|
| 14 |
|
| 15 |
**Interactive LLM serving simulator and SLO-aware capacity planner — written in Python, executed entirely in the browser.**
|
| 16 |
|
|
|
|
| 21 |
The public Hugging Face Space uses **no server CPU, no GPU, no API key, and no paid inference provider**. Hugging Face serves static files; Pyodide executes the same Python simulator used by the local test suite inside a Web Worker on the visitor's ordinary CPU.
|
| 22 |
|
| 23 |
> [!IMPORTANT]
|
| 24 |
+
> v0.2 ships with **analytical reference latency profiles**, not measured GPU calibration data. The simulator is useful for studying serving-system dynamics and relative scheduler behavior, but its absolute latency predictions must not be presented as benchmark measurements. A future calibrated profile bundle can replace the analytical backend without changing the simulator.
|
| 25 |
|
| 26 |
## Why simulation?
|
| 27 |
|
|
|
|
| 31 |
|
| 32 |
InferScale-Sim is intentionally narrower: a compact, inspectable Python implementation focused on workload dynamics, scheduling, KV-cache behavior, SLOs, and interactive configuration search.
|
| 33 |
|
| 34 |
+
## v0.2 capabilities
|
| 35 |
|
| 36 |
- deterministic **constant, Poisson, and bursty** arrival processes
|
| 37 |
- log-normal prompt/output-length distributions
|
|
|
|
| 47 |
- throughput, output-token throughput, **goodput**, and SLO attainment
|
| 48 |
- live queue / decode / KV-cache timeline
|
| 49 |
- scheduler arena using an identical deterministic workload
|
| 50 |
+
- conservative binary **capacity search** where every repetition must meet the target and drain
|
| 51 |
+
- repetition min/mean/max SLO evidence for searched rates
|
| 52 |
+
- simulator-side **bottleneck diagnosis** from queueing, SLO misses, busy fraction, unfinished work, and KV pressure
|
| 53 |
+
- exposed burst stress controls
|
| 54 |
+
- copy-to-clipboard and CSV export for Arena / Planner tables
|
| 55 |
+
- expandable full-screen charts
|
| 56 |
+
- JSON experiment copy/export
|
| 57 |
- completely client-side Hugging Face deployment via **Pyodide Web Worker**
|
| 58 |
|
| 59 |
## Architecture
|
|
|
|
| 190 |
|
| 191 |
## Roadmap
|
| 192 |
|
| 193 |
+
**v0.2** keeps the single-accelerator serving loop while improving inspectability and planner robustness. It adds bottleneck explanations, repetition-aware capacity evidence, burst controls, copy/CSV tooling, expandable charts, and a restrained engineering-oriented interface.
|
| 194 |
|
| 195 |
+
Potential v0.3 extensions, only after v0.2 validation:
|
| 196 |
|
| 197 |
- empirical calibration profile import
|
| 198 |
- prefill/decode disaggregation + KV transfer
|
app.js
CHANGED
|
@@ -1,19 +1,24 @@
|
|
| 1 |
const $ = (id) => document.getElementById(id);
|
| 2 |
-
|
| 3 |
const runtimePill = $("runtimePill");
|
| 4 |
const runtimeText = $("runtimeText");
|
|
|
|
|
|
|
| 5 |
let lastResult = null;
|
| 6 |
-
let
|
|
|
|
| 7 |
let requestId = 0;
|
| 8 |
-
|
| 9 |
|
| 10 |
-
const
|
|
|
|
|
|
|
| 11 |
worker.addEventListener("message", (event) => {
|
| 12 |
const data = event.data || {};
|
| 13 |
if (data.type === "ready") {
|
| 14 |
runtimePill.classList.add("ready");
|
| 15 |
runtimeText.textContent = "Python runtime ready";
|
| 16 |
-
["runBtn",
|
|
|
|
| 17 |
return;
|
| 18 |
}
|
| 19 |
if (data.type === "fatal") {
|
|
@@ -23,193 +28,59 @@ worker.addEventListener("message", (event) => {
|
|
| 23 |
return;
|
| 24 |
}
|
| 25 |
if (!pending.has(data.id)) return;
|
| 26 |
-
const {
|
| 27 |
pending.delete(data.id);
|
| 28 |
data.error ? reject(new Error(data.error)) : resolve(data.result);
|
| 29 |
});
|
| 30 |
|
| 31 |
-
function callPython(action,
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
}
|
| 38 |
-
|
| 39 |
-
function
|
| 40 |
-
function
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
slo_ttft_ms: num("sloTtft"),
|
| 59 |
-
slo_e2e_ms: num("sloE2e"),
|
| 60 |
-
slo_attainment_target: num("targetSlo"),
|
| 61 |
-
};
|
| 62 |
-
}
|
| 63 |
-
|
| 64 |
-
function fmt(value, digits = 1) {
|
| 65 |
-
if (!Number.isFinite(value)) return "—";
|
| 66 |
-
return value.toLocaleString(undefined, { maximumFractionDigits: digits });
|
| 67 |
-
}
|
| 68 |
-
function pct(v, digits = 1) { return `${fmt(v * 100, digits)}%`; }
|
| 69 |
-
|
| 70 |
-
function destroyChart(name) {
|
| 71 |
-
if (charts[name]) { charts[name].destroy(); delete charts[name]; }
|
| 72 |
-
}
|
| 73 |
-
|
| 74 |
-
const chartDefaults = {
|
| 75 |
-
color: "#8f9aab",
|
| 76 |
-
borderColor: "rgba(150,160,180,.13)",
|
| 77 |
-
};
|
| 78 |
-
Chart.defaults.color = chartDefaults.color;
|
| 79 |
-
Chart.defaults.borderColor = chartDefaults.borderColor;
|
| 80 |
-
Chart.defaults.font.family = getComputedStyle(document.body).fontFamily;
|
| 81 |
-
|
| 82 |
-
function renderSimulation(result) {
|
| 83 |
-
lastResult = result;
|
| 84 |
-
$("emptyState").classList.add("hidden");
|
| 85 |
-
$("resultContent").classList.remove("hidden");
|
| 86 |
-
$("exportBtn").disabled = false;
|
| 87 |
-
const s = result.summary, l = result.latency, r = result.resource;
|
| 88 |
-
$("mTtft").textContent = `${fmt(l.ttft_ms.p95)} ms`;
|
| 89 |
-
$("mE2e").textContent = `${fmt(l.e2e_ms.p95)} ms`;
|
| 90 |
-
$("mGoodput").textContent = `${fmt(s.goodput_rps, 2)} req/s`;
|
| 91 |
-
$("mSlo").textContent = pct(s.slo_attainment);
|
| 92 |
-
$("mReq").textContent = `${fmt(s.request_throughput_rps, 2)} req/s`;
|
| 93 |
-
$("mKv").textContent = `${fmt(r.peak_kv_gb, 2)} GB`;
|
| 94 |
-
const tag = $("runState");
|
| 95 |
-
tag.textContent = `${s.requests_completed}/${s.requests_generated} completed`;
|
| 96 |
-
tag.className = `tag ${s.slo_attainment >= 0.99 && s.requests_unfinished === 0 ? "good" : "bad"}`;
|
| 97 |
-
|
| 98 |
-
destroyChart("latency");
|
| 99 |
-
charts.latency = new Chart($("latencyChart"), {
|
| 100 |
-
type: "bar",
|
| 101 |
-
data: {
|
| 102 |
-
labels: ["TTFT p50", "TTFT p95", "E2E p50", "E2E p95", "Queue p95"],
|
| 103 |
-
datasets: [{ label: "milliseconds", data: [l.ttft_ms.p50,l.ttft_ms.p95,l.e2e_ms.p50,l.e2e_ms.p95,l.queue_ms.p95], backgroundColor: ["#766ef0","#8b7cff","#3b82f6","#55c2ff","#63d9a5"], borderRadius: 5 }]
|
| 104 |
-
},
|
| 105 |
-
options: { responsive:true, maintainAspectRatio:false, plugins:{legend:{display:false}}, scales:{y:{beginAtZero:true}} }
|
| 106 |
-
});
|
| 107 |
-
|
| 108 |
-
destroyChart("timeline");
|
| 109 |
-
const timeline = result.timeline;
|
| 110 |
-
charts.timeline = new Chart($("timelineChart"), {
|
| 111 |
-
type: "line",
|
| 112 |
-
data: {
|
| 113 |
-
labels: timeline.map(x => Number(x.time_s.toFixed(2))),
|
| 114 |
-
datasets: [
|
| 115 |
-
{label:"Waiting",data:timeline.map(x=>x.waiting),borderColor:"#ffad66",pointRadius:0,tension:.15},
|
| 116 |
-
{label:"Decoding",data:timeline.map(x=>x.decoding),borderColor:"#8b7cff",pointRadius:0,tension:.15},
|
| 117 |
-
{label:"KV GB",data:timeline.map(x=>x.kv_used_gb),borderColor:"#63d9a5",pointRadius:0,tension:.15,yAxisID:"y1"}
|
| 118 |
-
]
|
| 119 |
-
},
|
| 120 |
-
options:{responsive:true,maintainAspectRatio:false,interaction:{mode:"index",intersect:false},scales:{x:{title:{display:true,text:"virtual time (s)"}},y:{beginAtZero:true,title:{display:true,text:"requests"}},y1:{beginAtZero:true,position:"right",grid:{drawOnChartArea:false},title:{display:true,text:"KV GB"}}}}
|
| 121 |
-
});
|
| 122 |
-
|
| 123 |
-
destroyChart("scatter");
|
| 124 |
-
const sample = result.requests || [];
|
| 125 |
-
charts.scatter = new Chart($("scatterChart"), {
|
| 126 |
-
type:"scatter",
|
| 127 |
-
data:{datasets:[{label:"requests",data:sample.map(x=>({x:x.prompt_tokens,y:x.ttft_ms})),backgroundColor:"rgba(85,194,255,.55)",pointRadius:2.5}]},
|
| 128 |
-
options:{responsive:true,maintainAspectRatio:false,plugins:{legend:{display:false}},scales:{x:{title:{display:true,text:"prompt tokens"}},y:{title:{display:true,text:"TTFT (ms)"},beginAtZero:true}}}
|
| 129 |
-
});
|
| 130 |
-
|
| 131 |
-
const warnings = $("warnings");
|
| 132 |
-
const allWarnings = [...(result.warnings || [])];
|
| 133 |
-
if (result.provenance?.profile_warning) allWarnings.unshift(result.provenance.profile_warning);
|
| 134 |
-
if (allWarnings.length) {
|
| 135 |
-
warnings.innerHTML = allWarnings.map(w=>`<div>• ${escapeHtml(w)}</div>`).join("");
|
| 136 |
-
warnings.classList.remove("hidden");
|
| 137 |
-
} else warnings.classList.add("hidden");
|
| 138 |
-
}
|
| 139 |
-
|
| 140 |
-
function escapeHtml(value) {
|
| 141 |
-
return String(value).replace(/[&<>'"]/g, c => ({"&":"&","<":"<",">":">","'":"'",'"':"""}[c]));
|
| 142 |
-
}
|
| 143 |
-
|
| 144 |
-
function setBusy(button, stateEl, busy, label) {
|
| 145 |
-
button.disabled = busy;
|
| 146 |
-
if (stateEl) {
|
| 147 |
-
stateEl.textContent = busy ? label : stateEl.textContent;
|
| 148 |
-
if (busy) stateEl.className = "tag neutral";
|
| 149 |
-
}
|
| 150 |
-
}
|
| 151 |
-
|
| 152 |
-
$("runBtn").addEventListener("click", async () => {
|
| 153 |
-
const btn = $("runBtn"), state = $("runState");
|
| 154 |
-
setBusy(btn,state,true,"Simulating…"); btn.textContent="Running Python simulation…";
|
| 155 |
-
try {
|
| 156 |
-
const result = await callPython("simulate", configFromUI());
|
| 157 |
-
renderSimulation(result);
|
| 158 |
-
} catch (err) {
|
| 159 |
-
state.textContent="Error"; state.className="tag bad"; alert(`Simulation failed: ${err.message}`);
|
| 160 |
-
} finally {
|
| 161 |
-
btn.textContent="Run simulation"; btn.disabled=false;
|
| 162 |
-
}
|
| 163 |
-
});
|
| 164 |
-
|
| 165 |
-
$("exportBtn").addEventListener("click", () => {
|
| 166 |
-
if (!lastResult) return;
|
| 167 |
-
const blob = new Blob([JSON.stringify(lastResult,null,2)], {type:"application/json"});
|
| 168 |
-
const a = document.createElement("a"); a.href=URL.createObjectURL(blob); a.download=`inferscale-${Date.now()}.json`; a.click(); URL.revokeObjectURL(a.href);
|
| 169 |
-
});
|
| 170 |
-
|
| 171 |
-
$("arenaBtn").addEventListener("click", async () => {
|
| 172 |
-
const btn=$("arenaBtn"); btn.disabled=true; btn.textContent="Comparing…";
|
| 173 |
-
try {
|
| 174 |
-
const result = await callPython("compare", {config:configFromUI()});
|
| 175 |
-
renderArena(result.rows);
|
| 176 |
-
} catch(err){ alert(`Scheduler comparison failed: ${err.message}`); }
|
| 177 |
-
finally { btn.disabled=false; btn.textContent="Compare schedulers"; }
|
| 178 |
-
});
|
| 179 |
-
|
| 180 |
-
function schedulerLabel(s) { return ({static_fcfs:"Static FCFS",continuous_fcfs:"Continuous FCFS",continuous_sjf:"Continuous SJF",continuous_slo:"Continuous SLO",chunked_slo:"Chunked SLO"})[s] || s; }
|
| 181 |
-
function renderArena(rows) {
|
| 182 |
-
$("arenaEmpty").classList.add("hidden"); $("arenaContent").classList.remove("hidden");
|
| 183 |
-
$("arenaRows").innerHTML = rows.map((r,i)=>`<tr><td>${i===0?"★ ":""}${schedulerLabel(r.scheduler)}</td><td>${fmt(r.goodput_rps,2)} req/s</td><td>${pct(r.slo_attainment)}</td><td>${fmt(r.p95_ttft_ms)} ms</td><td>${fmt(r.p95_e2e_ms)} ms</td><td>${pct(r.peak_kv_utilization)}</td><td>${fmt(r.unfinished,0)}</td></tr>`).join("");
|
| 184 |
-
destroyChart("arena");
|
| 185 |
-
charts.arena = new Chart($("arenaChart"),{type:"bar",data:{labels:rows.map(r=>schedulerLabel(r.scheduler)),datasets:[{label:"Goodput (req/s)",data:rows.map(r=>r.goodput_rps),backgroundColor:"#8b7cff",borderRadius:5},{label:"Raw throughput (req/s)",data:rows.map(r=>r.request_throughput_rps),backgroundColor:"#3f7ee8",borderRadius:5}]},options:{responsive:true,maintainAspectRatio:false,scales:{y:{beginAtZero:true}}}});
|
| 186 |
}
|
| 187 |
|
| 188 |
-
$("
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
}
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
tab.addEventListener("click", () => {
|
| 211 |
-
document.querySelectorAll(".tab").forEach(t=>t.classList.remove("active"));
|
| 212 |
-
document.querySelectorAll(".tab-panel").forEach(p=>p.classList.remove("active"));
|
| 213 |
-
tab.classList.add("active"); $(tab.dataset.tab).classList.add("active");
|
| 214 |
-
});
|
| 215 |
-
}
|
|
|
|
| 1 |
const $ = (id) => document.getElementById(id);
|
|
|
|
| 2 |
const runtimePill = $("runtimePill");
|
| 3 |
const runtimeText = $("runtimeText");
|
| 4 |
+
const charts = {};
|
| 5 |
+
const pending = new Map();
|
| 6 |
let lastResult = null;
|
| 7 |
+
let lastArenaRows = [];
|
| 8 |
+
let lastCapacityTrace = [];
|
| 9 |
let requestId = 0;
|
| 10 |
+
let toastTimer = null;
|
| 11 |
|
| 12 |
+
const COLORS = {blue:"#79a7ff", blue2:"#5d8fe9", steel:"#92a2b5", green:"#69c99a", amber:"#dfb966", grid:"rgba(140,155,175,.14)"};
|
| 13 |
+
|
| 14 |
+
const worker = new Worker("./worker.mjs", {type:"module"});
|
| 15 |
worker.addEventListener("message", (event) => {
|
| 16 |
const data = event.data || {};
|
| 17 |
if (data.type === "ready") {
|
| 18 |
runtimePill.classList.add("ready");
|
| 19 |
runtimeText.textContent = "Python runtime ready";
|
| 20 |
+
["runBtn","arenaBtn","capacityBtn"].forEach((id) => $(id).disabled = false);
|
| 21 |
+
window.setTimeout(() => document.body.classList.add("runtime-ready"), 650);
|
| 22 |
return;
|
| 23 |
}
|
| 24 |
if (data.type === "fatal") {
|
|
|
|
| 28 |
return;
|
| 29 |
}
|
| 30 |
if (!pending.has(data.id)) return;
|
| 31 |
+
const {resolve,reject} = pending.get(data.id);
|
| 32 |
pending.delete(data.id);
|
| 33 |
data.error ? reject(new Error(data.error)) : resolve(data.result);
|
| 34 |
});
|
| 35 |
|
| 36 |
+
function callPython(action,payload){const id=++requestId;return new Promise((resolve,reject)=>{pending.set(id,{resolve,reject});worker.postMessage({id,action,payload});});}
|
| 37 |
+
function num(id){return Number($(id).value);}
|
| 38 |
+
function configFromUI(overrides={}){return {model:$("model").value,accelerator:$("accelerator").value,scheduler:$("scheduler").value,quantization:$("quantization").value,arrival_process:$("arrival").value,request_rate_rps:num("rate"),duration_s:num("duration"),prompt_tokens_mean:num("promptMean"),prompt_tokens_cv:num("promptCv"),output_tokens_mean:num("outputMean"),output_tokens_cv:num("outputCv"),max_batch_size:num("maxBatch"),max_batch_tokens:num("maxBatchTokens"),chunk_size:num("chunkSize"),kv_block_tokens:num("kvBlock"),burst_multiplier:num("burstMultiplier"),burst_period_s:num("burstPeriod"),seed:num("seed"),slo_ttft_ms:num("sloTtft"),slo_e2e_ms:num("sloE2e"),slo_attainment_target:.99,...overrides};}
|
| 39 |
+
function fmt(value,digits=1){if(!Number.isFinite(value))return "N/A";return value.toLocaleString(undefined,{maximumFractionDigits:digits});}
|
| 40 |
+
function pct(value,digits=1){return `${fmt(value*100,digits)}%`;}
|
| 41 |
+
function escapeHtml(value){return String(value).replace(/[&<>'"]/g,(c)=>({"&":"&","<":"<",">":">","'":"'",'"':"""})[c]);}
|
| 42 |
+
function destroyChart(name){if(charts[name]){charts[name].destroy();delete charts[name];}}
|
| 43 |
+
function showToast(message){const toast=$("toast");toast.textContent=message;toast.classList.add("show");if(toastTimer)clearTimeout(toastTimer);toastTimer=setTimeout(()=>toast.classList.remove("show"),1800);}
|
| 44 |
+
async function copyText(text,message){try{await navigator.clipboard.writeText(text);showToast(message);}catch{const area=document.createElement("textarea");area.value=text;area.style.position="fixed";area.style.opacity="0";document.body.appendChild(area);area.select();document.execCommand("copy");area.remove();showToast(message);}}
|
| 45 |
+
function csvCell(value){const text=String(value??"");return /[",\n]/.test(text)?`"${text.replaceAll('"','""')}"`:text;}
|
| 46 |
+
function tableText(headers,rows,separator="\t"){return [headers,...rows].map((row)=>row.join(separator)).join("\n");}
|
| 47 |
+
function downloadCsv(filename,headers,rows){const csv=[headers,...rows].map((row)=>row.map(csvCell).join(",")).join("\n");const blob=new Blob([csv],{type:"text/csv;charset=utf-8"});const url=URL.createObjectURL(blob);const a=document.createElement("a");a.href=url;a.download=filename;a.click();URL.revokeObjectURL(url);}
|
| 48 |
+
|
| 49 |
+
Chart.defaults.color="#929dab";Chart.defaults.borderColor=COLORS.grid;Chart.defaults.font.family=getComputedStyle(document.body).fontFamily;Chart.defaults.animation.duration=180;
|
| 50 |
+
function commonChartOptions(){return {responsive:true,maintainAspectRatio:false,plugins:{legend:{labels:{boxWidth:12,boxHeight:8}}}};}
|
| 51 |
+
|
| 52 |
+
function renderSimulation(result){
|
| 53 |
+
lastResult=result;$("emptyState").classList.add("hidden");$("resultContent").classList.remove("hidden");$("exportBtn").disabled=false;$("copyResultBtn").disabled=false;
|
| 54 |
+
const s=result.summary,l=result.latency,r=result.resource,d=result.diagnostics||{};
|
| 55 |
+
$("mTtft").textContent=`${fmt(l.ttft_ms.p95)} ms`;$("mE2e").textContent=`${fmt(l.e2e_ms.p95)} ms`;$("mGoodput").textContent=`${fmt(s.goodput_rps,2)} req/s`;$("mSlo").textContent=pct(s.slo_attainment);$("mReq").textContent=`${fmt(s.request_throughput_rps,2)} req/s`;$("mKv").textContent=`${fmt(r.peak_kv_gb,2)} GB`;
|
| 56 |
+
const tag=$("runState");tag.textContent=`${s.requests_completed}/${s.requests_generated} completed`;tag.className=`tag ${s.slo_attainment>=.99&&s.requests_unfinished===0?"good":"bad"}`;
|
| 57 |
+
$("mBottleneck").textContent=d.label||"N/A";$("mDiagnosis").textContent=d.explanation||"No simulator diagnosis available.";$("mRecommendation").textContent=d.recommendation?`Next check: ${d.recommendation}`:"";
|
| 58 |
+
const e=d.evidence||{};$("mEvidence").innerHTML=[`busy ${pct(e.busy_fraction??s.busy_fraction)}`,`KV ${pct(e.peak_kv_utilization??r.peak_kv_utilization)}`,`TTFT pass ${pct(e.ttft_slo_attainment??s.ttft_slo_attainment)}`,`E2E pass ${pct(e.e2e_slo_attainment??s.e2e_slo_attainment)}`,`queue p95 ${fmt(e.queue_p95_ms??l.queue_ms.p95)} ms`,`target 99%`].map((x)=>`<span>${escapeHtml(x)}</span>`).join("");
|
| 59 |
+
destroyChart("latency");charts.latency=new Chart($("latencyChart"),{type:"bar",data:{labels:["TTFT p50","TTFT p95","E2E p50","E2E p95","Queue p95"],datasets:[{label:"Milliseconds",data:[l.ttft_ms.p50,l.ttft_ms.p95,l.e2e_ms.p50,l.e2e_ms.p95,l.queue_ms.p95],backgroundColor:[COLORS.steel,COLORS.blue,"#8196b1",COLORS.blue2,COLORS.amber]}]},options:{...commonChartOptions(),plugins:{legend:{display:false}},scales:{y:{beginAtZero:true}}}});
|
| 60 |
+
const timeline=result.timeline||[];destroyChart("timeline");charts.timeline=new Chart($("timelineChart"),{type:"line",data:{datasets:[{label:"Waiting",data:timeline.map((x)=>({x:x.time_s,y:x.waiting})),borderColor:COLORS.amber,pointRadius:0,tension:.08,yAxisID:"y"},{label:"Decoding",data:timeline.map((x)=>({x:x.time_s,y:x.decoding})),borderColor:COLORS.blue,pointRadius:0,tension:.08,yAxisID:"y"},{label:"KV GB",data:timeline.map((x)=>({x:x.time_s,y:x.kv_used_gb})),borderColor:COLORS.green,pointRadius:0,tension:.08,yAxisID:"y1"}]},options:{...commonChartOptions(),parsing:false,interaction:{mode:"nearest",intersect:false},scales:{x:{type:"linear",title:{display:true,text:"Virtual time (s)"}},y:{beginAtZero:true,title:{display:true,text:"Requests"}},y1:{beginAtZero:true,position:"right",grid:{drawOnChartArea:false},title:{display:true,text:"KV GB"}}}}});
|
| 61 |
+
const sample=result.requests||[];destroyChart("scatter");charts.scatter=new Chart($("scatterChart"),{type:"scatter",data:{datasets:[{label:"Requests",data:sample.map((x)=>({x:x.prompt_tokens,y:x.ttft_ms})),backgroundColor:"rgba(121,167,255,.52)",pointRadius:2.2}]},options:{...commonChartOptions(),plugins:{legend:{display:false}},scales:{x:{title:{display:true,text:"Prompt tokens"}},y:{title:{display:true,text:"TTFT (ms)"},beginAtZero:true}}}});
|
| 62 |
+
const warnings=$("warnings"),all=[...(result.warnings||[])];if(result.provenance?.profile_warning)all.unshift(result.provenance.profile_warning);if(all.length){warnings.innerHTML=all.map((w)=>`<div>${escapeHtml(w)}</div>`).join("");warnings.classList.remove("hidden");}else warnings.classList.add("hidden");
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 63 |
}
|
| 64 |
|
| 65 |
+
$("runBtn").addEventListener("click",async()=>{const b=$("runBtn"),s=$("runState");b.disabled=true;b.textContent="Running simulation...";s.textContent="Simulating...";s.className="tag neutral";try{renderSimulation(await callPython("simulate",configFromUI()));}catch(e){s.textContent="Error";s.className="tag bad";alert(`Simulation failed: ${e.message}`);}finally{b.textContent="Run simulation";b.disabled=false;}});
|
| 66 |
+
$("copyResultBtn").addEventListener("click",()=>{if(lastResult)copyText(JSON.stringify(lastResult,null,2),"Result JSON copied");});
|
| 67 |
+
$("exportBtn").addEventListener("click",()=>{if(!lastResult)return;const blob=new Blob([JSON.stringify(lastResult,null,2)],{type:"application/json"});const url=URL.createObjectURL(blob);const a=document.createElement("a");a.href=url;a.download=`inferscale-run-${Date.now()}.json`;a.click();URL.revokeObjectURL(url);});
|
| 68 |
+
|
| 69 |
+
function schedulerLabel(s){return ({static_fcfs:"Static FCFS",continuous_fcfs:"Continuous FCFS",continuous_sjf:"Continuous SJF",continuous_slo:"Continuous SLO",chunked_slo:"Chunked SLO"})[s]||s;}
|
| 70 |
+
$("arenaBtn").addEventListener("click",async()=>{const b=$("arenaBtn");b.disabled=true;b.textContent="Comparing...";try{renderArena((await callPython("compare",{config:configFromUI()})).rows);}catch(e){alert(`Scheduler comparison failed: ${e.message}`);}finally{b.disabled=false;b.textContent="Compare schedulers";}});
|
| 71 |
+
function arenaTableRows(rows){return rows.map((r)=>[schedulerLabel(r.scheduler),`${fmt(r.goodput_rps,2)} req/s`,pct(r.slo_attainment),`${fmt(r.p95_ttft_ms)} ms`,`${fmt(r.p95_e2e_ms)} ms`,pct(r.peak_kv_utilization),fmt(r.unfinished,0),r.bottleneck||"N/A"]);}
|
| 72 |
+
function renderArena(rows){lastArenaRows=rows;$("arenaEmpty").classList.add("hidden");$("arenaContent").classList.remove("hidden");$("arenaCopyBtn").disabled=false;$("arenaCsvBtn").disabled=false;$("arenaRows").innerHTML=rows.map((r,i)=>`<tr><td>${escapeHtml(schedulerLabel(r.scheduler))}${i===0?'<span class="best-label">Best</span>':""}</td><td>${fmt(r.goodput_rps,2)} req/s</td><td>${pct(r.slo_attainment)}</td><td>${fmt(r.p95_ttft_ms)} ms</td><td>${fmt(r.p95_e2e_ms)} ms</td><td>${pct(r.peak_kv_utilization)}</td><td>${fmt(r.unfinished,0)}</td><td>${escapeHtml(r.bottleneck||"N/A")}</td></tr>`).join("");destroyChart("arena");charts.arena=new Chart($("arenaChart"),{type:"bar",data:{labels:rows.map((r)=>schedulerLabel(r.scheduler)),datasets:[{label:"Goodput (req/s)",data:rows.map((r)=>r.goodput_rps),backgroundColor:COLORS.blue},{label:"Raw throughput (req/s)",data:rows.map((r)=>r.request_throughput_rps),backgroundColor:"#53677f"}]},options:{...commonChartOptions(),scales:{y:{beginAtZero:true}}}});}
|
| 73 |
+
const arenaHeaders=["Scheduler","Goodput","SLO attainment","p95 TTFT","p95 E2E","KV peak","Unfinished","Diagnosis"];
|
| 74 |
+
$("arenaCopyBtn").addEventListener("click",()=>copyText(tableText(arenaHeaders,arenaTableRows(lastArenaRows)),"Scheduler table copied"));$("arenaCsvBtn").addEventListener("click",()=>downloadCsv("inferscale-scheduler-arena.csv",arenaHeaders,arenaTableRows(lastArenaRows)));
|
| 75 |
+
|
| 76 |
+
$("capacityBtn").addEventListener("click",async()=>{const b=$("capacityBtn"),s=$("plannerState");b.disabled=true;b.textContent="Searching...";s.textContent="Running simulations...";s.className="tag neutral";try{renderCapacity(await callPython("capacity",{config:configFromUI({slo_attainment_target:num("targetSlo")}),min_rate:num("minRate"),max_rate:num("maxRate"),iterations:num("searchIter"),repetitions:num("repetitions"),headroom:num("headroom")}));}catch(e){s.textContent="Error";s.className="tag bad";alert(`Capacity search failed: ${e.message}`);}finally{b.disabled=false;b.textContent="Find sustainable capacity";}});
|
| 77 |
+
function capacityTableRows(trace){return trace.map((r)=>[`${fmt(r.rate_rps,2)} req/s`,r.passed?"PASS":"FAIL",pct(r.slo_attainment),`${pct(r.slo_attainment_min)} - ${pct(r.slo_attainment_max)}`,`${fmt(r.goodput_rps,2)} req/s`,`${fmt(r.p95_ttft_ms)} ms`,`${fmt(r.p95_e2e_ms)} ms`]);}
|
| 78 |
+
function renderCapacity(result){lastCapacityTrace=result.trace||[];$("plannerEmpty").classList.add("hidden");$("plannerContent").classList.remove("hidden");$("capacityCopyBtn").disabled=false;$("capacityCsvBtn").disabled=false;$("pCapacity").textContent=`${fmt(result.capacity_rps,2)} req/s`;$("pRecommended").textContent=`${fmt(result.recommended_rps,2)} req/s`;$("pHeadroom").textContent=pct(result.headroom??num("headroom"));$("pStatus").textContent=result.status.replaceAll("_"," ");const state=$("plannerState");state.textContent=result.status==="ok"?"Search complete":result.status.replaceAll("_"," ");state.className=`tag ${result.capacity_rps>0?"good":"bad"}`;$("capacityRows").innerHTML=lastCapacityTrace.map((r)=>`<tr><td>${fmt(r.rate_rps,2)} req/s</td><td class="${r.passed?"pass":"fail"}">${r.passed?"PASS":"FAIL"}</td><td>${pct(r.slo_attainment)}</td><td>${pct(r.slo_attainment_min)} - ${pct(r.slo_attainment_max)}</td><td>${fmt(r.goodput_rps,2)} req/s</td><td>${fmt(r.p95_ttft_ms)} ms</td><td>${fmt(r.p95_e2e_ms)} ms</td></tr>`).join("");const target=num("targetSlo"),mean=lastCapacityTrace.map((r)=>({x:r.rate_rps,y:r.slo_attainment})),worst=lastCapacityTrace.map((r)=>({x:r.rate_rps,y:r.slo_attainment_min})),xs=lastCapacityTrace.map((r)=>r.rate_rps),minX=Math.min(...xs),maxX=Math.max(...xs);destroyChart("capacity");charts.capacity=new Chart($("capacityChart"),{type:"line",data:{datasets:[{label:"Mean SLO attainment",data:mean,borderColor:COLORS.blue,backgroundColor:"rgba(121,167,255,.10)",fill:true,tension:.08,pointRadius:3},{label:"Worst repetition",data:worst,borderColor:COLORS.amber,borderDash:[5,4],pointRadius:2,tension:.08},{label:"Target",data:[{x:minX,y:target},{x:maxX,y:target}],borderColor:COLORS.green,borderDash:[6,5],pointRadius:0}]},options:{...commonChartOptions(),parsing:false,scales:{x:{type:"linear",title:{display:true,text:"Offered load (req/s)"},ticks:{maxTicksLimit:8}},y:{min:0,max:1,ticks:{callback:(v)=>`${Math.round(v*100)}%`}}}}});}
|
| 79 |
+
const capacityHeaders=["Rate","Pass","Mean SLO","SLO range","Goodput","p95 TTFT","p95 E2E"];
|
| 80 |
+
$("capacityCopyBtn").addEventListener("click",()=>copyText(tableText(capacityHeaders,capacityTableRows(lastCapacityTrace)),"Capacity table copied"));$("capacityCsvBtn").addEventListener("click",()=>downloadCsv("inferscale-capacity-trace.csv",capacityHeaders,capacityTableRows(lastCapacityTrace)));
|
| 81 |
+
|
| 82 |
+
$("arrival").addEventListener("change",()=>$("burstControls").classList.toggle("hidden",$("arrival").value!=="bursty"));
|
| 83 |
+
for(const tab of document.querySelectorAll(".tab")){tab.addEventListener("click",()=>{document.querySelectorAll(".tab").forEach((t)=>t.classList.remove("active"));document.querySelectorAll(".tab-panel").forEach((p)=>p.classList.remove("active"));tab.classList.add("active");$(tab.dataset.tab).classList.add("active");setTimeout(()=>Object.values(charts).forEach((chart)=>chart.resize()),20);});}
|
| 84 |
+
function closeExpandedChart(){const card=document.querySelector(".chart-card.chart-expanded");if(!card)return;card.classList.remove("chart-expanded");const b=card.querySelector(".chart-expand");if(b)b.textContent="Expand";document.body.classList.remove("chart-open");setTimeout(()=>Chart.getChart(card.querySelector("canvas"))?.resize(),20);}
|
| 85 |
+
for(const b of document.querySelectorAll(".chart-expand")){b.addEventListener("click",()=>{const card=b.closest(".chart-card"),was=card.classList.contains("chart-expanded");closeExpandedChart();if(!was){card.classList.add("chart-expanded");b.textContent="Close";document.body.classList.add("chart-open");setTimeout(()=>Chart.getChart(card.querySelector("canvas"))?.resize(),20);}});}
|
| 86 |
+
document.addEventListener("keydown",(e)=>{if(e.key==="Escape")closeExpandedChart();});
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
docs/methodology.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Methodology and limitations
|
| 2 |
|
| 3 |
-
## What v0.
|
| 4 |
|
| 5 |
InferScale models request arrival, queueing, admission, prefill, autoregressive decode, dynamic batch membership, KV-cache memory and request completion. Metrics are computed from per-request virtual timestamps.
|
| 6 |
|
|
@@ -34,7 +34,7 @@ This produces useful qualitative dynamics but is **not empirically calibrated**.
|
|
| 34 |
|
| 35 |
## Quantization
|
| 36 |
|
| 37 |
-
INT8 and INT4 alter model weight footprint and apply a conservative compute-overhead multiplier. v0.
|
| 38 |
|
| 39 |
## KV-cache
|
| 40 |
|
|
@@ -48,13 +48,22 @@ for K and V with FP16 KV state. Paged allocation rounds live sequence lengths to
|
|
| 48 |
|
| 49 |
## Capacity search
|
| 50 |
|
| 51 |
-
Capacity search evaluates an offered request rate over multiple deterministic seeds and marks the rate feasible when:
|
| 52 |
|
| 53 |
-
1.
|
| 54 |
-
2. no requests
|
| 55 |
|
| 56 |
-
A bounded binary search then estimates the highest feasible offered rate.
|
| 57 |
|
| 58 |
## Validation status
|
| 59 |
|
| 60 |
-
v0.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Methodology and limitations
|
| 2 |
|
| 3 |
+
## What v0.2 simulates
|
| 4 |
|
| 5 |
InferScale models request arrival, queueing, admission, prefill, autoregressive decode, dynamic batch membership, KV-cache memory and request completion. Metrics are computed from per-request virtual timestamps.
|
| 6 |
|
|
|
|
| 34 |
|
| 35 |
## Quantization
|
| 36 |
|
| 37 |
+
INT8 and INT4 alter model weight footprint and apply a conservative compute-overhead multiplier. v0.2 does not claim a specific kernel implementation, quantization scheme, or quality impact.
|
| 38 |
|
| 39 |
## KV-cache
|
| 40 |
|
|
|
|
| 48 |
|
| 49 |
## Capacity search
|
| 50 |
|
| 51 |
+
Capacity search evaluates an offered request rate over multiple deterministic seeds and marks the rate feasible only when **every repetition**:
|
| 52 |
|
| 53 |
+
1. reaches the configured SLO-attainment target; and
|
| 54 |
+
2. leaves no requests unfinished after the simulated drain.
|
| 55 |
|
| 56 |
+
A bounded binary search then estimates the highest feasible offered rate. Mean/min/max attainment are retained for evidence, and the recommended rate applies user-configured safety headroom.
|
| 57 |
|
| 58 |
## Validation status
|
| 59 |
|
| 60 |
+
v0.2 validates software invariants and expected qualitative behavior through unit tests. It does **not** yet provide held-out GPU calibration error. Empirical profile calibration and validation remain future work rather than being fabricated for this release.
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
## v0.2 bottleneck diagnosis
|
| 64 |
+
|
| 65 |
+
InferScale derives a compact explanation from the simulator's own state: component SLO attainment, p95 queue latency, virtual device busy fraction, peak KV utilization, and unfinished requests. These are heuristic simulator diagnoses, not hardware-counter diagnoses of a real GPU.
|
| 66 |
+
|
| 67 |
+
## v0.2 robust capacity criterion
|
| 68 |
+
|
| 69 |
+
For each searched load, the planner repeats the workload with deterministic seed offsets. A rate passes only when every repetition reaches the configured SLO target and fully drains. Mean attainment is plotted while minimum and maximum attainment remain visible in the trace.
|
docs/research.md
CHANGED
|
@@ -30,8 +30,8 @@ Project inspiration: profile-based hardware abstraction and future disaggregatio
|
|
| 30 |
|
| 31 |
Frontier models modern serving structures including co-location, prefill/decode disaggregation, Attention-FFN disaggregation, runtime optimizations and stateful workloads. Its May 2026 paper reports average throughput error below 4% on a 16-H800 testbed and large improvements in end-to-end latency error over baseline simulators.
|
| 32 |
|
| 33 |
-
Project inspiration for
|
| 34 |
|
| 35 |
## Scope boundary
|
| 36 |
|
| 37 |
-
InferScale-Sim v0.
|
|
|
|
| 30 |
|
| 31 |
Frontier models modern serving structures including co-location, prefill/decode disaggregation, Attention-FFN disaggregation, runtime optimizations and stateful workloads. Its May 2026 paper reports average throughput error below 4% on a 16-H800 testbed and large improvements in end-to-end latency error over baseline simulators.
|
| 32 |
|
| 33 |
+
Project inspiration for future work: prefill/decode disaggregation, KV transfer and workload-state dependencies.
|
| 34 |
|
| 35 |
## Scope boundary
|
| 36 |
|
| 37 |
+
InferScale-Sim v0.2 is not intended to compete with these research systems on fidelity or scale. Its contribution is an inspectable, dependency-light Python implementation and a zero-backend interactive interface for exploring the underlying serving dynamics.
|
docs/validation.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# v0.
|
| 2 |
|
| 3 |
The release check is designed to prevent deployment and provenance mistakes rather than pretend the analytical profile has empirical accuracy.
|
| 4 |
|
|
@@ -13,6 +13,10 @@ The release check is designed to prevent deployment and provenance mistakes rath
|
|
| 13 |
- static vs continuous batching behavioral difference
|
| 14 |
- capacity-search output sanity
|
| 15 |
- scheduler-arena output sanity
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
- Hugging Face `short_description` <= 60 characters
|
| 17 |
- `sdk: static` metadata
|
| 18 |
- canonical Python source == browser mirror
|
|
@@ -20,7 +24,7 @@ The release check is designed to prevent deployment and provenance mistakes rath
|
|
| 20 |
- JavaScript syntax parse
|
| 21 |
- Python compilation
|
| 22 |
|
| 23 |
-
## Not claimed in v0.
|
| 24 |
|
| 25 |
- empirical L4/A10G/A100 latency accuracy
|
| 26 |
- exact vLLM/SGLang scheduler equivalence
|
|
|
|
| 1 |
+
# v0.2 validation checklist
|
| 2 |
|
| 3 |
The release check is designed to prevent deployment and provenance mistakes rather than pretend the analytical profile has empirical accuracy.
|
| 4 |
|
|
|
|
| 13 |
- static vs continuous batching behavioral difference
|
| 14 |
- capacity-search output sanity
|
| 15 |
- scheduler-arena output sanity
|
| 16 |
+
- component TTFT/E2E SLO-attainment accounting
|
| 17 |
+
- simulator bottleneck diagnosis provenance
|
| 18 |
+
- capacity repetition range / conservative pass criterion
|
| 19 |
+
- ASCII-only public UI labels for reliable rendering
|
| 20 |
- Hugging Face `short_description` <= 60 characters
|
| 21 |
- `sdk: static` metadata
|
| 22 |
- canonical Python source == browser mirror
|
|
|
|
| 24 |
- JavaScript syntax parse
|
| 25 |
- Python compilation
|
| 26 |
|
| 27 |
+
## Not claimed in v0.2
|
| 28 |
|
| 29 |
- empirical L4/A10G/A100 latency accuracy
|
| 30 |
- exact vLLM/SGLang scheduler equivalence
|
index.html
CHANGED
|
@@ -10,198 +10,52 @@
|
|
| 10 |
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.5.1/dist/chart.umd.min.js"></script>
|
| 11 |
</head>
|
| 12 |
<body>
|
| 13 |
-
<header class="topbar">
|
| 14 |
-
<div class="brand-wrap">
|
| 15 |
-
|
| 16 |
-
<div>
|
| 17 |
-
<div class="brand">InferScale-Sim</div>
|
| 18 |
-
<div class="subtitle">LLM serving systems laboratory</div>
|
| 19 |
-
</div>
|
| 20 |
-
</div>
|
| 21 |
-
<div class="runtime-pill" id="runtimePill"><span class="dot"></span><span id="runtimeText">Loading Python runtime…</span></div>
|
| 22 |
</header>
|
| 23 |
|
| 24 |
<main class="shell">
|
| 25 |
-
<section class="
|
| 26 |
-
<div>
|
| 27 |
-
|
| 28 |
-
<h1>Explore LLM serving dynamics without provisioning a GPU.</h1>
|
| 29 |
-
<p>Generate workloads, compare schedulers, inspect queueing and KV pressure, then search for the maximum SLO-compliant request rate. The simulator runs locally in your browser through Pyodide.</p>
|
| 30 |
-
</div>
|
| 31 |
-
<div class="hero-stat-grid">
|
| 32 |
-
<div class="hero-stat"><span>Backend</span><strong>None</strong></div>
|
| 33 |
-
<div class="hero-stat"><span>Runtime</span><strong>Python / WASM</strong></div>
|
| 34 |
-
<div class="hero-stat"><span>Data sent</span><strong>0 bytes</strong></div>
|
| 35 |
-
<div class="hero-stat"><span>Profile type</span><strong>Analytical</strong></div>
|
| 36 |
-
</div>
|
| 37 |
-
</section>
|
| 38 |
-
|
| 39 |
-
<div class="notice">
|
| 40 |
-
<strong>Reference-profile mode.</strong> v0.1 uses analytical hardware/model profiles to study systems behavior. Absolute latency values are predictions, not measured GPU benchmarks.
|
| 41 |
-
</div>
|
| 42 |
-
|
| 43 |
-
<nav class="tabs" aria-label="InferScale sections">
|
| 44 |
-
<button class="tab active" data-tab="lab">Serving Lab</button>
|
| 45 |
-
<button class="tab" data-tab="arena">Scheduler Arena</button>
|
| 46 |
-
<button class="tab" data-tab="planner">Capacity Planner</button>
|
| 47 |
-
<button class="tab" data-tab="method">Methodology</button>
|
| 48 |
-
</nav>
|
| 49 |
-
|
| 50 |
-
<section id="lab" class="tab-panel active">
|
| 51 |
-
<div class="workspace">
|
| 52 |
-
<aside class="panel controls-panel">
|
| 53 |
-
<div class="panel-title-row"><h2>Experiment</h2><span class="tag">Deterministic seed</span></div>
|
| 54 |
-
|
| 55 |
-
<div class="field-grid two">
|
| 56 |
-
<label>Model<select id="model"><option>Qwen2.5-3B</option><option>Llama-3.1-8B</option><option>Mistral-7B-v0.3</option></select></label>
|
| 57 |
-
<label>Accelerator<select id="accelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label>
|
| 58 |
-
</div>
|
| 59 |
-
|
| 60 |
-
<div class="field-grid two">
|
| 61 |
-
<label>Scheduler<select id="scheduler"><option value="continuous_fcfs">Continuous · FCFS</option><option value="continuous_sjf">Continuous · SJF</option><option value="continuous_slo">Continuous · SLO-aware</option><option value="chunked_slo">Chunked prefill · SLO-aware</option><option value="static_fcfs">Static batching · FCFS</option></select></label>
|
| 62 |
-
<label>Weight precision<select id="quantization"><option value="fp16">FP16</option><option value="int8" selected>INT8 scenario</option><option value="int4">INT4 scenario</option></select></label>
|
| 63 |
-
</div>
|
| 64 |
-
|
| 65 |
-
<hr />
|
| 66 |
-
<div class="section-kicker">Workload</div>
|
| 67 |
-
<div class="field-grid two">
|
| 68 |
-
<label>Arrival process<select id="arrival"><option value="poisson">Poisson</option><option value="constant">Constant</option><option value="bursty">Bursty</option></select></label>
|
| 69 |
-
<label>Request rate<input id="rate" type="number" min="0.1" step="0.1" value="4" /><span class="unit">req/s</span></label>
|
| 70 |
-
<label>Duration<input id="duration" type="number" min="2" step="1" value="30" /><span class="unit">simulated s</span></label>
|
| 71 |
-
<label>Seed<input id="seed" type="number" step="1" value="7" /></label>
|
| 72 |
-
</div>
|
| 73 |
-
<div class="field-grid two">
|
| 74 |
-
<label>Prompt mean<input id="promptMean" type="number" min="16" value="512" /><span class="unit">tokens</span></label>
|
| 75 |
-
<label>Prompt CV<input id="promptCv" type="number" min="0" max="2" step="0.05" value="0.50" /></label>
|
| 76 |
-
<label>Output mean<input id="outputMean" type="number" min="1" value="64" /><span class="unit">tokens</span></label>
|
| 77 |
-
<label>Output CV<input id="outputCv" type="number" min="0" max="2" step="0.05" value="0.60" /></label>
|
| 78 |
-
</div>
|
| 79 |
-
|
| 80 |
-
<hr />
|
| 81 |
-
<div class="section-kicker">Serving controls</div>
|
| 82 |
-
<div class="field-grid two">
|
| 83 |
-
<label>Max batch size<input id="maxBatch" type="number" min="1" max="128" value="16" /></label>
|
| 84 |
-
<label>Max batch tokens<input id="maxBatchTokens" type="number" min="128" step="128" value="8192" /></label>
|
| 85 |
-
<label>Prefill chunk<input id="chunkSize" type="number" min="64" step="64" value="512" /><span class="unit">tokens</span></label>
|
| 86 |
-
<label>KV block<input id="kvBlock" type="number" min="1" value="16" /><span class="unit">tokens</span></label>
|
| 87 |
-
</div>
|
| 88 |
-
|
| 89 |
-
<hr />
|
| 90 |
-
<div class="section-kicker">SLO</div>
|
| 91 |
-
<div class="field-grid two">
|
| 92 |
-
<label>TTFT limit<input id="sloTtft" type="number" min="1" value="500" /><span class="unit">ms</span></label>
|
| 93 |
-
<label>E2E limit<input id="sloE2e" type="number" min="100" value="5000" /><span class="unit">ms</span></label>
|
| 94 |
-
</div>
|
| 95 |
-
|
| 96 |
-
<button id="runBtn" class="primary" disabled>Run simulation</button>
|
| 97 |
-
<button id="exportBtn" class="secondary" disabled>Export last result</button>
|
| 98 |
-
</aside>
|
| 99 |
-
|
| 100 |
-
<div class="results-column">
|
| 101 |
-
<section class="panel result-panel">
|
| 102 |
-
<div class="panel-title-row"><h2>Run summary</h2><span id="runState" class="tag neutral">Waiting</span></div>
|
| 103 |
-
<div id="emptyState" class="empty-state"><div class="empty-icon">↗</div><h3>Configure a workload and run it</h3><p>The Python simulator will execute in a Web Worker and return request-level virtual timestamps.</p></div>
|
| 104 |
-
<div id="resultContent" class="hidden">
|
| 105 |
-
<div class="metric-grid">
|
| 106 |
-
<div class="metric"><span>p95 TTFT</span><strong id="mTtft">—</strong></div>
|
| 107 |
-
<div class="metric"><span>p95 E2E</span><strong id="mE2e">—</strong></div>
|
| 108 |
-
<div class="metric"><span>Goodput</span><strong id="mGoodput">—</strong></div>
|
| 109 |
-
<div class="metric"><span>SLO attainment</span><strong id="mSlo">—</strong></div>
|
| 110 |
-
<div class="metric"><span>Throughput</span><strong id="mReq">—</strong></div>
|
| 111 |
-
<div class="metric"><span>Peak KV</span><strong id="mKv">—</strong></div>
|
| 112 |
-
</div>
|
| 113 |
-
<div class="chart-grid">
|
| 114 |
-
<div class="chart-card"><div class="chart-title">Latency percentiles</div><canvas id="latencyChart"></canvas></div>
|
| 115 |
-
<div class="chart-card"><div class="chart-title">Queue & decode timeline</div><canvas id="timelineChart"></canvas></div>
|
| 116 |
-
</div>
|
| 117 |
-
<div class="chart-card full"><div class="chart-title">Request TTFT vs prompt length</div><canvas id="scatterChart"></canvas></div>
|
| 118 |
-
<div id="warnings" class="warnings hidden"></div>
|
| 119 |
-
</div>
|
| 120 |
-
</section>
|
| 121 |
-
</div>
|
| 122 |
-
</div>
|
| 123 |
-
</section>
|
| 124 |
-
|
| 125 |
-
<section id="arena" class="tab-panel">
|
| 126 |
-
<div class="panel wide-panel">
|
| 127 |
-
<div class="panel-title-row">
|
| 128 |
-
<div><div class="section-kicker">Same workload · same seed</div><h2>Scheduler Arena</h2><p class="muted">Run every v0.1 scheduler against the current Serving Lab configuration and rank by SLO attainment then goodput.</p></div>
|
| 129 |
-
<button id="arenaBtn" class="primary compact" disabled>Compare schedulers</button>
|
| 130 |
-
</div>
|
| 131 |
-
<div id="arenaEmpty" class="empty-state small"><h3>No comparison yet</h3><p>Your Serving Lab controls are reused automatically.</p></div>
|
| 132 |
-
<div id="arenaContent" class="hidden">
|
| 133 |
-
<div class="chart-card full"><canvas id="arenaChart"></canvas></div>
|
| 134 |
-
<div class="table-wrap"><table><thead><tr><th>Scheduler</th><th>Goodput</th><th>SLO attainment</th><th>p95 TTFT</th><th>p95 E2E</th><th>KV peak</th><th>Unfinished</th></tr></thead><tbody id="arenaRows"></tbody></table></div>
|
| 135 |
-
</div>
|
| 136 |
-
</div>
|
| 137 |
-
</section>
|
| 138 |
-
|
| 139 |
-
<section id="planner" class="tab-panel">
|
| 140 |
-
<div class="workspace planner-grid">
|
| 141 |
-
<aside class="panel controls-panel">
|
| 142 |
-
<div class="panel-title-row"><h2>Capacity search</h2><span class="tag">Binary search</span></div>
|
| 143 |
-
<p class="muted">Find the highest offered load that reaches the target SLO attainment and drains all generated requests.</p>
|
| 144 |
-
<label>Required SLO attainment<input id="targetSlo" type="number" min="0.5" max="1" step="0.001" value="0.99" /></label>
|
| 145 |
-
<div class="field-grid two">
|
| 146 |
-
<label>Minimum rate<input id="minRate" type="number" min="0.05" step="0.1" value="0.25" /><span class="unit">req/s</span></label>
|
| 147 |
-
<label>Maximum rate<input id="maxRate" type="number" min="0.1" step="1" value="20" /><span class="unit">req/s</span></label>
|
| 148 |
-
<label>Search iterations<input id="searchIter" type="number" min="2" max="12" value="7" /></label>
|
| 149 |
-
<label>Repetitions / rate<input id="repetitions" type="number" min="1" max="5" value="2" /></label>
|
| 150 |
-
</div>
|
| 151 |
-
<label>Safety headroom<input id="headroom" type="number" min="0" max="0.8" step="0.05" value="0.20" /><span class="unit">fraction</span></label>
|
| 152 |
-
<button id="capacityBtn" class="primary" disabled>Find sustainable capacity</button>
|
| 153 |
-
</aside>
|
| 154 |
-
<section class="panel result-panel">
|
| 155 |
-
<div class="panel-title-row"><h2>Planner result</h2><span id="plannerState" class="tag neutral">Waiting</span></div>
|
| 156 |
-
<div id="plannerEmpty" class="empty-state"><h3>No search yet</h3><p>The planner repeatedly runs the simulator at different offered loads.</p></div>
|
| 157 |
-
<div id="plannerContent" class="hidden">
|
| 158 |
-
<div class="metric-grid four">
|
| 159 |
-
<div class="metric emphasis"><span>Estimated capacity</span><strong id="pCapacity">—</strong></div>
|
| 160 |
-
<div class="metric"><span>Recommended load</span><strong id="pRecommended">—</strong></div>
|
| 161 |
-
<div class="metric"><span>Safety headroom</span><strong id="pHeadroom">—</strong></div>
|
| 162 |
-
<div class="metric"><span>Status</span><strong id="pStatus">—</strong></div>
|
| 163 |
-
</div>
|
| 164 |
-
<div class="chart-card full"><div class="chart-title">SLO attainment across searched rates</div><canvas id="capacityChart"></canvas></div>
|
| 165 |
-
<div class="table-wrap"><table><thead><tr><th>Rate</th><th>Pass</th><th>SLO attainment</th><th>Goodput</th><th>p95 TTFT</th><th>p95 E2E</th></tr></thead><tbody id="capacityRows"></tbody></table></div>
|
| 166 |
-
</div>
|
| 167 |
-
</section>
|
| 168 |
-
</div>
|
| 169 |
-
</section>
|
| 170 |
-
|
| 171 |
-
<section id="method" class="tab-panel">
|
| 172 |
-
<div class="method-grid">
|
| 173 |
-
<article class="panel prose">
|
| 174 |
-
<div class="section-kicker">What is simulated?</div>
|
| 175 |
-
<h2>A serving loop, not CUDA kernels.</h2>
|
| 176 |
-
<p>InferScale advances virtual time through request arrivals, queueing, prefill, autoregressive decode, batch membership changes, KV allocation and completion. A predicted 40 ms decode step becomes <code>simulated_time += 0.040</code>; the browser never waits 40 ms.</p>
|
| 177 |
-
<p>The latency backend is roofline-inspired and intentionally replaceable. It combines model size, attention shape, accelerator peak throughput and memory bandwidth with conservative efficiency factors. v0.1 uses it to expose systems interactions, not to claim measured L4/A100 latency.</p>
|
| 178 |
-
</article>
|
| 179 |
-
<article class="panel prose">
|
| 180 |
-
<div class="section-kicker">Why goodput?</div>
|
| 181 |
-
<h2>Throughput can reward overload.</h2>
|
| 182 |
-
<p>InferScale defines goodput as the number of completed requests that satisfy both the TTFT and end-to-end latency SLOs divided by simulated makespan. A system can increase raw throughput while simultaneously becoming less useful to latency-sensitive applications.</p>
|
| 183 |
-
<div class="formula">goodput = SLO-compliant completions / simulated time</div>
|
| 184 |
-
</article>
|
| 185 |
-
<article class="panel prose wide-method">
|
| 186 |
-
<div class="section-kicker">Research lineage</div>
|
| 187 |
-
<h2>From Vidur to modern disaggregated simulators.</h2>
|
| 188 |
-
<div class="paper-grid">
|
| 189 |
-
<div><strong>Vidur · 2024</strong><span>Predictive profiling + deployment search. Reported a LLaMA2-70B search in ~1 CPU-hour versus an estimated 42K GPU-hours (~$218K).</span></div>
|
| 190 |
-
<div><strong>TokenSim · 2025</strong><span>Extensible hardware/software exploration with scheduler and memory-management modeling.</span></div>
|
| 191 |
-
<div><strong>Revati · 2026</strong><span>GPU-free time-warp emulation that executes real serving control logic rather than reimplementing it.</span></div>
|
| 192 |
-
<div><strong>LLMServingSim 2.0 · 2026</strong><span>Heterogeneous/disaggregated infrastructure with runtime-driven batching, routing, memory and power.</span></div>
|
| 193 |
-
<div><strong>Frontier · 2026</strong><span>Modern disaggregation, runtime optimizations, stateful workloads and large-scale configuration exploration.</span></div>
|
| 194 |
-
</div>
|
| 195 |
-
</article>
|
| 196 |
-
</div>
|
| 197 |
</section>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
</main>
|
| 199 |
-
|
| 200 |
-
<
|
| 201 |
-
<span>InferScale-Sim v0.1.0</span>
|
| 202 |
-
<span>Simulation executes locally in your browser. No experiment data is transmitted.</span>
|
| 203 |
-
</footer>
|
| 204 |
-
|
| 205 |
<script type="module" src="app.js"></script>
|
| 206 |
</body>
|
| 207 |
</html>
|
|
|
|
| 10 |
<script src="https://cdn.jsdelivr.net/npm/chart.js@4.5.1/dist/chart.umd.min.js"></script>
|
| 11 |
</head>
|
| 12 |
<body>
|
| 13 |
+
<header class="topbar" id="topbar">
|
| 14 |
+
<div class="brand-wrap"><div class="logo">IS</div><div><div class="brand">InferScale-Sim</div><div class="subtitle">LLM serving systems laboratory</div></div></div>
|
| 15 |
+
<div class="runtime-pill" id="runtimePill"><span class="dot"></span><span id="runtimeText">Loading Python runtime...</span></div>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 16 |
</header>
|
| 17 |
|
| 18 |
<main class="shell">
|
| 19 |
+
<section class="intro">
|
| 20 |
+
<div class="intro-copy"><div class="eyebrow">InferScale-Sim / v0.2.0</div><h1>Explore LLM serving dynamics without provisioning a GPU.</h1><p>Generate workloads, compare schedulers, inspect queueing and KV pressure, and search for the highest SLO-compliant request rate. The simulator is Python running locally in a Pyodide Web Worker.</p></div>
|
| 21 |
+
<dl class="project-facts"><div><dt>Backend</dt><dd>None</dd></div><div><dt>Runtime</dt><dd>Python / WASM</dd></div><div><dt>Network data</dt><dd>None after load</dd></div><div><dt>Profiles</dt><dd>Analytical reference</dd></div></dl>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
</section>
|
| 23 |
+
<div class="reference-note"><strong>Reference-profile mode.</strong> Absolute L4, A10G, and A100 latency values are analytical predictions, not measured GPU benchmarks. Scheduler, queueing, KV-cache, and SLO behavior is simulated live.</div>
|
| 24 |
+
|
| 25 |
+
<nav class="tabs" aria-label="InferScale sections"><button class="tab active" data-tab="lab">Serving Lab</button><button class="tab" data-tab="arena">Scheduler Arena</button><button class="tab" data-tab="planner">Capacity Planner</button><button class="tab" data-tab="method">Methodology</button></nav>
|
| 26 |
+
|
| 27 |
+
<section id="lab" class="tab-panel active"><div class="workspace">
|
| 28 |
+
<aside class="panel controls-panel">
|
| 29 |
+
<div class="panel-title-row"><h2>Experiment</h2><span class="tag">Deterministic seed</span></div>
|
| 30 |
+
<div class="field-grid two"><label>Model<select id="model"><option>Qwen2.5-3B</option><option>Llama-3.1-8B</option><option>Mistral-7B-v0.3</option></select></label><label>Accelerator<select id="accelerator"><option value="L4">NVIDIA L4</option><option value="A10G">NVIDIA A10G</option><option value="A100-40GB">NVIDIA A100 40GB</option></select></label></div>
|
| 31 |
+
<div class="field-grid two"><label>Scheduler<select id="scheduler"><option value="continuous_fcfs">Continuous - FCFS</option><option value="continuous_sjf">Continuous - SJF</option><option value="continuous_slo">Continuous - SLO-aware</option><option value="chunked_slo">Chunked prefill - SLO-aware</option><option value="static_fcfs">Static batching - FCFS</option></select></label><label>Weight precision<select id="quantization"><option value="fp16">FP16</option><option value="int8" selected>INT8 scenario</option><option value="int4">INT4 scenario</option></select></label></div>
|
| 32 |
+
<hr /><div class="section-kicker">Workload</div>
|
| 33 |
+
<div class="field-grid two"><label>Arrival process<select id="arrival"><option value="poisson">Poisson</option><option value="constant">Constant</option><option value="bursty">Bursty</option></select></label><label>Request rate<input id="rate" type="number" min="0.1" step="0.1" value="4" /><span class="unit">req/s</span></label><label>Duration<input id="duration" type="number" min="2" step="1" value="30" /><span class="unit">sim s</span></label><label>Seed<input id="seed" type="number" step="1" value="7" /></label></div>
|
| 34 |
+
<div id="burstControls" class="field-grid two hidden"><label>Burst multiplier<input id="burstMultiplier" type="number" min="1" step="0.25" value="3" /><span class="unit">x</span></label><label>Burst period<input id="burstPeriod" type="number" min="0.5" step="0.5" value="10" /><span class="unit">s</span></label></div>
|
| 35 |
+
<div class="field-grid two"><label>Prompt mean<input id="promptMean" type="number" min="16" value="512" /><span class="unit">tokens</span></label><label>Prompt CV<input id="promptCv" type="number" min="0" max="2" step="0.05" value="0.50" /></label><label>Output mean<input id="outputMean" type="number" min="1" value="64" /><span class="unit">tokens</span></label><label>Output CV<input id="outputCv" type="number" min="0" max="2" step="0.05" value="0.60" /></label></div>
|
| 36 |
+
<hr /><div class="section-kicker">Serving controls</div>
|
| 37 |
+
<div class="field-grid two"><label>Max batch size<input id="maxBatch" type="number" min="1" max="128" value="16" /></label><label>Max batch tokens<input id="maxBatchTokens" type="number" min="128" step="128" value="8192" /></label><label>Prefill chunk<input id="chunkSize" type="number" min="64" step="64" value="512" /><span class="unit">tokens</span></label><label>KV block<input id="kvBlock" type="number" min="1" value="16" /><span class="unit">tokens</span></label></div>
|
| 38 |
+
<hr /><div class="section-kicker">SLO</div><div class="field-grid two"><label>TTFT limit<input id="sloTtft" type="number" min="1" value="500" /><span class="unit">ms</span></label><label>E2E limit<input id="sloE2e" type="number" min="100" value="5000" /><span class="unit">ms</span></label></div>
|
| 39 |
+
<button id="runBtn" class="primary" disabled>Run simulation</button><div class="button-row"><button id="copyResultBtn" class="secondary" disabled>Copy JSON</button><button id="exportBtn" class="secondary" disabled>Download JSON</button></div>
|
| 40 |
+
</aside>
|
| 41 |
+
<div class="results-column"><section class="panel result-panel"><div class="panel-title-row"><h2>Run summary</h2><span id="runState" class="tag neutral">Waiting</span></div><div id="emptyState" class="empty-state"><h3>Configure a workload and run it</h3><p>The Python simulator executes in a background Web Worker and returns request-level virtual timestamps.</p></div><div id="resultContent" class="hidden">
|
| 42 |
+
<div class="metric-grid"><div class="metric"><span>p95 TTFT</span><strong id="mTtft">N/A</strong></div><div class="metric"><span>p95 E2E</span><strong id="mE2e">N/A</strong></div><div class="metric"><span>Goodput</span><strong id="mGoodput">N/A</strong></div><div class="metric"><span>SLO attainment</span><strong id="mSlo">N/A</strong></div><div class="metric"><span>Throughput</span><strong id="mReq">N/A</strong></div><div class="metric"><span>Peak KV</span><strong id="mKv">N/A</strong></div></div>
|
| 43 |
+
<section class="diagnostic-card" id="diagnosticCard"><div class="diagnostic-head"><span>Simulator diagnosis</span><strong id="mBottleneck">N/A</strong></div><p id="mDiagnosis">Run a simulation to generate a bottleneck explanation.</p><p class="diagnostic-action" id="mRecommendation"></p><div class="evidence-row" id="mEvidence"></div></section>
|
| 44 |
+
<div class="chart-grid"><div class="chart-card" data-chart-card><div class="chart-head"><div class="chart-title">Latency percentiles</div><button class="chart-expand" type="button">Expand</button></div><div class="chart-body"><canvas id="latencyChart"></canvas></div></div><div class="chart-card" data-chart-card><div class="chart-head"><div class="chart-title">Queue, decode, and KV timeline</div><button class="chart-expand" type="button">Expand</button></div><div class="chart-body"><canvas id="timelineChart"></canvas></div></div></div>
|
| 45 |
+
<div class="chart-card full" data-chart-card><div class="chart-head"><div class="chart-title">Request TTFT vs prompt length</div><button class="chart-expand" type="button">Expand</button></div><div class="chart-body"><canvas id="scatterChart"></canvas></div></div><div id="warnings" class="warnings hidden"></div>
|
| 46 |
+
</div></section></div>
|
| 47 |
+
</div></section>
|
| 48 |
+
|
| 49 |
+
<section id="arena" class="tab-panel"><div class="panel wide-panel"><div class="panel-title-row arena-title-row"><div><div class="section-kicker">Same workload - same seed</div><h2>Scheduler Arena</h2><p class="muted">Run every scheduler against the current Serving Lab configuration and rank by SLO attainment, then goodput.</p></div><button id="arenaBtn" class="primary compact" disabled>Compare schedulers</button></div><div id="arenaEmpty" class="empty-state small"><h3>No comparison yet</h3><p>Your Serving Lab controls are reused automatically.</p></div><div id="arenaContent" class="hidden"><div class="chart-card full" data-chart-card><div class="chart-head"><div class="chart-title">Useful vs raw request throughput</div><button class="chart-expand" type="button">Expand</button></div><div class="chart-body large"><canvas id="arenaChart"></canvas></div></div><div class="table-toolbar"><span>Scheduler comparison</span><div><button id="arenaCopyBtn" class="mini-button" disabled>Copy table</button><button id="arenaCsvBtn" class="mini-button" disabled>Download CSV</button></div></div><div class="table-wrap"><table><thead><tr><th>Scheduler</th><th>Goodput</th><th>SLO attainment</th><th>p95 TTFT</th><th>p95 E2E</th><th>KV peak</th><th>Unfinished</th><th>Diagnosis</th></tr></thead><tbody id="arenaRows"></tbody></table></div></div></div></section>
|
| 50 |
+
|
| 51 |
+
<section id="planner" class="tab-panel"><div class="workspace planner-grid"><aside class="panel controls-panel"><div class="panel-title-row"><h2>Capacity search</h2><span class="tag">Robust binary search</span></div><p class="muted">Find the highest offered load where every repetition meets the SLO target and fully drains.</p><label>Required SLO attainment<input id="targetSlo" type="number" min="0.5" max="1" step="0.001" value="0.99" /></label><div class="field-grid two"><label>Minimum rate<input id="minRate" type="number" min="0.05" step="0.1" value="0.25" /><span class="unit">req/s</span></label><label>Maximum rate<input id="maxRate" type="number" min="0.1" step="1" value="20" /><span class="unit">req/s</span></label><label>Search iterations<input id="searchIter" type="number" min="2" max="12" value="7" /></label><label>Repetitions / rate<input id="repetitions" type="number" min="1" max="5" value="2" /></label></div><label>Safety headroom<input id="headroom" type="number" min="0" max="0.8" step="0.05" value="0.20" /><span class="unit">fraction</span></label><button id="capacityBtn" class="primary" disabled>Find sustainable capacity</button></aside>
|
| 52 |
+
<section class="panel result-panel"><div class="panel-title-row"><h2>Planner result</h2><span id="plannerState" class="tag neutral">Waiting</span></div><div id="plannerEmpty" class="empty-state"><h3>No search yet</h3><p>The planner repeatedly runs the simulator at different offered loads.</p></div><div id="plannerContent" class="hidden"><div class="metric-grid four"><div class="metric emphasis"><span>Estimated capacity</span><strong id="pCapacity">N/A</strong></div><div class="metric"><span>Recommended load</span><strong id="pRecommended">N/A</strong></div><div class="metric"><span>Safety headroom</span><strong id="pHeadroom">N/A</strong></div><div class="metric"><span>Status</span><strong id="pStatus">N/A</strong></div></div><div class="planner-note" id="plannerCriterion">A rate passes only if every repetition meets the target and drains all generated requests.</div><div class="chart-card full" data-chart-card><div class="chart-head"><div class="chart-title">SLO attainment across searched rates</div><button class="chart-expand" type="button">Expand</button></div><div class="chart-body large"><canvas id="capacityChart"></canvas></div></div><div class="table-toolbar"><span>Search trace</span><div><button id="capacityCopyBtn" class="mini-button" disabled>Copy table</button><button id="capacityCsvBtn" class="mini-button" disabled>Download CSV</button></div></div><div class="table-wrap"><table><thead><tr><th>Rate</th><th>Pass</th><th>Mean SLO</th><th>SLO range</th><th>Goodput</th><th>p95 TTFT</th><th>p95 E2E</th></tr></thead><tbody id="capacityRows"></tbody></table></div></div></section>
|
| 53 |
+
</div></section>
|
| 54 |
+
|
| 55 |
+
<section id="method" class="tab-panel"><div class="method-grid"><article class="panel prose"><div class="section-kicker">What is simulated?</div><h2>A serving loop, not CUDA kernels.</h2><p>InferScale advances virtual time through request arrivals, queueing, prefill, autoregressive decode, batch membership changes, KV allocation, and completion. A predicted 40 ms decode step becomes <code>simulated_time += 0.040</code>; the browser never waits 40 ms.</p><p>The latency backend is roofline-inspired and replaceable. v0.2 uses it to expose systems interactions, not to claim measured L4/A100 latency.</p></article><article class="panel prose"><div class="section-kicker">Why goodput?</div><h2>Throughput can reward overload.</h2><p>Goodput counts completed requests satisfying both TTFT and E2E SLOs per simulated second.</p><div class="formula">goodput = SLO-compliant completions / simulated time</div></article><article class="panel prose"><div class="section-kicker">v0.2 analysis</div><h2>Explain the limiting regime.</h2><p>Each run derives a simulator-side diagnosis from SLO failures, queue latency, busy fraction, unfinished work, and KV utilization. These labels explain simulated state, not hardware counters.</p><div class="formula">healthy | queue/prefill | decode/E2E | KV pressure | over-capacity</div></article><article class="panel prose"><div class="section-kicker">Capacity criterion</div><h2>Do not average away a bad repetition.</h2><p>A searched rate passes only when every configured repetition reaches the SLO target and fully drains. Mean, minimum, and maximum attainment remain visible in the trace.</p></article><article class="panel prose wide-method"><div class="section-kicker">Research lineage</div><h2>From Vidur to modern disaggregated simulators.</h2><div class="paper-grid"><div><strong>Vidur - 2024</strong><span>Predictive profiling and deployment search. Reported about 1 CPU-hour versus an estimated 42K GPU-hours (about $218K) for one LLaMA2-70B search.</span></div><div><strong>TokenSim - 2025</strong><span>Extensible hardware/software exploration with scheduler and memory-management modeling.</span></div><div><strong>Revati - 2026</strong><span>GPU-free time-warp emulation using serving control logic.</span></div><div><strong>LLMServingSim 2.0 - 2026</strong><span>Heterogeneous and disaggregated serving with runtime-driven batching, routing, memory, and power.</span></div><div><strong>Frontier - 2026</strong><span>Modern disaggregation, runtime optimizations, stateful workloads, and large-scale configuration exploration.</span></div></div></article></div></section>
|
| 56 |
</main>
|
| 57 |
+
<footer><span>InferScale-Sim v0.2.0</span><span>Simulation executes locally in your browser. Experiment data is not transmitted.</span></footer>
|
| 58 |
+
<div id="toast" class="toast" role="status" aria-live="polite"></div>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
<script type="module" src="app.js"></script>
|
| 60 |
</body>
|
| 61 |
</html>
|
py/inferscale/__init__.py
CHANGED
|
@@ -12,4 +12,4 @@ __all__ = [
|
|
| 12 |
"run_simulation",
|
| 13 |
]
|
| 14 |
|
| 15 |
-
__version__ = "0.
|
|
|
|
| 12 |
"run_simulation",
|
| 13 |
]
|
| 14 |
|
| 15 |
+
__version__ = "0.2.0"
|
py/inferscale/api.py
CHANGED
|
@@ -7,7 +7,7 @@ from .simulator import SCHEDULERS, run_simulation
|
|
| 7 |
|
| 8 |
def metadata() -> dict:
|
| 9 |
return {
|
| 10 |
-
"version": "0.
|
| 11 |
"models": list(MODELS.keys()),
|
| 12 |
"accelerators": list(ACCELERATORS.keys()),
|
| 13 |
"schedulers": sorted(SCHEDULERS),
|
|
|
|
| 7 |
|
| 8 |
def metadata() -> dict:
|
| 9 |
return {
|
| 10 |
+
"version": "0.2.0",
|
| 11 |
"models": list(MODELS.keys()),
|
| 12 |
"accelerators": list(ACCELERATORS.keys()),
|
| 13 |
"schedulers": sorted(SCHEDULERS),
|
py/inferscale/diagnostics.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
from typing import Any
|
| 4 |
+
|
| 5 |
+
from .models import SimulationConfig
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
def diagnose_run(summary: dict[str, Any], latency: dict[str, Any], resource: dict[str, Any], cfg: SimulationConfig) -> dict[str, Any]:
|
| 9 |
+
"""Explain the simulator's limiting regime from its own state."""
|
| 10 |
+
target = cfg.slo_attainment_target
|
| 11 |
+
attainment = float(summary.get("slo_attainment", 0.0))
|
| 12 |
+
ttft_attainment = float(summary.get("ttft_slo_attainment", 0.0))
|
| 13 |
+
e2e_attainment = float(summary.get("e2e_slo_attainment", 0.0))
|
| 14 |
+
unfinished = int(summary.get("requests_unfinished", 0))
|
| 15 |
+
busy = float(summary.get("busy_fraction", 0.0))
|
| 16 |
+
kv_util = float(resource.get("peak_kv_utilization", 0.0))
|
| 17 |
+
queue_p95 = float(latency.get("queue_ms", {}).get("p95", 0.0))
|
| 18 |
+
|
| 19 |
+
code = "mixed_pressure"
|
| 20 |
+
label = "Mixed SLO pressure"
|
| 21 |
+
explanation = "Several constraints are active; inspect queueing, KV pressure, and tail latency together."
|
| 22 |
+
recommendation = "Compare schedulers and run the capacity planner to isolate the limiting regime."
|
| 23 |
+
|
| 24 |
+
if unfinished > 0 and kv_util >= 0.85:
|
| 25 |
+
code, label = "kv_pressure", "KV memory pressure"
|
| 26 |
+
explanation = "Requests remain unfinished while the simulated KV cache approaches its configured capacity."
|
| 27 |
+
recommendation = "Reduce sequence pressure, lower offered load, use a smaller KV block, or choose a larger-memory profile."
|
| 28 |
+
elif unfinished > 0 and busy >= 0.92:
|
| 29 |
+
code, label = "over_capacity", "Offered load above capacity"
|
| 30 |
+
explanation = "The virtual device remains busy while the workload fails to drain."
|
| 31 |
+
recommendation = "Reduce offered load or use the capacity planner to find a sustainable operating point."
|
| 32 |
+
elif unfinished > 0:
|
| 33 |
+
code, label = "admission_stall", "Admission stall"
|
| 34 |
+
explanation = "Some requests cannot be admitted or completed under the current serving constraints."
|
| 35 |
+
recommendation = "Inspect batch-token and KV-cache limits, then compare a less restrictive scheduler configuration."
|
| 36 |
+
elif attainment >= target:
|
| 37 |
+
if busy >= 0.90:
|
| 38 |
+
code, label = "healthy_near_saturation", "Healthy, little headroom"
|
| 39 |
+
explanation = "The workload meets the configured SLO target, but simulated device busy time is already high."
|
| 40 |
+
recommendation = "Run the capacity planner before increasing traffic; the current point is close to saturation."
|
| 41 |
+
else:
|
| 42 |
+
code, label = "healthy", "Healthy"
|
| 43 |
+
explanation = "The workload drains and meets the configured SLO-attainment target with visible headroom."
|
| 44 |
+
recommendation = "Use the capacity planner to quantify how much additional offered load the profile can absorb."
|
| 45 |
+
elif kv_util >= 0.90:
|
| 46 |
+
code, label = "kv_pressure", "KV memory pressure"
|
| 47 |
+
explanation = "Peak simulated KV utilization is high enough to constrain admission and batching flexibility."
|
| 48 |
+
recommendation = "Reduce sequence pressure, use smaller KV blocks, or select a larger-memory accelerator profile."
|
| 49 |
+
elif ttft_attainment < target and queue_p95 >= max(25.0, cfg.slo_ttft_ms * 0.20):
|
| 50 |
+
code, label = "queue_prefill_pressure", "Queue / prefill pressure"
|
| 51 |
+
explanation = "TTFT misses coincide with substantial queueing before prefill begins."
|
| 52 |
+
recommendation = "Lower offered load, increase safe batching capacity, or try chunked prefill / SLO-aware scheduling."
|
| 53 |
+
elif ttft_attainment < target:
|
| 54 |
+
code, label = "ttft_pressure", "TTFT pressure"
|
| 55 |
+
explanation = "First-token latency is the dominant SLO failure even though queueing is not the only contributor."
|
| 56 |
+
recommendation = "Compare prefill-oriented schedulers and reduce prompt or offered-load pressure."
|
| 57 |
+
elif e2e_attainment < target:
|
| 58 |
+
code, label = "decode_pressure", "Decode / E2E pressure"
|
| 59 |
+
explanation = "Most first tokens arrive within target, but end-to-end latency still misses the configured SLO."
|
| 60 |
+
recommendation = "Reduce output-length pressure or offered load; larger decode batches may help when memory allows."
|
| 61 |
+
elif busy >= 0.95:
|
| 62 |
+
code, label = "compute_saturation", "Compute saturation"
|
| 63 |
+
explanation = "The simulated device is effectively saturated even though no single SLO component dominates."
|
| 64 |
+
recommendation = "Reduce load or switch to a higher-throughput accelerator profile."
|
| 65 |
+
|
| 66 |
+
return {
|
| 67 |
+
"code": code,
|
| 68 |
+
"label": label,
|
| 69 |
+
"explanation": explanation,
|
| 70 |
+
"recommendation": recommendation,
|
| 71 |
+
"evidence": {
|
| 72 |
+
"slo_attainment": attainment,
|
| 73 |
+
"ttft_slo_attainment": ttft_attainment,
|
| 74 |
+
"e2e_slo_attainment": e2e_attainment,
|
| 75 |
+
"busy_fraction": busy,
|
| 76 |
+
"peak_kv_utilization": kv_util,
|
| 77 |
+
"queue_p95_ms": queue_p95,
|
| 78 |
+
"unfinished_requests": unfinished,
|
| 79 |
+
},
|
| 80 |
+
"provenance": "heuristic-simulator-diagnosis",
|
| 81 |
+
}
|
py/inferscale/latency.py
CHANGED
|
@@ -29,7 +29,7 @@ class AnalyticalLatencyModel:
|
|
| 29 |
return self.model.params_b * self.weight_bytes_per_param
|
| 30 |
|
| 31 |
def kv_bytes_per_token(self) -> float:
|
| 32 |
-
# K + V, all layers, KV heads only. KV state is assumed fp16 in v0.
|
| 33 |
return (
|
| 34 |
2
|
| 35 |
* self.model.layers
|
|
|
|
| 29 |
return self.model.params_b * self.weight_bytes_per_param
|
| 30 |
|
| 31 |
def kv_bytes_per_token(self) -> float:
|
| 32 |
+
# K + V, all layers, KV heads only. KV state is assumed fp16 in v0.2.
|
| 33 |
return (
|
| 34 |
2
|
| 35 |
* self.model.layers
|
py/inferscale/metrics.py
CHANGED
|
@@ -57,14 +57,23 @@ def summarize(completed: list[Request], cfg: SimulationConfig, makespan_s: float
|
|
| 57 |
queue = [m.queue_ms for m in metrics]
|
| 58 |
total_output = sum(r.output_tokens for r in completed)
|
| 59 |
met = sum(m.met_all_slos for m in metrics)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
duration = max(makespan_s, 1e-9)
|
|
|
|
| 61 |
|
| 62 |
summary = {
|
| 63 |
"requests_completed": len(completed),
|
| 64 |
"request_throughput_rps": len(completed) / duration,
|
| 65 |
"output_throughput_tps": total_output / duration,
|
| 66 |
"goodput_rps": met / duration,
|
| 67 |
-
"slo_attainment": met /
|
|
|
|
|
|
|
|
|
|
| 68 |
"simulated_makespan_s": makespan_s,
|
| 69 |
"busy_fraction": min(1.0, busy_time_s / duration),
|
| 70 |
"mean_prompt_tokens": mean([r.prompt_tokens for r in completed]) if completed else 0.0,
|
|
|
|
| 57 |
queue = [m.queue_ms for m in metrics]
|
| 58 |
total_output = sum(r.output_tokens for r in completed)
|
| 59 |
met = sum(m.met_all_slos for m in metrics)
|
| 60 |
+
met_ttft = sum(m.met_ttft_slo for m in metrics)
|
| 61 |
+
met_e2e = sum(m.met_e2e_slo for m in metrics)
|
| 62 |
+
ttft_only_fail = sum((not m.met_ttft_slo) and m.met_e2e_slo for m in metrics)
|
| 63 |
+
e2e_only_fail = sum(m.met_ttft_slo and (not m.met_e2e_slo) for m in metrics)
|
| 64 |
+
both_fail = sum((not m.met_ttft_slo) and (not m.met_e2e_slo) for m in metrics)
|
| 65 |
duration = max(makespan_s, 1e-9)
|
| 66 |
+
count = len(metrics)
|
| 67 |
|
| 68 |
summary = {
|
| 69 |
"requests_completed": len(completed),
|
| 70 |
"request_throughput_rps": len(completed) / duration,
|
| 71 |
"output_throughput_tps": total_output / duration,
|
| 72 |
"goodput_rps": met / duration,
|
| 73 |
+
"slo_attainment": met / count if count else 0.0,
|
| 74 |
+
"ttft_slo_attainment": met_ttft / count if count else 0.0,
|
| 75 |
+
"e2e_slo_attainment": met_e2e / count if count else 0.0,
|
| 76 |
+
"slo_failure_breakdown": {"ttft_only": ttft_only_fail, "e2e_only": e2e_only_fail, "both": both_fail},
|
| 77 |
"simulated_makespan_s": makespan_s,
|
| 78 |
"busy_fraction": min(1.0, busy_time_s / duration),
|
| 79 |
"mean_prompt_tokens": mean([r.prompt_tokens for r in completed]) if completed else 0.0,
|
py/inferscale/models.py
CHANGED
|
@@ -122,6 +122,7 @@ class SimulationResult:
|
|
| 122 |
summary: dict[str, Any]
|
| 123 |
latency: dict[str, Any]
|
| 124 |
resource: dict[str, Any]
|
|
|
|
| 125 |
requests: list[dict[str, Any]] = field(default_factory=list)
|
| 126 |
timeline: list[dict[str, Any]] = field(default_factory=list)
|
| 127 |
warnings: list[str] = field(default_factory=list)
|
|
|
|
| 122 |
summary: dict[str, Any]
|
| 123 |
latency: dict[str, Any]
|
| 124 |
resource: dict[str, Any]
|
| 125 |
+
diagnostics: dict[str, Any] = field(default_factory=dict)
|
| 126 |
requests: list[dict[str, Any]] = field(default_factory=list)
|
| 127 |
timeline: list[dict[str, Any]] = field(default_factory=list)
|
| 128 |
warnings: list[str] = field(default_factory=list)
|
py/inferscale/optimizer.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
from __future__ import annotations
|
| 2 |
|
| 3 |
from copy import deepcopy
|
| 4 |
-
from statistics import mean
|
| 5 |
|
| 6 |
from .models import SimulationConfig
|
| 7 |
from .simulator import run_simulation
|
|
@@ -14,24 +14,43 @@ def evaluate_rate(base: SimulationConfig, rate: float, repetitions: int = 3) ->
|
|
| 14 |
cfg.request_rate_rps = rate
|
| 15 |
cfg.seed = base.seed + rep * 101
|
| 16 |
runs.append(run_simulation(cfg.to_dict()))
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
return {
|
| 24 |
"rate_rps": rate,
|
| 25 |
"passed": passed,
|
| 26 |
-
"slo_attainment":
|
| 27 |
-
"
|
| 28 |
-
"
|
| 29 |
-
"
|
| 30 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
}
|
| 32 |
|
| 33 |
|
| 34 |
-
def capacity_search(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
base = SimulationConfig.from_dict(config)
|
| 36 |
low = max(0.01, min_rate)
|
| 37 |
high = max(low * 1.01, max_rate)
|
|
@@ -44,18 +63,21 @@ def capacity_search(config: dict, min_rate: float = 0.25, max_rate: float = 32.0
|
|
| 44 |
"status": "no_feasible_rate",
|
| 45 |
"capacity_rps": 0.0,
|
| 46 |
"recommended_rps": 0.0,
|
|
|
|
|
|
|
| 47 |
"trace": trace,
|
| 48 |
}
|
| 49 |
|
| 50 |
high_eval = evaluate_rate(base, high, repetitions)
|
| 51 |
trace.append(high_eval)
|
| 52 |
if high_eval["passed"]:
|
| 53 |
-
capacity = high
|
| 54 |
return {
|
| 55 |
"status": "upper_bound_still_feasible",
|
| 56 |
-
"capacity_rps":
|
| 57 |
-
"recommended_rps":
|
| 58 |
-
"
|
|
|
|
|
|
|
| 59 |
}
|
| 60 |
|
| 61 |
best = low
|
|
@@ -74,7 +96,8 @@ def capacity_search(config: dict, min_rate: float = 0.25, max_rate: float = 32.0
|
|
| 74 |
"capacity_rps": best,
|
| 75 |
"recommended_rps": best * (1.0 - headroom),
|
| 76 |
"headroom": headroom,
|
| 77 |
-
"
|
|
|
|
| 78 |
}
|
| 79 |
|
| 80 |
|
|
@@ -92,15 +115,18 @@ def compare_schedulers(config: dict, schedulers: list[str] | None = None) -> dic
|
|
| 92 |
cfg = deepcopy(base)
|
| 93 |
cfg.scheduler = scheduler
|
| 94 |
result = run_simulation(cfg.to_dict())
|
| 95 |
-
rows.append(
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
|
|
|
|
|
|
|
|
|
| 106 |
return {"rows": rows}
|
|
|
|
| 1 |
from __future__ import annotations
|
| 2 |
|
| 3 |
from copy import deepcopy
|
| 4 |
+
from statistics import mean, pstdev
|
| 5 |
|
| 6 |
from .models import SimulationConfig
|
| 7 |
from .simulator import run_simulation
|
|
|
|
| 14 |
cfg.request_rate_rps = rate
|
| 15 |
cfg.seed = base.seed + rep * 101
|
| 16 |
runs.append(run_simulation(cfg.to_dict()))
|
| 17 |
+
|
| 18 |
+
attainments = [run["summary"]["slo_attainment"] for run in runs]
|
| 19 |
+
goodputs = [run["summary"]["goodput_rps"] for run in runs]
|
| 20 |
+
ttfts = [run["latency"]["ttft_ms"]["p95"] for run in runs]
|
| 21 |
+
e2es = [run["latency"]["e2e_ms"]["p95"] for run in runs]
|
| 22 |
+
unfinished_values = [run["summary"]["requests_unfinished"] for run in runs]
|
| 23 |
+
|
| 24 |
+
passed = all(
|
| 25 |
+
attainment >= base.slo_attainment_target and unfinished == 0
|
| 26 |
+
for attainment, unfinished in zip(attainments, unfinished_values, strict=True)
|
| 27 |
+
)
|
| 28 |
return {
|
| 29 |
"rate_rps": rate,
|
| 30 |
"passed": passed,
|
| 31 |
+
"slo_attainment": mean(attainments),
|
| 32 |
+
"slo_attainment_min": min(attainments),
|
| 33 |
+
"slo_attainment_max": max(attainments),
|
| 34 |
+
"slo_attainment_std": pstdev(attainments) if len(attainments) > 1 else 0.0,
|
| 35 |
+
"goodput_rps": mean(goodputs),
|
| 36 |
+
"goodput_std": pstdev(goodputs) if len(goodputs) > 1 else 0.0,
|
| 37 |
+
"p95_ttft_ms": mean(ttfts),
|
| 38 |
+
"p95_e2e_ms": mean(e2es),
|
| 39 |
+
"mean_unfinished": mean(unfinished_values),
|
| 40 |
+
"max_unfinished": max(unfinished_values),
|
| 41 |
+
"repetitions": repetitions,
|
| 42 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 43 |
}
|
| 44 |
|
| 45 |
|
| 46 |
+
def capacity_search(
|
| 47 |
+
config: dict,
|
| 48 |
+
min_rate: float = 0.25,
|
| 49 |
+
max_rate: float = 32.0,
|
| 50 |
+
iterations: int = 8,
|
| 51 |
+
repetitions: int = 2,
|
| 52 |
+
headroom: float = 0.20,
|
| 53 |
+
) -> dict:
|
| 54 |
base = SimulationConfig.from_dict(config)
|
| 55 |
low = max(0.01, min_rate)
|
| 56 |
high = max(low * 1.01, max_rate)
|
|
|
|
| 63 |
"status": "no_feasible_rate",
|
| 64 |
"capacity_rps": 0.0,
|
| 65 |
"recommended_rps": 0.0,
|
| 66 |
+
"headroom": headroom,
|
| 67 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 68 |
"trace": trace,
|
| 69 |
}
|
| 70 |
|
| 71 |
high_eval = evaluate_rate(base, high, repetitions)
|
| 72 |
trace.append(high_eval)
|
| 73 |
if high_eval["passed"]:
|
|
|
|
| 74 |
return {
|
| 75 |
"status": "upper_bound_still_feasible",
|
| 76 |
+
"capacity_rps": high,
|
| 77 |
+
"recommended_rps": high * (1.0 - headroom),
|
| 78 |
+
"headroom": headroom,
|
| 79 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 80 |
+
"trace": sorted(trace, key=lambda item: item["rate_rps"]),
|
| 81 |
}
|
| 82 |
|
| 83 |
best = low
|
|
|
|
| 96 |
"capacity_rps": best,
|
| 97 |
"recommended_rps": best * (1.0 - headroom),
|
| 98 |
"headroom": headroom,
|
| 99 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 100 |
+
"trace": sorted(trace, key=lambda item: item["rate_rps"]),
|
| 101 |
}
|
| 102 |
|
| 103 |
|
|
|
|
| 115 |
cfg = deepcopy(base)
|
| 116 |
cfg.scheduler = scheduler
|
| 117 |
result = run_simulation(cfg.to_dict())
|
| 118 |
+
rows.append(
|
| 119 |
+
{
|
| 120 |
+
"scheduler": scheduler,
|
| 121 |
+
"request_throughput_rps": result["summary"]["request_throughput_rps"],
|
| 122 |
+
"goodput_rps": result["summary"]["goodput_rps"],
|
| 123 |
+
"slo_attainment": result["summary"]["slo_attainment"],
|
| 124 |
+
"p95_ttft_ms": result["latency"]["ttft_ms"]["p95"],
|
| 125 |
+
"p95_e2e_ms": result["latency"]["e2e_ms"]["p95"],
|
| 126 |
+
"peak_kv_utilization": result["resource"]["peak_kv_utilization"],
|
| 127 |
+
"unfinished": result["summary"]["requests_unfinished"],
|
| 128 |
+
"bottleneck": result["diagnostics"]["label"],
|
| 129 |
+
}
|
| 130 |
+
)
|
| 131 |
+
rows.sort(key=lambda row: (row["slo_attainment"], row["goodput_rps"]), reverse=True)
|
| 132 |
return {"rows": rows}
|
py/inferscale/simulator.py
CHANGED
|
@@ -2,6 +2,7 @@ from __future__ import annotations
|
|
| 2 |
|
| 3 |
from dataclasses import asdict
|
| 4 |
|
|
|
|
| 5 |
from .kv_cache import KVCacheModel
|
| 6 |
from .latency import AnalyticalLatencyModel
|
| 7 |
from .metrics import summarize
|
|
@@ -296,9 +297,11 @@ class Simulator:
|
|
| 296 |
"e2e_ms": (req.completion_time - req.arrival_time) * 1000.0 if req.completion_time is not None else None,
|
| 297 |
})
|
| 298 |
|
|
|
|
|
|
|
| 299 |
provenance = {
|
| 300 |
"simulator": "InferScale-Sim",
|
| 301 |
-
"version": "0.
|
| 302 |
"latency_profile_type": "analytical-reference",
|
| 303 |
"profile_warning": "Reference profiles are analytical proxies, not measured hardware benchmarks.",
|
| 304 |
"model_profile_source": self.model.source,
|
|
@@ -311,6 +314,7 @@ class Simulator:
|
|
| 311 |
summary=summary,
|
| 312 |
latency=latency,
|
| 313 |
resource=resource,
|
|
|
|
| 314 |
requests=request_rows,
|
| 315 |
timeline=[asdict(p) for p in self.timeline],
|
| 316 |
warnings=self.warnings,
|
|
|
|
| 2 |
|
| 3 |
from dataclasses import asdict
|
| 4 |
|
| 5 |
+
from .diagnostics import diagnose_run
|
| 6 |
from .kv_cache import KVCacheModel
|
| 7 |
from .latency import AnalyticalLatencyModel
|
| 8 |
from .metrics import summarize
|
|
|
|
| 297 |
"e2e_ms": (req.completion_time - req.arrival_time) * 1000.0 if req.completion_time is not None else None,
|
| 298 |
})
|
| 299 |
|
| 300 |
+
diagnostics = diagnose_run(summary, latency, resource, self.cfg)
|
| 301 |
+
|
| 302 |
provenance = {
|
| 303 |
"simulator": "InferScale-Sim",
|
| 304 |
+
"version": "0.2.0",
|
| 305 |
"latency_profile_type": "analytical-reference",
|
| 306 |
"profile_warning": "Reference profiles are analytical proxies, not measured hardware benchmarks.",
|
| 307 |
"model_profile_source": self.model.source,
|
|
|
|
| 314 |
summary=summary,
|
| 315 |
latency=latency,
|
| 316 |
resource=resource,
|
| 317 |
+
diagnostics=diagnostics,
|
| 318 |
requests=request_rows,
|
| 319 |
timeline=[asdict(p) for p in self.timeline],
|
| 320 |
warnings=self.warnings,
|
pyproject.toml
CHANGED
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "inferscale-sim"
|
| 7 |
-
version = "0.
|
| 8 |
description = "Interactive LLM serving simulator and SLO capacity planner"
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
|
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "inferscale-sim"
|
| 7 |
+
version = "0.2.0"
|
| 8 |
description = "Interactive LLM serving simulator and SLO capacity planner"
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
scripts/release_check.py
CHANGED
|
@@ -9,7 +9,7 @@ README = (ROOT / "README.md").read_text()
|
|
| 9 |
SRC = ROOT / "src"
|
| 10 |
sys.path.insert(0, str(SRC))
|
| 11 |
|
| 12 |
-
from inferscale import run_simulation # noqa: E402
|
| 13 |
|
| 14 |
errors: list[str] = []
|
| 15 |
|
|
@@ -24,20 +24,29 @@ else:
|
|
| 24 |
|
| 25 |
if "sdk: static" not in README:
|
| 26 |
errors.append("README metadata must use sdk: static")
|
| 27 |
-
if
|
| 28 |
-
errors.append("
|
| 29 |
-
if not (ROOT / "worker.mjs").exists():
|
| 30 |
-
errors.append("worker.mjs missing")
|
| 31 |
|
| 32 |
src_files = sorted((ROOT / "src" / "inferscale").glob("*.py"))
|
| 33 |
web_files = sorted((ROOT / "py" / "inferscale").glob("*.py"))
|
| 34 |
-
if [
|
| 35 |
errors.append("browser Python mirror is stale; run python scripts/sync_web_python.py")
|
| 36 |
else:
|
| 37 |
-
for src, web in zip(src_files, web_files):
|
| 38 |
if src.read_bytes() != web.read_bytes():
|
| 39 |
errors.append(f"browser mirror differs for {src.name}; run sync_web_python.py")
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
smoke_cfg = {
|
| 42 |
"model": "Qwen2.5-3B",
|
| 43 |
"accelerator": "L4",
|
|
@@ -53,6 +62,10 @@ try:
|
|
| 53 |
errors.append("simulation smoke test completed zero requests")
|
| 54 |
if smoke["provenance"]["latency_profile_type"] != "analytical-reference":
|
| 55 |
errors.append("profile provenance guard is missing")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
except Exception as exc: # pragma: no cover - release diagnostic
|
| 57 |
errors.append(f"simulation smoke test raised: {exc}")
|
| 58 |
|
|
@@ -63,7 +76,9 @@ if errors:
|
|
| 63 |
raise SystemExit(1)
|
| 64 |
|
| 65 |
print("InferScale release check: PASS")
|
|
|
|
| 66 |
print(f"HF short_description: {len(short)}/60 characters")
|
| 67 |
print(f"Python modules mirrored: {len(src_files)}")
|
| 68 |
print(f"Smoke requests completed: {smoke['summary']['requests_completed']}")
|
| 69 |
print(f"Profile provenance: {smoke['provenance']['latency_profile_type']}")
|
|
|
|
|
|
| 9 |
SRC = ROOT / "src"
|
| 10 |
sys.path.insert(0, str(SRC))
|
| 11 |
|
| 12 |
+
from inferscale import __version__, run_simulation # noqa: E402
|
| 13 |
|
| 14 |
errors: list[str] = []
|
| 15 |
|
|
|
|
| 24 |
|
| 25 |
if "sdk: static" not in README:
|
| 26 |
errors.append("README metadata must use sdk: static")
|
| 27 |
+
if __version__ != "0.2.0":
|
| 28 |
+
errors.append(f"package version is {__version__}; expected 0.2.0")
|
|
|
|
|
|
|
| 29 |
|
| 30 |
src_files = sorted((ROOT / "src" / "inferscale").glob("*.py"))
|
| 31 |
web_files = sorted((ROOT / "py" / "inferscale").glob("*.py"))
|
| 32 |
+
if [path.name for path in src_files] != [path.name for path in web_files]:
|
| 33 |
errors.append("browser Python mirror is stale; run python scripts/sync_web_python.py")
|
| 34 |
else:
|
| 35 |
+
for src, web in zip(src_files, web_files, strict=True):
|
| 36 |
if src.read_bytes() != web.read_bytes():
|
| 37 |
errors.append(f"browser mirror differs for {src.name}; run sync_web_python.py")
|
| 38 |
|
| 39 |
+
worker_text = (ROOT / "worker.mjs").read_text()
|
| 40 |
+
for src in src_files:
|
| 41 |
+
if f'"{src.name}"' not in worker_text:
|
| 42 |
+
errors.append(f"worker module list is missing {src.name}")
|
| 43 |
+
|
| 44 |
+
for ui_file in (ROOT / "index.html", ROOT / "app.js"):
|
| 45 |
+
try:
|
| 46 |
+
ui_file.read_text().encode("ascii")
|
| 47 |
+
except UnicodeEncodeError:
|
| 48 |
+
errors.append(f"{ui_file.name} contains non-ASCII UI glyphs; use text labels for reliable rendering")
|
| 49 |
+
|
| 50 |
smoke_cfg = {
|
| 51 |
"model": "Qwen2.5-3B",
|
| 52 |
"accelerator": "L4",
|
|
|
|
| 62 |
errors.append("simulation smoke test completed zero requests")
|
| 63 |
if smoke["provenance"]["latency_profile_type"] != "analytical-reference":
|
| 64 |
errors.append("profile provenance guard is missing")
|
| 65 |
+
if smoke["diagnostics"].get("provenance") != "heuristic-simulator-diagnosis":
|
| 66 |
+
errors.append("v0.2 diagnosis provenance missing")
|
| 67 |
+
if "ttft_slo_attainment" not in smoke["summary"] or "e2e_slo_attainment" not in smoke["summary"]:
|
| 68 |
+
errors.append("component SLO metrics missing")
|
| 69 |
except Exception as exc: # pragma: no cover - release diagnostic
|
| 70 |
errors.append(f"simulation smoke test raised: {exc}")
|
| 71 |
|
|
|
|
| 76 |
raise SystemExit(1)
|
| 77 |
|
| 78 |
print("InferScale release check: PASS")
|
| 79 |
+
print(f"Version: {__version__}")
|
| 80 |
print(f"HF short_description: {len(short)}/60 characters")
|
| 81 |
print(f"Python modules mirrored: {len(src_files)}")
|
| 82 |
print(f"Smoke requests completed: {smoke['summary']['requests_completed']}")
|
| 83 |
print(f"Profile provenance: {smoke['provenance']['latency_profile_type']}")
|
| 84 |
+
print(f"Diagnosis: {smoke['diagnostics']['label']}")
|
src/inferscale/__init__.py
CHANGED
|
@@ -12,4 +12,4 @@ __all__ = [
|
|
| 12 |
"run_simulation",
|
| 13 |
]
|
| 14 |
|
| 15 |
-
__version__ = "0.
|
|
|
|
| 12 |
"run_simulation",
|
| 13 |
]
|
| 14 |
|
| 15 |
+
__version__ = "0.2.0"
|
src/inferscale/api.py
CHANGED
|
@@ -7,7 +7,7 @@ from .simulator import SCHEDULERS, run_simulation
|
|
| 7 |
|
| 8 |
def metadata() -> dict:
|
| 9 |
return {
|
| 10 |
-
"version": "0.
|
| 11 |
"models": list(MODELS.keys()),
|
| 12 |
"accelerators": list(ACCELERATORS.keys()),
|
| 13 |
"schedulers": sorted(SCHEDULERS),
|
|
|
|
| 7 |
|
| 8 |
def metadata() -> dict:
|
| 9 |
return {
|
| 10 |
+
"version": "0.2.0",
|
| 11 |
"models": list(MODELS.keys()),
|
| 12 |
"accelerators": list(ACCELERATORS.keys()),
|
| 13 |
"schedulers": sorted(SCHEDULERS),
|
src/inferscale/diagnostics.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
from typing import Any
|
| 4 |
+
|
| 5 |
+
from .models import SimulationConfig
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
def diagnose_run(summary: dict[str, Any], latency: dict[str, Any], resource: dict[str, Any], cfg: SimulationConfig) -> dict[str, Any]:
|
| 9 |
+
"""Explain the simulator's limiting regime from its own state."""
|
| 10 |
+
target = cfg.slo_attainment_target
|
| 11 |
+
attainment = float(summary.get("slo_attainment", 0.0))
|
| 12 |
+
ttft_attainment = float(summary.get("ttft_slo_attainment", 0.0))
|
| 13 |
+
e2e_attainment = float(summary.get("e2e_slo_attainment", 0.0))
|
| 14 |
+
unfinished = int(summary.get("requests_unfinished", 0))
|
| 15 |
+
busy = float(summary.get("busy_fraction", 0.0))
|
| 16 |
+
kv_util = float(resource.get("peak_kv_utilization", 0.0))
|
| 17 |
+
queue_p95 = float(latency.get("queue_ms", {}).get("p95", 0.0))
|
| 18 |
+
|
| 19 |
+
code = "mixed_pressure"
|
| 20 |
+
label = "Mixed SLO pressure"
|
| 21 |
+
explanation = "Several constraints are active; inspect queueing, KV pressure, and tail latency together."
|
| 22 |
+
recommendation = "Compare schedulers and run the capacity planner to isolate the limiting regime."
|
| 23 |
+
|
| 24 |
+
if unfinished > 0 and kv_util >= 0.85:
|
| 25 |
+
code, label = "kv_pressure", "KV memory pressure"
|
| 26 |
+
explanation = "Requests remain unfinished while the simulated KV cache approaches its configured capacity."
|
| 27 |
+
recommendation = "Reduce sequence pressure, lower offered load, use a smaller KV block, or choose a larger-memory profile."
|
| 28 |
+
elif unfinished > 0 and busy >= 0.92:
|
| 29 |
+
code, label = "over_capacity", "Offered load above capacity"
|
| 30 |
+
explanation = "The virtual device remains busy while the workload fails to drain."
|
| 31 |
+
recommendation = "Reduce offered load or use the capacity planner to find a sustainable operating point."
|
| 32 |
+
elif unfinished > 0:
|
| 33 |
+
code, label = "admission_stall", "Admission stall"
|
| 34 |
+
explanation = "Some requests cannot be admitted or completed under the current serving constraints."
|
| 35 |
+
recommendation = "Inspect batch-token and KV-cache limits, then compare a less restrictive scheduler configuration."
|
| 36 |
+
elif attainment >= target:
|
| 37 |
+
if busy >= 0.90:
|
| 38 |
+
code, label = "healthy_near_saturation", "Healthy, little headroom"
|
| 39 |
+
explanation = "The workload meets the configured SLO target, but simulated device busy time is already high."
|
| 40 |
+
recommendation = "Run the capacity planner before increasing traffic; the current point is close to saturation."
|
| 41 |
+
else:
|
| 42 |
+
code, label = "healthy", "Healthy"
|
| 43 |
+
explanation = "The workload drains and meets the configured SLO-attainment target with visible headroom."
|
| 44 |
+
recommendation = "Use the capacity planner to quantify how much additional offered load the profile can absorb."
|
| 45 |
+
elif kv_util >= 0.90:
|
| 46 |
+
code, label = "kv_pressure", "KV memory pressure"
|
| 47 |
+
explanation = "Peak simulated KV utilization is high enough to constrain admission and batching flexibility."
|
| 48 |
+
recommendation = "Reduce sequence pressure, use smaller KV blocks, or select a larger-memory accelerator profile."
|
| 49 |
+
elif ttft_attainment < target and queue_p95 >= max(25.0, cfg.slo_ttft_ms * 0.20):
|
| 50 |
+
code, label = "queue_prefill_pressure", "Queue / prefill pressure"
|
| 51 |
+
explanation = "TTFT misses coincide with substantial queueing before prefill begins."
|
| 52 |
+
recommendation = "Lower offered load, increase safe batching capacity, or try chunked prefill / SLO-aware scheduling."
|
| 53 |
+
elif ttft_attainment < target:
|
| 54 |
+
code, label = "ttft_pressure", "TTFT pressure"
|
| 55 |
+
explanation = "First-token latency is the dominant SLO failure even though queueing is not the only contributor."
|
| 56 |
+
recommendation = "Compare prefill-oriented schedulers and reduce prompt or offered-load pressure."
|
| 57 |
+
elif e2e_attainment < target:
|
| 58 |
+
code, label = "decode_pressure", "Decode / E2E pressure"
|
| 59 |
+
explanation = "Most first tokens arrive within target, but end-to-end latency still misses the configured SLO."
|
| 60 |
+
recommendation = "Reduce output-length pressure or offered load; larger decode batches may help when memory allows."
|
| 61 |
+
elif busy >= 0.95:
|
| 62 |
+
code, label = "compute_saturation", "Compute saturation"
|
| 63 |
+
explanation = "The simulated device is effectively saturated even though no single SLO component dominates."
|
| 64 |
+
recommendation = "Reduce load or switch to a higher-throughput accelerator profile."
|
| 65 |
+
|
| 66 |
+
return {
|
| 67 |
+
"code": code,
|
| 68 |
+
"label": label,
|
| 69 |
+
"explanation": explanation,
|
| 70 |
+
"recommendation": recommendation,
|
| 71 |
+
"evidence": {
|
| 72 |
+
"slo_attainment": attainment,
|
| 73 |
+
"ttft_slo_attainment": ttft_attainment,
|
| 74 |
+
"e2e_slo_attainment": e2e_attainment,
|
| 75 |
+
"busy_fraction": busy,
|
| 76 |
+
"peak_kv_utilization": kv_util,
|
| 77 |
+
"queue_p95_ms": queue_p95,
|
| 78 |
+
"unfinished_requests": unfinished,
|
| 79 |
+
},
|
| 80 |
+
"provenance": "heuristic-simulator-diagnosis",
|
| 81 |
+
}
|
src/inferscale/latency.py
CHANGED
|
@@ -29,7 +29,7 @@ class AnalyticalLatencyModel:
|
|
| 29 |
return self.model.params_b * self.weight_bytes_per_param
|
| 30 |
|
| 31 |
def kv_bytes_per_token(self) -> float:
|
| 32 |
-
# K + V, all layers, KV heads only. KV state is assumed fp16 in v0.
|
| 33 |
return (
|
| 34 |
2
|
| 35 |
* self.model.layers
|
|
|
|
| 29 |
return self.model.params_b * self.weight_bytes_per_param
|
| 30 |
|
| 31 |
def kv_bytes_per_token(self) -> float:
|
| 32 |
+
# K + V, all layers, KV heads only. KV state is assumed fp16 in v0.2.
|
| 33 |
return (
|
| 34 |
2
|
| 35 |
* self.model.layers
|
src/inferscale/metrics.py
CHANGED
|
@@ -57,14 +57,23 @@ def summarize(completed: list[Request], cfg: SimulationConfig, makespan_s: float
|
|
| 57 |
queue = [m.queue_ms for m in metrics]
|
| 58 |
total_output = sum(r.output_tokens for r in completed)
|
| 59 |
met = sum(m.met_all_slos for m in metrics)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
duration = max(makespan_s, 1e-9)
|
|
|
|
| 61 |
|
| 62 |
summary = {
|
| 63 |
"requests_completed": len(completed),
|
| 64 |
"request_throughput_rps": len(completed) / duration,
|
| 65 |
"output_throughput_tps": total_output / duration,
|
| 66 |
"goodput_rps": met / duration,
|
| 67 |
-
"slo_attainment": met /
|
|
|
|
|
|
|
|
|
|
| 68 |
"simulated_makespan_s": makespan_s,
|
| 69 |
"busy_fraction": min(1.0, busy_time_s / duration),
|
| 70 |
"mean_prompt_tokens": mean([r.prompt_tokens for r in completed]) if completed else 0.0,
|
|
|
|
| 57 |
queue = [m.queue_ms for m in metrics]
|
| 58 |
total_output = sum(r.output_tokens for r in completed)
|
| 59 |
met = sum(m.met_all_slos for m in metrics)
|
| 60 |
+
met_ttft = sum(m.met_ttft_slo for m in metrics)
|
| 61 |
+
met_e2e = sum(m.met_e2e_slo for m in metrics)
|
| 62 |
+
ttft_only_fail = sum((not m.met_ttft_slo) and m.met_e2e_slo for m in metrics)
|
| 63 |
+
e2e_only_fail = sum(m.met_ttft_slo and (not m.met_e2e_slo) for m in metrics)
|
| 64 |
+
both_fail = sum((not m.met_ttft_slo) and (not m.met_e2e_slo) for m in metrics)
|
| 65 |
duration = max(makespan_s, 1e-9)
|
| 66 |
+
count = len(metrics)
|
| 67 |
|
| 68 |
summary = {
|
| 69 |
"requests_completed": len(completed),
|
| 70 |
"request_throughput_rps": len(completed) / duration,
|
| 71 |
"output_throughput_tps": total_output / duration,
|
| 72 |
"goodput_rps": met / duration,
|
| 73 |
+
"slo_attainment": met / count if count else 0.0,
|
| 74 |
+
"ttft_slo_attainment": met_ttft / count if count else 0.0,
|
| 75 |
+
"e2e_slo_attainment": met_e2e / count if count else 0.0,
|
| 76 |
+
"slo_failure_breakdown": {"ttft_only": ttft_only_fail, "e2e_only": e2e_only_fail, "both": both_fail},
|
| 77 |
"simulated_makespan_s": makespan_s,
|
| 78 |
"busy_fraction": min(1.0, busy_time_s / duration),
|
| 79 |
"mean_prompt_tokens": mean([r.prompt_tokens for r in completed]) if completed else 0.0,
|
src/inferscale/models.py
CHANGED
|
@@ -122,6 +122,7 @@ class SimulationResult:
|
|
| 122 |
summary: dict[str, Any]
|
| 123 |
latency: dict[str, Any]
|
| 124 |
resource: dict[str, Any]
|
|
|
|
| 125 |
requests: list[dict[str, Any]] = field(default_factory=list)
|
| 126 |
timeline: list[dict[str, Any]] = field(default_factory=list)
|
| 127 |
warnings: list[str] = field(default_factory=list)
|
|
|
|
| 122 |
summary: dict[str, Any]
|
| 123 |
latency: dict[str, Any]
|
| 124 |
resource: dict[str, Any]
|
| 125 |
+
diagnostics: dict[str, Any] = field(default_factory=dict)
|
| 126 |
requests: list[dict[str, Any]] = field(default_factory=list)
|
| 127 |
timeline: list[dict[str, Any]] = field(default_factory=list)
|
| 128 |
warnings: list[str] = field(default_factory=list)
|
src/inferscale/optimizer.py
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
from __future__ import annotations
|
| 2 |
|
| 3 |
from copy import deepcopy
|
| 4 |
-
from statistics import mean
|
| 5 |
|
| 6 |
from .models import SimulationConfig
|
| 7 |
from .simulator import run_simulation
|
|
@@ -14,24 +14,43 @@ def evaluate_rate(base: SimulationConfig, rate: float, repetitions: int = 3) ->
|
|
| 14 |
cfg.request_rate_rps = rate
|
| 15 |
cfg.seed = base.seed + rep * 101
|
| 16 |
runs.append(run_simulation(cfg.to_dict()))
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
return {
|
| 24 |
"rate_rps": rate,
|
| 25 |
"passed": passed,
|
| 26 |
-
"slo_attainment":
|
| 27 |
-
"
|
| 28 |
-
"
|
| 29 |
-
"
|
| 30 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
}
|
| 32 |
|
| 33 |
|
| 34 |
-
def capacity_search(
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
base = SimulationConfig.from_dict(config)
|
| 36 |
low = max(0.01, min_rate)
|
| 37 |
high = max(low * 1.01, max_rate)
|
|
@@ -44,18 +63,21 @@ def capacity_search(config: dict, min_rate: float = 0.25, max_rate: float = 32.0
|
|
| 44 |
"status": "no_feasible_rate",
|
| 45 |
"capacity_rps": 0.0,
|
| 46 |
"recommended_rps": 0.0,
|
|
|
|
|
|
|
| 47 |
"trace": trace,
|
| 48 |
}
|
| 49 |
|
| 50 |
high_eval = evaluate_rate(base, high, repetitions)
|
| 51 |
trace.append(high_eval)
|
| 52 |
if high_eval["passed"]:
|
| 53 |
-
capacity = high
|
| 54 |
return {
|
| 55 |
"status": "upper_bound_still_feasible",
|
| 56 |
-
"capacity_rps":
|
| 57 |
-
"recommended_rps":
|
| 58 |
-
"
|
|
|
|
|
|
|
| 59 |
}
|
| 60 |
|
| 61 |
best = low
|
|
@@ -74,7 +96,8 @@ def capacity_search(config: dict, min_rate: float = 0.25, max_rate: float = 32.0
|
|
| 74 |
"capacity_rps": best,
|
| 75 |
"recommended_rps": best * (1.0 - headroom),
|
| 76 |
"headroom": headroom,
|
| 77 |
-
"
|
|
|
|
| 78 |
}
|
| 79 |
|
| 80 |
|
|
@@ -92,15 +115,18 @@ def compare_schedulers(config: dict, schedulers: list[str] | None = None) -> dic
|
|
| 92 |
cfg = deepcopy(base)
|
| 93 |
cfg.scheduler = scheduler
|
| 94 |
result = run_simulation(cfg.to_dict())
|
| 95 |
-
rows.append(
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
|
|
|
|
|
|
|
|
|
| 106 |
return {"rows": rows}
|
|
|
|
| 1 |
from __future__ import annotations
|
| 2 |
|
| 3 |
from copy import deepcopy
|
| 4 |
+
from statistics import mean, pstdev
|
| 5 |
|
| 6 |
from .models import SimulationConfig
|
| 7 |
from .simulator import run_simulation
|
|
|
|
| 14 |
cfg.request_rate_rps = rate
|
| 15 |
cfg.seed = base.seed + rep * 101
|
| 16 |
runs.append(run_simulation(cfg.to_dict()))
|
| 17 |
+
|
| 18 |
+
attainments = [run["summary"]["slo_attainment"] for run in runs]
|
| 19 |
+
goodputs = [run["summary"]["goodput_rps"] for run in runs]
|
| 20 |
+
ttfts = [run["latency"]["ttft_ms"]["p95"] for run in runs]
|
| 21 |
+
e2es = [run["latency"]["e2e_ms"]["p95"] for run in runs]
|
| 22 |
+
unfinished_values = [run["summary"]["requests_unfinished"] for run in runs]
|
| 23 |
+
|
| 24 |
+
passed = all(
|
| 25 |
+
attainment >= base.slo_attainment_target and unfinished == 0
|
| 26 |
+
for attainment, unfinished in zip(attainments, unfinished_values, strict=True)
|
| 27 |
+
)
|
| 28 |
return {
|
| 29 |
"rate_rps": rate,
|
| 30 |
"passed": passed,
|
| 31 |
+
"slo_attainment": mean(attainments),
|
| 32 |
+
"slo_attainment_min": min(attainments),
|
| 33 |
+
"slo_attainment_max": max(attainments),
|
| 34 |
+
"slo_attainment_std": pstdev(attainments) if len(attainments) > 1 else 0.0,
|
| 35 |
+
"goodput_rps": mean(goodputs),
|
| 36 |
+
"goodput_std": pstdev(goodputs) if len(goodputs) > 1 else 0.0,
|
| 37 |
+
"p95_ttft_ms": mean(ttfts),
|
| 38 |
+
"p95_e2e_ms": mean(e2es),
|
| 39 |
+
"mean_unfinished": mean(unfinished_values),
|
| 40 |
+
"max_unfinished": max(unfinished_values),
|
| 41 |
+
"repetitions": repetitions,
|
| 42 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 43 |
}
|
| 44 |
|
| 45 |
|
| 46 |
+
def capacity_search(
|
| 47 |
+
config: dict,
|
| 48 |
+
min_rate: float = 0.25,
|
| 49 |
+
max_rate: float = 32.0,
|
| 50 |
+
iterations: int = 8,
|
| 51 |
+
repetitions: int = 2,
|
| 52 |
+
headroom: float = 0.20,
|
| 53 |
+
) -> dict:
|
| 54 |
base = SimulationConfig.from_dict(config)
|
| 55 |
low = max(0.01, min_rate)
|
| 56 |
high = max(low * 1.01, max_rate)
|
|
|
|
| 63 |
"status": "no_feasible_rate",
|
| 64 |
"capacity_rps": 0.0,
|
| 65 |
"recommended_rps": 0.0,
|
| 66 |
+
"headroom": headroom,
|
| 67 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 68 |
"trace": trace,
|
| 69 |
}
|
| 70 |
|
| 71 |
high_eval = evaluate_rate(base, high, repetitions)
|
| 72 |
trace.append(high_eval)
|
| 73 |
if high_eval["passed"]:
|
|
|
|
| 74 |
return {
|
| 75 |
"status": "upper_bound_still_feasible",
|
| 76 |
+
"capacity_rps": high,
|
| 77 |
+
"recommended_rps": high * (1.0 - headroom),
|
| 78 |
+
"headroom": headroom,
|
| 79 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 80 |
+
"trace": sorted(trace, key=lambda item: item["rate_rps"]),
|
| 81 |
}
|
| 82 |
|
| 83 |
best = low
|
|
|
|
| 96 |
"capacity_rps": best,
|
| 97 |
"recommended_rps": best * (1.0 - headroom),
|
| 98 |
"headroom": headroom,
|
| 99 |
+
"criterion": "all-repetitions-meet-target-and-drain",
|
| 100 |
+
"trace": sorted(trace, key=lambda item: item["rate_rps"]),
|
| 101 |
}
|
| 102 |
|
| 103 |
|
|
|
|
| 115 |
cfg = deepcopy(base)
|
| 116 |
cfg.scheduler = scheduler
|
| 117 |
result = run_simulation(cfg.to_dict())
|
| 118 |
+
rows.append(
|
| 119 |
+
{
|
| 120 |
+
"scheduler": scheduler,
|
| 121 |
+
"request_throughput_rps": result["summary"]["request_throughput_rps"],
|
| 122 |
+
"goodput_rps": result["summary"]["goodput_rps"],
|
| 123 |
+
"slo_attainment": result["summary"]["slo_attainment"],
|
| 124 |
+
"p95_ttft_ms": result["latency"]["ttft_ms"]["p95"],
|
| 125 |
+
"p95_e2e_ms": result["latency"]["e2e_ms"]["p95"],
|
| 126 |
+
"peak_kv_utilization": result["resource"]["peak_kv_utilization"],
|
| 127 |
+
"unfinished": result["summary"]["requests_unfinished"],
|
| 128 |
+
"bottleneck": result["diagnostics"]["label"],
|
| 129 |
+
}
|
| 130 |
+
)
|
| 131 |
+
rows.sort(key=lambda row: (row["slo_attainment"], row["goodput_rps"]), reverse=True)
|
| 132 |
return {"rows": rows}
|
src/inferscale/simulator.py
CHANGED
|
@@ -2,6 +2,7 @@ from __future__ import annotations
|
|
| 2 |
|
| 3 |
from dataclasses import asdict
|
| 4 |
|
|
|
|
| 5 |
from .kv_cache import KVCacheModel
|
| 6 |
from .latency import AnalyticalLatencyModel
|
| 7 |
from .metrics import summarize
|
|
@@ -296,9 +297,11 @@ class Simulator:
|
|
| 296 |
"e2e_ms": (req.completion_time - req.arrival_time) * 1000.0 if req.completion_time is not None else None,
|
| 297 |
})
|
| 298 |
|
|
|
|
|
|
|
| 299 |
provenance = {
|
| 300 |
"simulator": "InferScale-Sim",
|
| 301 |
-
"version": "0.
|
| 302 |
"latency_profile_type": "analytical-reference",
|
| 303 |
"profile_warning": "Reference profiles are analytical proxies, not measured hardware benchmarks.",
|
| 304 |
"model_profile_source": self.model.source,
|
|
@@ -311,6 +314,7 @@ class Simulator:
|
|
| 311 |
summary=summary,
|
| 312 |
latency=latency,
|
| 313 |
resource=resource,
|
|
|
|
| 314 |
requests=request_rows,
|
| 315 |
timeline=[asdict(p) for p in self.timeline],
|
| 316 |
warnings=self.warnings,
|
|
|
|
| 2 |
|
| 3 |
from dataclasses import asdict
|
| 4 |
|
| 5 |
+
from .diagnostics import diagnose_run
|
| 6 |
from .kv_cache import KVCacheModel
|
| 7 |
from .latency import AnalyticalLatencyModel
|
| 8 |
from .metrics import summarize
|
|
|
|
| 297 |
"e2e_ms": (req.completion_time - req.arrival_time) * 1000.0 if req.completion_time is not None else None,
|
| 298 |
})
|
| 299 |
|
| 300 |
+
diagnostics = diagnose_run(summary, latency, resource, self.cfg)
|
| 301 |
+
|
| 302 |
provenance = {
|
| 303 |
"simulator": "InferScale-Sim",
|
| 304 |
+
"version": "0.2.0",
|
| 305 |
"latency_profile_type": "analytical-reference",
|
| 306 |
"profile_warning": "Reference profiles are analytical proxies, not measured hardware benchmarks.",
|
| 307 |
"model_profile_source": self.model.source,
|
|
|
|
| 314 |
summary=summary,
|
| 315 |
latency=latency,
|
| 316 |
resource=resource,
|
| 317 |
+
diagnostics=diagnostics,
|
| 318 |
requests=request_rows,
|
| 319 |
timeline=[asdict(p) for p in self.timeline],
|
| 320 |
warnings=self.warnings,
|
styles.css
CHANGED
|
@@ -1,79 +1,12 @@
|
|
| 1 |
-
:root
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
--shadow: 0 22px 70px rgba(0,0,0,.28);
|
| 14 |
-
}
|
| 15 |
-
* { box-sizing: border-box; }
|
| 16 |
-
html { background: var(--bg); color-scheme: dark; }
|
| 17 |
-
body { margin: 0; font-family: Inter, ui-sans-serif, system-ui, -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; color: var(--text); background: radial-gradient(circle at 70% -10%, rgba(82, 80, 210, .17), transparent 30rem), var(--bg); }
|
| 18 |
-
button, input, select { font: inherit; }
|
| 19 |
-
.topbar { height: 76px; padding: 0 34px; border-bottom: 1px solid rgba(255,255,255,.06); display:flex; align-items:center; justify-content:space-between; position:sticky; top:0; z-index:20; background:rgba(8,11,18,.88); backdrop-filter: blur(16px); }
|
| 20 |
-
.brand-wrap { display:flex; align-items:center; gap:13px; }
|
| 21 |
-
.logo { width:38px; height:38px; display:grid; place-items:center; border-radius:10px; background:linear-gradient(135deg,var(--accent),var(--accent2)); color:white; font-weight:800; letter-spacing:-.04em; }
|
| 22 |
-
.brand { font-weight:750; letter-spacing:-.02em; }
|
| 23 |
-
.subtitle { color:var(--muted); font-size:12px; margin-top:2px; }
|
| 24 |
-
.runtime-pill { display:flex; gap:8px; align-items:center; padding:8px 12px; border:1px solid var(--line); border-radius:99px; color:var(--muted); font-size:12px; background:rgba(255,255,255,.025); }
|
| 25 |
-
.dot { width:7px; height:7px; border-radius:50%; background:var(--warn); box-shadow:0 0 14px currentColor; }
|
| 26 |
-
.runtime-pill.ready .dot { background:var(--good); }
|
| 27 |
-
.runtime-pill.error .dot { background:var(--danger); }
|
| 28 |
-
.shell { width:min(1480px, calc(100% - 44px)); margin:0 auto; padding:42px 0 60px; }
|
| 29 |
-
.hero { display:grid; grid-template-columns:minmax(0,1.45fr) minmax(400px,.75fr); gap:44px; align-items:end; padding:26px 4px 34px; }
|
| 30 |
-
.eyebrow,.section-kicker { font-size:11px; letter-spacing:.12em; text-transform:uppercase; color:#9f96ff; font-weight:750; }
|
| 31 |
-
h1 { font-size:clamp(34px,4vw,60px); line-height:1.03; letter-spacing:-.045em; max-width:940px; margin:10px 0 18px; }
|
| 32 |
-
.hero p { max-width:830px; color:var(--muted); line-height:1.65; font-size:16px; margin:0; }
|
| 33 |
-
.hero-stat-grid { display:grid; grid-template-columns:1fr 1fr; border:1px solid var(--line); border-radius:16px; overflow:hidden; background:rgba(255,255,255,.025); }
|
| 34 |
-
.hero-stat { padding:17px 18px; min-height:78px; border-right:1px solid var(--line); border-bottom:1px solid var(--line); }
|
| 35 |
-
.hero-stat:nth-child(2n){border-right:0}.hero-stat:nth-child(n+3){border-bottom:0}
|
| 36 |
-
.hero-stat span { display:block; color:var(--muted); font-size:11px; margin-bottom:8px; }
|
| 37 |
-
.hero-stat strong { font-size:15px; }
|
| 38 |
-
.notice { border:1px solid rgba(245,199,110,.27); background:rgba(245,199,110,.07); color:#d9cfb8; border-radius:12px; padding:13px 16px; font-size:13px; line-height:1.5; }
|
| 39 |
-
.tabs { display:flex; gap:8px; padding:26px 0 16px; overflow:auto; }
|
| 40 |
-
.tab { border:0; color:var(--muted); background:transparent; padding:10px 14px; cursor:pointer; border-radius:8px; font-weight:650; }
|
| 41 |
-
.tab:hover { color:var(--text); background:rgba(255,255,255,.035); }
|
| 42 |
-
.tab.active { color:var(--text); background:rgba(139,124,255,.12); }
|
| 43 |
-
.tab-panel { display:none; }.tab-panel.active { display:block; }
|
| 44 |
-
.workspace { display:grid; grid-template-columns:400px minmax(0,1fr); gap:16px; align-items:start; }
|
| 45 |
-
.panel { background:linear-gradient(180deg,rgba(19,25,37,.96),rgba(14,19,29,.96)); border:1px solid var(--line); border-radius:16px; box-shadow:var(--shadow); }
|
| 46 |
-
.controls-panel { padding:20px; position:sticky; top:94px; }
|
| 47 |
-
.result-panel,.wide-panel { padding:22px; }
|
| 48 |
-
.panel-title-row { display:flex; align-items:flex-start; justify-content:space-between; gap:14px; margin-bottom:18px; }
|
| 49 |
-
.panel-title-row h2 { margin:0; font-size:18px; letter-spacing:-.02em; }
|
| 50 |
-
.panel-title-row p { margin:6px 0 0; }
|
| 51 |
-
.tag { display:inline-flex; align-items:center; border:1px solid var(--line); background:rgba(255,255,255,.03); color:var(--muted); border-radius:99px; padding:5px 8px; font-size:10px; white-space:nowrap; }
|
| 52 |
-
.tag.good { color:var(--good); border-color:rgba(99,217,165,.3); background:rgba(99,217,165,.07); }.tag.bad { color:var(--danger); border-color:rgba(255,123,138,.28); }.tag.neutral{color:var(--muted)}
|
| 53 |
-
label { display:block; color:#bdc5d2; font-size:11px; font-weight:650; position:relative; }
|
| 54 |
-
select,input { width:100%; margin-top:7px; height:38px; border:1px solid var(--line); background:#0b1018; color:var(--text); border-radius:8px; padding:0 10px; outline:none; }
|
| 55 |
-
select:focus,input:focus { border-color:#625bd0; box-shadow:0 0 0 3px rgba(98,91,208,.12); }
|
| 56 |
-
.unit { position:absolute; right:9px; bottom:11px; color:#697488; font-size:9px; pointer-events:none; }
|
| 57 |
-
.field-grid { display:grid; gap:11px; margin-bottom:11px; }.field-grid.two { grid-template-columns:1fr 1fr; }
|
| 58 |
-
hr { border:0; border-top:1px solid var(--line); margin:18px 0; }
|
| 59 |
-
.section-kicker { margin-bottom:11px; }
|
| 60 |
-
button.primary,button.secondary { width:100%; border-radius:9px; height:42px; border:1px solid transparent; margin-top:12px; font-weight:750; cursor:pointer; transition:.16s ease; }
|
| 61 |
-
button.primary { background:linear-gradient(135deg,#7568f5,#4775f2); color:white; box-shadow:0 10px 26px rgba(94,85,222,.20); }
|
| 62 |
-
button.primary:hover:not(:disabled){transform:translateY(-1px);filter:brightness(1.06)}
|
| 63 |
-
button.secondary { background:transparent; color:#c7cfdb; border-color:var(--line); }
|
| 64 |
-
button:disabled { opacity:.45; cursor:not-allowed; }.compact { width:auto !important; min-width:180px; padding:0 18px; margin-top:0 !important; }
|
| 65 |
-
.empty-state { min-height:420px; display:grid; place-content:center; text-align:center; color:var(--muted); padding:30px; }.empty-state.small { min-height:260px; }.empty-state h3{color:#cfd6e2;margin:10px 0 6px;font-size:16px}.empty-state p{max-width:480px;margin:0;line-height:1.55;font-size:13px}.empty-icon{width:44px;height:44px;border:1px solid var(--line);border-radius:12px;display:grid;place-items:center;margin:0 auto;font-size:20px;color:var(--accent2)}
|
| 66 |
-
.hidden { display:none !important; }
|
| 67 |
-
.metric-grid { display:grid; grid-template-columns:repeat(6,1fr); gap:10px; margin-bottom:16px; }.metric-grid.four{grid-template-columns:repeat(4,1fr)}
|
| 68 |
-
.metric { min-height:88px; padding:14px; border:1px solid var(--line); border-radius:11px; background:#0c111a; }.metric span{display:block;color:var(--muted);font-size:10px;margin-bottom:10px}.metric strong{font-size:19px;letter-spacing:-.03em}.metric.emphasis{border-color:rgba(139,124,255,.38);background:rgba(139,124,255,.06)}
|
| 69 |
-
.chart-grid { display:grid; grid-template-columns:1fr 1fr; gap:12px; }.chart-card { border:1px solid var(--line); border-radius:12px; background:#0c111a; padding:14px; min-height:310px; }.chart-card.full{margin-top:12px}.chart-title{color:#cfd6e2;font-size:11px;font-weight:700;margin-bottom:12px}
|
| 70 |
-
canvas { max-height:280px; }
|
| 71 |
-
.warnings { margin-top:12px; border:1px solid rgba(245,199,110,.24); padding:11px 13px; background:rgba(245,199,110,.06); border-radius:9px; color:#d8cdaF; font-size:12px; }
|
| 72 |
-
.muted { color:var(--muted); font-size:13px; line-height:1.55; }
|
| 73 |
-
.table-wrap { overflow:auto; margin-top:14px; border:1px solid var(--line); border-radius:10px; }
|
| 74 |
-
table { width:100%; border-collapse:collapse; font-size:12px; min-width:760px; } th,td{padding:11px 12px;text-align:right;border-bottom:1px solid var(--line)}th:first-child,td:first-child{text-align:left}th{color:#7f8ba0;font-size:10px;text-transform:uppercase;letter-spacing:.06em;background:#0a0f17}td{color:#cbd3df}tbody tr:last-child td{border-bottom:0}.pass{color:var(--good)}.fail{color:var(--danger)}
|
| 75 |
-
.planner-grid { grid-template-columns:370px minmax(0,1fr); }
|
| 76 |
-
.method-grid { display:grid; grid-template-columns:1fr 1fr; gap:16px; }.prose{padding:26px}.prose h2{font-size:22px;margin:8px 0 12px}.prose p{color:var(--muted);line-height:1.7;font-size:14px}.prose code{color:#b7b0ff}.formula{font-family:ui-monospace,SFMono-Regular,Menlo,monospace;color:#b9c6dc;padding:12px;border:1px solid var(--line);border-radius:8px;background:#0b1018;font-size:12px}.wide-method{grid-column:1/-1}.paper-grid{display:grid;grid-template-columns:repeat(5,1fr);gap:10px;margin-top:20px}.paper-grid>div{padding:14px;border:1px solid var(--line);border-radius:10px;background:#0c111a}.paper-grid strong{display:block;font-size:12px;margin-bottom:8px}.paper-grid span{color:var(--muted);font-size:11px;line-height:1.55;display:block}
|
| 77 |
-
footer { width:min(1480px,calc(100% - 44px)); margin:0 auto; padding:22px 0 36px; border-top:1px solid rgba(255,255,255,.055); display:flex; justify-content:space-between; gap:20px; color:#657085; font-size:11px; }
|
| 78 |
-
@media (max-width:1100px){.hero{grid-template-columns:1fr}.workspace,.planner-grid{grid-template-columns:1fr}.controls-panel{position:static}.metric-grid{grid-template-columns:repeat(3,1fr)}.paper-grid{grid-template-columns:1fr 1fr}}
|
| 79 |
-
@media (max-width:720px){.topbar{padding:0 18px}.subtitle{display:none}.runtime-pill{max-width:170px}.shell{width:min(100% - 24px,1480px);padding-top:20px}.hero{padding-top:8px;gap:20px}.hero-stat-grid{grid-template-columns:1fr 1fr}.field-grid.two,.chart-grid,.method-grid{grid-template-columns:1fr}.metric-grid,.metric-grid.four{grid-template-columns:1fr 1fr}.paper-grid{grid-template-columns:1fr}.panel-title-row{flex-direction:column}.compact{width:100%!important}.wide-method{grid-column:auto}footer{width:calc(100% - 24px);flex-direction:column}}
|
|
|
|
| 1 |
+
:root{--bg:#0b0e13;--surface:#11161e;--surface2:#0d1219;--line:#29313d;--line2:#202732;--text:#e8edf3;--muted:#929dab;--faint:#66717f;--accent:#79a7ff;--good:#69c99a;--warn:#dfb966;--danger:#e47e87}
|
| 2 |
+
*{box-sizing:border-box}html{background:var(--bg);color-scheme:dark}body{margin:0;font-family:ui-sans-serif,system-ui,-apple-system,BlinkMacSystemFont,"Segoe UI",sans-serif;color:var(--text);background:var(--bg);font-size:14px}button,input,select{font:inherit}
|
| 3 |
+
.topbar{height:68px;padding:0 28px;border-bottom:1px solid var(--line2);display:flex;align-items:center;justify-content:space-between;position:fixed;top:0;left:0;right:0;z-index:50;background:#0c1016;transition:transform .22s ease}body.runtime-ready .topbar{transform:translateY(calc(-100% + 6px))}body.runtime-ready .topbar:hover,body.runtime-ready .topbar:focus-within{transform:translateY(0)}.brand-wrap{display:flex;align-items:center;gap:11px}.logo{width:34px;height:34px;display:grid;place-items:center;border:1px solid #455164;border-radius:6px;color:#d9e2ee;font:700 12px/1 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}.brand{font-weight:700;letter-spacing:-.015em}.subtitle{color:var(--muted);font-size:11px;margin-top:2px}.runtime-pill{display:flex;gap:8px;align-items:center;padding:7px 10px;border:1px solid var(--line);border-radius:6px;color:var(--muted);font:11px/1.2 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;background:var(--surface2)}.dot{width:7px;height:7px;border-radius:50%;background:var(--warn)}.runtime-pill.ready .dot{background:var(--good)}.runtime-pill.error .dot{background:var(--danger)}
|
| 4 |
+
.shell{width:min(1460px,calc(100% - 40px));margin:0 auto;padding:100px 0 56px;transition:padding-top .22s ease}body.runtime-ready .shell{padding-top:30px}.intro{display:grid;grid-template-columns:minmax(0,1.45fr) minmax(380px,.55fr);gap:48px;align-items:end;padding:20px 2px 26px}.eyebrow,.section-kicker{font:700 10px/1.2 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;letter-spacing:.08em;text-transform:uppercase;color:#8ea6c7}h1{font-size:clamp(32px,3.2vw,48px);line-height:1.06;letter-spacing:-.035em;max-width:900px;margin:10px 0 14px;font-weight:720}.intro p{max-width:840px;color:var(--muted);line-height:1.65;margin:0;font-size:15px}.project-facts{margin:0;border-top:1px solid var(--line);border-bottom:1px solid var(--line)}.project-facts div{display:grid;grid-template-columns:130px 1fr;gap:14px;padding:9px 0;border-bottom:1px solid var(--line2)}.project-facts div:last-child{border-bottom:0}.project-facts dt{color:var(--faint);font-size:11px}.project-facts dd{margin:0;color:#cbd4df;font-size:12px;font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}.reference-note{border-left:3px solid #8b7445;background:#12151a;color:#bcb5a5;padding:11px 13px;font-size:12px;line-height:1.5}.tabs{display:flex;gap:2px;padding:22px 0 12px;overflow:auto;border-bottom:1px solid var(--line2);margin-bottom:14px}.tab{border:0;border-bottom:2px solid transparent;color:var(--muted);background:transparent;padding:9px 12px;cursor:pointer;font-weight:650}.tab:hover{color:var(--text)}.tab.active{color:var(--text);border-bottom-color:var(--accent)}.tab-panel{display:none}.tab-panel.active{display:block}
|
| 5 |
+
.workspace{display:grid;grid-template-columns:390px minmax(0,1fr);gap:14px;align-items:start}.panel{background:var(--surface);border:1px solid var(--line);border-radius:8px}.controls-panel{padding:18px;position:sticky;top:18px}.result-panel,.wide-panel{padding:20px}.panel-title-row{display:flex;align-items:flex-start;justify-content:space-between;gap:14px;margin-bottom:16px}.panel-title-row h2{margin:0;font-size:17px;letter-spacing:-.02em}.panel-title-row p{margin:6px 0 0}.tag{display:inline-flex;align-items:center;border:1px solid var(--line);background:var(--surface2);color:var(--muted);border-radius:5px;padding:5px 7px;font-size:10px;white-space:nowrap}.tag.good{color:var(--good);border-color:#315e4a}.tag.bad{color:var(--danger);border-color:#643943}label{display:block;color:#b5bfcb;font-size:11px;font-weight:650;position:relative}select,input{width:100%;margin-top:7px;height:37px;border:1px solid var(--line);background:#0b1016;color:var(--text);border-radius:5px;padding:0 9px;outline:none}select:focus,input:focus{border-color:#5475a8;box-shadow:0 0 0 2px rgba(84,117,168,.17)}.unit{position:absolute;right:9px;bottom:11px;color:#697687;font-size:9px;pointer-events:none}.field-grid{display:grid;gap:10px;margin-bottom:10px}.field-grid.two{grid-template-columns:1fr 1fr}hr{border:0;border-top:1px solid var(--line2);margin:17px 0}.section-kicker{margin-bottom:10px}
|
| 6 |
+
button.primary,button.secondary{height:40px;border-radius:5px;font-weight:700;cursor:pointer;transition:.13s ease}button.primary{width:100%;border:1px solid #668fd5;background:#4f7fce;color:white;margin-top:11px}button.primary:hover:not(:disabled){background:#5b8bd8}button.secondary{border:1px solid var(--line);background:#0d1219;color:#c4ccd7}button.secondary:hover:not(:disabled),.mini-button:hover:not(:disabled),.chart-expand:hover{border-color:#536174;background:#151b24}button:disabled{opacity:.42;cursor:not-allowed}.compact{width:auto!important;min-width:178px;padding:0 17px;margin-top:0!important}.button-row{display:grid;grid-template-columns:1fr 1fr;gap:8px;margin-top:8px}
|
| 7 |
+
.empty-state{min-height:380px;display:grid;place-content:center;text-align:center;color:var(--muted);padding:30px}.empty-state.small{min-height:240px}.empty-state h3{color:#ccd4de;margin:0 0 6px;font-size:15px}.empty-state p{max-width:500px;margin:0;line-height:1.55;font-size:12px}.hidden{display:none!important}.metric-grid{display:grid;grid-template-columns:repeat(6,1fr);gap:8px;margin-bottom:12px}.metric-grid.four{grid-template-columns:repeat(4,1fr)}.metric{min-height:82px;padding:13px;border:1px solid var(--line2);background:var(--surface2)}.metric span{display:block;color:var(--muted);font-size:10px;margin-bottom:9px}.metric strong{font-size:18px;letter-spacing:-.025em}.metric.emphasis{border-color:#486da4;background:#101824}
|
| 8 |
+
.diagnostic-card{margin-bottom:12px;border:1px solid var(--line);background:#0e131a;padding:14px}.diagnostic-head{display:flex;align-items:baseline;justify-content:space-between;gap:16px}.diagnostic-head span{color:var(--muted);font-size:10px;text-transform:uppercase;letter-spacing:.06em}.diagnostic-head strong{font-size:14px}.diagnostic-card p{margin:8px 0 0;color:#b8c1cd;font-size:12px;line-height:1.55}.diagnostic-card .diagnostic-action{color:var(--muted)}.evidence-row{display:flex;flex-wrap:wrap;gap:6px;margin-top:10px}.evidence-row span{border:1px solid var(--line2);background:#0a0f15;padding:4px 6px;color:#8390a0;font:10px/1.2 ui-monospace,SFMono-Regular,Menlo,Consolas,monospace}
|
| 9 |
+
.chart-grid{display:grid;grid-template-columns:1fr 1fr;gap:10px}.chart-card{border:1px solid var(--line);background:var(--surface2);min-height:300px;position:relative}.chart-card.full{margin-top:10px}.chart-head{height:42px;display:flex;align-items:center;justify-content:space-between;gap:12px;padding:0 12px;border-bottom:1px solid var(--line2)}.chart-title{color:#c7d0db;font-size:11px;font-weight:700}.chart-expand,.mini-button{border:1px solid var(--line);background:#0b1016;color:var(--muted);border-radius:4px;cursor:pointer;height:28px;padding:0 9px;font-size:10px}.chart-body{height:260px;padding:10px 12px 12px}.chart-body.large{height:340px}.chart-card.chart-expanded{position:fixed;inset:18px;z-index:100;background:#0b1016;border-color:#4c596b;box-shadow:0 0 0 9999px rgba(0,0,0,.72);min-height:0;margin:0}.chart-card.chart-expanded .chart-body{height:calc(100vh - 84px)}body.chart-open{overflow:hidden}canvas{width:100%!important;height:100%!important;max-height:none}.warnings{margin-top:10px;border-left:3px solid #7b6840;padding:9px 11px;background:#15140f;color:#c4bca9;font-size:11px;line-height:1.5}.muted{color:var(--muted);font-size:12px;line-height:1.55}
|
| 10 |
+
.table-toolbar{display:flex;align-items:center;justify-content:space-between;gap:12px;margin-top:14px;color:#aab4c1;font-size:11px}.table-toolbar>div{display:flex;gap:6px}.table-wrap{overflow:auto;margin-top:7px;border:1px solid var(--line)}table{width:100%;border-collapse:collapse;font-size:12px;min-width:900px}th,td{padding:10px 11px;text-align:right;border-bottom:1px solid var(--line2);white-space:nowrap}th:first-child,td:first-child{text-align:left}th{color:#7f8b9b;font-size:9px;text-transform:uppercase;letter-spacing:.06em;background:#0a0f15;position:sticky;top:0}td{color:#c7d0db}tbody tr:last-child td{border-bottom:0}tbody tr:hover td{background:#121821}.pass{color:var(--good)}.fail{color:var(--danger)}.best-label{display:inline-block;margin-left:7px;border:1px solid #4a6386;color:#9ab8df;padding:2px 5px;border-radius:3px;font-size:9px;text-transform:uppercase}.planner-grid{grid-template-columns:360px minmax(0,1fr)}.planner-note{color:var(--muted);border-left:2px solid #445a77;padding:7px 9px;margin-bottom:2px;font-size:11px;line-height:1.5}
|
| 11 |
+
.method-grid{display:grid;grid-template-columns:1fr 1fr;gap:12px}.prose{padding:22px}.prose h2{font-size:20px;margin:8px 0 11px}.prose p{color:var(--muted);line-height:1.7;font-size:13px}.prose code{color:#a8bfe4}.formula{font-family:ui-monospace,SFMono-Regular,Menlo,Consolas,monospace;color:#b7c3d2;padding:10px;border:1px solid var(--line);background:#0b1016;font-size:11px}.wide-method{grid-column:1/-1}.paper-grid{display:grid;grid-template-columns:repeat(5,1fr);gap:8px;margin-top:18px}.paper-grid>div{padding:12px;border:1px solid var(--line2);background:#0d1219}.paper-grid strong{display:block;font-size:11px;margin-bottom:7px}.paper-grid span{color:var(--muted);font-size:10px;line-height:1.55;display:block}footer{width:min(1460px,calc(100% - 40px));margin:0 auto;padding:20px 0 34px;border-top:1px solid var(--line2);display:flex;justify-content:space-between;gap:20px;color:var(--faint);font-size:10px}.toast{position:fixed;right:20px;bottom:20px;z-index:130;background:#17202b;border:1px solid #465467;color:#d5dce5;padding:9px 12px;border-radius:5px;font-size:11px;opacity:0;transform:translateY(8px);pointer-events:none;transition:.16s ease}.toast.show{opacity:1;transform:translateY(0)}
|
| 12 |
+
@media(max-width:1100px){.intro{grid-template-columns:1fr;gap:24px}.workspace,.planner-grid{grid-template-columns:1fr}.controls-panel{position:static}.metric-grid{grid-template-columns:repeat(3,1fr)}.paper-grid{grid-template-columns:1fr 1fr}}@media(max-width:720px){.topbar{padding:0 15px}.subtitle{display:none}.runtime-pill{max-width:170px}.shell{width:calc(100% - 20px);padding-top:88px}body.runtime-ready .shell{padding-top:20px}.intro{padding-top:10px}.project-facts div{grid-template-columns:100px 1fr}.field-grid.two,.chart-grid,.method-grid{grid-template-columns:1fr}.metric-grid,.metric-grid.four{grid-template-columns:1fr 1fr}.paper-grid{grid-template-columns:1fr}.panel-title-row{flex-direction:column}.arena-title-row .compact{width:100%!important}.wide-method{grid-column:auto}footer{width:calc(100% - 20px);flex-direction:column}.chart-card.chart-expanded{inset:6px}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
tests/test_diagnostics.py
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from inferscale import run_simulation
|
| 2 |
+
|
| 3 |
+
|
| 4 |
+
def test_healthy_run_has_diagnosis():
|
| 5 |
+
result = run_simulation(
|
| 6 |
+
{
|
| 7 |
+
"duration_s": 6,
|
| 8 |
+
"request_rate_rps": 0.5,
|
| 9 |
+
"prompt_tokens_mean": 128,
|
| 10 |
+
"output_tokens_mean": 8,
|
| 11 |
+
"slo_ttft_ms": 2000,
|
| 12 |
+
"slo_e2e_ms": 20000,
|
| 13 |
+
"slo_attainment_target": 0.95,
|
| 14 |
+
}
|
| 15 |
+
)
|
| 16 |
+
assert result["diagnostics"]["code"] in {"healthy", "healthy_near_saturation"}
|
| 17 |
+
assert result["diagnostics"]["provenance"] == "heuristic-simulator-diagnosis"
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def test_summary_exposes_component_slo_attainment():
|
| 21 |
+
result = run_simulation(
|
| 22 |
+
{
|
| 23 |
+
"duration_s": 6,
|
| 24 |
+
"request_rate_rps": 2,
|
| 25 |
+
"prompt_tokens_mean": 128,
|
| 26 |
+
"output_tokens_mean": 8,
|
| 27 |
+
}
|
| 28 |
+
)
|
| 29 |
+
summary = result["summary"]
|
| 30 |
+
assert 0 <= summary["ttft_slo_attainment"] <= 1
|
| 31 |
+
assert 0 <= summary["e2e_slo_attainment"] <= 1
|
| 32 |
+
assert set(summary["slo_failure_breakdown"]) == {"ttft_only", "e2e_only", "both"}
|
tests/test_optimizer.py
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
-
from inferscale.
|
|
|
|
| 2 |
|
| 3 |
|
| 4 |
def test_capacity_search_returns_trace():
|
|
@@ -13,8 +14,16 @@ def test_capacity_search_returns_trace():
|
|
| 13 |
out = capacity_search(cfg, min_rate=0.25, max_rate=3, iterations=3, repetitions=1)
|
| 14 |
assert out["trace"]
|
| 15 |
assert out["capacity_rps"] >= 0
|
|
|
|
| 16 |
|
| 17 |
|
| 18 |
-
def
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
out = compare_schedulers({"duration_s": 4, "request_rate_rps": 1, "output_tokens_mean": 8})
|
| 20 |
assert len(out["rows"]) == 5
|
|
|
|
|
|
| 1 |
+
from inferscale.models import SimulationConfig
|
| 2 |
+
from inferscale.optimizer import capacity_search, compare_schedulers, evaluate_rate
|
| 3 |
|
| 4 |
|
| 5 |
def test_capacity_search_returns_trace():
|
|
|
|
| 14 |
out = capacity_search(cfg, min_rate=0.25, max_rate=3, iterations=3, repetitions=1)
|
| 15 |
assert out["trace"]
|
| 16 |
assert out["capacity_rps"] >= 0
|
| 17 |
+
assert out["criterion"] == "all-repetitions-meet-target-and-drain"
|
| 18 |
|
| 19 |
|
| 20 |
+
def test_rate_evaluation_exposes_repetition_range():
|
| 21 |
+
out = evaluate_rate(SimulationConfig(duration_s=4, request_rate_rps=1), 1.0, repetitions=2)
|
| 22 |
+
assert out["slo_attainment_min"] <= out["slo_attainment"] <= out["slo_attainment_max"]
|
| 23 |
+
assert out["repetitions"] == 2
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def test_scheduler_compare_has_rows_and_diagnostics():
|
| 27 |
out = compare_schedulers({"duration_s": 4, "request_rate_rps": 1, "output_tokens_mean": 8})
|
| 28 |
assert len(out["rows"]) == 5
|
| 29 |
+
assert all(row["bottleneck"] for row in out["rows"])
|
worker.mjs
CHANGED
|
@@ -3,6 +3,7 @@ import { loadPyodide } from "https://cdn.jsdelivr.net/pyodide/v314.0.5/full/pyod
|
|
| 3 |
const MODULES = [
|
| 4 |
"__init__.py",
|
| 5 |
"api.py",
|
|
|
|
| 6 |
"kv_cache.py",
|
| 7 |
"latency.py",
|
| 8 |
"metrics.py",
|
|
|
|
| 3 |
const MODULES = [
|
| 4 |
"__init__.py",
|
| 5 |
"api.py",
|
| 6 |
+
"diagnostics.py",
|
| 7 |
"kv_cache.py",
|
| 8 |
"latency.py",
|
| 9 |
"metrics.py",
|