Generate workloads, compare schedulers, model KV pressure and prefix reuse, separate prefill from decode, and search the SLO-constrained design space. The simulator is Python running locally in a Pyodide Web Worker.
-The Python simulator executes in a background Web Worker and returns request-level virtual timestamps.
Run a simulation to generate a bottleneck explanation.
- - -Run every colocated scheduler against the current Serving Lab workload and rank by SLO attainment, then goodput.
Your Serving Lab workload and model controls are reused automatically.
| Scheduler | Goodput | SLO attainment | p95 TTFT | p95 E2E | KV peak | Unfinished | Diagnosis |
|---|
The planner repeatedly runs the simulator at different offered loads.
| Rate | Pass | Mean SLO | Worst repetition | Target | SLO range | Goodput | p95 TTFT | p95 E2E |
|---|
Compare colocated and prefill/decode-disaggregated serving, each with and without the current shared-prefix reuse scenario. P/D worker counts and interconnect settings come from Serving Lab.
Set shared-prefix and P/D parameters in Serving Lab, then run this controlled comparison.
| Scenario | GPU instances | Goodput | Goodput / GPU | SLO attainment | p95 TTFT | p95 E2E | p95 KV transfer | Cache hit | Prefill saved | Diagnosis |
|---|
Sweep 10 colocated candidates plus six P/D/cache variants, then identify the non-dominated goodput/TTFT frontier. This is an interactive design study, not an exhaustive optimizer.
The current workload, model, SLO, and P/D parameters are reused from Serving Lab.
| Candidate | Perf Pareto | Efficiency Pareto | SLO pass | GPU instances | Goodput | Goodput / GPU | p95 TTFT | p95 E2E | KV peak | Diagnosis |
|---|
Baseline and treatment use the same random seed on every repetition. The reported confidence interval is over paired deltas, not unrelated runs.
| Metric | Baseline mean | Treatment mean | Mean delta | 95% bootstrap CI | Relative change | Treatment win rate | CI excludes zero |
|---|
Shared multiplicative perturbations are applied to prefill, decode, and transfer latency proxies. This tests sensitivity to profile error; it is not a probability distribution over real hardware.
| Sample | Prefill scale | Decode scale | Transfer scale | Baseline SLO | Treatment SLO | Goodput delta | TTFT delta | E2E delta |
|---|
Each session has ordered LLM turns separated by sampled tool gaps. A replica processes one turn at a time in this mode so cache/routing effects remain isolated from the batching experiments in Serving Lab.
All candidates replay the identical generated program trace.
Compare stateless load balancing, retention without affinity, TTL with affinity, and full retention with affinity.
| Policy | p95 turn TTFT | p95 session E2E | Cache hit | Route locality | Recomputed history | Mean KV | Peak KV | HBM GB-s | Evictions |
|---|
Sweep the KV retention horizon on the same program trace and expose the latency vs memory-residency frontier.
The sweep varies retention from immediate eviction through long-lived state while keeping session arrivals, turns, tool gaps, and token lengths fixed.
| TTL | Pareto | p95 turn TTFT | p95 session E2E | Cache hit | Recomputed history | Mean KV | HBM GB-s | TTL evictions | Pressure evictions |
|---|
Replay one program trace across stateless, TTL, bounded-affinity, host-offload, and gap-aware policies, then deliberately shrink the per-replica HBM KV budget to expose eviction pressure.
Host offload preserves reusable KV outside HBM and pays an explicit transfer cost on restore. Gap-aware tiering is reported as an oracle upper bound because it uses the realized simulated tool gap.
| Policy | p95 turn TTFT | p95 session E2E | Session SLO | Reuse | HBM hit | Host hit | Mean HBM | Mean host | Transfer p95 | Recomputed |
|---|
| Policy | Budget / replica | Reference x | p95 TTFT | Session SLO | Reuse | Mean HBM | Mean host | Pressure evictions | Failed turns |
|---|
Bounded affinity keeps a session on its cached replica only while the estimated queue penalty stays within a configurable slack. Sweep that slack on one common program trace.
Low slack behaves closer to least-load routing; high slack increasingly prioritizes KV locality. This exposes when cache reuse begins to overload a hot replica.
| Affinity slack | p95 turn TTFT | p95 session E2E | Route locality | Cache hit | Recomputed | Session SLO | Pressure evictions |
|---|
Replace the oracle tool-gap policy with online estimates learned only from completed tool calls. The study changes slow-tool latency partway through one common program trace, then measures whether global and per-tool predictors adapt without future-duration leakage.
The adaptive policies use exponentially weighted tool-duration history. The oracle candidate is retained only as an upper bound; it can see the realized future gap while adaptive candidates cannot.
| Policy | p95 turn TTFT | p95 session E2E | Session SLO | Reuse | Mean HBM | Mean host | Prediction MAE | Oracle agreement | Post-shift MAE | Recomputed |
|---|
| Alpha | Pre-shift MAE | Post-shift MAE | Oracle agreement | p95 TTFT | p95 session E2E | Session SLO | Mean HBM | Mean host |
|---|
Agent-role transitions are revealed only when the next workflow step becomes ready. Learned policies may prefetch likely static prefixes from the modeled host tier before that happens.
Compare no prefetch, cumulative transition counts, exponentially decayed counts, and a clairvoyant next-role upper bound on exactly the same workflows.
The study separates transition-model quality from serving outcomes such as TTFT, cache hits, HBM residency, and wrong-step prefetch traffic.
| Policy | p95 TTFT | p95 workflow E2E | Post-shift top-1 | Prefix hit | Prefetch precision | Coverage | Mean HBM | Wrong prefetch | Saved prefill |
|---|
Sweep when a learned next-role prediction is confident enough to justify prefetch. Lower thresholds raise coverage but can pollute a small prefix cache with wrong-step transfers.
The same learned transition model is replayed at multiple decision thresholds on one common shifted workflow trace.
| Threshold | p95 TTFT | Post-shift top-1 | Precision | Coverage | Prefix hit | Wrong prefetch | Mean HBM |
|---|
Sweep exponential forgetting in the transition matrix after the workflow changes. The goal is not merely the most accurate predictor, but the best downstream serving behavior under cache and transfer constraints.
Decay = 1.0 is cumulative history. Smaller values forget old transitions faster and can react more quickly to the regime shift.
| Decay | Pre-shift top-1 | Post-shift top-1 | p95 TTFT | Prefix hit | Precision | Coverage | Wrong prefetch |
|---|
Compare one-step prediction with multi-step forecasting, a utility-aware planner, and a clairvoyant future-set information bound on one common shifted workflow trace.
This experiment separates next-role accuracy from forecast-set recall, cache utilization, unused prefetch traffic, and actual serving latency.
| Policy | p95 TTFT | Future-role recall@K | Prefix hit | Utilization | Next-step precision | Mean HBM | Unused prefetch | Saved prefill | ECE |
|---|
Sweep a utility-aware planner from one-step prediction to longer rollout horizons. Longer forecasts can reveal reuse but can also waste bandwidth or evict nearer-term prefixes.
Each horizon replays the exact same shifted workflow trace and uses the same cache budget and learned transition model.
| Horizon | p95 TTFT | Future-role recall@K | Prefix hit | Utilization | Unused prefetch | Mean HBM | Saved prefill |
|---|
Compare top-1, multi-step, and utility-aware planning while shrinking or expanding the shared prefix-cache budget. The useful policy can change when speculative prefixes compete for HBM.
The study executes 12 controlled simulations: three planning policies across four working-set-relative HBM budgets.
| Budget | Policy | p95 TTFT | Future-role recall@K | Prefix hit | Utilization | Mean HBM | Unused prefetch | Pressure evictions |
|---|
Nominal winners can move with one workload trace. This study reports bootstrap uncertainty, per-seed TTFT wins, Pareto stability, worst-seed latency, and regret to the bounded offline oracle.
Run the matched-seed experiment from the left panel. The default executes four deployable policies plus an offline oracle search on each seed.
| Rank | Policy | Median TTFT | 95% CI of mean | TTFT wins | Pareto stable | Oracle regret | Worst seed | Unused prefetch | Mean HBM |
|---|
Fit robust global prefill/decode timing multipliers on training measurements, then evaluate the analytical simulator on held-out benchmark cases. This can correct global scale bias; it cannot by itself validate unseen schedulers or hardware.
This public demo ships no fake benchmark truth. Import your own vLLM/SGLang measurements or an InferScale validation-case bundle.
| Case | Metric | Measured | Baseline pred. | Calibrated pred. | Baseline APE | Calibrated APE |
|---|
Requests are generated from deterministic workload distributions and advanced through virtual time. Colocated runs model admission, prefill, paged KV allocation, dynamic decode batches, and completion. P/D runs use separate prefill and decode worker pools plus an explicit serialized KV-transfer link.
The simulator models a single shared prompt prefix with configurable length and reuse fraction. Cache hits avoid redundant prefill work and share one persistent KV allocation. It is deliberately a controlled what-if abstraction, not a claim to reproduce SGLang's full RadixAttention policy.
Prefill and decode have separate accelerator profiles and worker counts. Prompt KV state crosses a modeled interconnect before decode admission. The simulator reports role utilization, transfer latency, and transfer pressure so the benefit of isolation can be weighed against data-movement overhead.
Agent Sessions preserves program identity and turn order, materializes tool-induced gaps, and tracks HBM retention, TTL expiry, host-memory offload/restore, recomputation, and pressure eviction. Strict affinity, least-load, and bounded-affinity routing expose the tension between cache locality and hot-replica queueing. The per-replica service model remains intentionally serial so state-management effects are not confounded with dynamic batching.
Adaptive tiering learns an exponentially weighted estimate from tool calls only after they complete. Global and per-tool predictors can be evaluated on the same non-stationary program trace, while an oracle policy that sees the realized future gap is kept separate as an upper bound. A shift experiment changes slower external-tool durations mid-trace to expose the stability-versus-adaptation trade-off.
Execution Learning updates a first-order role-transition model only after the next role is observed. The learned matrix can be rolled forward for multi-step forecasts without future-trace access. A utility-aware planner weighs expected prefill savings against host-transfer cost and forecast-weighted eviction cost, while calibration, horizon, confidence, forgetting, and cache-budget studies expose where better predictions do or do not improve serving outcomes.
The Design Explorer reports both a raw-performance frontier and a resource-normalized frontier using goodput per accelerator. That prevents a multi-GPU P/D layout from looking unconditionally better merely because it uses more simulated hardware.
Poisson, constant, and bursty workloads are open-loop: arrivals are scheduled independently of response completion, so queueing delay is not hidden by client backpressure. Exact CSV/JSON traces can be replayed with their original arrival times and token lengths.
Research Studies use common random numbers: baseline and treatment receive identical seeds, reducing workload variance in the paired difference. Research Summary extends that idea across matched seeds, bootstrap uncertainty, Pareto stability, and regret to a bounded full-trace serving oracle.
The measurement bridge imports serving-benchmark artifacts, fits simple prefill/decode scale factors on training cases only, and evaluates held-out residuals. Current vLLM serving benchmarks can save JSON results, while SGLang bench_serving supports JSONL output; InferScale normalizes both into explicit validation cases rather than treating imported measurements as hidden truth.
Vidur established the value of simulation for avoiding expensive deployment sweeps. Recent systems have pushed toward heterogeneous and disaggregated serving, communication-aware modeling, stateful workloads, trace replay, and SLA-dependent design-space exploration. InferScale-Sim remains intentionally smaller and inspectable, with paired experiments and sensitivity analysis built into the workflow.
Discrete-event Python model for queueing, batching, KV state, prefill/decode disaggregation, and stateful agent workloads. Device timings use analytical reference profiles unless measurements are imported.
The Python simulator executes in a background Web Worker and returns request-level virtual timestamps.
Run a simulation to generate a bottleneck explanation.
+ + +