docs/research.md

# Research context

InferScale-Sim is a portfolio-scale implementation situated within the research line on replacing expensive deployment sweeps with modeling, simulation, emulation, and statistically careful load testing.

## Vidur (2024)

Vidur combines experimental profiling, predictive models, and end-to-end inference simulation. Its paper reports inference-latency estimates below 9% error in the evaluated range and a configuration-search example where LLaMA2-70B exploration took about one CPU-hour versus an estimated 42K GPU-hours (~$218K) for deployment-based exploration.

InferScale inspiration: explicit performance-profile provenance, workload-aware search, and separation between latency prediction and event simulation.

Source: https://arxiv.org/abs/2405.05465

## SGLang / RadixAttention (2024)

SGLang introduced RadixAttention for automatic KV-cache reuse across shared prefixes. InferScale does not implement a radix tree; it uses a controlled exact-prefix abstraction so the compute and memory consequences can be isolated and paired against an identical workload.

Source: https://arxiv.org/abs/2312.07104

## TokenSim (2025)

TokenSim emphasizes extensible scheduling and memory-management exploration.

InferScale inspiration: modular scheduler/memory experiments rather than a single hard-coded serving policy.

Source: https://arxiv.org/abs/2503.08415

## Revati (2026)

Revati executes real serving control paths while virtualizing GPU time, addressing a fidelity limitation of simulators that reimplement rapidly evolving runtime logic.

InferScale lesson: scheduler semantics must be documented as explicit approximations rather than presented as framework equivalence.

Source: https://arxiv.org/abs/2601.00397

## LLMServingSim 2.0 (2026)

LLMServingSim 2.0 models heterogeneous and disaggregated serving while integrating runtime decisions with batching, routing, memory, communication, and power. The paper reports 0.97% average error in its validation.

InferScale inspiration: role-specific resources and explicit data movement rather than a monolithic-replica abstraction.

Source: https://arxiv.org/abs/2602.23036

## Frontier (May 2026)

Frontier models co-location, Prefill-Decode Disaggregation, Attention-FFN Disaggregation, runtime optimizations such as speculative decoding, and stateful workloads. It reports average throughput error below 4% on its H800 evaluation and supports SLA-dependent Pareto exploration.

InferScale inspiration:

- separate prefill/decode workers;
- communication/KV-transfer cost;
- role-specific bottleneck telemetry;
- resource-aware Pareto design studies.

Source: https://arxiv.org/abs/2605.21312

## AgentServeSim (June 2026)

AgentServeSim extends serving simulation to multi-turn programs with tool-induced gaps, session-aware routing, cache locality, and KV residency. It reports reproducing real-system behavior within 6% across its evaluated metrics while executing on CPUs.

InferScale now implements a smaller program-level analogue: session identity, ordered turns, explicit tool gaps, cross-turn KV residency, session-affinity routing, and TTL/pressure eviction. Its per-replica service station is intentionally simpler than AGENTSERVESIM and remains analytically profiled rather than empirically validated.

Source: https://arxiv.org/abs/2606.09613

## Vanguard / Load Testing for ML Serving Systems at Scale (June 2026)

Vanguard highlights two methodology points that directly influence InferScale:

1. **open-loop replay** avoids coordinated omission by scheduling requests independently of response time;
2. repeated load-test analysis benefits from reproducibility statistics and bootstrap confidence intervals.

InferScale's generated workloads are open-loop, trace replay preserves externally supplied arrivals, and Research Studies use repeated paired simulations with bootstrap intervals.

Source: https://arxiv.org/abs/2606.22013

## HeteroPanacea / When Does Disaggregation Pay? (August 2026)

HeteroPanacea pushes design exploration toward stage-specific heterogeneous hardware, quantization, parallelism, and Prefill/Decode/Attention/FFN disaggregation. The paper emphasizes that disaggregation gains are conditional on workload and hardware design rather than universally beneficial.

InferScale's smaller P/D model already separates role hardware, worker counts, interconnect cost, and resource-normalized goodput. It does not model custom NPUs or Attention-FFN disaggregation.

Source: https://arxiv.org/abs/2608.03741

## Scope boundary

InferScale-Sim is not intended to compete with these research systems on fidelity, hardware scale, or runtime compatibility. Its aim is an inspectable Python implementation with a zero-backend interactive interface, explicit uncertainty, controlled experiments, and a path to external empirical validation.


## CacheTTL (2026)

CacheTTL studies multi-turn agent scheduling under variable tool-call gaps and treats KV retention as a time-to-live decision: holding state improves future reuse but occupies accelerator memory while the program is waiting on tools.

InferScale inspiration: explicit TTL retention, expiry events, memory-time accounting, and a TTL frontier that makes latency versus retained KV visible on a common program trace.

Source: https://arxiv.org/abs/2511.02230

## AgentSysBench (August 2026)

AgentSysBench characterizes agentic applications as long-running, stateful workflows with heterogeneous components, shifting bottlenecks, and idle state that can persist between active steps. The paper reports that non-LLM components can dominate latency in several applications and that state/offloading choices materially affect resource use.

InferScale inspiration: model tool gaps and whole-session latency rather than treating every LLM call as an independent request.

Source: https://arxiv.org/abs/2608.15127

## CacheScout / agent-aware KV management (2026)

Recent agent-aware KV work argues that future reuse is governed by program execution semantics rather than recency alone and uses predicted transitions to guide retention/prefetch. InferScale does not implement learned prediction, but its session-affinity and TTL experiments provide an inspectable baseline for studying why agent identity changes cache decisions.

Source: https://arxiv.org/abs/2608.14624

## IdleKV / tool-call idle-window offloading (June 2026)

Recent agent-serving work explicitly exploits tool-call idle windows to move KV state out of scarce GPU memory, because agent tool gaps can be long enough to hide part of the offload cost. InferScale models a smaller version of this idea with an explicit host-memory tier, offload/restore bandwidth, base transfer latency, host residency, and exposed restore delay when the state is needed again.

The model is intentionally not packet- or DMA-level: it is a transparent what-if abstraction for comparing recomputation, HBM retention, and host offload.

Source: https://arxiv.org/abs/2606.00866

## SMetric / locality-aware scheduling under load (July 2026)

SMetric reports that aggressively routing agent turns to replicas that already cache their KV can overload a subset of replicas and leave others underused. This motivates treating cache locality and load balancing as a joint scheduling problem rather than using strict affinity everywhere.

InferScale's **bounded-affinity** policy is a deliberately simple baseline: it follows the cached replica only while its estimated queue penalty remains within a configurable slack. The Affinity Frontier then sweeps that slack on one common trace.

Source: https://arxiv.org/abs/2607.08565

## Continuum / adaptive TTL for tool calls (May 2026)

Continuum treats tool-call duration as a first-class signal for KV retention. It records historical tool-call information, estimates whether a program is likely to return within a candidate TTL, and balances the benefit of avoiding reload/prefill and queueing delay against the opportunity cost of pinning GPU memory. The paper also emphasizes robustness to highly variable tool durations rather than assuming clairvoyant tool completion times.

InferScale inspiration: the new adaptive policy learns tool-duration estimates only from completed tool calls, keeps its oracle policy visibly separate, and exposes an explicit non-stationary shift experiment. InferScale's EWMA predictor is deliberately simpler than Continuum's cost/TTL model.

Source: https://arxiv.org/abs/2511.02230

## Predictive multi-tier KV memory (April/August 2026)

Recent predictive KV-memory work combines a multi-tier memory hierarchy with online reuse prediction instead of relying only on reactive eviction. One 2026 design uses a Bayesian reuse predictor with per-class priors to guide eviction and prefetching across a much richer storage hierarchy than InferScale models.

InferScale inspiration: measure prediction quality and systems outcomes together, and keep HBM/host placement decisions explicit. InferScale currently models only HBM plus host DRAM and uses transparent EWMA duration prediction; it does not reproduce the paper's six-tier hierarchy or Bayesian predictor.

Source: https://arxiv.org/abs/2604.26968

## CacheScout / learning agent execution online (July 2026)

CacheScout argues that future KV reuse in multi-agent systems is governed by execution semantics rather than recency alone. It learns agent execution transitions online without requiring predefined workflow graphs or offline training, then uses the learned model for eviction and proactive prefetching.

InferScale inspiration: move beyond an oracle or fixed TTL toward a policy that actually learns from runtime history. The current Predictive Tiering Lab learns tool-duration statistics, not workflow-transition graphs or prefetch decisions, so CacheScout remains a direction for deeper future work rather than a claimed reproduction.

Source: https://arxiv.org/abs/2608.14624