Deepen InferScale simulation research workflow

README.md

@@ -71,7 +71,7 @@ arrival_time,prompt_tokens,output_tokens
 - prefix-cache hit rate / saved prefill work
 - P/D transfer latency / transfer volume / role utilization
 - heuristic bottleneck diagnoses with explicit simulator provenance
-- cross-turn KV hit rate, recomputed history tokens, routing locality, HBM GB-seconds, and TTL/pressure evictions for stateful sessions
+- cross-turn KV hit rate, recomputed history tokens, routing locality, HBM/host GB-seconds, TTL/pressure evictions, and explicit host-tier transfer latency for stateful sessions
 
 ## Interactive experiments
 
@@ -109,11 +109,14 @@ Run a bounded live sweep across scheduler, batch size, prefix caching, and P/D w
 
 Model stateful multi-turn programs rather than isolated requests. Each program has ordered LLM turns separated by tool-call gaps. The simulator tracks cross-turn KV residency on multiple replicas, routing locality, TTL expiry, pressure eviction, recomputation after cache loss, and memory-time residency.
 
-Three live experiments are included:
+Six live experiments are included:
 
 1. **Session run:** inspect one retention/routing policy on a generated program trace.
-2. **Session Policy Arena:** replay the exact same program trace through stateless least-load, retained least-load, TTL+affinity, and retained+affinity policies.
+2. **Session Policy Arena:** replay the exact same program trace through simple stateless/retention baselines.
 3. **TTL frontier:** sweep KV retention time under common random numbers and expose the non-dominated trade-off between p95 turn TTFT and mean resident KV.
+4. **Agent Memory Lab:** compare HBM retention, bounded affinity, host-memory offload, and an explicitly labeled oracle gap-aware tiering upper bound.
+5. **Finite-HBM budget stress:** derive per-replica KV budgets from the trace's unconstrained working set and rerun tiering policies under deliberate memory pressure.
+6. **Affinity Frontier:** sweep the queue-delay slack that bounded-affinity routing is willing to pay for cache locality.
 
 Agent mode intentionally uses a serial service station per replica. The existing Serving Lab models batching; Agent Sessions isolates program dependencies, cache locality, tool gaps, and state-management decisions so those effects are interpretable rather than conflated.
 
@@ -216,12 +219,26 @@ Agent-session mode follows a program-level event trace:
 session arrival -> turn 1 -> tool gap -> turn 2 -> ... -> completion
 ```
 
-When a turn becomes ready, it is routed either to the least-loaded replica or to a replica already holding that session's KV state. A cache hit prefills only newly appended tokens; a miss recomputes the accumulated history. KV can be evicted immediately, retained until the session ends, or retained under a TTL. Per-replica capacity uses the same analytical KV-byte model as the request simulator and falls back to LRU-style pressure eviction when the retained working set exceeds the configured budget.
+When a turn becomes ready, it is routed either to the least-loaded replica or to a replica already holding that session's KV state. A cache hit prefills only newly appended tokens; a miss recomputes the accumulated history. KV can be evicted immediately, retained until the session ends, retained under a TTL, or moved to a modeled host-memory tier. Host offload/restore pays an explicit bandwidth + base-latency transfer cost. A bounded-affinity router only follows cached state while the estimated queue imbalance stays under a configurable slack; this avoids treating cache locality as unconditionally preferable. Per-replica capacity uses the same analytical KV-byte model as the request simulator and falls back to LRU-style pressure eviction when the retained working set exceeds the configured budget.
 
-The public app reports **HBM GB-seconds** in addition to peak/mean KV occupancy. This makes the cost of keeping state resident during tool gaps visible rather than treating cache hits as free.
+The public app reports **HBM GB-seconds** and **host GB-seconds** in addition to peak/mean occupancy and transfer volume. This makes both retaining and offloading state visible rather than treating cache hits or a host tier as free.
 
 This module is inspired by recent work on agentic serving, including AGENTSERVESIM's program/tool-gap/session-routing abstraction and CacheTTL's observation that retention across variable tool gaps is a latency-vs-memory decision. It is intentionally smaller and does not claim their measured-system fidelity.
 
+### Host-tier offload and bounded affinity
+
+For `offload` retention, idle session KV is moved from the modeled HBM tier into finite host memory. The reference transfer model is explicit:
+
+```text
+host_transfer = base_latency + KV_size / host_bandwidth
+```
+
+If the next agent turn returns before offload has completed, the remaining transfer time becomes exposed before restore. Host residency, offloaded/restored bytes, and p95 transfer latency are reported separately from HBM residency.
+
+`bounded_affinity` is a transparent middle ground between strict session affinity and least-load routing. InferScale follows the replica holding the session KV only while its estimated extra queue penalty is below `affinity_slack_ms`. This is meant to study the locality/load-balance tension, not reproduce a particular production scheduler.
+
+`gap_aware` tiering uses the **realized simulated tool gap** to retain short-gap state in HBM and offload longer-gap state. Because a real serving system does not know the future perfectly, InferScale labels this policy as an **oracle upper bound** rather than a deployable predictor.
+
 ## Empirical validation hook
 
 The public project does **not** ship invented "measured" GPU results. Instead, the repository includes an explicit validation path for future real measurements:

app.js

@@ -13,6 +13,9 @@ let lastRobustStudy = null;
 let lastAgentRun = null;
 let lastAgentCompare = null;
 let lastAgentTtl = null;
+let lastAgentMemory = null;
+let lastAgentBudget = null;
+let lastAgentAffinity = null;
 let traceRequests = [];
 
 const COLORS = {
@@ -22,6 +25,7 @@ const COLORS = {
   green: "#69c99a",
   amber: "#dfb966",
   red: "#df7d89",
+  purple: "#9b8cff",
   gray: "#667384",
   grid: "rgba(140,155,175,.14)",
 };
@@ -35,7 +39,7 @@ worker.addEventListener("message", (event) => {
   if (data.type === "ready") {
     runtimePill.classList.add("ready");
     runtimeText.textContent = "Python runtime ready";
-    ["runBtn", "arenaBtn", "capacityBtn", "topologyBtn", "designBtn", "pairedStudyBtn", "robustStudyBtn", "agentRunBtn", "agentCompareBtn", "agentTtlBtn"].forEach((id) => { $(id).disabled = false; });
+    ["runBtn", "arenaBtn", "capacityBtn", "topologyBtn", "designBtn", "pairedStudyBtn", "robustStudyBtn", "agentRunBtn", "agentCompareBtn", "agentTtlBtn", "agentMemoryCompareBtn", "agentBudgetBtn", "agentAffinityBtn"].forEach((id) => { $(id).disabled = false; });
     syncConditionalControls();
     window.setTimeout(() => document.body.classList.add("runtime-ready"), 650);
     return;
@@ -771,6 +775,11 @@ function agentConfigFromUI(overrides = {}) {
     routing_policy: $("agentRouting").value,
     kv_ttl_s: num("agentTtl"),
     kv_memory_fraction: num("agentKvFraction"),
+    affinity_slack_ms: num("agentAffinitySlack"),
+    gap_aware_threshold_s: num("agentGapThreshold"),
+    host_memory_gb: num("agentHostMemory"),
+    host_bandwidth_gbps: num("agentHostBandwidth"),
+    host_transfer_base_ms: num("agentHostBase"),
     slo_turn_ttft_ms: num("agentTtftSlo"),
     slo_session_e2e_ms: num("agentSessionSlo"),
     ...overrides,
@@ -786,14 +795,16 @@ function renderAgentRun(result) {
   const sm = result.summary;
   $("agentTtft").textContent = `${fmt(l.turn_ttft_ms.p95)} ms`;
   $("agentSessionE2e").textContent = `${fmt(l.session_e2e_ms.p95)} ms`;
+  $("agentSessionSloValue").textContent = pct(sm.session_slo_attainment);
   $("agentCacheHit").textContent = pct(r.cross_turn_cache_hit_rate);
   $("agentRecompute").textContent = `${fmt(r.recomputed_history_tokens, 0)} tok`;
   $("agentPeakKv").textContent = `${fmt(r.peak_kv_gb, 3)} GB`;
-  $("agentMeanKv").textContent = `${fmt(r.mean_kv_gb, 3)} GB`;
+  $("agentHostMeanKv").textContent = `${fmt(r.mean_host_kv_gb, 3)} GB`;
+  $("agentHostTransfer").textContent = `${fmt(r.p95_host_transfer_ms)} ms`;
   const state = $("agentRunState");
   state.textContent = `${sm.sessions_completed}/${sm.sessions_generated} sessions`;
   state.className = `tag ${sm.sessions_completed === sm.sessions_generated && sm.turns_failed === 0 ? "good" : "bad"}`;
-  $("agentRunSummary").innerHTML = `<strong>${escapeHtml(result.config.retention_policy)} retention + ${escapeHtml(result.config.routing_policy.replaceAll("_", " "))} routing.</strong> ${sm.turns_completed} turns completed across ${sm.sessions_completed} sessions. Cross-turn reuse avoided ${fmt(r.recomputed_history_tokens, 0)} history tokens of recomputation only when state was resident on the selected replica; the reported HBM residency is a simulator-side memory-time accounting metric.`;
+  $("agentRunSummary").innerHTML = `<strong>${escapeHtml(result.config.retention_policy)} retention + ${escapeHtml(result.config.routing_policy.replaceAll("_", " "))} routing.</strong> ${sm.turns_completed} turns completed across ${sm.sessions_completed} sessions. Reuse hit ${pct(r.cross_turn_cache_hit_rate)} of eligible turns; ${pct(r.host_cache_hit_rate)} came from the host tier. ${fmt(r.recomputed_history_tokens, 0)} history tokens were recomputed. HBM and host residency are simulator-side memory-time accounting metrics.`;
 
   const timeline = result.timeline || [];
   destroyChart("agentTimeline");
@@ -801,7 +812,8 @@ function renderAgentRun(result) {
     type: "line",
     data: {
       datasets: [
-        { label: "KV GB", data: timeline.map((x) => ({ x: x.time_s, y: x.kv_used_gb })), borderColor: COLORS.green, backgroundColor: COLORS.green, pointRadius: 0, tension: .08, yAxisID: "yKv" },
+        { label: "HBM KV GB", data: timeline.map((x) => ({ x: x.time_s, y: x.kv_used_gb })), borderColor: COLORS.green, backgroundColor: COLORS.green, pointRadius: 0, tension: .08, yAxisID: "yKv" },
+        { label: "Host KV GB", data: timeline.map((x) => ({ x: x.time_s, y: x.host_kv_gb || 0 })), borderColor: COLORS.steel, backgroundColor: COLORS.steel, pointRadius: 0, borderDash: [6, 4], tension: .08, yAxisID: "yKv" },
         { label: "Busy replicas", data: timeline.map((x) => ({ x: x.time_s, y: x.busy_replicas })), borderColor: COLORS.blue, backgroundColor: COLORS.blue, pointRadius: 0, tension: .08, yAxisID: "yCount" },
         { label: "Queued turns", data: timeline.map((x) => ({ x: x.time_s, y: x.queued_turns })), borderColor: COLORS.amber, backgroundColor: COLORS.amber, pointRadius: 0, tension: .08, yAxisID: "yCount" },
       ],
@@ -884,12 +896,112 @@ $("agentTtlBtn").addEventListener("click", async () => {
 $("agentTtlCopy").addEventListener("click", () => { if (lastAgentTtl) copyText(tableText(agentTtlHeaders, agentTtlTableRows(lastAgentTtl)), "TTL sweep copied"); });
 $("agentTtlCsv").addEventListener("click", () => { if (lastAgentTtl) downloadCsv(`inferscale_agent-ttl-frontier_${stamp()}.csv`, agentTtlHeaders, agentTtlTableRows(lastAgentTtl)); });
 
+
+const agentMemoryHeaders = ["Policy", "p95 turn TTFT", "p95 session E2E", "Session SLO", "Reuse", "HBM hit", "Host hit", "Mean HBM", "Mean host", "Transfer p95", "Recomputed"];
+function agentMemoryTableRows(result) {
+  return result.rows.map((r) => [r.label, `${fmt(r.p95_turn_ttft_ms)} ms`, `${fmt(r.p95_session_e2e_ms)} ms`, pct(r.session_slo_attainment), pct(r.cache_hit_rate), pct(r.hbm_hit_rate), pct(r.host_hit_rate), `${fmt(r.mean_hbm_gb, 3)} GB`, `${fmt(r.mean_host_gb, 3)} GB`, `${fmt(r.p95_host_transfer_ms)} ms`, `${fmt(r.recomputed_history_tokens, 0)} tok`]);
+}
+function renderAgentMemory(result) {
+  lastAgentMemory = result;
+  $("agentMemoryEmpty").classList.add("hidden");
+  $("agentMemoryContent").classList.remove("hidden");
+  $("agentMemoryCompareBlock").classList.remove("hidden");
+  ["agentMemoryCopy", "agentMemoryCsv"].forEach((id) => { $(id).disabled = false; });
+  $("agentMemoryRows").innerHTML = result.rows.map((r) => `<tr><td>${escapeHtml(r.label)}</td><td>${fmt(r.p95_turn_ttft_ms)} ms</td><td>${fmt(r.p95_session_e2e_ms)} ms</td><td>${pct(r.session_slo_attainment)}</td><td>${pct(r.cache_hit_rate)}</td><td>${pct(r.hbm_hit_rate)}</td><td>${pct(r.host_hit_rate)}</td><td>${fmt(r.mean_hbm_gb, 3)} GB</td><td>${fmt(r.mean_host_gb, 3)} GB</td><td>${fmt(r.p95_host_transfer_ms)} ms</td><td>${fmt(r.recomputed_history_tokens, 0)} tok</td></tr>`).join("");
+  const palette = [COLORS.gray, COLORS.amber, COLORS.green, COLORS.blue, COLORS.purple];
+  destroyChart("agentMemory");
+  charts.agentMemory = new Chart($("agentMemoryChart"), {
+    type: "scatter",
+    data: { datasets: result.rows.map((r, i) => ({ label: r.label, data: [{ x: r.mean_hbm_gb, y: r.p95_turn_ttft_ms }], backgroundColor: palette[i], borderColor: palette[i], pointRadius: 7, pointHoverRadius: 9 })) },
+    options: { ...commonChartOptions(), plugins: { legend: pointLegend() }, scales: { x: { beginAtZero: true, title: { display: true, text: "Mean HBM KV (GB)" } }, y: { beginAtZero: true, title: { display: true, text: "p95 turn TTFT (ms)" } } } },
+  });
+}
+$("agentMemoryCompareBtn").addEventListener("click", async () => {
+  const button = $("agentMemoryCompareBtn"); button.disabled = true; button.textContent = "Comparing policies...";
+  try { renderAgentMemory(await callPython("agent_memory_compare", { config: agentConfigFromUI() })); }
+  catch (error) { alert(`Memory policy comparison failed: ${error.message}`); }
+  finally { button.disabled = false; button.textContent = "Compare memory policies"; }
+});
+$("agentMemoryCopy").addEventListener("click", () => { if (lastAgentMemory) copyText(tableText(agentMemoryHeaders, agentMemoryTableRows(lastAgentMemory)), "Memory-policy table copied"); });
+$("agentMemoryCsv").addEventListener("click", () => { if (lastAgentMemory) downloadCsv(`inferscale_agent-memory-policies_${stamp()}.csv`, agentMemoryHeaders, agentMemoryTableRows(lastAgentMemory)); });
+
+const agentBudgetHeaders = ["Policy", "Budget / replica", "Reference x", "p95 TTFT", "Session SLO", "Reuse", "Mean HBM", "Mean host", "Pressure evictions", "Failed turns"];
+function agentBudgetTableRows(result) {
+  return result.rows.map((r) => [r.policy, `${fmt(r.budget_gb_per_replica, 4)} GB`, `${fmt(r.budget_multiplier, 2)}x`, `${fmt(r.p95_turn_ttft_ms)} ms`, pct(r.session_slo_attainment), pct(r.cache_hit_rate), `${fmt(r.mean_hbm_gb, 4)} GB`, `${fmt(r.mean_host_gb, 4)} GB`, r.pressure_evictions, r.turns_failed]);
+}
+function renderAgentBudget(result) {
+  lastAgentBudget = result;
+  $("agentMemoryEmpty").classList.add("hidden");
+  $("agentMemoryContent").classList.remove("hidden");
+  $("agentBudgetBlock").classList.remove("hidden");
+  ["agentBudgetCopy", "agentBudgetCsv"].forEach((id) => { $(id).disabled = false; });
+  $("agentBudgetCaption").textContent = `HBM budget sweep (reference peak ${fmt(result.reference_peak_replica_kv_gb, 4)} GB / replica)`;
+  $("agentBudgetRows").innerHTML = result.rows.map((r) => `<tr><td>${escapeHtml(r.policy)}</td><td>${fmt(r.budget_gb_per_replica, 4)} GB</td><td>${fmt(r.budget_multiplier, 2)}x</td><td>${fmt(r.p95_turn_ttft_ms)} ms</td><td>${pct(r.session_slo_attainment)}</td><td>${pct(r.cache_hit_rate)}</td><td>${fmt(r.mean_hbm_gb, 4)} GB</td><td>${fmt(r.mean_host_gb, 4)} GB</td><td>${r.pressure_evictions}</td><td>${r.turns_failed}</td></tr>`).join("");
+  const policies = [...new Set(result.rows.map((r) => r.policy))];
+  const palette = [COLORS.green, COLORS.blue, COLORS.purple];
+  destroyChart("agentBudget");
+  charts.agentBudget = new Chart($("agentBudgetChart"), {
+    type: "line",
+    data: { datasets: policies.map((policy, i) => ({ label: policy, data: result.rows.filter((r) => r.policy === policy).sort((a, b) => a.budget_gb_per_replica - b.budget_gb_per_replica).map((r) => ({ x: r.budget_gb_per_replica, y: r.p95_turn_ttft_ms })), borderColor: palette[i], backgroundColor: palette[i], pointRadius: 4, tension: .12, fill: false })) },
+    options: { ...commonChartOptions(), parsing: false, plugins: { legend: lineLegend() }, scales: { x: { type: "linear", beginAtZero: true, title: { display: true, text: "HBM KV budget per replica (GB)" } }, y: { beginAtZero: true, title: { display: true, text: "p95 turn TTFT (ms)" } } } },
+  });
+}
+$("agentBudgetBtn").addEventListener("click", async () => {
+  const button = $("agentBudgetBtn"); button.disabled = true; button.textContent = "Stressing HBM budget...";
+  try { renderAgentBudget(await callPython("agent_memory_sweep", { config: agentConfigFromUI() })); }
+  catch (error) { alert(`HBM budget study failed: ${error.message}`); }
+  finally { button.disabled = false; button.textContent = "Stress HBM budget"; }
+});
+$("agentBudgetCopy").addEventListener("click", () => { if (lastAgentBudget) copyText(tableText(agentBudgetHeaders, agentBudgetTableRows(lastAgentBudget)), "HBM budget table copied"); });
+$("agentBudgetCsv").addEventListener("click", () => { if (lastAgentBudget) downloadCsv(`inferscale_agent-hbm-budget_${stamp()}.csv`, agentBudgetHeaders, agentBudgetTableRows(lastAgentBudget)); });
+
+const agentAffinityHeaders = ["Affinity slack", "p95 turn TTFT", "p95 session E2E", "Route locality", "Cache hit", "Recomputed", "Session SLO", "Pressure evictions"];
+function agentAffinityTableRows(result) {
+  return result.rows.map((r) => [`${fmt(r.affinity_slack_ms, 0)} ms`, `${fmt(r.p95_turn_ttft_ms)} ms`, `${fmt(r.p95_session_e2e_ms)} ms`, pct(r.routing_locality_rate), pct(r.cache_hit_rate), `${fmt(r.recomputed_history_tokens, 0)} tok`, pct(r.session_slo_attainment), r.pressure_evictions]);
+}
+function renderAgentAffinity(result) {
+  lastAgentAffinity = result;
+  $("agentAffinityEmpty").classList.add("hidden");
+  $("agentAffinityContent").classList.remove("hidden");
+  ["agentAffinityCopy", "agentAffinityCsv"].forEach((id) => { $(id).disabled = false; });
+  $("agentAffinityRows").innerHTML = result.rows.map((r) => `<tr><td>${fmt(r.affinity_slack_ms, 0)} ms</td><td>${fmt(r.p95_turn_ttft_ms)} ms</td><td>${fmt(r.p95_session_e2e_ms)} ms</td><td>${pct(r.routing_locality_rate)}</td><td>${pct(r.cache_hit_rate)}</td><td>${fmt(r.recomputed_history_tokens, 0)} tok</td><td>${pct(r.session_slo_attainment)}</td><td>${r.pressure_evictions}</td></tr>`).join("");
+  destroyChart("agentAffinity");
+  charts.agentAffinity = new Chart($("agentAffinityChart"), {
+    type: "line",
+    data: { datasets: [
+      { label: "p95 session E2E", data: result.rows.map((r) => ({ x: r.affinity_slack_ms, y: r.p95_session_e2e_ms })), borderColor: COLORS.blue, backgroundColor: COLORS.blue, pointRadius: 4, tension: .12, yAxisID: "yLatency", fill: false },
+      { label: "Routing locality", data: result.rows.map((r) => ({ x: r.affinity_slack_ms, y: r.routing_locality_rate * 100 })), borderColor: COLORS.green, backgroundColor: COLORS.green, pointRadius: 4, tension: .12, yAxisID: "yLocality", fill: false },
+    ] },
+    options: { ...commonChartOptions(), parsing: false, plugins: { legend: lineLegend() }, scales: { x: { type: "linear", beginAtZero: true, title: { display: true, text: "Affinity slack (ms)" } }, yLatency: { position: "left", beginAtZero: true, title: { display: true, text: "p95 session E2E (ms)" } }, yLocality: { position: "right", min: 0, max: 100, grid: { drawOnChartArea: false }, title: { display: true, text: "Routing locality (%)" } } } },
+  });
+}
+$("agentAffinityBtn").addEventListener("click", async () => {
+  const button = $("agentAffinityBtn"); button.disabled = true; button.textContent = "Sweeping affinity...";
+  try { renderAgentAffinity(await callPython("agent_affinity_sweep", { config: agentConfigFromUI() })); }
+  catch (error) { alert(`Affinity sweep failed: ${error.message}`); }
+  finally { button.disabled = false; button.textContent = "Run affinity sweep"; }
+});
+$("agentAffinityCopy").addEventListener("click", () => { if (lastAgentAffinity) copyText(tableText(agentAffinityHeaders, agentAffinityTableRows(lastAgentAffinity)), "Affinity sweep copied"); });
+$("agentAffinityCsv").addEventListener("click", () => { if (lastAgentAffinity) downloadCsv(`inferscale_agent-affinity-frontier_${stamp()}.csv`, agentAffinityHeaders, agentAffinityTableRows(lastAgentAffinity)); });
+
 function syncAgentControls() {
-  const ttl = $("agentRetention").value === "ttl";
+  const retention = $("agentRetention").value;
+  const routing = $("agentRouting").value;
+  const ttl = retention === "ttl";
+  const hostTier = retention === "offload" || retention === "gap_aware";
+  const gapAware = retention === "gap_aware";
+  const bounded = routing === "bounded_affinity";
   $("agentTtl").disabled = !ttl;
   $("agentTtlLabel").classList.toggle("field-disabled", !ttl);
+  $("agentAffinitySlack").disabled = !bounded;
+  $("agentAffinitySlackLabel").classList.toggle("field-disabled", !bounded);
+  $("agentGapThreshold").disabled = !gapAware;
+  $("agentGapThresholdLabel").classList.toggle("field-disabled", !gapAware);
+  ["agentHostMemory", "agentHostBandwidth", "agentHostBase"].forEach((id) => { $(id).disabled = !hostTier; });
+  ["agentHostMemoryLabel", "agentHostBandwidthLabel", "agentHostBaseLabel"].forEach((id) => { $(id).classList.toggle("field-disabled", !hostTier); });
 }
 $("agentRetention").addEventListener("change", syncAgentControls);
+$("agentRouting").addEventListener("change", syncAgentControls);
 syncAgentControls();
 
 function normalizeTraceRows(rows) {

docs/architecture.md

@@ -85,3 +85,21 @@ The simulator therefore runs away from the browser main thread and uses the same
 - `kv_bytes_per_token()`
 
 without rewriting workload generation, scheduling, cache logic, P/D orchestration, SLO metrics, or research protocols.
+
+## Agent memory tiering
+
+Stateful Agent Sessions has an additional memory path that is independent from the stateless/P-D simulator:
+
+```text
+turn completes
+   |
+   +-- retain HBM --------------------------+
+   |                                        |
+   +-- TTL -> expire / pressure evict       | next turn
+   |                                        |
+   +-- host offload -> host KV -> restore --+
+   |                                        |
+   +-- evict -> history recomputation ------+
+```
+
+A global host tier models capacity, residency, offload/restore volume, and transfer latency. A bounded-affinity router can trade cached-replica locality against estimated queue imbalance.

docs/methodology.md

@@ -150,3 +150,36 @@ The TTL sweep replays one identical agent-program trace for every TTL and report
 `validation.py` accepts externally measured cases and compares them against simulator predictions. Supported observations include p95 TTFT, p95 E2E, goodput, request throughput, and SLO attainment. The report contains per-case residuals, MAPE, median APE, and maximum APE.
 
 No measured benchmark values are bundled as truth with the project.
+
+## Stateful host-tier KV model
+
+Agent-session experiments can move idle cross-turn KV from the modeled HBM tier to a host-memory tier. Transfer time is represented as:
+
+```text
+transfer_time = base_latency + KV_size / host_bandwidth
+```
+
+Offload begins when an LLM turn completes and can overlap the simulated tool gap. If the next turn becomes ready before offload completion, the remaining offload time is exposed before restore. Restore is serialized with the turn service model in this reference implementation. Host capacity is finite and uses LRU-style pressure eviction.
+
+This is a deliberately transparent what-if model, not a PCIe/NVLink/NIXL/DMA simulator. Its purpose is to compare three costs under the same program trace:
+
+- recompute history after eviction;
+- retain KV in scarce HBM during tool gaps;
+- offload KV and pay data movement on reuse.
+
+## Bounded-affinity routing
+
+Strict session affinity always routes a turn to the replica that already holds its HBM KV. Least-load routing ignores locality. Bounded affinity interpolates between them:
+
+1. find the least-loaded replica;
+2. find the replica holding the session KV, if any;
+3. estimate the extra queue/busy-horizon penalty of following locality;
+4. keep affinity only if that penalty is at most `affinity_slack_ms`.
+
+The estimator is intentionally simple and labeled as such. The Affinity Frontier sweeps the slack on one common program trace to show where additional locality stops being worth the load imbalance.
+
+## Finite-HBM budget stress
+
+The memory-budget experiment first runs a full-retention reference trace and records the unconstrained peak per-replica KV working set. It then expresses stress budgets as multiples of that trace-specific peak rather than arbitrary fractions of total accelerator VRAM. This makes the experiment meaningful even for small models whose default VRAM headroom would otherwise be far larger than the generated KV working set.
+
+Failed turns and incomplete sessions count against SLO attainment; latency percentiles remain conditional on turns/sessions that actually complete and are accompanied by failure counts in the experiment table.

docs/research.md

@@ -106,3 +106,19 @@ Source: https://arxiv.org/abs/2608.15127
 Recent agent-aware KV work argues that future reuse is governed by program execution semantics rather than recency alone and uses predicted transitions to guide retention/prefetch. InferScale does not implement learned prediction, but its session-affinity and TTL experiments provide an inspectable baseline for studying why agent identity changes cache decisions.
 
 Source: https://arxiv.org/abs/2608.14624
+
+## IdleKV / tool-call idle-window offloading (June 2026)
+
+Recent agent-serving work explicitly exploits tool-call idle windows to move KV state out of scarce GPU memory, because agent tool gaps can be long enough to hide part of the offload cost. InferScale models a smaller version of this idea with an explicit host-memory tier, offload/restore bandwidth, base transfer latency, host residency, and exposed restore delay when the state is needed again.
+
+The model is intentionally not packet- or DMA-level: it is a transparent what-if abstraction for comparing recomputation, HBM retention, and host offload.
+
+Source: https://arxiv.org/abs/2606.00866
+
+## SMetric / locality-aware scheduling under load (July 2026)
+
+SMetric reports that aggressively routing agent turns to replicas that already cache their KV can overload a subset of replicas and leave others underused. This motivates treating cache locality and load balancing as a joint scheduling problem rather than using strict affinity everywhere.
+
+InferScale's **bounded-affinity** policy is a deliberately simple baseline: it follows the cached replica only while its estimated queue penalty remains within a configurable slack. The Affinity Frontier then sweeps that slack on one common trace.
+
+Source: https://arxiv.org/abs/2607.08565

index.html

@@ -296,7 +296,7 @@
             <label>Samples<input id="robustSamples" type="number" min="4" max="96" value="32" /></label>
             <label>Latency uncertainty<input id="robustUncertainty" type="number" min="0" max="0.75" step="0.05" value="0.20" /><span class="unit">fraction</span></label>
           </div>
-          <button id="robustStudyBtn" class="secondary research-secondary" disabled>Stress-test selected hypothesis</button>
+          <button id="robustStudyBtn" class="primary" disabled>Stress-test selected hypothesis</button>
         </aside>
 
         <div class="research-results">
@@ -363,10 +363,15 @@
           <hr />
           <div class="section-kicker">State policy</div>
           <div class="field-grid two">
-            <label>KV retention<select id="agentRetention"><option value="evict">Evict after every turn</option><option value="ttl" selected>TTL retention</option><option value="retain">Retain until session ends</option></select></label>
-            <label>Routing<select id="agentRouting"><option value="least_load">Least-load</option><option value="session_affinity" selected>Session affinity</option></select></label>
+            <label>KV retention<select id="agentRetention"><option value="evict">Evict after every turn</option><option value="ttl" selected>TTL retention</option><option value="retain">Retain until session ends</option><option value="offload">Offload idle KV to host</option><option value="gap_aware">Oracle gap-aware HBM / host tiering</option></select></label>
+            <label>Routing<select id="agentRouting"><option value="least_load">Least-load</option><option value="session_affinity" selected>Strict session affinity</option><option value="bounded_affinity">Bounded affinity</option></select></label>
             <label id="agentTtlLabel">KV TTL<input id="agentTtl" type="number" min="0" step="0.25" value="3" /><span class="unit">s</span></label>
-            <label>KV memory fraction<input id="agentKvFraction" type="number" min="0.05" max="0.98" step="0.05" value="0.85" /><span class="unit">fraction</span></label>
+            <label>KV memory fraction<input id="agentKvFraction" type="number" min="0.01" max="0.98" step="0.05" value="0.85" /><span class="unit">fraction</span></label>
+            <label id="agentAffinitySlackLabel">Affinity slack<input id="agentAffinitySlack" type="number" min="0" step="25" value="150" /><span class="unit">ms</span></label>
+            <label id="agentGapThresholdLabel">Gap-aware HBM threshold<input id="agentGapThreshold" type="number" min="0" step="0.25" value="1.5" /><span class="unit">s</span></label>
+            <label id="agentHostMemoryLabel">Host KV capacity<input id="agentHostMemory" type="number" min="0" step="4" value="32" /><span class="unit">GB</span></label>
+            <label id="agentHostBandwidthLabel">Host transfer bandwidth<input id="agentHostBandwidth" type="number" min="0.1" step="1" value="32" /><span class="unit">GB/s</span></label>
+            <label id="agentHostBaseLabel">Transfer base latency<input id="agentHostBase" type="number" min="0" step="0.05" value="0.15" /><span class="unit">ms</span></label>
           </div>
           <div class="field-grid two">
             <label>Turn TTFT SLO<input id="agentTtftSlo" type="number" min="1" value="500" /><span class="unit">ms</span></label>
@@ -380,13 +385,15 @@
             <div class="panel-title-row"><div><div class="section-kicker">Program execution</div><h2>Session run</h2></div><span id="agentRunState" class="tag neutral">Waiting</span></div>
             <div id="agentRunEmpty" class="empty-state small"><h3>No stateful run yet</h3><p>Each session has ordered LLM turns separated by sampled tool gaps. A replica processes one turn at a time in this mode so cache/routing effects remain isolated from the batching experiments in Serving Lab.</p></div>
             <div id="agentRunContent" class="hidden">
-              <div class="metric-grid six-agent">
+              <div class="metric-grid eight-agent">
                 <div class="metric emphasis"><span>p95 turn TTFT</span><strong id="agentTtft">N/A</strong></div>
                 <div class="metric"><span>p95 session E2E</span><strong id="agentSessionE2e">N/A</strong></div>
-                <div class="metric"><span>Cross-turn cache hit</span><strong id="agentCacheHit">N/A</strong></div>
+                <div class="metric"><span>Session SLO</span><strong id="agentSessionSloValue">N/A</strong></div>
+                <div class="metric"><span>Cross-turn reuse</span><strong id="agentCacheHit">N/A</strong></div>
                 <div class="metric"><span>Recomputed history</span><strong id="agentRecompute">N/A</strong></div>
-                <div class="metric"><span>Peak KV</span><strong id="agentPeakKv">N/A</strong></div>
-                <div class="metric"><span>Mean resident KV</span><strong id="agentMeanKv">N/A</strong></div>
+                <div class="metric"><span>Peak HBM KV</span><strong id="agentPeakKv">N/A</strong></div>
+                <div class="metric"><span>Mean host KV</span><strong id="agentHostMeanKv">N/A</strong></div>
+                <div class="metric"><span>p95 host transfer</span><strong id="agentHostTransfer">N/A</strong></div>
               </div>
               <div class="study-summary" id="agentRunSummary"></div>
               <div class="chart-grid">
@@ -427,6 +434,42 @@
               <div class="table-wrap"><table><thead><tr><th>TTL</th><th>Pareto</th><th>p95 turn TTFT</th><th>p95 session E2E</th><th>Cache hit</th><th>Recomputed history</th><th>Mean KV</th><th>HBM GB-s</th><th>TTL evictions</th><th>Pressure evictions</th></tr></thead><tbody id="agentTtlRows"></tbody></table></div>
             </div>
           </section>
+          <section class="panel research-panel">
+            <div class="panel-title-row"><div><div class="section-kicker">Tiered-memory experiment</div><h2>Agent Memory Lab</h2><p class="muted">Replay one program trace across stateless, TTL, bounded-affinity, host-offload, and gap-aware policies, then deliberately shrink the per-replica HBM KV budget to expose eviction pressure.</p></div><div class="stacked-actions"><button id="agentMemoryCompareBtn" class="primary compact" disabled>Compare memory policies</button><button id="agentBudgetBtn" class="primary compact" disabled>Stress HBM budget</button></div></div>
+            <div id="agentMemoryEmpty" class="empty-state small"><h3>No tiered-memory experiment yet</h3><p>Host offload preserves reusable KV outside HBM and pays an explicit transfer cost on restore. Gap-aware tiering is reported as an oracle upper bound because it uses the realized simulated tool gap.</p></div>
+            <div id="agentMemoryContent" class="hidden">
+              <div id="agentMemoryCompareBlock" class="hidden">
+                <div class="chart-card full" data-chart-card data-chart-name="agent-memory-policy-tradeoff">
+                  <div class="chart-head"><div class="chart-title">Mean HBM residency vs p95 turn TTFT</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
+                  <div class="chart-body research-chart"><canvas id="agentMemoryChart"></canvas></div>
+                </div>
+                <div class="table-toolbar"><span>Memory-policy comparison</span><div><button id="agentMemoryCopy" class="mini-button" disabled>Copy table</button><button id="agentMemoryCsv" class="mini-button" disabled>Download CSV</button></div></div>
+                <div class="table-wrap"><table><thead><tr><th>Policy</th><th>p95 turn TTFT</th><th>p95 session E2E</th><th>Session SLO</th><th>Reuse</th><th>HBM hit</th><th>Host hit</th><th>Mean HBM</th><th>Mean host</th><th>Transfer p95</th><th>Recomputed</th></tr></thead><tbody id="agentMemoryRows"></tbody></table></div>
+              </div>
+              <div id="agentBudgetBlock" class="hidden experiment-subsection">
+                <div class="chart-card full" data-chart-card data-chart-name="agent-hbm-budget-stress">
+                  <div class="chart-head"><div class="chart-title">Finite HBM budget stress</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
+                  <div class="chart-body research-chart"><canvas id="agentBudgetChart"></canvas></div>
+                </div>
+                <div class="table-toolbar"><span id="agentBudgetCaption">HBM budget sweep</span><div><button id="agentBudgetCopy" class="mini-button" disabled>Copy table</button><button id="agentBudgetCsv" class="mini-button" disabled>Download CSV</button></div></div>
+                <div class="table-wrap"><table><thead><tr><th>Policy</th><th>Budget / replica</th><th>Reference x</th><th>p95 TTFT</th><th>Session SLO</th><th>Reuse</th><th>Mean HBM</th><th>Mean host</th><th>Pressure evictions</th><th>Failed turns</th></tr></thead><tbody id="agentBudgetRows"></tbody></table></div>
+              </div>
+            </div>
+          </section>
+
+          <section class="panel research-panel">
+            <div class="panel-title-row"><div><div class="section-kicker">Locality vs balance</div><h2>Affinity Frontier</h2><p class="muted">Bounded affinity keeps a session on its cached replica only while the estimated queue penalty stays within a configurable slack. Sweep that slack on one common program trace.</p></div><button id="agentAffinityBtn" class="primary compact" disabled>Run affinity sweep</button></div>
+            <div id="agentAffinityEmpty" class="empty-state small"><h3>No routing sweep yet</h3><p>Low slack behaves closer to least-load routing; high slack increasingly prioritizes KV locality. This exposes when cache reuse begins to overload a hot replica.</p></div>
+            <div id="agentAffinityContent" class="hidden">
+              <div class="chart-card full" data-chart-card data-chart-name="agent-affinity-routing-frontier">
+                <div class="chart-head"><div class="chart-title">Routing slack vs latency and locality</div><div class="chart-actions"><button class="chart-download" type="button">Download PNG</button><button class="chart-expand" type="button">Expand</button></div></div>
+                <div class="chart-body research-chart"><canvas id="agentAffinityChart"></canvas></div>
+              </div>
+              <div class="table-toolbar"><span>Bounded-affinity sweep</span><div><button id="agentAffinityCopy" class="mini-button" disabled>Copy table</button><button id="agentAffinityCsv" class="mini-button" disabled>Download CSV</button></div></div>
+              <div class="table-wrap"><table><thead><tr><th>Affinity slack</th><th>p95 turn TTFT</th><th>p95 session E2E</th><th>Route locality</th><th>Cache hit</th><th>Recomputed</th><th>Session SLO</th><th>Pressure evictions</th></tr></thead><tbody id="agentAffinityRows"></tbody></table></div>
+            </div>
+          </section>
+
         </div>
       </div>
     </section>
@@ -436,11 +479,11 @@
         <article class="panel prose"><div class="section-kicker">Simulation core</div><h2>What is actually simulated?</h2><p>Requests are generated from deterministic workload distributions and advanced through virtual time. Colocated runs model admission, prefill, paged KV allocation, dynamic decode batches, and completion. P/D runs use separate prefill and decode worker pools plus an explicit serialized KV-transfer link.</p><div class="formula">request -> queue -> prefill -> [KV transfer] -> decode -> completion</div></article>
         <article class="panel prose"><div class="section-kicker">Prefix reuse</div><h2>Cache without pretending to implement a radix tree</h2><p>The simulator models a single shared prompt prefix with configurable length and reuse fraction. Cache hits avoid redundant prefill work and share one persistent KV allocation. It is deliberately a controlled what-if abstraction, not a claim to reproduce SGLang's full RadixAttention policy.</p></article>
         <article class="panel prose"><div class="section-kicker">P/D disaggregation</div><h2>Role-specific resources and transfer cost</h2><p>Prefill and decode have separate accelerator profiles and worker counts. Prompt KV state crosses a modeled interconnect before decode admission. The simulator reports role utilization, transfer latency, and transfer pressure so the benefit of isolation can be weighed against data-movement overhead.</p></article>
-        <article class="panel prose"><div class="section-kicker">Stateful agent sessions</div><h2>Tool gaps turn KV into a residency decision</h2><p>Agent Sessions preserves program identity and turn order, materializes tool-induced gaps, and tracks whether cross-turn KV remains resident, expires under a TTL, or is recomputed after eviction. Session-affinity routing can trade load balance for cache locality. The per-replica service model is intentionally serial in this mode so those state-management effects are not confounded with batching.</p></article>
+        <article class="panel prose"><div class="section-kicker">Stateful agent sessions</div><h2>Tool gaps turn KV into a residency and routing decision</h2><p>Agent Sessions preserves program identity and turn order, materializes tool-induced gaps, and tracks HBM retention, TTL expiry, host-memory offload/restore, recomputation, and pressure eviction. Strict affinity, least-load, and bounded-affinity routing expose the tension between cache locality and hot-replica queueing. The per-replica service model remains intentionally serial so state-management effects are not confounded with dynamic batching.</p></article>
         <article class="panel prose"><div class="section-kicker">Design search</div><h2>Pareto, not one magic configuration</h2><p>The Design Explorer reports both a raw-performance frontier and a resource-normalized frontier using goodput per accelerator. That prevents a multi-GPU P/D layout from looking unconditionally better merely because it uses more simulated hardware.</p></article>
         <article class="panel prose"><div class="section-kicker">Trace methodology</div><h2>Generated load or exact replay</h2><p>Poisson, constant, and bursty workloads are open-loop: arrivals are scheduled independently of response completion, so queueing delay is not hidden by client backpressure. Exact CSV/JSON traces can be replayed with their original arrival times and token lengths.</p></article>
         <article class="panel prose"><div class="section-kicker">Research protocol</div><h2>Paired conclusions, not one lucky seed</h2><p>Research Studies use common random numbers: baseline and treatment receive identical seeds, reducing workload variance in the paired difference. A bootstrap interval summarizes the repeated effect, while a separate sensitivity study perturbs analytical latency scales to reveal conclusions that depend too strongly on one reference profile.</p></article>
-        <article class="panel prose wide-method"><div class="section-kicker">Research lineage</div><h2>Research lineage and scope</h2><p>Vidur established the value of simulation for avoiding expensive deployment sweeps. Recent systems have pushed toward heterogeneous and disaggregated serving, communication-aware modeling, stateful workloads, trace replay, and SLA-dependent design-space exploration. InferScale-Sim remains intentionally smaller and inspectable, with paired experiments and sensitivity analysis built into the workflow.</p><div class="paper-grid"><div><strong>Vidur / 2024</strong><span>Predictive profiling, workload-aware serving simulation, configuration search.</span></div><div><strong>TokenSim / 2025</strong><span>Extensible scheduling and memory-management simulation.</span></div><div><strong>Revati / 2026</strong><span>GPU-free time-warp emulation of serving control logic.</span></div><div><strong>LLMServingSim 2.0 / 2026</strong><span>Heterogeneous and disaggregated infrastructure, memory and communication.</span></div><div><strong>Frontier / May 2026</strong><span>P/D disaggregation, runtime optimizations, stateful workloads, Pareto exploration.</span></div><div><strong>HeteroPanacea / Aug 2026</strong><span>Heterogeneous stage specialization motivates resource-aware P/D comparison.</span></div><div><strong>Vanguard / Jun 2026</strong><span>Open-loop replay avoids coordinated omission when studying latency under load.</span></div><div><strong>AgentServeSim / Jun 2026</strong><span>Stateful multi-turn serving motivates future session-aware workload modeling.</span></div><div><strong>SGLang / RadixAttention</strong><span>Automatic shared-prefix KV reuse motivates the controlled cache scenario.</span></div></div></article>
+        <article class="panel prose wide-method"><div class="section-kicker">Research lineage</div><h2>Research lineage and scope</h2><p>Vidur established the value of simulation for avoiding expensive deployment sweeps. Recent systems have pushed toward heterogeneous and disaggregated serving, communication-aware modeling, stateful workloads, trace replay, and SLA-dependent design-space exploration. InferScale-Sim remains intentionally smaller and inspectable, with paired experiments and sensitivity analysis built into the workflow.</p><div class="paper-grid"><div><strong>Vidur / 2024</strong><span>Predictive profiling, workload-aware serving simulation, configuration search.</span></div><div><strong>TokenSim / 2025</strong><span>Extensible scheduling and memory-management simulation.</span></div><div><strong>Revati / 2026</strong><span>GPU-free time-warp emulation of serving control logic.</span></div><div><strong>LLMServingSim 2.0 / 2026</strong><span>Heterogeneous and disaggregated infrastructure, memory and communication.</span></div><div><strong>Frontier / May 2026</strong><span>P/D disaggregation, runtime optimizations, stateful workloads, Pareto exploration.</span></div><div><strong>HeteroPanacea / Aug 2026</strong><span>Heterogeneous stage specialization motivates resource-aware P/D comparison.</span></div><div><strong>Vanguard / Jun 2026</strong><span>Open-loop replay avoids coordinated omission when studying latency under load.</span></div><div><strong>AgentServeSim / Jun 2026</strong><span>Stateful multi-turn serving motivates session-aware workload modeling.</span></div><div><strong>IdleKV / Jun 2026</strong><span>Tool-call idle windows motivate explicit HBM-to-host KV offload experiments.</span></div><div><strong>SMetric / Jul 2026</strong><span>Cache-local routing can overload hot replicas, motivating bounded affinity.</span></div><div><strong>SGLang / RadixAttention</strong><span>Automatic shared-prefix KV reuse motivates the controlled cache scenario.</span></div></div></article>
       </div>
     </section>
   </main>

py/inferscale/__init__.py

@@ -1,4 +1,11 @@
-from .agentic import compare_agent_policies, run_agent_session_simulation, ttl_retention_sweep
+from .agentic import (
+    agent_affinity_sweep,
+    agent_memory_budget_sweep,
+    compare_agent_memory_policies,
+    compare_agent_policies,
+    run_agent_session_simulation,
+    ttl_retention_sweep,
+)
 from .api import execute, metadata
 from .models import SimulationConfig
 from .optimizer import capacity_search, compare_schedulers, compare_topologies, design_space_search
@@ -8,6 +15,9 @@ from .validation import validate_cases
 
 __all__ = [
     "SimulationConfig",
+    "agent_affinity_sweep",
+    "agent_memory_budget_sweep",
+    "compare_agent_memory_policies",
     "compare_agent_policies",
     "capacity_search",
     "compare_schedulers",
@@ -25,4 +35,4 @@ __all__ = [
 
 # Internal package metadata only; the public project intentionally avoids
 # release/version branding in the interface and documentation.
-__version__ = "0.5.0"
+__version__ = "0.6.0"

py/inferscale/agentic.py

@@ -12,8 +12,8 @@ from .metrics import percentile
 from .profiles import get_accelerator, get_model
 
 
-ROUTING_POLICIES = {"least_load", "session_affinity"}
-RETENTION_POLICIES = {"evict", "retain", "ttl"}
+ROUTING_POLICIES = {"least_load", "session_affinity", "bounded_affinity"}
+RETENTION_POLICIES = {"evict", "retain", "ttl", "offload", "gap_aware"}
 
 
 @dataclass
@@ -38,6 +38,12 @@ class AgentSessionConfig:
     kv_ttl_s: float = 3.0
     routing_policy: str = "session_affinity"
     kv_memory_fraction: float = 0.85
+    kv_capacity_override_gb: float = 0.0
+    host_memory_gb: float = 32.0
+    host_bandwidth_gbps: float = 32.0
+    host_transfer_base_ms: float = 0.15
+    affinity_slack_ms: float = 150.0
+    gap_aware_threshold_s: float = 1.5
     slo_turn_ttft_ms: float = 500.0
     slo_session_e2e_ms: float = 30000.0
     timeline_points: int = 240
@@ -54,7 +60,13 @@ class AgentSessionConfig:
         cfg.session_rate_rps = max(float(cfg.session_rate_rps), 0.01)
         cfg.duration_s = max(float(cfg.duration_s), 1.0)
         cfg.kv_ttl_s = max(float(cfg.kv_ttl_s), 0.0)
-        cfg.kv_memory_fraction = min(max(float(cfg.kv_memory_fraction), 0.05), 0.98)
+        cfg.kv_memory_fraction = min(max(float(cfg.kv_memory_fraction), 0.01), 0.98)
+        cfg.kv_capacity_override_gb = max(float(cfg.kv_capacity_override_gb), 0.0)
+        cfg.host_memory_gb = max(float(cfg.host_memory_gb), 0.0)
+        cfg.host_bandwidth_gbps = max(float(cfg.host_bandwidth_gbps), 0.1)
+        cfg.host_transfer_base_ms = max(float(cfg.host_transfer_base_ms), 0.0)
+        cfg.affinity_slack_ms = max(float(cfg.affinity_slack_ms), 0.0)
+        cfg.gap_aware_threshold_s = max(float(cfg.gap_aware_threshold_s), 0.0)
         return cfg
 
     def to_dict(self) -> dict[str, Any]:
@@ -86,6 +98,8 @@ class CacheEntry:
     last_access: float
     expiry_time: float | None
     generation: int
+    source_replica: int | None = None
+    available_time: float = 0.0
 
 
 @dataclass
@@ -175,8 +189,14 @@ class AgentSessionSimulator:
         self.accelerator = get_accelerator(cfg.accelerator)
         self.latency = AnalyticalLatencyModel(self.model, self.accelerator, cfg.quantization)
         remaining = max(0.0, self.accelerator.vram_gb - self.latency.model_weight_gb - 1.2)
-        self.kv_capacity_gb = remaining * cfg.kv_memory_fraction
+        automatic_capacity = remaining * cfg.kv_memory_fraction
+        self.kv_capacity_gb = (
+            min(automatic_capacity, cfg.kv_capacity_override_gb)
+            if cfg.kv_capacity_override_gb > 0.0
+            else automatic_capacity
+        )
         self.replicas = [ReplicaState(i) for i in range(cfg.replicas)]
+        self.host_cache: dict[int, CacheEntry] = {}
         specs = sessions if sessions is not None else generate_agent_sessions(cfg)
         self.sessions = {s.session_id: SessionRuntime(s, s.initial_prompt_tokens) for s in specs}
         self.events: list[tuple[float, int, str, tuple[Any, ...]]] = []
@@ -185,10 +205,18 @@ class AgentSessionSimulator:
         self.turn_rows: list[dict[str, Any]] = []
         self.timeline: list[dict[str, Any]] = []
         self.peak_kv_gb = 0.0
+        self.peak_replica_kv_gb = 0.0
         self.hbm_gb_seconds = 0.0
+        self.host_gb_seconds = 0.0
+        self.peak_host_gb = 0.0
         self.last_memory_time = 0.0
         self.pressure_evictions = 0
         self.ttl_evictions = 0
+        self.host_pressure_evictions = 0
+        self.host_cache_hits = 0
+        self.offload_bytes = 0.0
+        self.restore_bytes = 0.0
+        self.host_transfer_latencies_ms: list[float] = []
         self.recompute_tokens = 0
         self.cache_hits = 0
         self.cache_eligible_turns = 0
@@ -207,12 +235,23 @@ class AgentSessionSimulator:
     def _used_gb(self) -> float:
         return sum(entry.size_gb for replica in self.replicas for entry in replica.cache.values())
 
+    def _used_host_gb(self) -> float:
+        return sum(entry.size_gb for entry in self.host_cache.values())
+
     def _integrate_memory(self, new_time: float) -> None:
         new_time = max(float(new_time), self.last_memory_time)
+        elapsed = new_time - self.last_memory_time
         used = self._used_gb()
-        self.hbm_gb_seconds += used * (new_time - self.last_memory_time)
+        host_used = self._used_host_gb()
+        self.hbm_gb_seconds += used * elapsed
+        self.host_gb_seconds += host_used * elapsed
         self.last_memory_time = new_time
         self.peak_kv_gb = max(self.peak_kv_gb, used)
+        self.peak_replica_kv_gb = max(
+            self.peak_replica_kv_gb,
+            max((sum(entry.size_gb for entry in replica.cache.values()) for replica in self.replicas), default=0.0),
+        )
+        self.peak_host_gb = max(self.peak_host_gb, host_used)
 
     def _record_timeline(self) -> None:
         max_points = max(int(self.cfg.timeline_points), 40)
@@ -224,7 +263,9 @@ class AgentSessionSimulator:
                 "queued_turns": sum(len(r.queue) for r in self.replicas),
                 "busy_replicas": sum(1 for r in self.replicas if r.busy),
                 "kv_used_gb": self._used_gb(),
+                "host_kv_gb": self._used_host_gb(),
                 "resident_sessions": sum(len(r.cache) for r in self.replicas),
+                "host_sessions": len(self.host_cache),
             }
         )
         if len(self.timeline) > max_points * 2:
@@ -239,15 +280,28 @@ class AgentSessionSimulator:
                 return replica
         return None
 
+    def _replica_load_key(self, replica: ReplicaState) -> tuple[int, float, int]:
+        return (len(replica.queue), max(replica.busy_until, self.now), replica.replica_id)
+
     def _route(self, session_id: int) -> ReplicaState:
         cached = self._cache_replica(session_id)
         if cached is not None:
             self.route_opportunities += 1
+        least_loaded = min(self.replicas, key=self._replica_load_key)
         if self.cfg.routing_policy == "session_affinity" and cached is not None:
             self.affinity_routes += 1
             return cached
-        # Approximate least-finish-time routing using queued work count then busy horizon.
-        return min(self.replicas, key=lambda r: (len(r.queue), max(r.busy_until, self.now), r.replica_id))
+        if self.cfg.routing_policy == "bounded_affinity" and cached is not None:
+            cached_ready = max(cached.busy_until, self.now)
+            best_ready = max(least_loaded.busy_until, self.now)
+            queue_gap = max(0, len(cached.queue) - len(least_loaded.queue))
+            # Queue length is converted to a modest reference penalty because
+            # exact queued service time is intentionally not known at routing time.
+            estimated_extra_ms = max(0.0, cached_ready - best_ready) * 1000.0 + queue_gap * 75.0
+            if estimated_extra_ms <= self.cfg.affinity_slack_ms:
+                self.affinity_routes += 1
+                return cached
+        return least_loaded
 
     def _remove_cache(self, replica: ReplicaState, session_id: int, reason: str) -> None:
         if session_id not in replica.cache:
@@ -260,6 +314,73 @@ class AgentSessionSimulator:
             self.ttl_evictions += 1
         self._record_timeline()
 
+    def _host_transfer_seconds(self, size_gb: float) -> float:
+        return self.cfg.host_transfer_base_ms / 1000.0 + size_gb / self.cfg.host_bandwidth_gbps
+
+    def _remove_host_cache(self, session_id: int) -> None:
+        if session_id not in self.host_cache:
+            return
+        self._integrate_memory(self.now)
+        del self.host_cache[session_id]
+        self._record_timeline()
+
+    def _ensure_host_capacity(self, session_id: int, target_gb: float) -> bool:
+        if target_gb > self.cfg.host_memory_gb + 1e-12:
+            return False
+        while self._used_host_gb() + target_gb > self.cfg.host_memory_gb + 1e-12:
+            victims = [entry for sid, entry in self.host_cache.items() if sid != session_id]
+            if not victims:
+                return False
+            victim = min(victims, key=lambda e: (e.last_access, e.session_id))
+            self._integrate_memory(self.now)
+            del self.host_cache[victim.session_id]
+            self.host_pressure_evictions += 1
+        return True
+
+    def _offload_cache(self, replica: ReplicaState, session_id: int, tokens: int) -> bool:
+        size_gb = tokens * self.latency.kv_bytes_per_token() / 1e9
+        if self.cfg.host_memory_gb <= 0.0 or not self._ensure_host_capacity(session_id, size_gb):
+            self._remove_cache(replica, session_id, "complete")
+            return False
+        transfer_s = self._host_transfer_seconds(size_gb)
+        self._integrate_memory(self.now)
+        replica.cache.pop(session_id, None)
+        self.host_cache[session_id] = CacheEntry(
+            session_id=session_id,
+            tokens=tokens,
+            size_gb=size_gb,
+            last_access=self.now,
+            expiry_time=None,
+            generation=1,
+            source_replica=replica.replica_id,
+            available_time=self.now + transfer_s,
+        )
+        self.offload_bytes += size_gb * 1e9
+        self.host_transfer_latencies_ms.append(transfer_s * 1000.0)
+        self.peak_host_gb = max(self.peak_host_gb, self._used_host_gb())
+        self._record_timeline()
+        return True
+
+    def _restore_host_entry(self, replica: ReplicaState, session_id: int) -> tuple[bool, float]:
+        entry = self.host_cache.get(session_id)
+        if entry is None:
+            return False, 0.0
+        if not self._ensure_capacity(replica, session_id, entry.size_gb):
+            return False, 0.0
+        wait_s = max(0.0, entry.available_time - self.now)
+        copy_s = self._host_transfer_seconds(entry.size_gb)
+        # The host entry remains resident while an unfinished offload or restore
+        # is exposed. Account for that residency explicitly because the event
+        # clock advances only when the turn completes.
+        self.host_gb_seconds += entry.size_gb * (wait_s + copy_s)
+        self._integrate_memory(self.now)
+        del self.host_cache[session_id]
+        self.restore_bytes += entry.size_gb * 1e9
+        self.host_transfer_latencies_ms.append((wait_s + copy_s) * 1000.0)
+        self.host_cache_hits += 1
+        self._record_timeline()
+        return True, wait_s + copy_s
+
     def _ensure_capacity(self, replica: ReplicaState, session_id: int, target_gb: float) -> bool:
         current = replica.cache.get(session_id)
         current_gb = current.size_gb if current else 0.0
@@ -289,6 +410,10 @@ class AgentSessionSimulator:
             expiry = self.now + self.cfg.kv_ttl_s
         replica.cache[session_id] = CacheEntry(session_id, tokens, size_gb, self.now, expiry, generation)
         self.peak_kv_gb = max(self.peak_kv_gb, self._used_gb())
+        self.peak_replica_kv_gb = max(
+            self.peak_replica_kv_gb,
+            sum(entry.size_gb for entry in replica.cache.values()),
+        )
         if expiry is not None:
             self._push(expiry, "cache_expire", replica.replica_id, session_id, generation)
         self._record_timeline()
@@ -300,16 +425,29 @@ class AgentSessionSimulator:
         session_id, turn, ready_time = replica.queue.pop(0)
         runtime = self.sessions[session_id]
         cache = self._entry(replica, session_id)
-        hit = turn.turn_index > 0 and cache is not None
+        hbm_hit = turn.turn_index > 0 and cache is not None
+        host_hit = False
+        restore_s = 0.0
         if turn.turn_index > 0:
             self.cache_eligible_turns += 1
-        if hit:
+        if hbm_hit:
             self.cache_hits += 1
             prefill_tokens = max(turn.append_tokens, 1)
+            cache_source = "hbm"
+        elif turn.turn_index > 0 and session_id in self.host_cache:
+            host_hit, restore_s = self._restore_host_entry(replica, session_id)
+            if host_hit:
+                prefill_tokens = max(turn.append_tokens, 1)
+                cache_source = "host"
+            else:
+                prefill_tokens = max(runtime.context_tokens + turn.append_tokens, 1)
+                self.recompute_tokens += runtime.context_tokens
+                cache_source = "miss"
         else:
             prefill_tokens = max(runtime.context_tokens + turn.append_tokens, 1)
             if turn.turn_index > 0:
                 self.recompute_tokens += runtime.context_tokens
+            cache_source = "miss"
 
         context_before_decode = runtime.context_tokens + turn.append_tokens
         projected_tokens = context_before_decode + turn.output_tokens
@@ -332,8 +470,8 @@ class AgentSessionSimulator:
         first_decode_s = self.latency.decode_step_seconds([context_before_decode])
         midpoint = context_before_decode + max(turn.output_tokens // 2, 1)
         decode_step_s = self.latency.decode_step_seconds([midpoint])
-        service_s = prefill_s + turn.output_tokens * decode_step_s
-        ttft_s = (self.now - ready_time) + prefill_s + first_decode_s
+        service_s = restore_s + prefill_s + turn.output_tokens * decode_step_s
+        ttft_s = (self.now - ready_time) + restore_s + prefill_s + first_decode_s
 
         replica.busy = True
         replica.current_session = session_id
@@ -345,7 +483,9 @@ class AgentSessionSimulator:
             turn.turn_index,
             replica.replica_id,
             ready_time,
-            hit,
+            hbm_hit or host_hit,
+            cache_source,
+            restore_s,
             prefill_tokens,
             ttft_s,
             service_s,
@@ -359,6 +499,8 @@ class AgentSessionSimulator:
         replica_id: int,
         ready_time: float,
         cache_hit: bool,
+        cache_source: str,
+        restore_s: float,
         prefill_tokens: int,
         ttft_s: float,
         service_s: float,
@@ -375,11 +517,31 @@ class AgentSessionSimulator:
         replica.busy_until = self.now
 
         is_final = turn_index == len(runtime.spec.turns) - 1
-        keep = (not is_final) and self.cfg.retention_policy != "evict"
-        if keep:
-            self._put_cache(replica, session_id, projected_tokens, keep=True)
-        else:
+        state_action = "evict"
+        if is_final:
+            self._remove_cache(replica, session_id, "complete")
+            self._remove_host_cache(session_id)
+        elif self.cfg.retention_policy == "evict":
             self._remove_cache(replica, session_id, "complete")
+            self._remove_host_cache(session_id)
+        elif self.cfg.retention_policy in {"retain", "ttl"}:
+            self._put_cache(replica, session_id, projected_tokens, keep=True)
+            state_action = "retain_hbm"
+        elif self.cfg.retention_policy == "offload":
+            if self._offload_cache(replica, session_id, projected_tokens):
+                state_action = "offload_host"
+            else:
+                state_action = "evict_host_full"
+        elif self.cfg.retention_policy == "gap_aware":
+            # This is intentionally an oracle upper-bound policy: the simulated
+            # tool gap is already known from the generated program trace.
+            if turn.tool_gap_after_s <= self.cfg.gap_aware_threshold_s:
+                self._put_cache(replica, session_id, projected_tokens, keep=True)
+                state_action = "retain_hbm_short_gap"
+            elif self._offload_cache(replica, session_id, projected_tokens):
+                state_action = "offload_host_long_gap"
+            else:
+                state_action = "evict_host_full"
 
         e2e_ms = (self.now - ready_time) * 1000.0
         self.turn_rows.append(
@@ -390,6 +552,9 @@ class AgentSessionSimulator:
                 "ready_time": ready_time,
                 "completion_time": self.now,
                 "cache_hit": cache_hit,
+                "cache_source": cache_source,
+                "restore_ms": restore_s * 1000.0,
+                "state_action": state_action,
                 "prefill_tokens": prefill_tokens,
                 "context_tokens_after": projected_tokens,
                 "output_tokens": turn.output_tokens,
@@ -421,7 +586,7 @@ class AgentSessionSimulator:
             elif kind == "turn_complete":
                 self._finish_turn(
                     int(payload[0]), int(payload[1]), int(payload[2]), float(payload[3]), bool(payload[4]),
-                    int(payload[5]), float(payload[6]), float(payload[7]), int(payload[8]),
+                    str(payload[5]), float(payload[6]), int(payload[7]), float(payload[8]), float(payload[9]), int(payload[10]),
                 )
             elif kind == "cache_expire":
                 replica_id, session_id, generation = map(int, payload)
@@ -452,17 +617,21 @@ class AgentSessionSimulator:
         ]
         session_slo = sum(1 for v in session_e2e if v <= self.cfg.slo_session_e2e_ms)
         turn_slo = sum(1 for v in ttfts if v <= self.cfg.slo_turn_ttft_ms)
+        turns_generated = sum(len(s.spec.turns) for s in self.sessions.values())
         completion_horizon = max([s.completion_time or 0.0 for s in self.sessions.values()] + [self.cfg.duration_s, 1e-9])
         mean_kv_gb = self.hbm_gb_seconds / completion_horizon
         summary = {
             "sessions_generated": len(self.sessions),
             "sessions_completed": len(completed_sessions),
+            "turns_generated": turns_generated,
             "turns_completed": len(successful_turns),
             "turns_failed": self.failed_turns,
+            "session_completion_rate": len(completed_sessions) / len(self.sessions) if self.sessions else 0.0,
+            "turn_completion_rate": len(successful_turns) / turns_generated if turns_generated else 0.0,
             "session_throughput_rps": len(completed_sessions) / completion_horizon,
             "turn_throughput_rps": len(successful_turns) / completion_horizon,
-            "turn_ttft_slo_attainment": turn_slo / len(successful_turns) if successful_turns else 0.0,
-            "session_slo_attainment": session_slo / len(completed_sessions) if completed_sessions else 0.0,
+            "turn_ttft_slo_attainment": turn_slo / turns_generated if turns_generated else 0.0,
+            "session_slo_attainment": session_slo / len(self.sessions) if self.sessions else 0.0,
             "simulated_makespan_s": completion_horizon,
         }
         latency = {
@@ -470,19 +639,33 @@ class AgentSessionSimulator:
             "turn_e2e_ms": {"p50": percentile(turn_e2e, 0.50), "p95": percentile(turn_e2e, 0.95), "p99": percentile(turn_e2e, 0.99)},
             "session_e2e_ms": {"p50": percentile(session_e2e, 0.50), "p95": percentile(session_e2e, 0.95), "p99": percentile(session_e2e, 0.99)},
         }
+        host_restore_p95_ms = percentile(self.host_transfer_latencies_ms, 0.95)
+        total_reuse_hits = self.cache_hits + self.host_cache_hits
         resource = {
             "replicas": self.cfg.replicas,
             "kv_capacity_gb_per_replica": self.kv_capacity_gb,
             "peak_kv_gb": self.peak_kv_gb,
+            "peak_replica_kv_gb": self.peak_replica_kv_gb,
             "mean_kv_gb": mean_kv_gb,
             "hbm_gb_seconds": self.hbm_gb_seconds,
-            "cross_turn_cache_hits": self.cache_hits,
+            "peak_host_kv_gb": self.peak_host_gb,
+            "mean_host_kv_gb": self.host_gb_seconds / completion_horizon,
+            "host_gb_seconds": self.host_gb_seconds,
+            "cross_turn_cache_hits": total_reuse_hits,
+            "hbm_cache_hits": self.cache_hits,
+            "host_cache_hits": self.host_cache_hits,
             "cross_turn_cache_eligible": self.cache_eligible_turns,
-            "cross_turn_cache_hit_rate": self.cache_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
+            "cross_turn_cache_hit_rate": total_reuse_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
+            "hbm_cache_hit_rate": self.cache_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
+            "host_cache_hit_rate": self.host_cache_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
             "routing_locality_rate": self.affinity_routes / self.route_opportunities if self.route_opportunities else 0.0,
             "recomputed_history_tokens": self.recompute_tokens,
             "pressure_evictions": self.pressure_evictions,
             "ttl_evictions": self.ttl_evictions,
+            "host_pressure_evictions": self.host_pressure_evictions,
+            "offloaded_gb": self.offload_bytes / 1e9,
+            "restored_gb": self.restore_bytes / 1e9,
+            "p95_host_transfer_ms": host_restore_p95_ms,
             "tool_gap_total_s": self.tool_gap_total_s,
         }
         return {
@@ -492,6 +675,8 @@ class AgentSessionSimulator:
                 "mode": "stateful-agent-session-simulation",
                 "latency_profile_type": "analytical-reference",
                 "agent_service_model": "serial-per-replica-reference",
+                "host_tier_model": "serialized-reference-transfer",
+                "gap_aware_policy": "oracle-upper-bound" if self.cfg.retention_policy == "gap_aware" else "not-active",
                 "warning": "Agent-session mode isolates routing/KV-retention effects and does not model dynamic batching within each replica.",
             },
             "summary": summary,
@@ -620,3 +805,155 @@ def ttl_retention_sweep(config: dict[str, Any], ttl_values: list[float] | None =
         "rows": rows,
         "pareto_count": len(unique_front),
     }
+
+
+def compare_agent_memory_policies(config: dict[str, Any]) -> dict[str, Any]:
+    """Compare cache-residency and routing strategies on one common agent trace."""
+    base = AgentSessionConfig.from_dict(config)
+    trace = _same_trace(base)
+    policies = [
+        ("Stateless / least-load", "evict", "least_load"),
+        ("TTL / strict affinity", "ttl", "session_affinity"),
+        ("TTL / bounded affinity", "ttl", "bounded_affinity"),
+        ("Host offload / bounded affinity", "offload", "bounded_affinity"),
+        ("Gap-aware tiering / bounded affinity", "gap_aware", "bounded_affinity"),
+    ]
+    rows: list[dict[str, Any]] = []
+    results: list[dict[str, Any]] = []
+    for label, retention, routing in policies:
+        cfg = AgentSessionConfig.from_dict(base.to_dict())
+        cfg.retention_policy = retention
+        cfg.routing_policy = routing
+        result = run_agent_session_simulation(cfg.to_dict(), trace)
+        results.append(result)
+        rows.append(
+            {
+                "label": label,
+                "retention": retention,
+                "routing": routing,
+                "p95_turn_ttft_ms": result["latency"]["turn_ttft_ms"]["p95"],
+                "p95_session_e2e_ms": result["latency"]["session_e2e_ms"]["p95"],
+                "turn_slo_attainment": result["summary"]["turn_ttft_slo_attainment"],
+                "session_slo_attainment": result["summary"]["session_slo_attainment"],
+                "cache_hit_rate": result["resource"]["cross_turn_cache_hit_rate"],
+                "hbm_hit_rate": result["resource"]["hbm_cache_hit_rate"],
+                "host_hit_rate": result["resource"]["host_cache_hit_rate"],
+                "routing_locality_rate": result["resource"]["routing_locality_rate"],
+                "recomputed_history_tokens": result["resource"]["recomputed_history_tokens"],
+                "mean_hbm_gb": result["resource"]["mean_kv_gb"],
+                "mean_host_gb": result["resource"]["mean_host_kv_gb"],
+                "hbm_gb_seconds": result["resource"]["hbm_gb_seconds"],
+                "host_gb_seconds": result["resource"]["host_gb_seconds"],
+                "p95_host_transfer_ms": result["resource"]["p95_host_transfer_ms"],
+                "offloaded_gb": result["resource"]["offloaded_gb"],
+                "pressure_evictions": result["resource"]["pressure_evictions"],
+                "host_pressure_evictions": result["resource"]["host_pressure_evictions"],
+                "turns_failed": result["summary"]["turns_failed"],
+            }
+        )
+    return {
+        "protocol": "common-agent-program-trace",
+        "study": "agent-memory-tiering",
+        "candidate_count": len(rows),
+        "rows": rows,
+        "results": results,
+        "note": "Gap-aware tiering uses realized simulated tool gaps and is an oracle upper bound, not a deployable predictor.",
+    }
+
+
+def agent_memory_budget_sweep(
+    config: dict[str, Any], budget_multipliers: list[float] | None = None
+) -> dict[str, Any]:
+    """Stress state policies under finite per-replica HBM KV budgets.
+
+    Budgets are derived from the unconstrained peak working set of the exact same
+    program trace, avoiding arbitrary fractions of total GPU VRAM that would be
+    too loose for small models.
+    """
+    base = AgentSessionConfig.from_dict(config)
+    trace = _same_trace(base)
+
+    reference_cfg = AgentSessionConfig.from_dict(base.to_dict())
+    reference_cfg.retention_policy = "retain"
+    reference_cfg.routing_policy = "session_affinity"
+    reference_cfg.kv_capacity_override_gb = 0.0
+    reference = run_agent_session_simulation(reference_cfg.to_dict(), trace)
+    reference_peak = max(float(reference["resource"]["peak_replica_kv_gb"]), 0.002)
+
+    multipliers = budget_multipliers or [0.35, 0.50, 0.75, 1.00, 1.50]
+    cleaned = sorted({max(0.10, min(float(v), 3.0)) for v in multipliers})
+    policies = [
+        ("TTL / bounded affinity", "ttl", "bounded_affinity"),
+        ("Host offload / bounded affinity", "offload", "bounded_affinity"),
+        ("Gap-aware tiering / bounded affinity", "gap_aware", "bounded_affinity"),
+    ]
+
+    rows: list[dict[str, Any]] = []
+    for multiplier in cleaned:
+        budget = reference_peak * multiplier
+        for label, retention, routing in policies:
+            cfg = AgentSessionConfig.from_dict(base.to_dict())
+            cfg.retention_policy = retention
+            cfg.routing_policy = routing
+            cfg.kv_capacity_override_gb = budget
+            result = run_agent_session_simulation(cfg.to_dict(), trace)
+            rows.append(
+                {
+                    "policy": label,
+                    "budget_multiplier": multiplier,
+                    "budget_gb_per_replica": budget,
+                    "p95_turn_ttft_ms": result["latency"]["turn_ttft_ms"]["p95"],
+                    "p95_session_e2e_ms": result["latency"]["session_e2e_ms"]["p95"],
+                    "turn_slo_attainment": result["summary"]["turn_ttft_slo_attainment"],
+                    "session_slo_attainment": result["summary"]["session_slo_attainment"],
+                    "cache_hit_rate": result["resource"]["cross_turn_cache_hit_rate"],
+                    "recomputed_history_tokens": result["resource"]["recomputed_history_tokens"],
+                    "mean_hbm_gb": result["resource"]["mean_kv_gb"],
+                    "mean_host_gb": result["resource"]["mean_host_kv_gb"],
+                    "p95_host_transfer_ms": result["resource"]["p95_host_transfer_ms"],
+                    "pressure_evictions": result["resource"]["pressure_evictions"],
+                    "host_pressure_evictions": result["resource"]["host_pressure_evictions"],
+                    "turns_failed": result["summary"]["turns_failed"],
+                }
+            )
+    return {
+        "protocol": "common-agent-program-trace",
+        "study": "finite-hbm-budget-stress",
+        "reference_peak_replica_kv_gb": reference_peak,
+        "rows": rows,
+        "policy_count": len(policies),
+        "budget_count": len(cleaned),
+    }
+
+
+def agent_affinity_sweep(config: dict[str, Any], slack_values_ms: list[float] | None = None) -> dict[str, Any]:
+    """Sweep how much queue imbalance the router tolerates for KV locality."""
+    base = AgentSessionConfig.from_dict(config)
+    base.retention_policy = "ttl"
+    base.routing_policy = "bounded_affinity"
+    trace = _same_trace(base)
+    values = slack_values_ms or [0.0, 25.0, 75.0, 150.0, 300.0, 600.0, 1200.0]
+    cleaned = sorted({max(0.0, min(float(v), 5000.0)) for v in values})
+    rows: list[dict[str, Any]] = []
+    for slack in cleaned:
+        cfg = AgentSessionConfig.from_dict(base.to_dict())
+        cfg.affinity_slack_ms = slack
+        result = run_agent_session_simulation(cfg.to_dict(), trace)
+        rows.append(
+            {
+                "affinity_slack_ms": slack,
+                "p95_turn_ttft_ms": result["latency"]["turn_ttft_ms"]["p95"],
+                "p95_session_e2e_ms": result["latency"]["session_e2e_ms"]["p95"],
+                "cache_hit_rate": result["resource"]["cross_turn_cache_hit_rate"],
+                "routing_locality_rate": result["resource"]["routing_locality_rate"],
+                "recomputed_history_tokens": result["resource"]["recomputed_history_tokens"],
+                "turn_slo_attainment": result["summary"]["turn_ttft_slo_attainment"],
+                "session_slo_attainment": result["summary"]["session_slo_attainment"],
+                "pressure_evictions": result["resource"]["pressure_evictions"],
+            }
+        )
+    return {
+        "protocol": "common-agent-program-trace",
+        "study": "bounded-affinity-routing-frontier",
+        "rows": rows,
+    }

py/inferscale/api.py

@@ -1,6 +1,13 @@
 from __future__ import annotations
 
-from .agentic import compare_agent_policies, run_agent_session_simulation, ttl_retention_sweep
+from .agentic import (
+    agent_affinity_sweep,
+    agent_memory_budget_sweep,
+    compare_agent_memory_policies,
+    compare_agent_policies,
+    run_agent_session_simulation,
+    ttl_retention_sweep,
+)
 from .optimizer import capacity_search, compare_schedulers, compare_topologies, design_space_search
 from .profiles import ACCELERATORS, MODELS
 from .research import STUDIES, paired_study, robustness_study
@@ -15,7 +22,10 @@ def metadata() -> dict:
         "topologies": ["colocated", "disaggregated_pd"],
         "research_studies": STUDIES,
         "profile_type": "analytical-reference",
-        "agentic_modes": ["session_simulation", "policy_compare", "ttl_sweep"],
+        "agentic_modes": [
+            "session_simulation", "policy_compare", "ttl_sweep",
+            "memory_policy_compare", "memory_budget_sweep", "affinity_sweep",
+        ],
     }
 
 
@@ -55,6 +65,12 @@ def execute(action: str, payload: dict) -> dict:
         return compare_agent_policies(payload.get("config", payload))
     if action == "agent_ttl_sweep":
         return ttl_retention_sweep(payload.get("config", payload), payload.get("ttl_values"))
+    if action == "agent_memory_compare":
+        return compare_agent_memory_policies(payload.get("config", payload))
+    if action == "agent_memory_sweep":
+        return agent_memory_budget_sweep(payload.get("config", payload), payload.get("budget_multipliers"))
+    if action == "agent_affinity_sweep":
+        return agent_affinity_sweep(payload.get("config", payload), payload.get("slack_values_ms"))
     if action == "robustness_study":
         config = payload.get("config", payload)
         return robustness_study(

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "inferscale-sim"
-version = "0.5.0"
+version = "0.6.0"
 description = "Interactive LLM serving simulator and SLO capacity planner"
 readme = "README.md"
 requires-python = ">=3.10"

scripts/release_check.py

@@ -14,6 +14,9 @@ inferscale = importlib.import_module("inferscale")
 internal_version = inferscale.__version__
 design_space_search = inferscale.design_space_search
 compare_agent_policies = inferscale.compare_agent_policies
+compare_agent_memory_policies = inferscale.compare_agent_memory_policies
+agent_memory_budget_sweep = inferscale.agent_memory_budget_sweep
+agent_affinity_sweep = inferscale.agent_affinity_sweep
 paired_study = inferscale.paired_study
 robustness_study = inferscale.robustness_study
 run_agent_session_simulation = inferscale.run_agent_session_simulation
@@ -34,8 +37,8 @@ else:
 
 if "sdk: static" not in README:
     errors.append("README metadata must use sdk: static")
-if internal_version != "0.5.0":
-    errors.append(f"internal package version is {internal_version}; expected 0.5.0")
+if internal_version != "0.6.0":
+    errors.append(f"internal package version is {internal_version}; expected 0.6.0")
 
 # Public-facing release/version branding is intentionally absent. Model names
 # such as Mistral-7B-v0.3 are allowed; project headings/badges are not.
@@ -74,7 +77,7 @@ if "Download PNG" not in index or ".chart-download" not in app:
     errors.append("chart PNG export controls are missing")
 if "Worst repetition" not in index or "Target" not in index:
     errors.append("capacity evidence columns are missing")
-for expected in ["Trace replay", "Research Studies", "Run paired study", "Stress-test selected hypothesis", "Agent Sessions", "Session Policy Arena", "TTL frontier"]:
+for expected in ["Trace replay", "Research Studies", "Run paired study", "Stress-test selected hypothesis", "Agent Sessions", "Session Policy Arena", "TTL frontier", "Agent Memory Lab", "Affinity Frontier", "Stress HBM budget"]:
     if expected not in index:
         errors.append(f"UI is missing research/trace feature: {expected}")
 
@@ -185,6 +188,15 @@ try:
     agent_ttl = ttl_retention_sweep(agent_cfg, [0, 1, 3])
     if len(agent_ttl.get("rows", [])) != 3 or agent_ttl.get("pareto_count", 0) < 1:
         errors.append("agent TTL frontier smoke test is incomplete")
+    agent_memory = compare_agent_memory_policies(agent_cfg | {"host_memory_gb": 4})
+    if agent_memory.get("candidate_count") != 5 or not any(row.get("host_hit_rate", 0) > 0 for row in agent_memory.get("rows", [])):
+        errors.append("agent tiered-memory comparison is incomplete")
+    agent_budget = agent_memory_budget_sweep(agent_cfg | {"host_memory_gb": 4}, [0.5, 1.0])
+    if len(agent_budget.get("rows", [])) != 6:
+        errors.append("finite HBM budget study is incomplete")
+    agent_affinity = agent_affinity_sweep(agent_cfg, [0, 150, 600])
+    if len(agent_affinity.get("rows", [])) != 3:
+        errors.append("bounded-affinity sweep is incomplete")
 except Exception as exc:  # pragma: no cover
     errors.append(f"agent-session smoke test raised: {exc}")
 
@@ -217,5 +229,8 @@ print(f"Robustness perturbations: {len(robust['rows'])}")
 print(f"Agent turns: {agent['summary']['turns_completed']}")
 print(f"Agent policy candidates: {agent_compare['candidate_count']}")
 print(f"Agent TTL candidates: {len(agent_ttl['rows'])}")
+print(f"Agent memory policies: {agent_memory['candidate_count']}")
+print(f"HBM budget study rows: {len(agent_budget['rows'])}")
+print(f"Affinity sweep points: {len(agent_affinity['rows'])}")
 print(f"Validation observations: {validation['observation_count']}")
 print(f"Profile provenance: {smoke['provenance']['latency_profile_type']}")

src/inferscale/__init__.py

@@ -1,4 +1,11 @@
-from .agentic import compare_agent_policies, run_agent_session_simulation, ttl_retention_sweep
+from .agentic import (
+    agent_affinity_sweep,
+    agent_memory_budget_sweep,
+    compare_agent_memory_policies,
+    compare_agent_policies,
+    run_agent_session_simulation,
+    ttl_retention_sweep,
+)
 from .api import execute, metadata
 from .models import SimulationConfig
 from .optimizer import capacity_search, compare_schedulers, compare_topologies, design_space_search
@@ -8,6 +15,9 @@ from .validation import validate_cases
 
 __all__ = [
     "SimulationConfig",
+    "agent_affinity_sweep",
+    "agent_memory_budget_sweep",
+    "compare_agent_memory_policies",
     "compare_agent_policies",
     "capacity_search",
     "compare_schedulers",
@@ -25,4 +35,4 @@ __all__ = [
 
 # Internal package metadata only; the public project intentionally avoids
 # release/version branding in the interface and documentation.
-__version__ = "0.5.0"
+__version__ = "0.6.0"

src/inferscale/agentic.py

@@ -12,8 +12,8 @@ from .metrics import percentile
 from .profiles import get_accelerator, get_model
 
 
-ROUTING_POLICIES = {"least_load", "session_affinity"}
-RETENTION_POLICIES = {"evict", "retain", "ttl"}
+ROUTING_POLICIES = {"least_load", "session_affinity", "bounded_affinity"}
+RETENTION_POLICIES = {"evict", "retain", "ttl", "offload", "gap_aware"}
 
 
 @dataclass
@@ -38,6 +38,12 @@ class AgentSessionConfig:
     kv_ttl_s: float = 3.0
     routing_policy: str = "session_affinity"
     kv_memory_fraction: float = 0.85
+    kv_capacity_override_gb: float = 0.0
+    host_memory_gb: float = 32.0
+    host_bandwidth_gbps: float = 32.0
+    host_transfer_base_ms: float = 0.15
+    affinity_slack_ms: float = 150.0
+    gap_aware_threshold_s: float = 1.5
     slo_turn_ttft_ms: float = 500.0
     slo_session_e2e_ms: float = 30000.0
     timeline_points: int = 240
@@ -54,7 +60,13 @@ class AgentSessionConfig:
         cfg.session_rate_rps = max(float(cfg.session_rate_rps), 0.01)
         cfg.duration_s = max(float(cfg.duration_s), 1.0)
         cfg.kv_ttl_s = max(float(cfg.kv_ttl_s), 0.0)
-        cfg.kv_memory_fraction = min(max(float(cfg.kv_memory_fraction), 0.05), 0.98)
+        cfg.kv_memory_fraction = min(max(float(cfg.kv_memory_fraction), 0.01), 0.98)
+        cfg.kv_capacity_override_gb = max(float(cfg.kv_capacity_override_gb), 0.0)
+        cfg.host_memory_gb = max(float(cfg.host_memory_gb), 0.0)
+        cfg.host_bandwidth_gbps = max(float(cfg.host_bandwidth_gbps), 0.1)
+        cfg.host_transfer_base_ms = max(float(cfg.host_transfer_base_ms), 0.0)
+        cfg.affinity_slack_ms = max(float(cfg.affinity_slack_ms), 0.0)
+        cfg.gap_aware_threshold_s = max(float(cfg.gap_aware_threshold_s), 0.0)
         return cfg
 
     def to_dict(self) -> dict[str, Any]:
@@ -86,6 +98,8 @@ class CacheEntry:
     last_access: float
     expiry_time: float | None
     generation: int
+    source_replica: int | None = None
+    available_time: float = 0.0
 
 
 @dataclass
@@ -175,8 +189,14 @@ class AgentSessionSimulator:
         self.accelerator = get_accelerator(cfg.accelerator)
         self.latency = AnalyticalLatencyModel(self.model, self.accelerator, cfg.quantization)
         remaining = max(0.0, self.accelerator.vram_gb - self.latency.model_weight_gb - 1.2)
-        self.kv_capacity_gb = remaining * cfg.kv_memory_fraction
+        automatic_capacity = remaining * cfg.kv_memory_fraction
+        self.kv_capacity_gb = (
+            min(automatic_capacity, cfg.kv_capacity_override_gb)
+            if cfg.kv_capacity_override_gb > 0.0
+            else automatic_capacity
+        )
         self.replicas = [ReplicaState(i) for i in range(cfg.replicas)]
+        self.host_cache: dict[int, CacheEntry] = {}
         specs = sessions if sessions is not None else generate_agent_sessions(cfg)
         self.sessions = {s.session_id: SessionRuntime(s, s.initial_prompt_tokens) for s in specs}
         self.events: list[tuple[float, int, str, tuple[Any, ...]]] = []
@@ -185,10 +205,18 @@ class AgentSessionSimulator:
         self.turn_rows: list[dict[str, Any]] = []
         self.timeline: list[dict[str, Any]] = []
         self.peak_kv_gb = 0.0
+        self.peak_replica_kv_gb = 0.0
         self.hbm_gb_seconds = 0.0
+        self.host_gb_seconds = 0.0
+        self.peak_host_gb = 0.0
         self.last_memory_time = 0.0
         self.pressure_evictions = 0
         self.ttl_evictions = 0
+        self.host_pressure_evictions = 0
+        self.host_cache_hits = 0
+        self.offload_bytes = 0.0
+        self.restore_bytes = 0.0
+        self.host_transfer_latencies_ms: list[float] = []
         self.recompute_tokens = 0
         self.cache_hits = 0
         self.cache_eligible_turns = 0
@@ -207,12 +235,23 @@ class AgentSessionSimulator:
     def _used_gb(self) -> float:
         return sum(entry.size_gb for replica in self.replicas for entry in replica.cache.values())
 
+    def _used_host_gb(self) -> float:
+        return sum(entry.size_gb for entry in self.host_cache.values())
+
     def _integrate_memory(self, new_time: float) -> None:
         new_time = max(float(new_time), self.last_memory_time)
+        elapsed = new_time - self.last_memory_time
         used = self._used_gb()
-        self.hbm_gb_seconds += used * (new_time - self.last_memory_time)
+        host_used = self._used_host_gb()
+        self.hbm_gb_seconds += used * elapsed
+        self.host_gb_seconds += host_used * elapsed
         self.last_memory_time = new_time
         self.peak_kv_gb = max(self.peak_kv_gb, used)
+        self.peak_replica_kv_gb = max(
+            self.peak_replica_kv_gb,
+            max((sum(entry.size_gb for entry in replica.cache.values()) for replica in self.replicas), default=0.0),
+        )
+        self.peak_host_gb = max(self.peak_host_gb, host_used)
 
     def _record_timeline(self) -> None:
         max_points = max(int(self.cfg.timeline_points), 40)
@@ -224,7 +263,9 @@ class AgentSessionSimulator:
                 "queued_turns": sum(len(r.queue) for r in self.replicas),
                 "busy_replicas": sum(1 for r in self.replicas if r.busy),
                 "kv_used_gb": self._used_gb(),
+                "host_kv_gb": self._used_host_gb(),
                 "resident_sessions": sum(len(r.cache) for r in self.replicas),
+                "host_sessions": len(self.host_cache),
             }
         )
         if len(self.timeline) > max_points * 2:
@@ -239,15 +280,28 @@ class AgentSessionSimulator:
                 return replica
         return None
 
+    def _replica_load_key(self, replica: ReplicaState) -> tuple[int, float, int]:
+        return (len(replica.queue), max(replica.busy_until, self.now), replica.replica_id)
+
     def _route(self, session_id: int) -> ReplicaState:
         cached = self._cache_replica(session_id)
         if cached is not None:
             self.route_opportunities += 1
+        least_loaded = min(self.replicas, key=self._replica_load_key)
         if self.cfg.routing_policy == "session_affinity" and cached is not None:
             self.affinity_routes += 1
             return cached
-        # Approximate least-finish-time routing using queued work count then busy horizon.
-        return min(self.replicas, key=lambda r: (len(r.queue), max(r.busy_until, self.now), r.replica_id))
+        if self.cfg.routing_policy == "bounded_affinity" and cached is not None:
+            cached_ready = max(cached.busy_until, self.now)
+            best_ready = max(least_loaded.busy_until, self.now)
+            queue_gap = max(0, len(cached.queue) - len(least_loaded.queue))
+            # Queue length is converted to a modest reference penalty because
+            # exact queued service time is intentionally not known at routing time.
+            estimated_extra_ms = max(0.0, cached_ready - best_ready) * 1000.0 + queue_gap * 75.0
+            if estimated_extra_ms <= self.cfg.affinity_slack_ms:
+                self.affinity_routes += 1
+                return cached
+        return least_loaded
 
     def _remove_cache(self, replica: ReplicaState, session_id: int, reason: str) -> None:
         if session_id not in replica.cache:
@@ -260,6 +314,73 @@ class AgentSessionSimulator:
             self.ttl_evictions += 1
         self._record_timeline()
 
+    def _host_transfer_seconds(self, size_gb: float) -> float:
+        return self.cfg.host_transfer_base_ms / 1000.0 + size_gb / self.cfg.host_bandwidth_gbps
+
+    def _remove_host_cache(self, session_id: int) -> None:
+        if session_id not in self.host_cache:
+            return
+        self._integrate_memory(self.now)
+        del self.host_cache[session_id]
+        self._record_timeline()
+
+    def _ensure_host_capacity(self, session_id: int, target_gb: float) -> bool:
+        if target_gb > self.cfg.host_memory_gb + 1e-12:
+            return False
+        while self._used_host_gb() + target_gb > self.cfg.host_memory_gb + 1e-12:
+            victims = [entry for sid, entry in self.host_cache.items() if sid != session_id]
+            if not victims:
+                return False
+            victim = min(victims, key=lambda e: (e.last_access, e.session_id))
+            self._integrate_memory(self.now)
+            del self.host_cache[victim.session_id]
+            self.host_pressure_evictions += 1
+        return True
+
+    def _offload_cache(self, replica: ReplicaState, session_id: int, tokens: int) -> bool:
+        size_gb = tokens * self.latency.kv_bytes_per_token() / 1e9
+        if self.cfg.host_memory_gb <= 0.0 or not self._ensure_host_capacity(session_id, size_gb):
+            self._remove_cache(replica, session_id, "complete")
+            return False
+        transfer_s = self._host_transfer_seconds(size_gb)
+        self._integrate_memory(self.now)
+        replica.cache.pop(session_id, None)
+        self.host_cache[session_id] = CacheEntry(
+            session_id=session_id,
+            tokens=tokens,
+            size_gb=size_gb,
+            last_access=self.now,
+            expiry_time=None,
+            generation=1,
+            source_replica=replica.replica_id,
+            available_time=self.now + transfer_s,
+        )
+        self.offload_bytes += size_gb * 1e9
+        self.host_transfer_latencies_ms.append(transfer_s * 1000.0)
+        self.peak_host_gb = max(self.peak_host_gb, self._used_host_gb())
+        self._record_timeline()
+        return True
+
+    def _restore_host_entry(self, replica: ReplicaState, session_id: int) -> tuple[bool, float]:
+        entry = self.host_cache.get(session_id)
+        if entry is None:
+            return False, 0.0
+        if not self._ensure_capacity(replica, session_id, entry.size_gb):
+            return False, 0.0
+        wait_s = max(0.0, entry.available_time - self.now)
+        copy_s = self._host_transfer_seconds(entry.size_gb)
+        # The host entry remains resident while an unfinished offload or restore
+        # is exposed. Account for that residency explicitly because the event
+        # clock advances only when the turn completes.
+        self.host_gb_seconds += entry.size_gb * (wait_s + copy_s)
+        self._integrate_memory(self.now)
+        del self.host_cache[session_id]
+        self.restore_bytes += entry.size_gb * 1e9
+        self.host_transfer_latencies_ms.append((wait_s + copy_s) * 1000.0)
+        self.host_cache_hits += 1
+        self._record_timeline()
+        return True, wait_s + copy_s
+
     def _ensure_capacity(self, replica: ReplicaState, session_id: int, target_gb: float) -> bool:
         current = replica.cache.get(session_id)
         current_gb = current.size_gb if current else 0.0
@@ -289,6 +410,10 @@ class AgentSessionSimulator:
             expiry = self.now + self.cfg.kv_ttl_s
         replica.cache[session_id] = CacheEntry(session_id, tokens, size_gb, self.now, expiry, generation)
         self.peak_kv_gb = max(self.peak_kv_gb, self._used_gb())
+        self.peak_replica_kv_gb = max(
+            self.peak_replica_kv_gb,
+            sum(entry.size_gb for entry in replica.cache.values()),
+        )
         if expiry is not None:
             self._push(expiry, "cache_expire", replica.replica_id, session_id, generation)
         self._record_timeline()
@@ -300,16 +425,29 @@ class AgentSessionSimulator:
         session_id, turn, ready_time = replica.queue.pop(0)
         runtime = self.sessions[session_id]
         cache = self._entry(replica, session_id)
-        hit = turn.turn_index > 0 and cache is not None
+        hbm_hit = turn.turn_index > 0 and cache is not None
+        host_hit = False
+        restore_s = 0.0
         if turn.turn_index > 0:
             self.cache_eligible_turns += 1
-        if hit:
+        if hbm_hit:
             self.cache_hits += 1
             prefill_tokens = max(turn.append_tokens, 1)
+            cache_source = "hbm"
+        elif turn.turn_index > 0 and session_id in self.host_cache:
+            host_hit, restore_s = self._restore_host_entry(replica, session_id)
+            if host_hit:
+                prefill_tokens = max(turn.append_tokens, 1)
+                cache_source = "host"
+            else:
+                prefill_tokens = max(runtime.context_tokens + turn.append_tokens, 1)
+                self.recompute_tokens += runtime.context_tokens
+                cache_source = "miss"
         else:
             prefill_tokens = max(runtime.context_tokens + turn.append_tokens, 1)
             if turn.turn_index > 0:
                 self.recompute_tokens += runtime.context_tokens
+            cache_source = "miss"
 
         context_before_decode = runtime.context_tokens + turn.append_tokens
         projected_tokens = context_before_decode + turn.output_tokens
@@ -332,8 +470,8 @@ class AgentSessionSimulator:
         first_decode_s = self.latency.decode_step_seconds([context_before_decode])
         midpoint = context_before_decode + max(turn.output_tokens // 2, 1)
         decode_step_s = self.latency.decode_step_seconds([midpoint])
-        service_s = prefill_s + turn.output_tokens * decode_step_s
-        ttft_s = (self.now - ready_time) + prefill_s + first_decode_s
+        service_s = restore_s + prefill_s + turn.output_tokens * decode_step_s
+        ttft_s = (self.now - ready_time) + restore_s + prefill_s + first_decode_s
 
         replica.busy = True
         replica.current_session = session_id
@@ -345,7 +483,9 @@ class AgentSessionSimulator:
             turn.turn_index,
             replica.replica_id,
             ready_time,
-            hit,
+            hbm_hit or host_hit,
+            cache_source,
+            restore_s,
             prefill_tokens,
             ttft_s,
             service_s,
@@ -359,6 +499,8 @@ class AgentSessionSimulator:
         replica_id: int,
         ready_time: float,
         cache_hit: bool,
+        cache_source: str,
+        restore_s: float,
         prefill_tokens: int,
         ttft_s: float,
         service_s: float,
@@ -375,11 +517,31 @@ class AgentSessionSimulator:
         replica.busy_until = self.now
 
         is_final = turn_index == len(runtime.spec.turns) - 1
-        keep = (not is_final) and self.cfg.retention_policy != "evict"
-        if keep:
-            self._put_cache(replica, session_id, projected_tokens, keep=True)
-        else:
+        state_action = "evict"
+        if is_final:
+            self._remove_cache(replica, session_id, "complete")
+            self._remove_host_cache(session_id)
+        elif self.cfg.retention_policy == "evict":
             self._remove_cache(replica, session_id, "complete")
+            self._remove_host_cache(session_id)
+        elif self.cfg.retention_policy in {"retain", "ttl"}:
+            self._put_cache(replica, session_id, projected_tokens, keep=True)
+            state_action = "retain_hbm"
+        elif self.cfg.retention_policy == "offload":
+            if self._offload_cache(replica, session_id, projected_tokens):
+                state_action = "offload_host"
+            else:
+                state_action = "evict_host_full"
+        elif self.cfg.retention_policy == "gap_aware":
+            # This is intentionally an oracle upper-bound policy: the simulated
+            # tool gap is already known from the generated program trace.
+            if turn.tool_gap_after_s <= self.cfg.gap_aware_threshold_s:
+                self._put_cache(replica, session_id, projected_tokens, keep=True)
+                state_action = "retain_hbm_short_gap"
+            elif self._offload_cache(replica, session_id, projected_tokens):
+                state_action = "offload_host_long_gap"
+            else:
+                state_action = "evict_host_full"
 
         e2e_ms = (self.now - ready_time) * 1000.0
         self.turn_rows.append(
@@ -390,6 +552,9 @@ class AgentSessionSimulator:
                 "ready_time": ready_time,
                 "completion_time": self.now,
                 "cache_hit": cache_hit,
+                "cache_source": cache_source,
+                "restore_ms": restore_s * 1000.0,
+                "state_action": state_action,
                 "prefill_tokens": prefill_tokens,
                 "context_tokens_after": projected_tokens,
                 "output_tokens": turn.output_tokens,
@@ -421,7 +586,7 @@ class AgentSessionSimulator:
             elif kind == "turn_complete":
                 self._finish_turn(
                     int(payload[0]), int(payload[1]), int(payload[2]), float(payload[3]), bool(payload[4]),
-                    int(payload[5]), float(payload[6]), float(payload[7]), int(payload[8]),
+                    str(payload[5]), float(payload[6]), int(payload[7]), float(payload[8]), float(payload[9]), int(payload[10]),
                 )
             elif kind == "cache_expire":
                 replica_id, session_id, generation = map(int, payload)
@@ -452,17 +617,21 @@ class AgentSessionSimulator:
         ]
         session_slo = sum(1 for v in session_e2e if v <= self.cfg.slo_session_e2e_ms)
         turn_slo = sum(1 for v in ttfts if v <= self.cfg.slo_turn_ttft_ms)
+        turns_generated = sum(len(s.spec.turns) for s in self.sessions.values())
         completion_horizon = max([s.completion_time or 0.0 for s in self.sessions.values()] + [self.cfg.duration_s, 1e-9])
         mean_kv_gb = self.hbm_gb_seconds / completion_horizon
         summary = {
             "sessions_generated": len(self.sessions),
             "sessions_completed": len(completed_sessions),
+            "turns_generated": turns_generated,
             "turns_completed": len(successful_turns),
             "turns_failed": self.failed_turns,
+            "session_completion_rate": len(completed_sessions) / len(self.sessions) if self.sessions else 0.0,
+            "turn_completion_rate": len(successful_turns) / turns_generated if turns_generated else 0.0,
             "session_throughput_rps": len(completed_sessions) / completion_horizon,
             "turn_throughput_rps": len(successful_turns) / completion_horizon,
-            "turn_ttft_slo_attainment": turn_slo / len(successful_turns) if successful_turns else 0.0,
-            "session_slo_attainment": session_slo / len(completed_sessions) if completed_sessions else 0.0,
+            "turn_ttft_slo_attainment": turn_slo / turns_generated if turns_generated else 0.0,
+            "session_slo_attainment": session_slo / len(self.sessions) if self.sessions else 0.0,
             "simulated_makespan_s": completion_horizon,
         }
         latency = {
@@ -470,19 +639,33 @@ class AgentSessionSimulator:
             "turn_e2e_ms": {"p50": percentile(turn_e2e, 0.50), "p95": percentile(turn_e2e, 0.95), "p99": percentile(turn_e2e, 0.99)},
             "session_e2e_ms": {"p50": percentile(session_e2e, 0.50), "p95": percentile(session_e2e, 0.95), "p99": percentile(session_e2e, 0.99)},
         }
+        host_restore_p95_ms = percentile(self.host_transfer_latencies_ms, 0.95)
+        total_reuse_hits = self.cache_hits + self.host_cache_hits
         resource = {
             "replicas": self.cfg.replicas,
             "kv_capacity_gb_per_replica": self.kv_capacity_gb,
             "peak_kv_gb": self.peak_kv_gb,
+            "peak_replica_kv_gb": self.peak_replica_kv_gb,
             "mean_kv_gb": mean_kv_gb,
             "hbm_gb_seconds": self.hbm_gb_seconds,
-            "cross_turn_cache_hits": self.cache_hits,
+            "peak_host_kv_gb": self.peak_host_gb,
+            "mean_host_kv_gb": self.host_gb_seconds / completion_horizon,
+            "host_gb_seconds": self.host_gb_seconds,
+            "cross_turn_cache_hits": total_reuse_hits,
+            "hbm_cache_hits": self.cache_hits,
+            "host_cache_hits": self.host_cache_hits,
             "cross_turn_cache_eligible": self.cache_eligible_turns,
-            "cross_turn_cache_hit_rate": self.cache_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
+            "cross_turn_cache_hit_rate": total_reuse_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
+            "hbm_cache_hit_rate": self.cache_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
+            "host_cache_hit_rate": self.host_cache_hits / self.cache_eligible_turns if self.cache_eligible_turns else 0.0,
             "routing_locality_rate": self.affinity_routes / self.route_opportunities if self.route_opportunities else 0.0,
             "recomputed_history_tokens": self.recompute_tokens,
             "pressure_evictions": self.pressure_evictions,
             "ttl_evictions": self.ttl_evictions,
+            "host_pressure_evictions": self.host_pressure_evictions,
+            "offloaded_gb": self.offload_bytes / 1e9,
+            "restored_gb": self.restore_bytes / 1e9,
+            "p95_host_transfer_ms": host_restore_p95_ms,
             "tool_gap_total_s": self.tool_gap_total_s,
         }
         return {
@@ -492,6 +675,8 @@ class AgentSessionSimulator:
                 "mode": "stateful-agent-session-simulation",
                 "latency_profile_type": "analytical-reference",
                 "agent_service_model": "serial-per-replica-reference",
+                "host_tier_model": "serialized-reference-transfer",
+                "gap_aware_policy": "oracle-upper-bound" if self.cfg.retention_policy == "gap_aware" else "not-active",
                 "warning": "Agent-session mode isolates routing/KV-retention effects and does not model dynamic batching within each replica.",
             },
             "summary": summary,
@@ -620,3 +805,155 @@ def ttl_retention_sweep(config: dict[str, Any], ttl_values: list[float] | None =
         "rows": rows,
         "pareto_count": len(unique_front),
     }
+
+
+def compare_agent_memory_policies(config: dict[str, Any]) -> dict[str, Any]:
+    """Compare cache-residency and routing strategies on one common agent trace."""
+    base = AgentSessionConfig.from_dict(config)
+    trace = _same_trace(base)
+    policies = [
+        ("Stateless / least-load", "evict", "least_load"),
+        ("TTL / strict affinity", "ttl", "session_affinity"),
+        ("TTL / bounded affinity", "ttl", "bounded_affinity"),
+        ("Host offload / bounded affinity", "offload", "bounded_affinity"),
+        ("Gap-aware tiering / bounded affinity", "gap_aware", "bounded_affinity"),
+    ]
+    rows: list[dict[str, Any]] = []
+    results: list[dict[str, Any]] = []
+    for label, retention, routing in policies:
+        cfg = AgentSessionConfig.from_dict(base.to_dict())
+        cfg.retention_policy = retention
+        cfg.routing_policy = routing
+        result = run_agent_session_simulation(cfg.to_dict(), trace)
+        results.append(result)
+        rows.append(
+            {
+                "label": label,
+                "retention": retention,
+                "routing": routing,
+                "p95_turn_ttft_ms": result["latency"]["turn_ttft_ms"]["p95"],
+                "p95_session_e2e_ms": result["latency"]["session_e2e_ms"]["p95"],
+                "turn_slo_attainment": result["summary"]["turn_ttft_slo_attainment"],
+                "session_slo_attainment": result["summary"]["session_slo_attainment"],
+                "cache_hit_rate": result["resource"]["cross_turn_cache_hit_rate"],
+                "hbm_hit_rate": result["resource"]["hbm_cache_hit_rate"],
+                "host_hit_rate": result["resource"]["host_cache_hit_rate"],
+                "routing_locality_rate": result["resource"]["routing_locality_rate"],
+                "recomputed_history_tokens": result["resource"]["recomputed_history_tokens"],
+                "mean_hbm_gb": result["resource"]["mean_kv_gb"],
+                "mean_host_gb": result["resource"]["mean_host_kv_gb"],
+                "hbm_gb_seconds": result["resource"]["hbm_gb_seconds"],
+                "host_gb_seconds": result["resource"]["host_gb_seconds"],
+                "p95_host_transfer_ms": result["resource"]["p95_host_transfer_ms"],
+                "offloaded_gb": result["resource"]["offloaded_gb"],
+                "pressure_evictions": result["resource"]["pressure_evictions"],
+                "host_pressure_evictions": result["resource"]["host_pressure_evictions"],
+                "turns_failed": result["summary"]["turns_failed"],
+            }
+        )
+    return {
+        "protocol": "common-agent-program-trace",
+        "study": "agent-memory-tiering",
+        "candidate_count": len(rows),
+        "rows": rows,
+        "results": results,
+        "note": "Gap-aware tiering uses realized simulated tool gaps and is an oracle upper bound, not a deployable predictor.",
+    }
+
+
+def agent_memory_budget_sweep(
+    config: dict[str, Any], budget_multipliers: list[float] | None = None
+) -> dict[str, Any]:
+    """Stress state policies under finite per-replica HBM KV budgets.
+
+    Budgets are derived from the unconstrained peak working set of the exact same
+    program trace, avoiding arbitrary fractions of total GPU VRAM that would be
+    too loose for small models.
+    """
+    base = AgentSessionConfig.from_dict(config)
+    trace = _same_trace(base)
+
+    reference_cfg = AgentSessionConfig.from_dict(base.to_dict())
+    reference_cfg.retention_policy = "retain"
+    reference_cfg.routing_policy = "session_affinity"
+    reference_cfg.kv_capacity_override_gb = 0.0
+    reference = run_agent_session_simulation(reference_cfg.to_dict(), trace)
+    reference_peak = max(float(reference["resource"]["peak_replica_kv_gb"]), 0.002)
+
+    multipliers = budget_multipliers or [0.35, 0.50, 0.75, 1.00, 1.50]
+    cleaned = sorted({max(0.10, min(float(v), 3.0)) for v in multipliers})
+    policies = [
+        ("TTL / bounded affinity", "ttl", "bounded_affinity"),
+        ("Host offload / bounded affinity", "offload", "bounded_affinity"),
+        ("Gap-aware tiering / bounded affinity", "gap_aware", "bounded_affinity"),
+    ]
+
+    rows: list[dict[str, Any]] = []
+    for multiplier in cleaned:
+        budget = reference_peak * multiplier
+        for label, retention, routing in policies:
+            cfg = AgentSessionConfig.from_dict(base.to_dict())
+            cfg.retention_policy = retention
+            cfg.routing_policy = routing
+            cfg.kv_capacity_override_gb = budget
+            result = run_agent_session_simulation(cfg.to_dict(), trace)
+            rows.append(
+                {
+                    "policy": label,
+                    "budget_multiplier": multiplier,
+                    "budget_gb_per_replica": budget,
+                    "p95_turn_ttft_ms": result["latency"]["turn_ttft_ms"]["p95"],
+                    "p95_session_e2e_ms": result["latency"]["session_e2e_ms"]["p95"],
+                    "turn_slo_attainment": result["summary"]["turn_ttft_slo_attainment"],
+                    "session_slo_attainment": result["summary"]["session_slo_attainment"],
+                    "cache_hit_rate": result["resource"]["cross_turn_cache_hit_rate"],
+                    "recomputed_history_tokens": result["resource"]["recomputed_history_tokens"],
+                    "mean_hbm_gb": result["resource"]["mean_kv_gb"],
+                    "mean_host_gb": result["resource"]["mean_host_kv_gb"],
+                    "p95_host_transfer_ms": result["resource"]["p95_host_transfer_ms"],
+                    "pressure_evictions": result["resource"]["pressure_evictions"],
+                    "host_pressure_evictions": result["resource"]["host_pressure_evictions"],
+                    "turns_failed": result["summary"]["turns_failed"],
+                }
+            )
+    return {
+        "protocol": "common-agent-program-trace",
+        "study": "finite-hbm-budget-stress",
+        "reference_peak_replica_kv_gb": reference_peak,
+        "rows": rows,
+        "policy_count": len(policies),
+        "budget_count": len(cleaned),
+    }
+
+
+def agent_affinity_sweep(config: dict[str, Any], slack_values_ms: list[float] | None = None) -> dict[str, Any]:
+    """Sweep how much queue imbalance the router tolerates for KV locality."""
+    base = AgentSessionConfig.from_dict(config)
+    base.retention_policy = "ttl"
+    base.routing_policy = "bounded_affinity"
+    trace = _same_trace(base)
+    values = slack_values_ms or [0.0, 25.0, 75.0, 150.0, 300.0, 600.0, 1200.0]
+    cleaned = sorted({max(0.0, min(float(v), 5000.0)) for v in values})
+    rows: list[dict[str, Any]] = []
+    for slack in cleaned:
+        cfg = AgentSessionConfig.from_dict(base.to_dict())
+        cfg.affinity_slack_ms = slack
+        result = run_agent_session_simulation(cfg.to_dict(), trace)
+        rows.append(
+            {
+                "affinity_slack_ms": slack,
+                "p95_turn_ttft_ms": result["latency"]["turn_ttft_ms"]["p95"],
+                "p95_session_e2e_ms": result["latency"]["session_e2e_ms"]["p95"],
+                "cache_hit_rate": result["resource"]["cross_turn_cache_hit_rate"],
+                "routing_locality_rate": result["resource"]["routing_locality_rate"],
+                "recomputed_history_tokens": result["resource"]["recomputed_history_tokens"],
+                "turn_slo_attainment": result["summary"]["turn_ttft_slo_attainment"],
+                "session_slo_attainment": result["summary"]["session_slo_attainment"],
+                "pressure_evictions": result["resource"]["pressure_evictions"],
+            }
+        )
+    return {
+        "protocol": "common-agent-program-trace",
+        "study": "bounded-affinity-routing-frontier",
+        "rows": rows,
+    }

src/inferscale/api.py

@@ -1,6 +1,13 @@
 from __future__ import annotations
 
-from .agentic import compare_agent_policies, run_agent_session_simulation, ttl_retention_sweep
+from .agentic import (
+    agent_affinity_sweep,
+    agent_memory_budget_sweep,
+    compare_agent_memory_policies,
+    compare_agent_policies,
+    run_agent_session_simulation,
+    ttl_retention_sweep,
+)
 from .optimizer import capacity_search, compare_schedulers, compare_topologies, design_space_search
 from .profiles import ACCELERATORS, MODELS
 from .research import STUDIES, paired_study, robustness_study
@@ -15,7 +22,10 @@ def metadata() -> dict:
         "topologies": ["colocated", "disaggregated_pd"],
         "research_studies": STUDIES,
         "profile_type": "analytical-reference",
-        "agentic_modes": ["session_simulation", "policy_compare", "ttl_sweep"],
+        "agentic_modes": [
+            "session_simulation", "policy_compare", "ttl_sweep",
+            "memory_policy_compare", "memory_budget_sweep", "affinity_sweep",
+        ],
     }
 
 
@@ -55,6 +65,12 @@ def execute(action: str, payload: dict) -> dict:
         return compare_agent_policies(payload.get("config", payload))
     if action == "agent_ttl_sweep":
         return ttl_retention_sweep(payload.get("config", payload), payload.get("ttl_values"))
+    if action == "agent_memory_compare":
+        return compare_agent_memory_policies(payload.get("config", payload))
+    if action == "agent_memory_sweep":
+        return agent_memory_budget_sweep(payload.get("config", payload), payload.get("budget_multipliers"))
+    if action == "agent_affinity_sweep":
+        return agent_affinity_sweep(payload.get("config", payload), payload.get("slack_values_ms"))
     if action == "robustness_study":
         config = payload.get("config", payload)
         return robustness_study(

styles.css

@@ -322,3 +322,9 @@ input[type="file"]::file-selector-button {
 @media (max-width: 1100px) { .agent-layout { grid-template-columns: 1fr; } .agent-controls { position: static; } }
 @media (max-width: 720px) { .metric-grid.six-agent { grid-template-columns: 1fr 1fr; } }
 .field-disabled { opacity: .55; }
+.metric-grid.eight-agent { grid-template-columns: repeat(4, minmax(0, 1fr)); }
+.stacked-actions { display: flex; gap: 8px; flex-wrap: wrap; justify-content: flex-end; }
+.stacked-actions .compact { min-width: 168px; }
+@media (max-width: 1250px) { .metric-grid.eight-agent { grid-template-columns: repeat(2, 1fr); } }
+@media (max-width: 720px) { .metric-grid.eight-agent { grid-template-columns: 1fr 1fr; } .stacked-actions { width: 100%; } .stacked-actions .compact { width: 100% !important; } }
+.experiment-subsection { margin-top: 18px; padding-top: 18px; border-top: 1px solid var(--line); }

tests/test_agentic.py

@@ -56,3 +56,49 @@ def test_ttl_sweep_reports_latency_memory_frontier():
     assert result["pareto_count"] >= 1
     assert len(result["rows"]) == 4
     assert any(row["pareto"] for row in result["rows"])
+
+
+def test_host_offload_reuses_state_without_hbm_retention():
+    from inferscale.agentic import compare_agent_memory_policies
+
+    result = compare_agent_memory_policies(BASE | {"host_memory_gb": 4, "host_bandwidth_gbps": 32})
+    rows = {row["label"]: row for row in result["rows"]}
+    offload = rows["Host offload / bounded affinity"]
+    stateless = rows["Stateless / least-load"]
+    assert offload["host_hit_rate"] > 0
+    assert offload["offloaded_gb"] > 0
+    assert offload["recomputed_history_tokens"] < stateless["recomputed_history_tokens"]
+
+
+def test_gap_aware_policy_is_marked_as_oracle_upper_bound():
+    result = run_agent_session_simulation(
+        BASE
+        | {
+            "retention_policy": "gap_aware",
+            "routing_policy": "bounded_affinity",
+            "gap_aware_threshold_s": 1.0,
+            "host_memory_gb": 4,
+        }
+    )
+    assert result["provenance"]["gap_aware_policy"] == "oracle-upper-bound"
+    assert result["resource"]["cross_turn_cache_hit_rate"] > 0
+
+
+def test_memory_budget_sweep_uses_common_trace_and_finite_budgets():
+    from inferscale.agentic import agent_memory_budget_sweep
+
+    result = agent_memory_budget_sweep(BASE, [0.5, 1.0])
+    assert result["protocol"] == "common-agent-program-trace"
+    assert result["budget_count"] == 2
+    assert result["policy_count"] == 3
+    assert len(result["rows"]) == 6
+    assert all(row["budget_gb_per_replica"] > 0 for row in result["rows"])
+
+
+def test_bounded_affinity_sweep_reports_locality_frontier_inputs():
+    from inferscale.agentic import agent_affinity_sweep
+
+    result = agent_affinity_sweep(BASE, [0, 100, 500])
+    assert result["study"] == "bounded-affinity-routing-frontier"
+    assert [row["affinity_slack_ms"] for row in result["rows"]] == [0.0, 100.0, 500.0]
+    assert all(0 <= row["routing_locality_rate"] <= 1 for row in result["rows"])