Spaces:

Chennzi
/

agentic-forecasting

Running

Deploy the website

d811be1 verified about 12 hours ago

15.5 kB

// Methods: how a task is defined and timed, what every system receives, continual agents,
// the kinds of systems, scoring, taxonomy and data.
import { GITHUB, RESULTS } from "./config.js";
import { KINDS, KIND_GROUP, KIND_LABEL, bySystem, h, md, sectionHead, systemCell, tex } from "./util.js";

const KIND_TEXT = {
  agent: "General-purpose agents with a shell, run unmodified in an ALE sandbox. They get the task's instruction, the run date and the open internet, and must find everything else themselves, the target's own history included.",
  expert: "The forecast an official agency or expert community publishes for the same target, taken as its first version after the cutoff, which no agent sees. Its earlier versions stay public. It differs per task and is described on each task page.",
  tsfm: "Curated TSFM: a time-series foundation model, zero-shot, given $X_d$, the target history plus the covariates the task's author chose. It measures what human prior knowledge and a strong model achieve without searching. A history-only variant isolates the value of the covariates.",
  reference: "Seasonal naive: the value on the same weekday in the latest observed week. Every scaled score divides by its error, so it scores exactly 1.",
};

const def = (...pairs) => h("dl", { class: "def" }, pairs.flatMap(([term, text]) => [h("dt", {}, md(term)), h("dd", {}, ...[text].flat().map((t) => (typeof t === "string" ? md(t) : t)))]));

function definition(index) {
  const zone = (t) => new Intl.DateTimeFormat("en-US", { timeZone: t.schedule.timezone, timeZoneName: "short" })
    .formatToParts(new Date(t.next_run)).find((p) => p.type === "timeZoneName")?.value;
  const days = (hs) => (hs.length > 1 ? `${d3.min(hs)}–${d3.max(hs)}` : `${hs[0]}`);
  return h("section", { class: "section" },
    sectionHead("definition", "Task definition", "Each part is set by the task's author, under rules that make every system forecast the same quantity, at the same time, from public information that does not yet contain the answer."),
    h("div", { class: "card" }, tex("\\tau = (y,\\ \\Delta,\\ H,\\ X,\\ B)", true), def(
      ["$y$ · target", "A series of daily values $y_u$, settled on the first value its publisher releases, at time $p_u$; later revisions never count. The instruction defines $y_u$ precisely enough to recompute it independently: the source report, the aggregation (ERCOT's peak is the largest hourly average, not an instantaneous dashboard reading), the local day, and how many observations a day needs to count."],
      ["$\\Delta$ · run days", "Run days are $d_0, d_0 + \\Delta, d_0 + 2\\Delta, \\dots$; every task here runs daily. Every system forecasts once on each run day. Actual values are recorded every day."],
      ["$s_d,\\ t_d$ · start and cutoff", "On run day $d$ the run starts at $s_d$ and forecasts are due at the cutoff $t_d$, fixed local clock times. They are not free choices: the timing rules below derive them from when $y$ and $B$ are published."],
      ["$H$ · horizons", "Each run day forecasts the dates $d + h$, $h \\in H$. $H$ is the set of dates $B$ forecasts, so every horizon has an expert comparison. $h = 0$ is allowed only when day $d$'s target period starts after the cutoff."],
      ["Quantiles", "Each target date gets the quantiles at levels $0.1, 0.2, \\dots, 0.9$, from which its CRPS is computed (see Scoring)."],
      ["$X$ · curated input", "$X_d$ holds the target history and the covariates the author chose, each as published before $s_d$: the snapshot is taken only on the day itself, so no later value or revision can enter. Where $B$ exists, $X_d$ also holds its latest issue before $s_d$, which only the *-expert TSFMs receive. $X_d$ goes to the TSFMs only; agents never see it."],
      ["$B$ · expert forecast", "The official or expert forecast of the same target, if one is public: its first issue after the cutoff, released at $b_d$. No agent can see it, and it is made from almost the same information as theirs. It is recorded the next day. Its earlier issues are public, and agents may use them."])),
    h("div", { class: "card" },
      h("div", { class: "card-title" }, "Timing rules"),
      h("ol", { class: "prose" },
        h("li", {}, md("Latest value first: $p_{d-1} < s_d$. The run starts after the last day before $d$ is settled, so every system can know it.")),
        h("li", {}, md("Room to work: $s_d + T + m \\le t_d$, with the agents' 90-minute limit $T$ and a margin $m$ for building and setting up the sandbox.")),
        h("li", {}, md("Nothing to see in the answer: $t_d$ precedes the target period of $d + \\min H$, so at the cutoff no observation of a forecast date is public.")),
        h("li", {}, md("Same information as $B$, where the schedule allows: $t_d < b_d$, a few minutes before the earliest time that issue of $B$ has been released, so agents and the expert forecast from nearly the same information and the agents never see it. $s_d$ then follows from rule 2."))),
      h("p", { class: "secondary small" }, md("Rules 1 to 3 are required; rule 4 is preferred. Where $B$ comes out too soon after the previous day's value ($p_{d-1} + T + m > b_d$) or in the same release, the run starts after the value anyway and is compared with the issue the agents can see; the table marks those tasks. The TSFMs' expert covariate is always the issue of $B$ before $B$'s own. Without an expert forecast, rules 1 to 3 decide the timing."))),
    h("div", { class: "card table-wrap" }, h("table", {},
      h("thead", {}, h("tr", {}, ["Task", "Runs every", "Start", "Cutoff", "Horizons (days)", "Expert forecast", "Agents see it"].map((c) => h("th", {}, c)))),
      h("tbody", {}, index.tasks.map((t) => h("tr", {},
        h("td", {}, h("a", { href: `#/t/${t.id}` }, t.title)),
        h("td", {}, t.schedule.every_days === 1 ? "1 day" : `${t.schedule.every_days} days`),
        h("td", {}, `${t.schedule.snapshot} ${zone(t)}`),
        h("td", {}, `${t.schedule.cutoff} ${zone(t)}`),
        h("td", {}, days(t.horizons)),
        h("td", {}, t.baseline ?? "none"),
        h("td", {}, t.baseline ? (t.baseline_visible ? "yes" : "no") : "")))))));
}

function continual(index) {
  return h("section", { class: "section" },
    sectionHead("continual", "Continual forecasting", "RQ3: does an agent that keeps its workspace and session from one run day to the next forecast better than a fresh agent?"),
    h("div", { class: "card" }, def(
      ["Chain", "For an agent $a$, a chain runs over consecutive run days $d_1, d_2 = d_1 + \\Delta, \\dots$. Day $d_1$ is an ordinary fresh episode. After each episode, ALE removes the Task's setup, tests, solution and credentials and saves the sandbox's disk as a snapshot $S_{d_i}$: the agent's files and its native session $c_{d_i}$, but no running processes."],
      ["Day $d_{i+1}$", ["A new sandbox boots from $S_{d_i}$. Setup moves the earlier outputs to /home/user/history/<run date>/ and writes the new task.json. The agent resumes session $c_{d_i}$ and receives the live instruction with one more paragraph, after its note on internet access:",
        h("blockquote", { class: "small" }, index.continual_note.replaceAll("`", ""))]],
      ["Information", ["Beyond what a fresh agent on $d_{i+1}$ has, the continual agent has its earlier submissions $f_{a,d_1}, \\dots, f_{a,d_i}$, its files and its whole earlier conversation:",
        tex("\\mathcal{F}^{A,\\text{cont}}_{d_{i+1}} = \\mathcal{F}^{A}_{d_{i+1}} \\cup \\{ f_{a,d_j},\\ S_{d_j} \\}_{j \\le i}", true),
        "It gets no scores and no actual values: the values published since, it looks up itself."]],
      ["Breaks", "A chain continues only from the previous run day. If that day's episode left no snapshot or no session, the next run day starts a new chain. Snapshots are kept, so any day of a chain can be continued again later."],
      ["Measure", ["Each continual agent is compared with the fresh agent of the same harness and model, on the run days and target dates both resolved:",
        tex("\\delta = \\text{scaled CRPS}_{\\text{continual}} - \\text{scaled CRPS}_{\\text{fresh}}", true),
        "Below 0 favors the continual agent. Each task page lists these pairs; the main leaderboards show fresh agents only."]]),
      h("p", { class: "secondary small" }, md("Chains began on October 28 and 29, 2026. Until November 2, ALE kept the sandbox running between days, and the paragraph above was part of the instruction from a chain's first day. From November 4, every day continues from a snapshot and the paragraph is added only on continued days; the chains restarted then. No continual run took place on November 3."))));
}

export function methods(app, index) {
  const row = (...cells) => h("tr", {}, cells.map((c) => h("td", {}, c)));
  const toc = [["definition", "Task definition"], ["protocol", "Protocol"], ["continual", "Continual"], ["systems", "Systems"], ["scoring", "Scoring"], ["taxonomy", "Taxonomy"], ["data", "Data"]];
  app.replaceChildren(
    h("header", { class: "page-head" },
      h("span", { class: "eyebrow" }, "Methods"),
      h("h1", {}, "How the benchmark works"),
      h("p", { class: "lead" }, "Each task is a real series with a precise settlement rule. On every run day, agents forecast it live, and the day is kept as an offline replay built from what they saved. Scores arrive when the values are published."),
      h("nav", { class: "methods-toc" }, toc.map(([id, text]) => h("a", { href: `#/methods?at=${id}` }, text)))),

    definition(index),

    h("section", { class: "section" },
      sectionHead("protocol", "Protocol", "What each system receives on run day d. Nothing is known about the outcome when forecasts are made, so live runs need no leakage control."),
      h("div", { class: "card table-wrap" }, h("table", {},
        h("thead", {}, h("tr", {}, h("th", {}, "System"), h("th", {}, "Live"), h("th", {}, "Offline replay"))),
        h("tbody", {},
          row(h("strong", {}, "Agents"), "Instruction, task.json (run date, target dates, cutoff, quantile levels), the open internet; no sudo, 90 minutes",
            "The same Task with the network cut and hosted web search disabled, plus every live agent's web materials from day d"),
          row(h("strong", {}, "Continual agents"), "The same, continuing from a snapshot of the sandbox and native session of the agent's previous run day; its earlier submissions are in /home/user/history/, and it is asked to learn from them (see Continual)",
            "Not replayed"),
          row(h("strong", {}, "Expert forecast"), "Its source's first version after the cutoff, which no agent sees", "Not replayed"),
          row(h("strong", {}, "TSFM baselines"), md("$X_d$, snapshotted outside the sandbox before the run"), "Not replayed; the same forecasts serve as reference"),
          row(h("strong", {}, "Seasonal naive"), md("The target history in $X_d$"), "—")))),
      h("div", { class: "card" }, h("div", { class: "prose" },
        h("p", {}, md("Every run day of a task becomes its own offline Task in ALE format, ercot-peak-load-mock-<date> for example. Its image carries the day's schedule and the live agents' materials, and its solution assets carry $X_d$, which ALE uploads only for the reference solution, never for an evaluated agent. Replaying it, on demand since November 2, 2026, separates finding information from using it.")),
        h("p", { class: "secondary" }, md("Until October 23, 2026, live agents also received $X_d$, and replays received $X_d$ with the materials. Run days of that first protocol are not shown."))))),

    continual(index),

    h("section", { class: "section" },
      sectionHead("systems", "Systems", "Four kinds of forecaster. Colors follow the system, never its rank."),
      h("div", { class: "kind-grid" }, KINDS.map((kind) => h("div", { class: "card" }, h("h3", {}, KIND_GROUP[kind]), h("p", { class: "secondary small" }, md(KIND_TEXT[kind]))))),
      h("div", { class: "card table-wrap", style: { marginTop: "14px" } }, h("table", {},
        h("thead", {}, h("tr", {}, h("th", {}, "System"), h("th", {}, "Kind"), h("th", {}, "Configuration"))),
        h("tbody", {}, [...index.systems].sort((a, b) => bySystem(a.id, b.id)).map((s) =>
          h("tr", {}, h("td", {}, systemCell(s.id)), h("td", {}, h("span", { class: "tag" }, KIND_LABEL[s.kind] || s.kind)), h("td", { class: "small secondary" }, s.description))))))),

    h("section", { class: "section" },
      sectionHead("scoring", "Scoring"),
      h("div", { class: "card" }, h("dl", { class: "def" },
        h("dt", {}, "Settlement"), h("dd", {}, "The first value the publisher releases for a date. Later revisions do not count."),
        h("dt", {}, "CRPS"), h("dd", {}, "Quantile CRPS: twice the mean pinball loss over the submitted quantiles, CRPS ≈ (2/K) Σₖ ρ_τₖ(y − q̂_τₖ). For a point forecast it equals the absolute error."),
        h("dt", {}, "Scaled CRPS"), h("dd", {}, "Σ CRPS / Σ |ŷ_naive − y| over the same pairs. Below 1 beats the seasonal naive forecast. The primary score."),
        h("dt", {}, "Scaled MAE"), h("dd", {}, "The same ratio for the median alone: the accuracy of the center."),
        h("dt", {}, "80% coverage"), h("dd", {}, "Share of actual values inside the 10–90% interval. Calibrated forecasters land near 80%."),
        h("dt", {}, "Common pairs"), h("dd", {}, "Leaderboards use only the (run day, target date) pairs that every listed system resolved, so all rows compare the same days."),
        h("dt", {}, "Late"), h("dd", {}, "Live submissions after the cutoff are kept but not scored.")))),

    h("section", { class: "section" },
      sectionHead("taxonomy", "Taxonomy", "Tasks are organised by the system that generates the series, with two facets across domains."),
      h("div", { class: "card" }, h("dl", { class: "def" },
        h("dt", {}, "Domains"), h("dd", {}, index.taxonomy.domains.map((d) => `${d.name} (${d.leaves.length} areas)`).join(" · ")),
        h("dt", {}, "Driver"), h("dd", {}, `The main source of variation public context can explain: ${Object.values(index.taxonomy.facets.driver).join(", ")}.`),
        h("dt", {}, "Tradable"), h("dd", {}, "Prices in liquid markets. By the semi-strong efficient-market hypothesis, public context adds little to their mean forecast, so they serve as controls.")),
        h("p", { class: "note", style: { marginTop: "10px" } }, "The taxonomy lives in the tasks repository (taxonomy.yaml) and is revised as tasks are added."))),

    h("section", { class: "section" },
      sectionHead("data", "Data"),
      h("div", { class: "card" }, h("dl", { class: "def" },
        h("dt", {}, "Results"), h("dd", {}, h("a", { href: `https://huggingface.co/datasets/${RESULTS}`, target: "_blank", rel: "noopener" }, `${RESULTS} ↗`),
          md(" · public: every forecast, actual value, $X_d$ and agent trajectory, plus this site's JSON")),
        h("dt", {}, "Tasks"), h("dd", {}, h("a", { href: GITHUB, target: "_blank", rel: "noopener" }, "agentic-forecasting-tasks ↗"),
          " · the live and mock ALE Tasks; their assets, which include agents' saved web pages, stay private"),
        h("dt", {}, "Layout"), h("dd", {}, h("code", {}, "runs/<task>/<run day>/{live,mock}/<system>.json"), ", ", h("code", {}, ".../ale/<mode>-<agent>/<episode>/trajectory.json"), ", ", h("code", {}, ".../x/"), ", ", h("code", {}, "runs/<task>/truth.csv"))))));
}

	// Methods: how a task is defined and timed, what every system receives, continual agents,
	// the kinds of systems, scoring, taxonomy and data.
	import { GITHUB, RESULTS } from "./config.js";
	import { KINDS, KIND_GROUP, KIND_LABEL, bySystem, h, md, sectionHead, systemCell, tex } from "./util.js";

	const KIND_TEXT = {
	agent: "General-purpose agents with a shell, run unmodified in an ALE sandbox. They get the task's instruction, the run date and the open internet, and must find everything else themselves, the target's own history included.",
	expert: "The forecast an official agency or expert community publishes for the same target, taken as its first version after the cutoff, which no agent sees. Its earlier versions stay public. It differs per task and is described on each task page.",
	tsfm: "Curated TSFM: a time-series foundation model, zero-shot, given $X_d$, the target history plus the covariates the task's author chose. It measures what human prior knowledge and a strong model achieve without searching. A history-only variant isolates the value of the covariates.",
	reference: "Seasonal naive: the value on the same weekday in the latest observed week. Every scaled score divides by its error, so it scores exactly 1.",
	};

	const def = (...pairs) => h("dl", { class: "def" }, pairs.flatMap(([term, text]) => [h("dt", {}, md(term)), h("dd", {}, ...[text].flat().map((t) => (typeof t === "string" ? md(t) : t)))]));

	function definition(index) {
	const zone = (t) => new Intl.DateTimeFormat("en-US", { timeZone: t.schedule.timezone, timeZoneName: "short" })
	.formatToParts(new Date(t.next_run)).find((p) => p.type === "timeZoneName")?.value;
	const days = (hs) => (hs.length > 1 ? `${d3.min(hs)}–${d3.max(hs)}` : `${hs[0]}`);
	return h("section", { class: "section" },
	sectionHead("definition", "Task definition", "Each part is set by the task's author, under rules that make every system forecast the same quantity, at the same time, from public information that does not yet contain the answer."),
	h("div", { class: "card" }, tex("\\tau = (y,\\ \\Delta,\\ H,\\ X,\\ B)", true), def(
	["$y$ · target", "A series of daily values $y_u$, settled on the first value its publisher releases, at time $p_u$; later revisions never count. The instruction defines $y_u$ precisely enough to recompute it independently: the source report, the aggregation (ERCOT's peak is the largest hourly average, not an instantaneous dashboard reading), the local day, and how many observations a day needs to count."],
	["$\\Delta$ · run days", "Run days are $d_0, d_0 + \\Delta, d_0 + 2\\Delta, \\dots$; every task here runs daily. Every system forecasts once on each run day. Actual values are recorded every day."],
	["$s_d,\\ t_d$ · start and cutoff", "On run day $d$ the run starts at $s_d$ and forecasts are due at the cutoff $t_d$, fixed local clock times. They are not free choices: the timing rules below derive them from when $y$ and $B$ are published."],
	["$H$ · horizons", "Each run day forecasts the dates $d + h$, $h \\in H$. $H$ is the set of dates $B$ forecasts, so every horizon has an expert comparison. $h = 0$ is allowed only when day $d$'s target period starts after the cutoff."],
	["Quantiles", "Each target date gets the quantiles at levels $0.1, 0.2, \\dots, 0.9$, from which its CRPS is computed (see Scoring)."],
	["$X$ · curated input", "$X_d$ holds the target history and the covariates the author chose, each as published before $s_d$: the snapshot is taken only on the day itself, so no later value or revision can enter. Where $B$ exists, $X_d$ also holds its latest issue before $s_d$, which only the *-expert TSFMs receive. $X_d$ goes to the TSFMs only; agents never see it."],
	["$B$ · expert forecast", "The official or expert forecast of the same target, if one is public: its first issue after the cutoff, released at $b_d$. No agent can see it, and it is made from almost the same information as theirs. It is recorded the next day. Its earlier issues are public, and agents may use them."])),
	h("div", { class: "card" },
	h("div", { class: "card-title" }, "Timing rules"),
	h("ol", { class: "prose" },
	h("li", {}, md("Latest value first: $p_{d-1} < s_d$. The run starts after the last day before $d$ is settled, so every system can know it.")),
	h("li", {}, md("Room to work: $s_d + T + m \\le t_d$, with the agents' 90-minute limit $T$ and a margin $m$ for building and setting up the sandbox.")),
	h("li", {}, md("Nothing to see in the answer: $t_d$ precedes the target period of $d + \\min H$, so at the cutoff no observation of a forecast date is public.")),
	h("li", {}, md("Same information as $B$, where the schedule allows: $t_d < b_d$, a few minutes before the earliest time that issue of $B$ has been released, so agents and the expert forecast from nearly the same information and the agents never see it. $s_d$ then follows from rule 2."))),
	h("p", { class: "secondary small" }, md("Rules 1 to 3 are required; rule 4 is preferred. Where $B$ comes out too soon after the previous day's value ($p_{d-1} + T + m > b_d$) or in the same release, the run starts after the value anyway and is compared with the issue the agents can see; the table marks those tasks. The TSFMs' expert covariate is always the issue of $B$ before $B$'s own. Without an expert forecast, rules 1 to 3 decide the timing."))),
	h("div", { class: "card table-wrap" }, h("table", {},
	h("thead", {}, h("tr", {}, ["Task", "Runs every", "Start", "Cutoff", "Horizons (days)", "Expert forecast", "Agents see it"].map((c) => h("th", {}, c)))),
	h("tbody", {}, index.tasks.map((t) => h("tr", {},
	h("td", {}, h("a", { href: `#/t/${t.id}` }, t.title)),
	h("td", {}, t.schedule.every_days === 1 ? "1 day" : `${t.schedule.every_days} days`),
	h("td", {}, `${t.schedule.snapshot} ${zone(t)}`),
	h("td", {}, `${t.schedule.cutoff} ${zone(t)}`),
	h("td", {}, days(t.horizons)),
	h("td", {}, t.baseline ?? "none"),
	h("td", {}, t.baseline ? (t.baseline_visible ? "yes" : "no") : "")))))));
	}

	function continual(index) {
	return h("section", { class: "section" },
	sectionHead("continual", "Continual forecasting", "RQ3: does an agent that keeps its workspace and session from one run day to the next forecast better than a fresh agent?"),
	h("div", { class: "card" }, def(
	["Chain", "For an agent $a$, a chain runs over consecutive run days $d_1, d_2 = d_1 + \\Delta, \\dots$. Day $d_1$ is an ordinary fresh episode. After each episode, ALE removes the Task's setup, tests, solution and credentials and saves the sandbox's disk as a snapshot $S_{d_i}$: the agent's files and its native session $c_{d_i}$, but no running processes."],
	["Day $d_{i+1}$", ["A new sandbox boots from $S_{d_i}$. Setup moves the earlier outputs to /home/user/history/<run date>/ and writes the new task.json. The agent resumes session $c_{d_i}$ and receives the live instruction with one more paragraph, after its note on internet access:",
	h("blockquote", { class: "small" }, index.continual_note.replaceAll("`", ""))]],
	["Information", ["Beyond what a fresh agent on $d_{i+1}$ has, the continual agent has its earlier submissions $f_{a,d_1}, \\dots, f_{a,d_i}$, its files and its whole earlier conversation:",
	tex("\\mathcal{F}^{A,\\text{cont}}_{d_{i+1}} = \\mathcal{F}^{A}_{d_{i+1}} \\cup \\{ f_{a,d_j},\\ S_{d_j} \\}_{j \\le i}", true),
	"It gets no scores and no actual values: the values published since, it looks up itself."]],
	["Breaks", "A chain continues only from the previous run day. If that day's episode left no snapshot or no session, the next run day starts a new chain. Snapshots are kept, so any day of a chain can be continued again later."],
	["Measure", ["Each continual agent is compared with the fresh agent of the same harness and model, on the run days and target dates both resolved:",
	tex("\\delta = \\text{scaled CRPS}_{\\text{continual}} - \\text{scaled CRPS}_{\\text{fresh}}", true),
	"Below 0 favors the continual agent. Each task page lists these pairs; the main leaderboards show fresh agents only."]]),
	h("p", { class: "secondary small" }, md("Chains began on October 28 and 29, 2026. Until November 2, ALE kept the sandbox running between days, and the paragraph above was part of the instruction from a chain's first day. From November 4, every day continues from a snapshot and the paragraph is added only on continued days; the chains restarted then. No continual run took place on November 3."))));
	}

	export function methods(app, index) {
	const row = (...cells) => h("tr", {}, cells.map((c) => h("td", {}, c)));
	const toc = [["definition", "Task definition"], ["protocol", "Protocol"], ["continual", "Continual"], ["systems", "Systems"], ["scoring", "Scoring"], ["taxonomy", "Taxonomy"], ["data", "Data"]];
	app.replaceChildren(
	h("header", { class: "page-head" },
	h("span", { class: "eyebrow" }, "Methods"),
	h("h1", {}, "How the benchmark works"),
	h("p", { class: "lead" }, "Each task is a real series with a precise settlement rule. On every run day, agents forecast it live, and the day is kept as an offline replay built from what they saved. Scores arrive when the values are published."),
	h("nav", { class: "methods-toc" }, toc.map(([id, text]) => h("a", { href: `#/methods?at=${id}` }, text)))),

	definition(index),

	h("section", { class: "section" },
	sectionHead("protocol", "Protocol", "What each system receives on run day d. Nothing is known about the outcome when forecasts are made, so live runs need no leakage control."),
	h("div", { class: "card table-wrap" }, h("table", {},
	h("thead", {}, h("tr", {}, h("th", {}, "System"), h("th", {}, "Live"), h("th", {}, "Offline replay"))),
	h("tbody", {},
	row(h("strong", {}, "Agents"), "Instruction, task.json (run date, target dates, cutoff, quantile levels), the open internet; no sudo, 90 minutes",
	"The same Task with the network cut and hosted web search disabled, plus every live agent's web materials from day d"),
	row(h("strong", {}, "Continual agents"), "The same, continuing from a snapshot of the sandbox and native session of the agent's previous run day; its earlier submissions are in /home/user/history/, and it is asked to learn from them (see Continual)",
	"Not replayed"),
	row(h("strong", {}, "Expert forecast"), "Its source's first version after the cutoff, which no agent sees", "Not replayed"),
	row(h("strong", {}, "TSFM baselines"), md("$X_d$, snapshotted outside the sandbox before the run"), "Not replayed; the same forecasts serve as reference"),
	row(h("strong", {}, "Seasonal naive"), md("The target history in $X_d$"), "—")))),
	h("div", { class: "card" }, h("div", { class: "prose" },
	h("p", {}, md("Every run day of a task becomes its own offline Task in ALE format, ercot-peak-load-mock-<date> for example. Its image carries the day's schedule and the live agents' materials, and its solution assets carry $X_d$, which ALE uploads only for the reference solution, never for an evaluated agent. Replaying it, on demand since November 2, 2026, separates finding information from using it.")),
	h("p", { class: "secondary" }, md("Until October 23, 2026, live agents also received $X_d$, and replays received $X_d$ with the materials. Run days of that first protocol are not shown."))))),

	continual(index),

	h("section", { class: "section" },
	sectionHead("systems", "Systems", "Four kinds of forecaster. Colors follow the system, never its rank."),
	h("div", { class: "kind-grid" }, KINDS.map((kind) => h("div", { class: "card" }, h("h3", {}, KIND_GROUP[kind]), h("p", { class: "secondary small" }, md(KIND_TEXT[kind]))))),
	h("div", { class: "card table-wrap", style: { marginTop: "14px" } }, h("table", {},
	h("thead", {}, h("tr", {}, h("th", {}, "System"), h("th", {}, "Kind"), h("th", {}, "Configuration"))),
	h("tbody", {}, [...index.systems].sort((a, b) => bySystem(a.id, b.id)).map((s) =>
	h("tr", {}, h("td", {}, systemCell(s.id)), h("td", {}, h("span", { class: "tag" }, KIND_LABEL[s.kind] \|\| s.kind)), h("td", { class: "small secondary" }, s.description))))))),

	h("section", { class: "section" },
	sectionHead("scoring", "Scoring"),
	h("div", { class: "card" }, h("dl", { class: "def" },
	h("dt", {}, "Settlement"), h("dd", {}, "The first value the publisher releases for a date. Later revisions do not count."),
	h("dt", {}, "CRPS"), h("dd", {}, "Quantile CRPS: twice the mean pinball loss over the submitted quantiles, CRPS ≈ (2/K) Σₖ ρ_τₖ(y − q̂_τₖ). For a point forecast it equals the absolute error."),
	h("dt", {}, "Scaled CRPS"), h("dd", {}, "Σ CRPS / Σ \|ŷ_naive − y\| over the same pairs. Below 1 beats the seasonal naive forecast. The primary score."),
	h("dt", {}, "Scaled MAE"), h("dd", {}, "The same ratio for the median alone: the accuracy of the center."),
	h("dt", {}, "80% coverage"), h("dd", {}, "Share of actual values inside the 10–90% interval. Calibrated forecasters land near 80%."),
	h("dt", {}, "Common pairs"), h("dd", {}, "Leaderboards use only the (run day, target date) pairs that every listed system resolved, so all rows compare the same days."),
	h("dt", {}, "Late"), h("dd", {}, "Live submissions after the cutoff are kept but not scored.")))),

	h("section", { class: "section" },
	sectionHead("taxonomy", "Taxonomy", "Tasks are organised by the system that generates the series, with two facets across domains."),
	h("div", { class: "card" }, h("dl", { class: "def" },
	h("dt", {}, "Domains"), h("dd", {}, index.taxonomy.domains.map((d) => `${d.name} (${d.leaves.length} areas)`).join(" · ")),
	h("dt", {}, "Driver"), h("dd", {}, `The main source of variation public context can explain: ${Object.values(index.taxonomy.facets.driver).join(", ")}.`),
	h("dt", {}, "Tradable"), h("dd", {}, "Prices in liquid markets. By the semi-strong efficient-market hypothesis, public context adds little to their mean forecast, so they serve as controls.")),
	h("p", { class: "note", style: { marginTop: "10px" } }, "The taxonomy lives in the tasks repository (taxonomy.yaml) and is revised as tasks are added."))),

	h("section", { class: "section" },
	sectionHead("data", "Data"),
	h("div", { class: "card" }, h("dl", { class: "def" },
	h("dt", {}, "Results"), h("dd", {}, h("a", { href: `https://huggingface.co/datasets/${RESULTS}`, target: "_blank", rel: "noopener" }, `${RESULTS} ↗`),
	md(" · public: every forecast, actual value, $X_d$ and agent trajectory, plus this site's JSON")),
	h("dt", {}, "Tasks"), h("dd", {}, h("a", { href: GITHUB, target: "_blank", rel: "noopener" }, "agentic-forecasting-tasks ↗"),
	" · the live and mock ALE Tasks; their assets, which include agents' saved web pages, stay private"),
	h("dt", {}, "Layout"), h("dd", {}, h("code", {}, "runs/<task>/<run day>/{live,mock}/<system>.json"), ", ", h("code", {}, ".../ale/<mode>-<agent>/<episode>/trajectory.json"), ", ", h("code", {}, ".../x/"), ", ", h("code", {}, "runs/<task>/truth.csv"))))));
	}