docs/RESEARCH.md

SYNAPSE-X Research Note

Abstract

SYNAPSE-X is an OpenEnv-compatible benchmark for operational decision-making under uncertainty, growing risk, limited resources, and deadlines. The goal is to evaluate agents that must anticipate future deterioration, not just react to the current observation. Two core mechanisms drive this behavior: ECHO, which exposes time-evolving risk, and PRISM, which models seeded execution uncertainty. Together they create a compact benchmark where stronger policies measurably outperform weaker ones.

SYNAPSE-X is positioned as a benchmark for evaluating predictive, uncertainty-aware decision policies rather than reactive control policies.

Research Question

Can we build a benchmark where success depends on balancing immediate reward against uncertainty, deadline pressure, and the future consequences of delay?

This benchmark was designed to capture the kinds of tradeoffs we see in real operational systems.

SYNAPSE-X answers this with a structured task environment in which the agent must repeatedly decide whether to act now, postpone, or restore resources.

Environment Framing

Each episode presents a fixed set of tasks. Every task has:

• priority
• risk
• uncertainty
• deadline
• resource cost

The agent chooses one of three actions at each step:

• execute
• delay
• reallocate

This makes the benchmark easy to understand while still preserving a meaningful planning problem.

Key Mechanisms

ECHO

ECHO predicts how risk evolves as time passes and surfaces future_risk and deadline_pressure directly in the observation.

This matters because a task that looks acceptable now may become far more dangerous after several delays.

PRISM

PRISM introduces seeded execution uncertainty. Outcomes are therefore uncertain during the episode while remaining reproducible under fixed seeds.

This matters because agents must reason about both expected value and confidence, not just raw priority.

Combined Effect

Together, ECHO and PRISM create the core benchmark tension:

• acting too early can waste resources on uncertain tasks
• acting too late increases future risk and deadline pressure

That tension is what makes SYNAPSE-X predictive rather than purely reactive.

Task Difficulty

The benchmark includes three preset task groups:

Difficulty	What It Tests
easy	basic execution ordering under low risk
medium	tradeoffs between reward, risk, and deadline pressure
hard	uncertainty-aware decision-making under scarce resources

Difficulty scaling is visible in the benchmark results and is reflected in the agent score drop from easy to hard.

Deterministic Evaluation

The grader converts an action trace into a final score using:

score = 0.5 * completion_rate + 0.3 * efficiency + 0.2 * reward_score

This gives the benchmark three useful properties:

• action quality matters
• full-trajectory behavior matters
• repeated evaluation is stable for the same action sequence

Benchmark Evidence

Measured with python scripts/evaluate.py:

Agent	Easy	Medium	Hard
Random	0.6621	0.0694	0.0907
Baseline	0.9714	0.3734	0.3617

The consistent gap between policies confirms that decision quality, not randomness, drives performance.

These results show two important things:

• the environment is not trivial, because scores fall sharply as difficulty increases
• the hard setting is intentionally punitive, because CASCADE-X compounds dependency failures and exposes headroom for stronger decision policies

On hard mode, dependency-aware policies now meaningfully outperform passive or random behavior. The seeded hard benchmark remains difficult, but it no longer collapses into an unwinnable first-step failure regime, which makes CASCADE-X a more informative test of planning quality.

Why This Benchmark Is Useful

SYNAPSE-X is small enough to inspect quickly but rich enough to capture realistic operational tradeoffs:

• uncertainty affects execution outcome
• waiting changes future task quality
• resource management affects what is feasible next
• success depends on trajectory quality, not isolated action quality

This makes it a strong fit for evaluating scheduling, operations, and decision-intelligence agents.

Conclusion

SYNAPSE-X provides a compact benchmark for predictive decision-making under uncertainty. Its value comes from combining a simple action interface with meaningful temporal and probabilistic structure that exposes real decision tradeoffs. That balance makes it practical for evaluation while still producing informative differences between policies.