OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport

Reference implementation for the paper:

OnTrack: Real-Time Monitoring and Intervention in LLM Agent Trajectories via Streaming Structure-Aware Optimal Transport Babak Barazandeh, Connor Swanson, Chinmay Kulkarni, Nikhil Mungel Cribl AI Research Lab arXiv:2610.12375

Overview

OnTrack is a streaming monitoring mechanism that compares an LLM agent's execution graph against reference solutions using optimal transport, in real time. It flags loops, stalls, hallucinations, causal inversions, and off-track behavior as the agent runs, with sub-millisecond per-event latency.

The monitor operates in three information regimes:

• References + tool schemas: full structural alignment, pre-execution gate, and per-step diagnostics
• Schemas only: loop/stall detection plus prerequisite checks (no plan deviation)
• Bare stream: loop and stall detection via L1 vital signs

Files

File	Description
ontrack.py	Core OnTrack algorithm — pure NumPy, no external OT dependency
test_ontrack.py	Controlled fault-injection test suite (8 scenarios, paper Section 6.1 and Table 2)
eval_swebench.py	Main SWE-bench evaluation on sailplane/swe-agent-trajs (Sections 6.2--6.6)
eval_presubmit.py	Pre-submission experiments: rival abort policies (Table 6), multi-seed checkpoint AUROC (Table 5), checkpoint ablation
eval_reviewer.py	Reviewer-requested ablations: component attribution (Table 4), length-controlled analysis, multi-seed robustness
eval_review3.py	L3 gate tests (Table 3), corrected AUPRC computation, PR curves

Installation

pip install -r requirements.txt

The core algorithm (ontrack.py) only requires NumPy. The evaluation scripts additionally need sentence-transformers, datasets, scikit-learn, scipy, and optionally matplotlib for figures.

Quick start

Run the controlled test suite (no downloads needed)

python test_ontrack.py

This runs all 8 fault-injection scenarios (Table 2) with synthetic embeddings and prints PASS/FAIL for each.

Run the SWE-bench evaluation

python eval_swebench.py

This downloads sailplane/swe-agent-trajs (2,294 trajectories) from HuggingFace, embeds steps with all-MiniLM-L6-v2, and runs the full experiment suite:

• EXP 1: Resolved vs. unresolved discrimination (AUROC)
• EXP 1b: Length-controlled variant
• EXP A: Early warning and cost savings (abort threshold sweep)
• EXP B: L1-only exit-status detection
• EXP E: In-distribution splice injection
• EXP 2: Fault injection
• EXP 3: Ablation

Results are saved to a timestamped JSON file.

Run additional experiments

python eval_presubmit.py   # Table 5, Table 6, checkpoint ablation
python eval_reviewer.py    # Table 4, multi-seed robustness
python eval_review3.py     # Table 3 (L3 gate tests), AUPRC correction

Using OnTrack in your own project

import numpy as np
from ontrack import Config, Event, Reference, OnTrack

# Build a reference trajectory (known-good execution)
ref = Reference(
    action_embs=np.stack([...]),  # [m, d] action embeddings
    tools=["search", "read", "edit", "test", "commit"],
    parents=[[], [0], [0], [1, 2], [3]],  # DAG structure
    args_embs=np.stack([...]),    # [m, d] argument embeddings (optional)
    effect_embs=np.stack([...]),  # [m, d] observation embeddings (optional)
)

# Create evaluator
evaluator = OnTrack([ref], Config())

# Stream agent events one at a time
for step in agent_stream:
    event = Event(
        step_id=step.id,
        action_emb=embed(step.action),
        tool=step.tool_name,
        args_emb=embed(step.arguments),
        effect_emb=embed(step.result),
        irreversible=step.is_destructive,
        requires=step.required_artifacts,
    )
    result = evaluator.append_event(event)

    # result["verdict"] in: OK, EXPLORING, WARN, LOOP, STALLED,
    #                        OFF_TRACK, CAUSAL_INVERSION, BLOCK
    if result["verdict"] == "BLOCK":
        prevent_execution(step)
    elif result["verdict"] in ("LOOP", "OFF_TRACK"):
        alert_operator(result)

# At end of run
final = evaluator.finalize()
print(f"Final score: {final['final_score']:.3f}")

Key hyperparameters

Parameter	Default	Description
alpha	0.45	Action embedding weight in node cost
beta	0.20	Argument embedding weight
delta	0.35	Tool substitution cost weight
theta	0.35	Structure (GW) vs. attribute trade-off
lam	0.2	KL marginal relaxation (calibrate per encoder)
kappa	0.30	Coverage threshold for frontier mask
h_age	3	Structural age weight ramp
theta_leak	0.60	Leakage threshold for EXPLORING verdict
grace_ttl	3	Grace period (events) for unresolved nodes

lam is the main parameter to calibrate: it controls how aggressively the solver destroys mass for unmatched steps. The paper uses lam=0.1 on synthetic embeddings and lam=0.3 on SWE-bench with all-MiniLM-L6-v2.

Citation

@article{barazandeh2026ontrack,
  title={OnTrack: Real-Time Monitoring and Intervention in {LLM} Agent Trajectories via Streaming Structure-Aware Optimal Transport},
  author={Barazandeh, Babak and Swanson, Connor and Kulkarni, Chinmay and Mungel, Nikhil},
  journal={arXiv preprint arXiv:2610.12375},
  year={2026}
}

License

Apache 2.0