tasks/Frontier-CS/ARCHITECTURE.md

Evaluation Architecture

This document describes how evaluation flows through the codebase and the design choices behind the current structure.

Audience

• Problem contributors (see CONTRIBUTING.md)
• Model submitters (see SUBMIT.md)
• General researchers using Frontier-CS to evaluate solutions

Goals

• Clear separation between single-problem and batch evaluation.
• Shared validation and config parsing across research backends.
• Predictable cleanup to avoid orphaned cloud resources.
• Explicit naming to avoid backend ambiguity.

Architecture at a Glance

• CLI:
  • frontier eval → SingleEvaluator
  • frontier batch → BatchEvaluator
  • frontier list / frontier show → problem discovery
• CI:
  • Validate Problems → scripts/validate_problems.py → SingleEvaluator
  • Weekly Batch Evaluation → scripts/run_eval.sh → BatchEvaluator

Components

SingleEvaluator

Unified API for single-problem evaluation:

• Selects a runner based on track and backend.
• Registers SIGINT/atexit hooks to tear down SkyPilot clusters on exit.

BatchEvaluator

Orchestrates batch evaluation with parallel workers and SkyPilot cluster pools:

• Work queues with resumable state and result aggregation.
• Resource-grouped cluster pools — pairs are grouped by ResourceSignature (cloud, accelerators, instance type) so that CPU-only and GPU problems run on separate pools.
• Hash-based resume — each result stores solution/problem hashes so stale results are automatically re-evaluated when source changes.

Runners

Runners execute evaluation. The class hierarchy:

Runner (ABC)
├── ResearchRunner              # shared: problem validation, config loading, uv install
│   ├── ResearchDockerRunner    # local container
│   └── ResearchSkyPilotRunner  # cloud via SkyPilot
├── AlgorithmicLocalRunner      # go-judge via HTTP
└── AlgorithmicSkyPilotRunner   # go-judge on SkyPilot

Solution Generation (gen/)

The gen/ module generates solutions by calling LLMs. It is independent of the evaluation pipeline — frontier eval and frontier batch do not depend on it. It provides an LLM interface, an API key pool for concurrent requests, and solution file formatting.

Design Decisions

• Single vs Batch: SingleEvaluator stays focused on one-off runs (simple API + cleanup hooks), while BatchEvaluator owns scheduling, resumable state, and cluster pools. This keeps single-run paths lightweight and batch runs scalable.
• Shared research helpers: input validation and config parsing live in the ResearchRunner base class so Docker and SkyPilot backends stay in sync.
• Cleanup strategy: research evaluations tear down clusters by default unless keep_cluster is set. SingleEvaluator cleans up via an active-cluster registry; BatchEvaluator manages its own pool lifecycle.
• Naming: runner class names encode track + backend (e.g., ResearchDockerRunner) to remove ambiguity in logs and docs.
• Score semantics: a score of 0 can mean the evaluator ran successfully; failures are reported via status/metadata rather than score alone.
• Reference solutions: problems ship with reference.cpp/reference.py so CI can verify end-to-end evaluation without model submissions.
• Results separation: evaluation outputs go to a dedicated results repository to keep the main repo lean and auditable.
• Internal vs public: internal test cases and tooling live in a private repo; public artifacts are kept minimal but compatible.
• Weekly vs local: weekly CI uses scripts/run_eval.sh with batch scheduling; local runs use the same script or frontier eval for quick iteration.
• Resource-grouped cluster pools: BatchEvaluator groups pairs by ResourceSignature (cloud × accelerators × instance type) and creates a separate pool per group, avoiding the waste of running CPU-only problems on GPU clusters.
• Hash-based resume: resuming a batch compares solution/problem hashes against stored results. Changed inputs are re-evaluated even when a prior result exists, preventing silently stale scores.
• Generation vs evaluation: solution generation (gen/) is fully decoupled from evaluation. Generated files are plain source files with no special metadata; the evaluator has no dependency on the generation pipeline.

Runner Flow (Research)

Both research runners share the same pre-evaluation steps (via ResearchRunner):

1. Validate solution file and .FAILED marker.
2. Verify the problem path exists.
3. Load config.yaml and runtime settings.
4. Build uv install command if uv_project is specified.

Execution diverges at the backend:

• Docker — launches a local container.
• SkyPilot — provisions a cloud VM and runs remotely.

Operations (Cleanup + CI)

• Cleanup: research evaluations tear down clusters by default unless keep_cluster=True. SingleEvaluator uses an active-cluster registry to clean up on SIGINT/atexit; BatchEvaluator manages its own cluster pool lifecycle.

• CI: problem validation runs single evals; the weekly batch job runs full evaluations on SkyPilot (typically GCP).