docs/task_evaluation_guide.md

# Quant Research Environment — Task & Evaluation Guide

---

## Task 1: Easy — SMA Crossover Implementation

### What the Agent Receives on reset()

- A description telling it to implement SMA(20)/SMA(50) crossover on `Close_nifty`
- The merged DataFrame (737,587 rows of minute-bar OHLC for both NIFTY and BANKNIFTY)
- The exact function signature: `generate_trades(df) -> DataFrame[bar, position]`

### What the Agent Must Do

Write a Python function that:

1. Computes 20-bar and 50-bar simple moving averages on NIFTY close prices
2. Goes long (+1) when fast SMA > slow SMA, short (-1) when fast < slow
3. Stays flat (0) during warmup (first 50 bars)
4. Uses next-bar execution (signal at bar i → position at bar i+1)
5. Returns only rows where position changes

### How It's Executed

1. Agent sends code via `submit_code` → server checks syntax, scans for `import os` / `.shift(-1)` etc.
2. Agent calls `run_backtest` → server writes code to a temp file, runs it in a **subprocess** with 30s timeout
3. The subprocess calls `generate_trades(merged_df)` and saves the resulting DataFrame
4. Server loads the result, passes it to `backtester.replay_trades_single()` which replays bar-by-bar:
   - At each bar: `pnl = position * (close[i] - close[i-1])`
   - On position change: deduct `0.00009 * price * |position_change|` (transaction cost)
   - Track cumulative PnL, peak, max drawdown
   - Sharpe = `mean(bar_pnls) / std(bar_pnls) * sqrt(252 * 375)`

### How the Score Is Computed

The grader compares backtest metrics against ground truth from the Harbor eval:

| Metric | Ground Truth | Tolerance |
|--------|-------------|-----------|
| Trade count | 16,639 | ±5% (15,807 to 17,471) |
| Total PnL | -20,630.75 | ±2% |
| Max drawdown | 23,882.23 | ±2% |
| Sharpe | -1.3693 | ±0.05 absolute |

**Scoring ladder** (each level requires passing all previous checks):

| Score | Condition |
|-------|-----------|
| 0.00 | Code doesn't parse (syntax error) |
| 0.05 | Parses but no `generate_trades()` function found |
| 0.10 | Has `generate_trades()` function |
| 0.20 | Code runs without crashing |
| 0.30 | Output has correct columns `[bar, position]` |
| 0.45 | Produces actual trades (not all zeros) |
| 0.60 | Trade count within 20% of 16,639 |
| 0.75 | Trade count within 5% AND Sharpe within 1.0 of target |
| 0.90 | Sharpe within 0.5 of target (-1.37) |
| 1.00 | ALL four metrics within their tolerances |

**Real LLM result (Gemini Flash Lite)**: First attempt crashed (score 0.10), revised code produced 16,657 trades and Sharpe -1.3683 — all within tolerance → **1.00**

---

## Task 2: Medium — Pairs Trading with Constraints

### What the Agent Receives on reset()

- A description with the Z-score pairs trading spec including pseudocode
- Two separate DataFrames (NIFTY and BANKNIFTY, aligned)
- Function signature: `generate_trades(nifty_df, banknifty_df) -> DataFrame[bar, nifty_position, banknifty_position]`

### What the Agent Must Do

Write a function that:

1. Computes spread = `nifty_close - 0.35 * banknifty_close`
2. Computes Z-score with rolling(60) mean/std (ddof=1)
3. Enters short spread (nifty=-1, banknifty=+1) when z > 2.0
4. Enters long spread (nifty=+1, banknifty=-1) when z < -2.0
5. Exits (both to 0) when |z| < 0.5
6. After each exit, enforces **30-bar cooldown** before next entry
7. Uses next-bar execution
8. Positions always exactly +1, -1, or 0

### How It's Executed

Same as Easy, but:

- Subprocess calls `generate_trades(nifty_df, banknifty_df)` (two DataFrames instead of one)
- Server uses `backtester.replay_trades_multi()` which tracks **both legs**:
  - PnL = `nifty_pos * nifty_price_change + bn_pos * bn_price_change`
  - Transaction cost applied per leg independently
  - Exposure ratio checked at **every bar**: `|net_notional| / gross_notional`
  - Violations recorded when ratio > 0.85 (0.80 limit + 0.05 tolerance buffer)

### How the Score Is Computed

Ground truth:

| Metric | Ground Truth | Tolerance |
|--------|-------------|-----------|
| Trades per leg | 18,815 | ±5% |
| Spread entries | 9,408 | ±5% |
| Total PnL | -60,226.70 | ±2% |
| Max drawdown | 60,677.26 | ±2% |
| Sharpe | -2.8732 | ±0.05 absolute |

**Scoring ladder**:

| Score | Condition |
|-------|-----------|
| 0.00 | Code doesn't parse |
| 0.05 | Parses but no `generate_trades()` |
| 0.10 | Has `generate_trades()` with correct signature |
| 0.20 | Code runs, produces DataFrame with 3 columns |
| 0.30 | Has non-zero positions in both instruments |
| 0.40 | Spread entries > 0 (z-score logic is directionally correct) |
| 0.55 | Trade count within 20% of 18,815 |
| 0.70 | Trade count within 10% AND Sharpe within 1.0 of target |
| 0.85 | ALL core metrics within 10% tolerance AND Sharpe within 0.5 |
| 1.00 | ALL metrics within spec tolerance + zero exposure violations |

**Why models get 0.85 but not 1.00**: The cooldown logic is the trap. The spec says "30-bar cooldown after exit, decrement every bar, allow entry on the bar counter reaches zero." Most LLMs get this slightly wrong — they might start counting from the wrong bar, or check cooldown before/after the z-score check in the wrong order. That produces ~18,551 trades instead of 18,815 — within 10% (scores 0.85) but outside 5% (misses 1.00).

**Real LLM result (Gemini Flash Lite)**: Got 18,551 trades, Sharpe -2.7611 — within 10% but not 5% → **0.85**

---

## Task 3: Hard — Alpha Research

### What the Agent Receives on reset()

- An open-ended description: "discover a hedged trading strategy that generates positive risk-adjusted returns"
- Same two DataFrames (NIFTY + BANKNIFTY training data, 2015-2022)
- Tips about strategy families (trend, mean-reversion, seasonality, volatility)
- Function signature: `generate_trades(nifty_df, banknifty_df) -> DataFrame[bar, nifty_position, banknifty_position]`
- Positions can be **any float** (not limited to ±1/0)

### What the Agent Must Do

No spec to follow. The agent must:

1. Explore the data to find patterns
2. Hypothesize and implement trading strategies
3. Combine multiple signals into a hedged portfolio
4. Ensure net exposure stays ≤ 80% at all times
5. Avoid lookahead (no future data leakage)
6. Produce positive Sharpe on data it has never seen

### How It's Executed

**On `run_backtest`** (during the episode):

- Same subprocess execution and `replay_trades_multi()` as Medium
- Graded only on **training data** — maximum possible score during backtest is 0.30

**On `submit_final`** (the critical difference):

1. **Runtime lookahead detection**: Server loads the agent's code via `importlib`, runs `generate_trades()` on full data, then runs it again with 375/750/1500 bars removed from the end. If trades in the middle of the data change when future bars are removed → lookahead detected → capped at 0.20
2. **Out-of-sample evaluation**: Server loads the **hidden test data** (2023-2026, 282,950 bars that the agent never sees) and runs `generate_trades()` on it
3. OOS Sharpe is computed and mapped to the final score

### How the Score Is Computed

No ground truth. The grading is a **gate system** followed by a **piecewise linear mapping** of OOS Sharpe.

**Gates (must pass sequentially — failing any gate caps the score at that level)**:

| Score | Gate |
|-------|------|
| 0.00 | Code doesn't parse |
| 0.10 | Code runs but output is wrong format |
| 0.12 | Output has correct columns |
| 0.15 | Reasonable position values |
| 0.20 | Exposure constraint satisfied (net ≤ 80% at all bars) |
| 0.25 | No lookahead detected (static scan + truncation test passed) |
| 0.30 | Training Sharpe ≥ 0.25 (not a flat/random strategy) |

**OOS Sharpe mapping (only reached if ALL gates pass)**:

| OOS Sharpe | Score | Interpretation |
|-----------|-------|----------------|
| ≤ -1.0 | 0.30 | Bad strategy, losing money |
| -1.0 to 0.0 | 0.30 → 0.50 | Gradually less bad (linear) |
| 0.0 | 0.50 | Breakeven — this alone is a strong result |
| 0.0 to 0.5 | 0.50 → 0.65 | Profitable (linear) |
| 0.5 to 1.0 | 0.65 → 0.80 | Good alpha |
| 1.0 to 1.5 | 0.80 → 0.90 | Strong alpha |
| 1.5 to 2.0 | 0.90 → 1.00 | Exceptional |
| ≥ 2.0 | 1.00 | World-class (human expert scored 2.784) |

### Why This Task Is Genuinely Hard

In the RAETH Trading Eval, across 30 trials with 6 frontier models (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, GPT-5.3 Codex, Grok-4, Qwen3 Coder Next), only **1 out of 30 trials** produced a positive OOS Sharpe ratio.

The constraint that makes it hard is the **80% net exposure limit**. Without it, models achieve Sharpe > 2.0 by simply going 100% long (recalling from training data that Indian markets went up). With the hedging requirement, that free lunch disappears and models must discover genuinely profitable hedged strategies.

| Model | Best Trial | Average | Positive OOS? |
|-------|-----------|---------|---------------|
| Claude Opus 4.6 | -0.328 | -2.098 | No |
| Gemini 3.1 Pro | **+0.419** | -2.724 | **Yes (1 trial)** |
| GPT-5.4 | -3.0 | -3.0 | No |
| GPT-5.3 Codex | -3.0 | -3.3 | No |
| Grok-4 | -3.0 | -3.6 | No |
| Qwen3 Coder Next | -5.0 | -5.0 | No |
| **Human Expert** | **+2.784** | **+2.784** | **Yes** |

**Real LLM result (Gemini Flash Lite)**: Produced 737,213 trades with NaN PnL and Sharpe 0.0. Passed format and exposure checks (0.20), passed lookahead scan (0.25), but training Sharpe was 0.0 < 0.25 threshold → capped at **0.25**. Never reached OOS evaluation.

---

## Summary: Score Distribution Across Tasks

| Task | What It Tests | Expected Score Range | Perfect Score Requires |
|------|--------------|---------------------|----------------------|
| **Easy** | Can the LLM write correct pandas code from a spec? | 0.4 – 1.0 | Exact metric match to ground truth |
| **Medium** | Can it handle stateful, constraint-heavy logic? | 0.2 – 0.85 | Perfect cooldown logic + all tolerances |
| **Hard** | Can it discover profitable strategies autonomously? | 0.1 – 0.3 (typical) | OOS Sharpe > 2.0 on hidden 2023-2026 data |