docs/task_evaluation_guide.md

Quant Research Environment — Task & Evaluation Guide

---

Task 1: Easy — SMA Crossover Implementation

What the Agent Receives on reset()

• A description telling it to implement SMA(20)/SMA(50) crossover on Close_nifty
• The merged DataFrame (737,587 rows of minute-bar OHLC for both NIFTY and BANKNIFTY)
• The exact function signature: generate_trades(df) -> DataFrame[bar, position]

What the Agent Must Do

Write a Python function that:

1. Computes 20-bar and 50-bar simple moving averages on NIFTY close prices
2. Goes long (+1) when fast SMA > slow SMA, short (-1) when fast < slow
3. Stays flat (0) during warmup (first 50 bars)
4. Uses next-bar execution (signal at bar i → position at bar i+1)
5. Returns only rows where position changes

How It's Executed

1. Agent sends code via submit_code → server checks syntax, scans for import os / .shift(-1) etc.
2. Agent calls run_backtest → server writes code to a temp file, runs it in a subprocess with 30s timeout
3. The subprocess calls generate_trades(merged_df) and saves the resulting DataFrame
4. Server loads the result, passes it to backtester.replay_trades_single() which replays bar-by-bar:
   • At each bar: pnl = position * (close[i] - close[i-1])
   • On position change: deduct 0.00009 * price * |position_change| (transaction cost)
   • Track cumulative PnL, peak, max drawdown
   • Sharpe = mean(bar_pnls) / std(bar_pnls) * sqrt(252 * 375)

How the Score Is Computed

The grader compares backtest metrics against ground truth from the Harbor eval:

Metric	Ground Truth	Tolerance
Trade count	16,639	±5% (15,807 to 17,471)
Total PnL	-20,630.75	±2%
Max drawdown	23,882.23	±2%
Sharpe	-1.3693	±0.05 absolute

Scoring ladder (each level requires passing all previous checks):

Score	Condition
0.00	Code doesn't parse (syntax error)
0.05	Parses but no generate_trades() function found
0.10	Has generate_trades() function
0.20	Code runs without crashing
0.30	Output has correct columns [bar, position]
0.45	Produces actual trades (not all zeros)
0.60	Trade count within 20% of 16,639
0.75	Trade count within 5% AND Sharpe within 1.0 of target
0.90	Sharpe within 0.5 of target (-1.37)
1.00	ALL four metrics within their tolerances

Real LLM result (Gemini Flash Lite): First attempt crashed (score 0.10), revised code produced 16,657 trades and Sharpe -1.3683 — all within tolerance → 1.00

---

Task 2: Medium — Pairs Trading with Constraints

What the Agent Receives on reset()

• A description with the Z-score pairs trading spec including pseudocode
• Two separate DataFrames (NIFTY and BANKNIFTY, aligned)
• Function signature: generate_trades(nifty_df, banknifty_df) -> DataFrame[bar, nifty_position, banknifty_position]

What the Agent Must Do

Write a function that:

1. Computes spread = nifty_close - 0.35 * banknifty_close
2. Computes Z-score with rolling(60) mean/std (ddof=1)
3. Enters short spread (nifty=-1, banknifty=+1) when z > 2.0
4. Enters long spread (nifty=+1, banknifty=-1) when z < -2.0
5. Exits (both to 0) when |z| < 0.5
6. After each exit, enforces 30-bar cooldown before next entry
7. Uses next-bar execution
8. Positions always exactly +1, -1, or 0

How It's Executed

Same as Easy, but:

• Subprocess calls generate_trades(nifty_df, banknifty_df) (two DataFrames instead of one)
• Server uses backtester.replay_trades_multi() which tracks both legs:
  • PnL = nifty_pos * nifty_price_change + bn_pos * bn_price_change
  • Transaction cost applied per leg independently
  • Exposure ratio checked at every bar: |net_notional| / gross_notional
  • Violations recorded when ratio > 0.85 (0.80 limit + 0.05 tolerance buffer)

How the Score Is Computed

Ground truth:

Metric	Ground Truth	Tolerance
Trades per leg	18,815	±5%
Spread entries	9,408	±5%
Total PnL	-60,226.70	±2%
Max drawdown	60,677.26	±2%
Sharpe	-2.8732	±0.05 absolute

Scoring ladder:

Score	Condition
0.00	Code doesn't parse
0.05	Parses but no generate_trades()
0.10	Has generate_trades() with correct signature
0.20	Code runs, produces DataFrame with 3 columns
0.30	Has non-zero positions in both instruments
0.40	Spread entries > 0 (z-score logic is directionally correct)
0.55	Trade count within 20% of 18,815
0.70	Trade count within 10% AND Sharpe within 1.0 of target
0.85	ALL core metrics within 10% tolerance AND Sharpe within 0.5
1.00	ALL metrics within spec tolerance + zero exposure violations

Why models get 0.85 but not 1.00: The cooldown logic is the trap. The spec says "30-bar cooldown after exit, decrement every bar, allow entry on the bar counter reaches zero." Most LLMs get this slightly wrong — they might start counting from the wrong bar, or check cooldown before/after the z-score check in the wrong order. That produces ~18,551 trades instead of 18,815 — within 10% (scores 0.85) but outside 5% (misses 1.00).

Real LLM result (Gemini Flash Lite): Got 18,551 trades, Sharpe -2.7611 — within 10% but not 5% → 0.85

---

Task 3: Hard — Alpha Research

What the Agent Receives on reset()

• An open-ended description: "discover a hedged trading strategy that generates positive risk-adjusted returns"
• Same two DataFrames (NIFTY + BANKNIFTY training data, 2015-2022)
• Tips about strategy families (trend, mean-reversion, seasonality, volatility)
• Function signature: generate_trades(nifty_df, banknifty_df) -> DataFrame[bar, nifty_position, banknifty_position]
• Positions can be any float (not limited to ±1/0)

What the Agent Must Do

No spec to follow. The agent must:

1. Explore the data to find patterns
2. Hypothesize and implement trading strategies
3. Combine multiple signals into a hedged portfolio
4. Ensure net exposure stays ≤ 80% at all times
5. Avoid lookahead (no future data leakage)
6. Produce positive Sharpe on data it has never seen

How It's Executed

On run_backtest (during the episode):

• Same subprocess execution and replay_trades_multi() as Medium
• Graded only on training data — maximum possible score during backtest is 0.30

On submit_final (the critical difference):

1. Runtime lookahead detection: Server loads the agent's code via importlib, runs generate_trades() on full data, then runs it again with 375/750/1500 bars removed from the end. If trades in the middle of the data change when future bars are removed → lookahead detected → capped at 0.20
2. Out-of-sample evaluation: Server loads the hidden test data (2023-2026, 282,950 bars that the agent never sees) and runs generate_trades() on it
3. OOS Sharpe is computed and mapped to the final score

How the Score Is Computed

No ground truth. The grading is a gate system followed by a piecewise linear mapping of OOS Sharpe.

Gates (must pass sequentially — failing any gate caps the score at that level):

Score	Gate
0.00	Code doesn't parse
0.10	Code runs but output is wrong format
0.12	Output has correct columns
0.15	Reasonable position values
0.20	Exposure constraint satisfied (net ≤ 80% at all bars)
0.25	No lookahead detected (static scan + truncation test passed)
0.30	Training Sharpe ≥ 0.25 (not a flat/random strategy)

OOS Sharpe mapping (only reached if ALL gates pass):

OOS Sharpe	Score	Interpretation
≤ -1.0	0.30	Bad strategy, losing money
-1.0 to 0.0	0.30 → 0.50	Gradually less bad (linear)
0.0	0.50	Breakeven — this alone is a strong result
0.0 to 0.5	0.50 → 0.65	Profitable (linear)
0.5 to 1.0	0.65 → 0.80	Good alpha
1.0 to 1.5	0.80 → 0.90	Strong alpha
1.5 to 2.0	0.90 → 1.00	Exceptional
≥ 2.0	1.00	World-class (human expert scored 2.784)

Why This Task Is Genuinely Hard

In the RAETH Trading Eval, across 30 trials with 6 frontier models (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro, GPT-5.3 Codex, Grok-4, Qwen3 Coder Next), only 1 out of 30 trials produced a positive OOS Sharpe ratio.

The constraint that makes it hard is the 80% net exposure limit. Without it, models achieve Sharpe > 2.0 by simply going 100% long (recalling from training data that Indian markets went up). With the hedging requirement, that free lunch disappears and models must discover genuinely profitable hedged strategies.

Model	Best Trial	Average	Positive OOS?
Claude Opus 4.6	-0.328	-2.098	No
Gemini 3.1 Pro	+0.419	-2.724	Yes (1 trial)
GPT-5.4	-3.0	-3.0	No
GPT-5.3 Codex	-3.0	-3.3	No
Grok-4	-3.0	-3.6	No
Qwen3 Coder Next	-5.0	-5.0	No
Human Expert	+2.784	+2.784	Yes

Real LLM result (Gemini Flash Lite): Produced 737,213 trades with NaN PnL and Sharpe 0.0. Passed format and exposure checks (0.20), passed lookahead scan (0.25), but training Sharpe was 0.0 < 0.25 threshold → capped at 0.25. Never reached OOS evaluation.

---

Summary: Score Distribution Across Tasks

Task	What It Tests	Expected Score Range	Perfect Score Requires
Easy	Can the LLM write correct pandas code from a spec?	0.4 – 1.0	Exact metric match to ground truth
Medium	Can it handle stateful, constraint-heavy logic?	0.2 – 0.85	Perfect cooldown logic + all tolerances
Hard	Can it discover profitable strategies autonomously?	0.1 – 0.3 (typical)	OOS Sharpe > 2.0 on hidden 2023-2026 data