docs/PROJECT_REQUIREMENT.md

AEPO — Autonomous Enterprise Payment Orchestrator

Master Project Baseline Document

Author: Umesh Maurya Competition: Meta × PyTorch OpenEnv Hackathon — Round 2 (Grand Finale, Onsite Apr 25–26, 2026) Organizer: Scaler School of Technology Status: Round 1 Winner → Grand Finale Theme: #3.1 — World Modeling (Professional Tasks) Dashboard: https://www.scaler.com/school-of-technology/meta-pytorch-hackathon/dashboard#study-1

---

Table of Contents

1. Project Overview
2. Hackathon Theme Alignment
3. Judging Criteria & Scoring Strategy
4. Minimum Submission Requirements
5. Environment Design
6. Technical Requirements — OpenEnv Compliance
7. Training Requirements
8. Inference Script Requirements
9. Deployment Requirements
10. Deliverables Checklist
11. Pre-Submission Validation
12. Technology Stack
13. Key Differentiators from Round 1
14. Resource Links
15. Hackathon Execution Guide

---

1. Project Overview

AEPO (Autonomous Enterprise Payment Orchestrator) is an OpenEnv-compliant reinforcement learning environment simulating enterprise payment routing decisions. It models a real-world problem: the organizational blind spot between Security/Fraud Operations and Infrastructure/SRE teams in Tier-1 payment processors.

1.1 Round 1 → Grand Finale Evolution

Dimension	UFRG (Round 1)	AEPO (Grand Finale)
Observation fields	5	10
Action fields	3 — MultiDiscrete [3,3,2]	6 — MultiDiscrete [3,2,3,2,2,3] — 216 combinations
Causal transitions	None (memoryless)	11 causal state transitions
Phase structure	None	4-phase task machine per episode
Dynamics model	None	LagPredictor MLP (PyTorch)
Training	None	Q-Table agent, 500 episodes, hard task PASS
Test suite	~30 tests	221 tests, 97% coverage

1.2 The Core Story: Blind Spot Discovery

At Episode 3, Step 42 of training, the Q-table agent discovered something no human SRE heuristic ever found: Reject+SkipVerify on high-risk transactions is the non-obvious optimal action. It saves 250 Kafka lag per step and earns a +0.04 bonus — but "high risk → full verification" always felt safe to human designers, so the heuristic never explored it.

Result: Trained hard task score 0.6650 vs heuristic 0.2955 — a 2.25× improvement. The agent learned something its creator missed.

---

2. Hackathon Theme Alignment

Primary: Theme #3.1 — World Modeling (Professional Tasks)

"Develop environments that require real interaction with tools, APIs, or dynamic systems where the model is expected to do real hard work instead of exploiting shortcuts. Learning enables agents to maintain consistent internal state, update beliefs based on outcomes, and orchestrate multi-step workflows."

Why AEPO fits:

• Models a real enterprise fintech system — not a game or toy. Payment routing with fraud risk, Kafka infrastructure, SLA compliance, and bank API status.
• 11 causal state transitions require the agent to maintain persistent internal world state and update beliefs as conditions evolve.
• LagPredictor PyTorch MLP is an explicit world model predicting future Kafka lag from current state + action.
• Anti-shortcut design: every naive policy produces poor scores — reward hacking is structurally defeated.

Secondary: Theme #4 — Self-Improvement

Adversary escalation mechanism: the environment gets harder as the agent improves (5-episode lag gate), creating a self-play style adaptive curriculum.

All Five Hackathon Themes (Reference)

Theme	Description	Example Environments
#1 Multi-Agent Interactions	Cooperation, competition, negotiation, coalition formation	Market simulations, compute-allocation negotiations, collaborative puzzle worlds
#2 Long-Horizon Planning	Multi-step reasoning with sparse/delayed rewards	Research-planning simulators, codebase refactoring, 300-instruction following
#3.1 World Modeling — Professional	Real interaction with tools, APIs, dynamic systems	Dynamic browser/API ecosystems, scientific workflow loops, economic simulations
#3.2 World Modeling — Personal	Personalized task handling (messages, scheduling, email)	Executive assistant, meeting planner, email replying, shopping
#4 Self-Improvement	Self-play, adaptive curricula, recursive skill amplification	Self-play negotiation arenas, evolving coding competitions, auto-generated math
#5 Wild Card	Out-of-box ideas that meaningfully add value to LLM training	Anything novel and ambitious

---

3. Judging Criteria & Scoring Strategy

Criterion	Weight	What It Means
Environment Innovation	40%	Novel, creative, genuinely challenging? Tests agent behavior in a way that hasn't been done before?
Storytelling & Presentation	30%	Can you clearly explain the problem, environment, and what the agent learned? Engaging for a non-technical audience?
Showing Improvement in Rewards	20%	Observable training progress: reward curves, before/after behavior, comparison against baseline.
Reward & Training Pipeline	10%	Coherent reward logic? Does the pipeline produce meaningful improvement in the trained agent's behavior?

AEPO's Angle for Each Criterion

Innovation (40%): The Asymmetric Risk Triad (Fraud + Infrastructure + SLA) encoded into a single RL surface is novel. No existing OpenEnv environment models enterprise payment routing with causally-structured, multi-dimensional risk. The blind spot discovery narrative is a concrete example of emergent agent behavior.

Storytelling (30%): The pitch centers on the Siloed Metrics problem — security and infrastructure teams operate in separate worlds, and AEPO is the training ground where AI learns to bridge them. The blind spot event (Episode 3, Step 42) is the story: the Q-table discovered what the human SRE never found.

Reward Improvement (20%): Hard task trained score 0.6650 vs heuristic 0.2955 — 2.25× improvement. The staircase reward curve (plateau → blind spot discovery → new plateau → harder adversary → adaptation) is the visual centerpiece.

Pipeline (10%): inference.py uses the OpenAI client, emits [START]/[STEP]/[END] logs, produces reproducible scores on all 3 tasks. train.py trains the Q-table + LagPredictor in ~3–4 seconds on 2 vCPU.

---

4. Minimum Submission Requirements

⚠️ These are non-negotiable. Missing any results in disqualification.

Requirement	Status	Notes
Use OpenEnv (latest release)	✅ Done	openenv-core 0.2.0+, validated via openenv validate
Working training script (Unsloth/TRL) in Colab	✅ Code Shipped · ⏳ E2E Run Pending	AEPO_Unsloth_GRPO.ipynb — TRL GRPOTrainer + Unsloth FastLanguageModel, Qwen2.5-7B (A10G) / 3B (T4) auto-detect. Code shipped; E2E run on Colab/HF-Space A10G must produce results/grpo_reward_curve.png before submission (see §7.4).
Evidence of training (loss & reward plots)	✅ Done (Q-table) · ⏳ Pending (GRPO)	results/reward_curve.png (Q-table, 500 episodes) committed and embedded in README. results/grpo_reward_curve.png produced by the notebook's Section 5; commit after Colab/A10G run.
Mini-blog on HF OR <2 min YouTube video	⬜ TODO	Create and link from README before submission deadline
Push environment to Hugging Face Space	✅ Done	Tagged openenv, port 7860
README with motivation, env description, results	✅ Done	README updated with embedded reward curve, baselines, and Colab link. Writeup link pending.

---

5. Environment Design

5.1 Real-World Task

AEPO simulates enterprise payment routing — a task Tier-1 payment processors (UPI, card networks) perform billions of times per month. The agent must simultaneously manage:

• Fraud risk — risk_score [0–100], adversary_threat_level [0–10]
• Infrastructure health — kafka_lag [0–10000], api_latency [0–5000ms], rolling_p99 [0–5000ms]
• Business SLAs — db_connection_pool, bank_api_status, merchant_tier

5.2 Observation Space (10 Fields)

All values normalized to [0.0, 1.0] in the observation. Raw values available in info["raw_obs"].

Field	Raw Range	Role / Key Threshold
transaction_type	{0, 1}	UPI vs Card rail
risk_score	[0–100]	>80 → catastrophe on Approve+SkipVerify
adversary_threat_level	[0–10]	Escalates after 5 episodes of high performance
system_entropy	[0–100]	>70 → random latency spike
kafka_lag	[0–10000]	>4000 → crash (reward=0, done=True)
api_latency	[0–5000ms]	Driven by lag + bank_status + entropy
rolling_p99	[0–5000ms]	EMA of api_latency; SLA gate at 800ms → −0.30
db_connection_pool	[0–100]	Pool saturation drives retry penalties
bank_api_status	{0, 1, 2}	Healthy / Degraded / Down
merchant_tier	{0, 1}	Small vs Enterprise; affects optimal app_priority

5.3 Action Space (6 Fields, 216 Combinations)

MultiDiscrete([3, 2, 3, 2, 2, 3])

Action	Choices	Key Failure Condition
risk_decision	0=Approve, 1=Reject, 2=Challenge	Approve + SkipVerify + risk>80 → fraud catastrophe (reward=0)
crypto_verify	0=FullVerify, 1=SkipVerify	SkipVerify on Reject+high-risk = optimal (the Blind Spot)
infra_routing	0=Normal, 1=Throttle, 2=CircuitBreaker	CircuitBreaker → −0.50/step penalty
db_retry_policy	0=Fail-Fast, 1=ExponentialBackoff	Backoff when pool<20 → −0.10 waste penalty
settlement_policy	0=StandardSync, 1=DeferredAsyncFallback	DeferredAsync during Normal → −0.15 penalty
app_priority	0=UPI, 1=Credit, 2=Balanced	Mismatch to merchant_tier → missed +0.02 bonus/step

5.4 Three Tasks with Deterministic Graders

Task	Phase Sequence	Grader Threshold	Seed
easy	Normal × 100 steps	≥ 0.75	42
medium	Normal × 40 → Spike × 60	≥ 0.45	43
hard	Normal × 20 → Spike × 20 → Attack × 40 → Recovery × 20	≥ 0.30	44

Each grader runs 10 episodes with a fixed seed. Scores are deterministic and reproducible. All scores in [0.0, 1.0].

5.5 Reward Function

Base reward = 0.8. Final = clamp(base + bonuses − penalties, 0.0, 1.0).

Catastrophic conditions (override everything):

• Approve + SkipVerify + risk_score > 80 → reward = 0.0, done = True
• kafka_lag > 4000 → reward = 0.0, done = True
• rolling_p99 > 800ms → −0.30 penalty

Anti-reward-hacking by design:

• Always CircuitBreaker → −0.50/step
• Always DeferredAsync → −0.15 or −0.20
• Always ExponentialBackoff when pool<20 → −0.10

Reward is dense — partial progress is rewarded at every step, not just at episode end.

5.6 Eleven Causal State Transitions

AEPO is not a memoryless simulator. These 11 transitions make it a world model:

1. Lag → Latency — lag >3000 compounds into api_latency next step
2. Throttle Relief Queue — queues −150 lag reductions at t+1 and t+2
3. Bank Coupling — Degraded bank + StandardSync → rolling_p99 += 200
4. DB Pressure — pool>80 + ExponentialBackoff → +100ms latency
5. DB Waste — pool<20 + ExponentialBackoff → −0.10 reward
6. Entropy Spike — system_entropy>70 → random +100–300ms latency
7. Adversary Escalation — 5-episode rolling avg gates adversary level changes
8. P99 EMA — α=0.2 EMA — cannot be corrected in a single step
9. Circuit-Breaker State Machine — open → half-open → closed
10. Bank API Markov Flapping — per-phase transition probabilities
11. Diurnal Clock Signal — sinusoidal lag modulation, unobservable by agent

---

6. Technical Requirements — OpenEnv Compliance

Requirement	Implementation
Typed Observation Pydantic model	AEPOObservation(BaseModel) — 10 fields with ge/le validators
Typed Action Pydantic model	AEPOAction(BaseModel) — 6 fields with integer range validators
step(action) → (obs, reward, done, info)	Returns 4-tuple — NOT 5-tuple (locked per OpenEnv spec)
reset() → (obs, dict)	Returns 2-tuple
state() → AEPOObservation	Returns current observation
openenv.yaml with task metadata	Present; tasks: easy, medium, hard
openenv validate passes	✅ Validated in strict mode

Info Dict Contract

Every step() returns a full info dict including:

• phase — current task phase (Normal / Spike / Attack / Recovery)
• curriculum_level — current difficulty level
• step_in_episode — step counter
• raw_obs — all 10 raw (un-normalized) values
• reward_breakdown — base + all penalty/bonus components
• termination_reason — why done=True was triggered (if applicable)
• adversary_threat_level_raw — raw adversary value
• blind_spot_triggered — boolean flag for Reject+SkipVerify event
• consecutive_deferred_async — counter for DeferredAsync abuse detection

---

7. Training Requirements

7.1 Q-Table Agent (Implemented)

Parameter	Value
Training episodes	500
State features	7: risk_score, kafka_lag, rolling_p99, db_connection_pool, bank_api_status, merchant_tier, adversary_threat_level
Discretization bins	4 → 4^7 = 16,384 reachable states
Curriculum advance (easy→medium)	5-episode rolling avg > 0.65
Curriculum advance (medium→hard)	5-episode rolling avg > 0.38
Training runtime	~3–4 seconds on 2 vCPU

7.2 Trained Scores (All PASS)

Task	Random Baseline	Heuristic (Human SRE)	Trained Agent	Threshold	Status
easy	~0.50	~0.76	~0.76+	≥ 0.75	✅ PASS
medium	~0.55	~0.41	~0.63+	≥ 0.45	✅ PASS
hard	~0.25	~0.30	~0.6650	≥ 0.30	✅ PASS (2.25×)

7.3 LagPredictor — PyTorch World Model

2-layer MLP: 16 inputs (10 obs normalized + 6 action scalars) → 1 output (next kafka_lag normalized). Final MSE = 0.007 on held-out transitions. Trains alongside the Q-table loop on collected (state, action, next_lag) transitions.

7.4 TRL + Unsloth Colab Notebook · ✅ Code Shipped · ⏳ E2E Run Pending

⚠️ Mandatory Round-2 deliverable. This section is the single source of truth for notebook status — §4 and §10 mirror it.

Artifact: AEPO_Unsloth_GRPO.ipynb (repo root, 16 cells). Implements GRPO end-to-end against the in-process AEPO env.

Sub-deliverable	Status	Evidence / Action
Notebook uses Unsloth	✅ Done	FastLanguageModel.from_pretrained(..., load_in_4bit=True, fast_inference=True) — Cell 3
Notebook uses TRL GRPO	✅ Done	from trl import GRPOConfig, GRPOTrainer — Cell 3, training in Cell 9
Connects to AEPO env	✅ Done	from unified_gateway import UnifiedFintechEnv, AEPOAction — Cells 1, 5
Hardware-aware: Qwen2.5-7B on A10G ≥22 GB; falls back to 3B on T4	✅ Done	_vram_gb branch — Cell 3, Cell 7, Cell 9
Reward function (env_reward_func) parses 6-int completion → AEPOAction → env.step() → float reward; logs Blind Spot #1 hits	✅ Done	Cell 5 (FIX-1..FIX-4 documented inline)
Dataset deterministically reconstructable: (seed_val, task_name) columns forwarded to reward func	✅ Done	Cell 7 — 50% hard / 33% medium / 17% easy split
Reward curve plot — saves results/grpo_reward_curve.png	✅ Code Shipped · ⏳ Run Pending	Cell 11 — runs only after trainer.train() completes on a GPU runtime
Before / after eval table (heuristic vs GRPO) on all 3 task tiers	✅ Code Shipped · ⏳ Run Pending	Cell 13
LoRA adapter saved + optionally pushed to HF Hub	✅ Code Shipped	Cell 15 — gated on HF_TOKEN env var
Re-runnable by judges	✅ Done	Cell 1 auto-detects HF Space vs Colab vs local; clones repo on Colab; installs all extras
Notebook URL added to README	⏳ Pending	Add an "Open in Colab" badge linking to the GitHub raw URL once the repo is public

E2E run procedure (must execute on a GPU runtime — cannot run on the dev laptop):

1. Open AEPO_Unsloth_GRPO.ipynb on Colab (T4 free, 25 min) or on the HF Space A10G (35 min).
2. Run all cells top-to-bottom. Cells 1, 3 install deps and load the model.
3. Cell 9 (trainer.train()) is the long step.
4. Cell 11 writes results/grpo_reward_curve.png — download and commit this file.
5. Cell 13 prints the heuristic-vs-GRPO score table — paste into the README under §8.
6. Re-export the notebook with outputs intact (File → Download .ipynb); overwrite the repo copy and commit.
7. Flip the three ⏳ rows above to ✅ in this section, in §4, and in §10. Single source of truth.

Reference recipes:

• Qwen2.5 (3B) GRPO: https://github.com/unslothai/notebooks/blob/main/nb/Qwen2.5_%283B%29-GRPO.ipynb
• TRL GRPO Trainer: https://huggingface.co/docs/trl/grpo_trainer
• TRL OpenEnv integration: https://huggingface.co/docs/trl/openenv

---

8. Inference Script Requirements

⚠️ The inference.py file must follow these rules exactly. Deviations cause scoring failure.

File rules:

• Must be named inference.py
• Must be placed in the root directory of the project
• Must use the OpenAI client (from openai import OpenAI) for all LLM calls

Environment variables:

Variable	Description
API_BASE_URL	The API endpoint for the LLM
MODEL_NAME	The model identifier to use for inference
HF_TOKEN	Your Hugging Face / API key

Required STDOUT Format (Strict)

[START] task=<task_name> env=<benchmark> model=<model_name>
[STEP]  step=<n> action=<action_str> reward=<0.00> done=<true|false> error=<msg|null>
[END]   success=<true|false> steps=<n> score=<score> rewards=<r1,r2,...,rn>

Rules:

• One [START] line at episode begin
• One [STEP] line per step, immediately after env.step() returns
• One [END] line after env.close(), always emitted (even on exception)
• reward and rewards formatted to 2 decimal places
• done and success are lowercase: true or false
• error is the raw error string or null
• All fields on a single line — no newlines within a line
• Each task score must be in [0, 1]

8.1 Time-Budget Resilience (Spec: inference < 20 min on 2 vCPU/8 GB)

Added 2026-04-26. Hardens inference.py against slow LLM providers, network blips, and rate-limit storms that would otherwise zero a task or burn the entire 20-minute budget on a single stuck call.

Three guard rails wired into inference.py:

Guard	Constant	Value	Where it acts
Per-call LLM timeout	LLM_CALL_TIMEOUT_SEC	5.0 s	OpenAI client timeout= (default would be 600 s)
OpenAI built-in retries	max_retries	0	Disables silent re-tries — heuristic fallback handles failures
Per-task wall budget	TASK_WALL_BUDGET_SEC	300.0 s	Loop guard — ends a task early so the other two still run

Failure-mode behavior:

Failure	Before fix	After fix
LLM call hangs	Single call burns ≤ 600 s of budget	Aborts at 5 s, falls back to heuristic for that step
LLM timeout / 503 / rate limit	Exception → task aborted with reward = 0	except Exception → heuristic action, episode continues
LLM returns malformed completion	parse_llm_action exception bubbles up	Same fallback path — heuristic completes the step
One task drags past 5 minutes	Eats budget for remaining tasks	Per-task budget breaks loop, emits error="task_wall_budget_exceeded" [STEP], moves on

Worst-case math: 3 tasks × 5 min/task = 15 min, leaving 5 min headroom for HF Space cold-start, dynamics-model load, and grader computation. Spec ceiling of 20 min is never breached even if every LLM call times out.

Why heuristic fallback specifically: the heuristic policy is deterministic, in-process, and produces non-catastrophic actions on every observation (zero LLM latency, zero network risk). Heuristic-mixed scores are degraded vs full-LLM, but degraded > zero on a 100-step episode.

Spec compliance: All three guards are passive — they only activate on failure. Healthy LLM runs see no behavior change. The [START]/[STEP]/[END] format is preserved on every code path, including the budget-exceeded path.

---

9. Deployment Requirements

Hugging Face Space

• Environment must be deployed as a Docker-based HF Space tagged openenv
• Must respond to POST /reset with HTTP 200 (automated ping checks this)
• Accessible at a stable public URL

API Endpoints

Endpoint	Method	Description
/	GET	Health check — returns {"status": "healthy"}
/reset	POST	Reset environment for a given task
/step	POST	Step environment with an action
/state	GET	Return current observation

Dockerfile Requirements

• Working Dockerfile in the repository root (or server/ directory)
• Must succeed with: docker build && docker run
• Base image: python:3.10-slim
• Must expose port 7860
• CMD: uvicorn server.app:app --host 0.0.0.0 --port 7860

Infrastructure Constraints

Constraint	Value
Max inference runtime	< 20 minutes
Target compute	2 vCPU, 8 GB RAM
Environment port	7860
Python version	3.10
OpenEnv step return	4-tuple (obs, reward, done, info) — NOT 5-tuple
LLM API client	OpenAI Python SDK (OpenAI-compatible interface)
Submission policy	One submission per team; no commits after deadline

---

10. Deliverables Checklist

Code & Environment

• unified_gateway.py — Core environment (AEPO v10) with all 11 causal transitions
• server/app.py — FastAPI wrapper exposing /reset, /step, /state
• inference.py — In root directory, uses OpenAI client, emits strict log format
• train.py — Q-table + LagPredictor training (500 episodes, ~3–4s on 2 vCPU)
• graders.py — Deterministic graders for easy/medium/hard (10 episodes, fixed seeds)
• openenv.yaml — Manifest with all 3 tasks
• Dockerfile — Working build + run
• requirements.txt — All dependencies pinned

Tests

• 221 tests across 14 files — all passing
• unified_gateway.py at ≥97% coverage
• pytest tests/ -v runs cleanly

Training Evidence

• results/reward_curve.png — staircase improvement curve committed to repo
• Reward curve embedded in README with caption
• Training comparison table (random vs heuristic vs trained) in README

TRL + Unsloth Colab Notebook · ✅ Code Shipped · ⏳ E2E Run Pending

(Mirrors §7.4 — single source of truth lives there. Update both sections together.)

• Colab notebook using Unsloth + TRL GRPO — AEPO_Unsloth_GRPO.ipynb
• Connects to AEPO environment (in-process import of UnifiedFintechEnv)
• Reward plot from an actual training run — results/grpo_reward_curve.png (run Cell 9 + 11 on Colab/A10G, commit the PNG)
• Plot committed to repo and linked from README (see §7.4 procedure step 4–5)
• Notebook link in README — add Open-in-Colab badge to the GitHub raw URL
• Notebook re-exported with outputs intact and committed — jupyter nbconvert --to notebook --execute is not acceptable; must show real GPU outputs

Writeup ⬜ Required for Round 2

• Mini-blog on Hugging Face OR <2 minute YouTube video
• Covers: problem statement, environment design, what the agent learned, results
• Link added to README

README

• Problem motivation (Siloed Metrics, Asymmetric Risk Triad)
• Environment description (observation/action spaces, phase structure)
• Embedded reward curve with caption
• Baseline scores table (random / heuristic / trained per task)
• Link to HF Space (live URL)
• Link to writeup (blog or video)
• Link to Colab training notebook
• Setup and usage instructions
• openenv validate passing confirmation

Deployment

• HF Space live and responds to POST /reset with HTTP 200
• Space tagged openenv
• docker build succeeds on submitted repo
• inference.py runs without error and produces [START]/[STEP]/[END] output

---

11. Pre-Submission Validation

Run the official validation script before submitting:

./validate-submission.sh <your-hf-space-url> [repo-dir]

This checks: (1) HF Space is live, (2) Docker build succeeds, (3) OpenEnv spec compliance.

Manual Verification Commands

# OpenEnv validation
openenv validate .

# Docker
docker build -t aepo .
docker run -p 7860:7860 aepo

# Full test suite (expect: 221 passed)
pytest tests/ -v --tb=short
pytest tests/ --cov=unified_gateway --cov-report=term-missing

# Training (expect: hard task ~0.67, PASS)
python train.py

# Inference dry run (heuristic agent — no LLM required)
DRY_RUN=true python inference.py

# Inference smoke test against a real LLM with strict time budgets active
# (5 s per LLM call, 5 min per task, 20 min total — see Section 8.1)
API_BASE_URL=... MODEL_NAME=... HF_TOKEN=... python inference.py

Disqualification Checklist

Check	Must Pass
HF Space deploys	Automated ping to Space URL — must return 200 and respond to reset()
OpenEnv spec compliance	openenv.yaml, typed models, step()/reset()/state() endpoints
Dockerfile builds	Automated docker build on submitted repo
Baseline reproduces	Inference script completes without error and produces scores
3+ tasks with graders	Graders enumerate tasks, run each, verify scores in [0.0, 1.0]

---

12. Technology Stack

Layer	Technology	Version	Role
Runtime	Python	3.10+	Core language
RL Framework	Gymnasium	0.29.1	gym.Env base class
Type Safety	Pydantic	v2.0+	Runtime validation of Observation/Action
Numerical	NumPy	1.26.4	Array backing for observation space
Dynamics Model	PyTorch	2.2.0	LagPredictor 2-layer MLP
API Server	FastAPI	Latest	Async HTTP endpoints
ASGI Server	Uvicorn	Latest	Serves FastAPI on port 7860
LLM Client	OpenAI SDK	1.0+	OpenAI-compatible client for inference
Containerization	Docker	python:3.10-slim	Hugging Face Spaces deployment
OpenEnv SDK	openenv-core	0.2.0+	openenv validate CLI
Deployment	HF Spaces	—	Persistent Docker container
RL Training	TRL	Latest	GRPO/PPO trainer (Colab notebook)
Efficiency	Unsloth	Latest	Fast RL fine-tuning (Colab notebook)

---

13. Key Differentiators from Round 1

Architectural Advances

• 10-field observation vs 5-field — doubles the signal richness
• 216 unique action combinations vs 18 — 12× larger policy space
• 11 causal state transitions vs 0 — transforms memoryless simulator into persistent world
• 4-phase task machine vs none (Normal, Spike, Attack, Recovery)

World Modeling

• LagPredictor PyTorch MLP trains on rollout transitions and predicts future Kafka lag
• Diurnal clock signal (sinusoidal modulation, unobservable) forces proactive hedging

The Learning Story

• Q-table discovers Blind Spot #1 at Episode 3, Step 42: Reject+SkipVerify on high-risk transactions is non-obvious optimal
• Saves 250 lag/step, earns +0.04 bonus — but feels unsafe to humans, so the SRE heuristic never found it
• Trained hard task score 0.6650 vs heuristic 0.2955: the agent learned something its creator missed

Anti-Reward-Hacking Design

• Every shortcut defeated: always CircuitBreaker → −0.50 penalty → 0.30/step net reward (base 0.8 − 0.50); always DeferredAsync → −0.15/−0.20
• Adversary escalation: performs well → environment gets harder → staircase learning curve

Engineering Quality

• 221 tests at 97% coverage vs ~30 tests
• Dual-mode architecture (standalone + FastAPI server, zero code changes)
• openenv validate strict-mode passing

---

14. Resource Links

AEPO Project

Resource	URL
HF Space (live environment)	(add URL before submission)
GitHub repo	(add URL before submission)
Mini-blog / video writeup	(⬜ Required — add URL before submission)
Training Colab notebook	Local: AEPO_Unsloth_GRPO.ipynb · Colab badge URL: (add public GitHub raw URL — see §7.4 step 7)
Competition dashboard	https://www.scaler.com/school-of-technology/meta-pytorch-hackathon/dashboard

OpenEnv

Resource	URL
GitHub	https://github.com/meta-pytorch/OpenEnv
Docs	https://meta-pytorch.org/OpenEnv/
HF Hub — Environments	https://huggingface.co/openenv
Tutorials	https://github.com/meta-pytorch/OpenEnv/tree/main/tutorial
Environment Examples	https://github.com/meta-pytorch/OpenEnv/tree/main/envs
Reward Design Guide	https://meta-pytorch.org/OpenEnv/guides/rewards.html
TRL Integration	https://huggingface.co/docs/trl/openenv

Training Stack

Resource	URL
TRL GRPO Trainer	https://huggingface.co/docs/trl/grpo_trainer
HF GRPO Cookbook	https://huggingface.co/learn/cookbook/fine_tuning_llm_grpo_trl
Unsloth Notebooks	https://github.com/unslothai/notebooks
Qwen2.5 (3B) GRPO	https://github.com/unslothai/notebooks/blob/main/nb/Qwen2.5_%283B%29-GRPO.ipynb
Gemma3 (1B) GRPO	https://github.com/unslothai/notebooks/blob/main/nb/Gemma3_%281B%29-GRPO.ipynb
Unsloth repo	https://github.com/unslothai/unsloth

Learning Videos (Hackathon Guide)

Module	URL	Content
Why OpenEnv (~7 min)	https://www.youtube.com/watch?v=1jU05MlENOI&t=482s	RL loop, fragmented env APIs, OpenEnv as universal interface
Using Existing Envs (~7.5 min)	https://www.youtube.com/watch?v=1jU05MlENOI&t=2133s	Hub org, env collections, Space interfaces, from_hub
Deploying Envs (~9 min)	https://www.youtube.com/watch?v=Jew4lhAiqnw&t=5400s	openenv init, scaffold, running locally, openenv push
Building Your Own (~6.5 min)	https://www.youtube.com/watch?v=1jU05MlENOI&t=2625s	Scaffold files, business logic, models, client, publishing
Training + TRL (~14 min)	https://www.youtube.com/watch?v=Jew4lhAiqnw&t=6800s	Wordle GRPO walkthrough — rollout, GRPOTrainer, live training
RL Mega Lecture (Recommended)	https://www.youtube.com/watch?v=Jew4lhAiqnw	Full lecture — start here
Workshop Full	https://www.youtube.com/watch?v=1jU05MlENOI	Full workshop
Live Session	https://www.youtube.com/live/kkCNMz0Ptd8	Live build session

Research Papers — Reward Engineering

• https://arxiv.org/abs/2408.10215
• https://arxiv.org/abs/2601.19100

---

15. Hackathon Execution Guide

15.1 1-Day Execution Plan

Phase	Task	Key Output
1 — Pick	Choose a narrow, verifiable environment	Clear problem statement with objective reward
2 — Build Env	Implement reset/step/state, get local loop working	Working environment with local test
3 — Build Rewards	Add 2–4 independent reward checks + timeout + anti-cheat	Multi-component reward function
4 — Deploy	Push to HF Space or run via container/Uvicorn	Shared environment accessible to teammates
5 — Train Small	Tiny TRL + Unsloth experiment, look at outputs	First reward curves (even if noisy)
6 — Inspect	Sample generations, check for globals/hacks/shortcuts	Confirmed no reward hacking
7 — Curriculum	Simplify tasks if model gets zero reward too often	Non-zero reward in early training
8 — Train Bigger	Increase scale, batch size, environment diversity	Stable learning curve
9 — Save & Demo	Export model correctly, test inference, show before/after	Final demo artifact

15.2 Recommended Team Split

Role	Responsibilities
Person A — Environment	Builds reset/step/state, adds timeouts and safety constraints, makes local + remote execution work
Person B — Verifier/Rewards	Writes multiple reward functions, adds anti-hacking checks, makes failure cases visible
Person C — Training	Sets up TRL + Unsloth, runs experiments, tracks metrics and generations
Person D — Demo/Product	Prepares Space demo, creates simple interface, records examples and final benchmarks

15.3 RL Core Concepts to Keep in Mind

The minimum RL loop:

1. Give the model a prompt
2. Let it generate an action, strategy, answer, or code
3. Execute that output in an environment or verifier
4. Convert the result into a reward
5. Update the model so higher-reward behavior becomes more likely

When to use SFT vs RL:

• Have a lot of good data → use SFT
• No data but can verify outputs → use RL
• Best of both: light SFT first for warm start, then RL for improvement

GRPO vs PPO: Prefer GRPO/RLVR for verifiable tasks — more efficient, no value model needed. Build the verifier first, then plug into RL training.

Inference bottleneck: In RL for LLMs, rollout generation often dominates runtime — not the optimizer step. Fast sampling and tight environment loops are critical (why Unsloth matters).

15.4 Common Mistakes to Avoid

• Task too hard — if success probability is zero, RL learns nothing. Start simple, add curriculum.
• Single reward function — easy to game. Use 2–4 independent checks.
• Not checking for reward hacking — inspect actual generations, not just average reward.
• Training before environment is stable — confirm reset/step/rewards work before scaling.
• Ignoring output quality — a rising reward means nothing if the model is exploiting bugs.
• Forgetting timeouts and sandbox limits — essential for preventing infinite loops.
• Saving LoRA/QLoRA models incorrectly — never upcast 4-bit to 16-bit naively before merging.

15.5 What Judges Find Most Compelling

A strong demo shows:

1. Baseline model attempt → reward/verifier output
2. Trained model attempt → measurable improvement
3. Short explanation of safeguards against reward hacking
4. Clear environment design with objective, non-gameable rewards
5. Reproducible deployment — judges can pull and run your environment

"A messy but ambitious environment with real training evidence beats a polished but boring one. Pick a problem that excites you — that energy comes through in the pitch."

---

Document End · AEPO Grand Finale — Master Baseline v1.0 Author: Umesh Maurya · Date: April 25, 2026 Sources: MASTER_PROJECT_REQUIREMENTS.md + Hackthon_guid.md + Hackthon_Themes.md + OpenEnv_Hackathon_Resources.docx