Spaces:
Sleeping
Sleeping
| title: AI Executive Operations Manager | |
| emoji: ⚡ | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| tags: | |
| - openenv | |
| # AI Executive Operations Manager | |
| An OpenEnv-compliant reinforcement learning environment where an AI agent plays the role of **Alex Rivera**, CEO of **NovaTech AI** - a fictional Series B AI infrastructure startup with ~80 employees. The agent manages a realistic executive inbox under time pressure, making triage decisions with real business consequences. | |
| --- | |
| ## What is this? | |
| The agent's job is to read emails, decide what to act on, and choose *how* to act. Every choice has consequences. The clock is ticking and there is always more to do than time allows. | |
| Each email has a **priority** (1-5) and a **decaying urgency** score. The agent picks one of four actions per step: | |
| | Action | When to use | | |
| |--------|-------------| | |
| | `reply` | CEO must personally handle - investor relations, legal deadlines, critical incidents | | |
| | `schedule` | Needs a meeting - strategic discussions, relationship-building | | |
| | `delegate` | Can be handled by the team - operational, low-stakes items | | |
| | `ignore` | Truly trivial - only safe when no critical items are unhandled | | |
| **Format:** `{"type": "reply|schedule|delegate|ignore", "email_id": "<id>"}` | |
| --- | |
| ## The Core Challenge | |
| Good decisions require real prioritization - not just doing the most urgent thing first, but understanding *what matters most* and *what can be handed off*. | |
| **You cannot do everything.** Every step costs time. The hardest scenario is designed so a perfect score is mathematically impossible - the agent must triage ruthlessly and decide what to sacrifice. | |
| ### Why it's hard | |
| - **Conflicting priorities** - Two fires at once, one CEO | |
| - **Urgency vs. importance** - Something loud is not always something critical | |
| - **Delegation trade-offs** - Some things only you can do; others you shouldn't touch | |
| - **Time decay** - Urgency drops 0.15 per step; waiting on critical items costs you | |
| - **Impossible situations** - The hard scenario cannot be perfectly solved. The score reflects *which* crises the agent chose to address | |
| --- | |
| ## The Three Scenarios | |
| ### Easy - "Monday Morning Catchup" | |
| 4 emails, 6 steps. Priorities are clear. Designed to establish a baseline - a competent agent handles everything well. | |
| Key decisions: | |
| - Approve a production deploy to fix a live memory leak costing $3K/hour | |
| - Sign off on a $1.2M GlobalBank enterprise contract before a competitor swoops in | |
| - Delegate new hire onboarding paperwork to HR | |
| - Ignore the office snack order | |
| **Expected score:** 0.70 - 1.00 | |
| --- | |
| ### Medium - "Investor Demo Day Prep" | |
| 7 emails, 8 steps. NovaTech's Series B pitch is today. Multiple high-priority items compete for the same limited time. Requires smart scheduling and knowing what to delegate. | |
| Key decisions: | |
| - Schedule a pre-demo call with the lead Sequoia investor | |
| - Align with the CTO on architecture before investors ask about scaling | |
| - Sign an NDA legally required before the presentation | |
| - Approve an offer letter for a senior ML engineer before they go to OpenAI | |
| - Manage the CFO's runway concerns (8.5 months left, down from 11) | |
| **Expected score:** 0.45 - 0.85 | |
| --- | |
| ### Hard - "Series B Crisis Day" | |
| 10 emails, 8 steps, 7 goals. Four simultaneous P5 crises. **A perfect score is intentionally impossible.** A top agent scores around 0.65-0.75. | |
| The crises hitting all at once: | |
| - Lead investor threatening to pull an $18M term sheet before 4PM | |
| - Production database corruption affecting 23 enterprise customers ($85K/hour revenue impact) | |
| - TechCrunch publishing a false security breach story in 90 minutes | |
| - Key engineer resigning to join Anthropic - mid Series B | |
| - GDPR data deletion deadline expiring at 5PM (20M euro fine risk) | |
| - AWS contract expiring at midnight (cost jumps from $180K to $340K/month) | |
| - Co-founder publicly disagreeing on strategy before the board meeting | |
| - Microsoft acquisition inquiry ($400-600M range) | |
| The agent must decide what to sacrifice. | |
| **Expected score:** 0.20 - 0.75 | |
| --- | |
| ## Observation Space | |
| Each observation is a JSON dict: | |
| ```json | |
| { | |
| "time": "10:00 AM", | |
| "step": 2, | |
| "max_steps": 8, | |
| "steps_remaining": 6, | |
| "inbox": [ | |
| { | |
| "id": "e1", | |
| "sender": "Sarah Chen <s.chen@novatech.ai>", | |
| "subject": "URGENT: Production deploy approval", | |
| "body": "...", | |
| "priority": 5, | |
| "urgency": 0.65, | |
| "handled": false, | |
| "action_taken": null | |
| } | |
| ], | |
| "calendar": [ | |
| {"time_slot": 3, "title": "Board Sync", "attendee": "Board of Directors", "locked": true} | |
| ], | |
| "goals": [ | |
| { | |
| "id": "g1", | |
| "description": "Approve production deploy to fix memory leak", | |
| "priority": 5, | |
| "required_action": "reply", | |
| "target_email_id": "e1", | |
| "completed": false | |
| } | |
| ], | |
| "pending_goals": [ | |
| { | |
| "id": "g1", | |
| "description": "Approve production deploy to fix memory leak", | |
| "priority": 5, | |
| "required_action": "reply", | |
| "target_email_id": "e1", | |
| "completed": false | |
| } | |
| ], | |
| "total_reward": 0.498, | |
| "done": false | |
| } | |
| ``` | |
| - `priority`: 1 (minimal) to 5 (critical) | |
| - `urgency`: 0.0-1.0, decays by 0.15 each step - act fast on high-urgency items | |
| - `goals`: all goals with completion status | |
| - `pending_goals`: only incomplete goals | |
| --- | |
| ## Reward Function | |
| Per-step continuous reward, clamped to `[-0.3, 0.5]`: | |
| | Signal | Value | | |
| |--------|-------| | |
| | Goal completion | `+0.30 x (priority / 5)` | | |
| | Urgency bonus | `+0.15 x urgency` (if urgency > 0.5, not ignoring) | | |
| | Correct action type | `+0.10` (matches goal's required action) | | |
| | Smart delegation (P1-P2) | `+0.05` | | |
| | Ignore critical (P4-P5) | `-0.25` | | |
| | Waste step on trivial when crises pending | `-0.10` | | |
| | Per-step cost | `-0.02` | | |
| --- | |
| ## Grader | |
| Deterministic. Score in [0.0, 1.0]: | |
| ``` | |
| score = 0.60 x goal_completion_rate (priority-weighted) | |
| + 0.25 x email_handling_rate (priority-weighted) | |
| + 0.15 x efficiency_bonus (only if all high-priority goals done) | |
| ``` | |
| Goals carry 60% of the weight because completing the right goal (saving an $18M term sheet) matters far more than handling any random email. Efficiency only rewards speed *after* critical goals are met - it never trumps correctness. | |
| --- | |
| ## What makes a good agent? | |
| 1. **Triage instinct** - Identify the two or three things that absolutely cannot wait | |
| 2. **Delegation confidence** - Know what doesn't need the CEO's personal attention | |
| 3. **Urgency sensitivity** - Act on time-decaying items before they expire | |
| 4. **Sacrifice awareness** - In the hard scenario, explicitly choose what to drop | |
| The environment rewards agents that think like an actual executive under pressure - not agents that just process emails in order. | |
| --- | |
| ## Setup & Running | |
| ### Docker (Recommended) | |
| ```bash | |
| docker build -t exec-ops-manager . | |
| docker run -p 7860:7860 exec-ops-manager | |
| # Open http://localhost:7860 | |
| ``` | |
| ### Local Development | |
| ```bash | |
| pip install -r requirements.txt | |
| uvicorn app:app --port 7860 --reload | |
| ``` | |
| ### Running Inference / Demo Script | |
| `inference.py` runs an LLM agent through all 3 tasks and prints structured logs. | |
| ```bash | |
| export API_BASE_URL="https://api.openai.com/v1" | |
| export MODEL_NAME="gpt-4o-mini" | |
| export HF_TOKEN="your-key" | |
| python inference.py | |
| ``` | |
| Output format: | |
| ``` | |
| [START] task=easy env=exec-ops model=gpt-4o-mini | |
| [STEP] step=1 action=reply('e1') reward=0.50 done=false error=null | |
| [STEP] step=2 action=reply('e2') reward=0.41 done=false error=null | |
| [STEP] step=3 action=delegate('e3') reward=0.26 done=true error=null | |
| [END] success=true steps=3 score=0.906 rewards=0.50,0.41,0.26 | |
| ``` | |
| --- | |
| ## API Endpoints | |
| | Endpoint | Method | Description | | |
| |----------|--------|-------------| | |
| | `/` | GET | Serves React dashboard (200 OK) | | |
| | `/reset?task=easy` | GET/POST | Reset environment, returns initial observation | | |
| | `/step` | POST | Execute one action, returns observation + reward | | |
| | `/state` | GET | Full current environment state | | |
| | `/grade` | GET | Current score (0.0-1.0) | | |
| | `/tasks` | GET | List all available tasks | | |
| --- | |
| ## Baseline Results | |
| *Model: `gpt-4o-mini` via OpenAI API* | |
| | Task | Score | Steps used | | |
| |------|-------|------------| | |
| | easy | 0.906 | 3 / 6 | | |
| | medium | 0.866 | 5 / 8 | | |
| | hard | 0.681 | 8 / 8 | | |
| | **Average** | **0.818** | | | |
| --- | |
| ## Architecture | |
| ``` | |
| Single Docker container (port 7860) | |
| ├── FastAPI (app.py) - API + static file serving | |
| ├── env/ - Pure Python RL environment | |
| │ ├── models.py - Pydantic data models | |
| │ ├── tasks.py - 3 task definitions | |
| │ ├── reward.py - Per-step reward function | |
| │ ├── grader.py - Deterministic final grader | |
| │ └── environment.py - ExecOpsEnv state machine | |
| ├── frontend/build/ - Pre-built React dashboard | |
| └── inference.py - LLM agent baseline script | |
| ``` | |