tasks/gaps.md

# SentinelOps Arena -- Gap Analysis & 4-Hour Action Plan

**Generated:** Sunday March 8, 2026 ~9:00 AM
**Deadline:** Sunday March 8, 2026 1:00 PM (4 hours remaining)
**Status:** Strong core implementation, missing 3 required submission deliverables

---

## EXECUTIVE SUMMARY

The environment implementation is solid: 3 agents, 3 systems, 4 attack types, reward functions, randomized attacker, security metrics, and a polished Gradio UI with cybersecurity theme. The code runs without errors and the trained vs untrained comparison shows meaningful differences (30.0 vs 25.0 worker score).

**However, the 3 required hackathon deliverables are NOT done:**
1. HuggingFace Spaces deployment -- NOT DEPLOYED
2. Google Colab training notebook -- DOES NOT EXIST
3. Demo video on YouTube -- NOT RECORDED

Without all three, the submission is invalid regardless of code quality.

---

## GAP LIST (Prioritized)

### BLOCKER -- Must fix or submission fails

| # | Gap | Details | Estimated Time |
|---|-----|---------|----------------|
| B1 | **No HuggingFace Spaces deployment** | README.md has correct frontmatter (sdk: gradio, sdk_version: 6.9.0, app_file: app.py). No HF remote configured. Need to create Space and push. requirements.txt exists but may need pandas added. | 30 min |
| B2 | **No Colab training notebook** | `training/` directory is empty. Submission requires "Minimal Training Script" as Colab notebook. `train.py` exists at root but is a standalone Python script, not a notebook. Must create `training/colab_training.ipynb`. | 60-90 min |
| B3 | **No demo video** | Submission requires YouTube demo video. Need to screen record the Gradio app, show episode replay, before/after comparison, and explain the 3-agent dynamic. SETUP.md says 1 minute length. | 30 min |
| B4 | **No `nihal` branch** | CLAUDE.md says push to `nihal` but only `main` exists. All code is on `main`. Need to create branch and push. | 5 min |

### HIGH -- Significantly improves judging score

| # | Gap | Details | Estimated Time |
|---|-----|---------|----------------|
| H1 | **Gradio app not verified to launch** | `tasks/todo.md` shows "Gradio app launches without errors" is UNCHECKED. Must verify `python app.py` works and the UI renders correctly. Fix any runtime errors. | 15 min |
| H2 | **requirements.txt missing pandas** | `requirements.txt` has 6 packages but `app.py` imports pandas (via chart_helpers.py, inspector.py). HF Spaces will fail to install. Must add `pandas>=2.0`. | 2 min |
| H3 | **SENTINELOPS_ARENA.md claims "80 ticks" and "OpenEnv 0.4"** | Environment actually uses 30 ticks and OpenEnv 0.2.x. Spec doc has aspirational content that doesn't match reality. Judges who read the spec will notice discrepancies. README.md is more accurate but should be cross-checked. | 15 min |
| H4 | **pyproject.toml version mismatch** | `pyproject.toml` says `gradio>=5.0.0`, README frontmatter says `sdk_version: 6.9.0`, `requirements.txt` says `gradio>=6.0.0`. Should be consistent. | 5 min |
| H5 | **train.py uses `datasets` and `trl` but these aren't in requirements.txt** | train.py has GPU-only dependencies that are correctly optional, but the Colab notebook needs them listed. Just awareness -- Colab notebook handles its own installs. | 0 min |
| H6 | **No `nihal` branch for pushing** | CLAUDE.md mandates pushing to `nihal`, but no such branch exists. | 2 min |

### MEDIUM -- Nice to have for judges

| # | Gap | Details | Estimated Time |
|---|-----|---------|----------------|
| M1 | **Colab notebook should show real training signal** | Even a few training steps with decreasing loss would impress judges (especially Daniel Han from Unsloth and Michael Han, Unsloth CTO). The reward_function in train.py is well-designed for this. | included in B2 |
| M2 | **About tab could link to Colab notebook and video** | Once created, add links to the About tab in app.py for judges to find easily. | 10 min |
| M3 | **No mcp_x/ gateway demo** | SENTINELOPS_ARENA.md describes MCP-X per-agent tool isolation, but it's not implemented. The MCP tools ARE defined in environment.py (19 tools), just no gateway layer. Not critical but was a differentiator in the spec. | SKIP |
| M4 | **hackathon_env/ directory is vestigial** | Contains old echo environment template. Should be in .gitignore or removed to avoid confusing judges. | 5 min |
| M5 | **README.md project structure shows files that don't exist** | Lists `mcp_tools.py` separately but MCP tools are inline in `environment.py`. Minor but sloppy. | 10 min |

### LOW -- Skip for the 4-hour window

| # | Gap | Details | Estimated Time |
|---|-----|---------|----------------|
| L1 | **No compound attacks** | Spec describes compound attacks (2-3 simultaneous), not implemented | 2+ hours |
| L2 | **No compliance drift attack type** | Spec describes it, not implemented (only 4 of 6 attack types exist) | 1+ hours |
| L3 | **A2A protocol not implemented** | Already marked as "Cut" in spec. Correct decision. | N/A |
| L4 | **No Docker support** | HF Spaces uses Gradio SDK, Docker was backup option. Not needed. | N/A |
| L5 | **SENTINELOPS_ARENA.md has unrealized training dynamics section** | Describes episodes 1-50, 50-200, 200-500, 500+ progression that hasn't been trained. This is aspirational/theoretical. | N/A |

---

## WHAT'S DONE AND WORKING (Assets to leverage)

- Core environment: `SentinelOpsArena(MCPEnvironment)` with step/reset/state -- WORKING
- 3 enterprise systems (CRM, Billing, Ticketing) with full CRUD -- WORKING
- 4 attack types (schema_drift, policy_drift, social_engineering, rate_limit) -- WORKING
- 3 reward functions matching spec tables exactly -- WORKING
- RandomizedAttacker with budget, probability, seeded RNG -- WORKING
- HeuristicWorker with trained/untrained modes -- WORKING
- HeuristicOversight with violation detection -- WORKING
- 19 MCP tools registered via FastMCP -- WORKING
- HTTP server via `create_app()` -- WORKING
- Security metrics: ASR, Benign Task Success, FPR, MTTD, Social Eng. Resistance -- WORKING
- Gradio UI with 4 tabs (Run Episode, Untrained vs Trained, Environment Inspector, About) -- EXISTS (needs verification)
- Custom cybersecurity theme (SentinelTheme) -- EXISTS
- Styled HTML replay renderer -- EXISTS
- Chart helpers for LinePlot/BarPlot -- EXISTS
- train.py with GRPO pipeline, env verification, data collection -- EXISTS (GPU-only)
- README.md with correct HF Spaces frontmatter -- EXISTS

---

## 4-HOUR ACTION PLAN

### Phase 1: Verify & Fix (0:00 - 0:45) -- 45 minutes

**Goal: Make sure everything that exists actually works**

1. **[5 min] Create `nihal` branch and push** (B4, H6)
   ```bash
   git checkout -b nihal
   git push origin nihal
   ```

2. **[2 min] Fix requirements.txt** (H2)
   - Add `pandas>=2.0` to requirements.txt
   - Verify `gradio>=6.0.0` (not 5.0.0)

3. **[15 min] Verify Gradio app launches** (H1)
   ```bash
   cd /Users/nihalnihalani/Desktop/Github/NexusEnv
   python app.py
   ```
   - Test all 4 tabs: Run Episode, Untrained vs Trained, Environment Inspector, About
   - Fix any import errors, rendering issues, or crashes
   - Take screenshots for the video

4. **[10 min] Fix pyproject.toml consistency** (H4)
   - Set `gradio>=6.0.0` in pyproject.toml
   - Verify `requires-python = ">=3.12"` matches reality

5. **[10 min] Clean up misleading claims** (H3, M4, M5)
   - Remove or gitignore `hackathon_env/` directory
   - Fix README.md project structure to match reality
   - Do NOT touch SENTINELOPS_ARENA.md (it's a spec doc, acceptable to be aspirational)

6. **[3 min] Commit and push everything**

### Phase 2: HuggingFace Spaces Deployment (0:45 - 1:15) -- 30 minutes

**Goal: Get a live public URL**

1. **[5 min] Create HuggingFace Space**
   - Go to huggingface.co/new-space
   - Name: `nihalnihalani/sentinelops-arena`
   - SDK: Gradio
   - Hardware: CPU Basic (free)

2. **[10 min] Configure and push**
   ```bash
   git remote add hf https://huggingface.co/spaces/nihalnihalani/sentinelops-arena
   git push hf nihal:main
   ```
   - If push fails, use HuggingFace Hub Python API:
   ```python
   from huggingface_hub import HfApi
   api = HfApi()
   api.upload_folder(folder_path=".", repo_id="nihalnihalani/sentinelops-arena", repo_type="space")
   ```

3. **[10 min] Verify Space builds and runs**
   - Watch build logs
   - Fix any dependency issues
   - Common issues: missing packages, port mismatch (must be 7860)

4. **[5 min] Test live URL**
   - Run an episode
   - Run untrained vs trained comparison
   - Verify Environment Inspector works

### Phase 3: Colab Training Notebook (1:15 - 2:30) -- 75 minutes

**Goal: Create a working Colab notebook that demonstrates GRPO training**

1. **[45 min] Create `training/colab_training.ipynb`** (B2)
   Cells:
   - Cell 1: Install dependencies
     ```python
     !pip install unsloth "trl>=0.15" transformers torch accelerate pydantic datasets
     !pip install openenv-core[core]>=0.2.0 fastmcp>=2.14.5 mcp>=1.26.0 httpx>=0.27
     ```
   - Cell 2: Clone repo and import environment
     ```python
     !git clone https://github.com/nihalnihalani/NexusEnv.git
     import sys; sys.path.insert(0, "NexusEnv")
     from sentinelops_arena.environment import SentinelOpsArena
     from sentinelops_arena.models import AgentRole, SentinelAction
     ```
   - Cell 3: Verify environment works (run 1 episode)
   - Cell 4: Collect training data (reuse `build_training_dataset` from train.py)
   - Cell 5: Load model with Unsloth
   - Cell 6: Define reward function (reuse from train.py)
   - Cell 7: Configure GRPO and train
   - Cell 8: Show results / save model

   **Key decisions:**
   - Use `Qwen/Qwen2.5-0.5B-Instruct` (smallest, fits free Colab T4)
   - Use Unsloth for model loading, vanilla TRL GRPOTrainer for training
   - If openenv-core fails on Colab Python version, inline the minimal env code
   - Even 5-10 training steps is enough to show the pipeline works

2. **[15 min] Test notebook runs (at least partially)**
   - Upload to Colab
   - Verify cells 1-4 work (env setup + data collection)
   - Cells 5-8 need GPU -- verify they at least don't crash on import

3. **[15 min] Polish and save**
   - Add markdown cells explaining each step
   - Add the SentinelOps Arena header/description
   - Mention partner tracks (Fleet AI, Patronus AI)
   - Save and get shareable link
   - Commit to repo

### Phase 4: Demo Video (2:30 - 3:00) -- 30 minutes

**Goal: 1-minute YouTube video demonstrating the environment**

1. **[5 min] Script the video**
   - 0-10s: Title card + what SentinelOps Arena is
   - 10-30s: Run an episode in Gradio, show attack/adapt/flag cycle
   - 30-45s: Show Untrained vs Trained comparison, highlight score difference
   - 45-55s: Show Environment Inspector (databases, task queue)
   - 55-60s: Mention partner tracks, training approach, link to Colab

2. **[15 min] Record**
   - Screen record the Gradio app (use HF Spaces URL if live, else local)
   - Voice narration or text overlay
   - Keep it to exactly 1 minute

3. **[10 min] Upload to YouTube**
   - Title: "SentinelOps Arena -- Multi-Agent RL for Enterprise Security | OpenEnv Hackathon"
   - Upload as unlisted
   - Get shareable link

### Phase 5: Final Polish & Submit (3:00 - 3:45) -- 45 minutes

1. **[10 min] Add links to About tab** (M2)
   - HF Spaces URL
   - YouTube demo link
   - Colab notebook link
   - GitHub repo link

2. **[10 min] Final push to both remotes**
   ```bash
   git add -A
   git commit -m "Final submission: add Colab notebook, update links"
   git push origin nihal
   git push hf nihal:main
   ```

3. **[10 min] Verify everything one last time**
   - HF Spaces loads and works
   - Colab notebook link is accessible
   - YouTube video plays
   - All links in About tab work

4. **[15 min] Submit**
   - Team Name: SentinelOps (or NexusEnv)
   - Project Description: (use draft from SENTINELOPS_ARENA.md)
   - HF Spaces Link: https://huggingface.co/spaces/nihalnihalani/sentinelops-arena
   - Demo Video: YouTube URL
   - Minimal Training Script: Colab link
   - Partner Tracks: Fleet AI (Scalable Oversight), Patronus AI (Schema Drift)

### Buffer: 15 minutes (3:45 - 4:00)

For unexpected issues, last-minute fixes, or submission form problems.

---

## CRITICAL PATH

The absolute minimum to submit (if everything goes wrong):

1. Fix requirements.txt (2 min)
2. Push to HF Spaces (15 min)
3. Create minimal Colab notebook that at least runs the environment (30 min)
4. Record 60-second screen capture (15 min)
5. Upload video + submit (10 min)

**Total critical path: ~72 minutes**

This leaves ~2.5 hours for polish, testing, and fixing issues.

---

## RISK REGISTER

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| HF Spaces build fails | Medium | BLOCKER | Test locally first. Have `huggingface_hub` upload as backup. Check Python version compat. |
| Colab Python version incompatible with openenv-core | Medium | HIGH | Bundle standalone env code in notebook (no openenv import needed for demo). |
| Gradio 6 has breaking changes on HF | Low | HIGH | Pin sdk_version in README frontmatter. Test specific version. |
| Video recording takes too long | Low | BLOCKER | Use simplest tool (QuickTime screen record). Keep to exactly 1 min. No editing. |
| Unsloth doesn't install on Colab | Medium | MEDIUM | Fall back to vanilla transformers (slower but works). Show pipeline, not convergence. |
| Submission form has unexpected fields | Low | LOW | Read form early, adapt. |

---

## WHAT NOT TO DO (Time traps)

- DO NOT try to implement compound attacks, compliance drift, or A2A protocol
- DO NOT try to actually train to convergence -- show the pipeline works, that's enough
- DO NOT refactor the codebase or clean up the spec doc
- DO NOT spend more than 30 min on the video -- 1 minute, simple screen recording
- DO NOT try to add Docker support
- DO NOT spend time on MCP-X gateway -- MCP tools in environment.py are sufficient
- DO NOT worry about the `hackathon_env/` directory during final push -- judges won't look at it unless it causes confusion