Spaces:
Sleeping
Sleeping
Commit ·
c2a35d8
1
Parent(s): 3b7ad8b
submission fixes
Browse files
README.md
CHANGED
|
@@ -42,7 +42,7 @@ Someone ran a cleanup script that deleted three products still referenced by act
|
|
| 42 |
A migration script died partway through. The incident log blames a timeout, but that is a red herring. The real problem is corrupted duplicate employee IDs in the source table that caused the crash. Clean the corruption, then finish the migration and preserve every salary mapping correctly.
|
| 43 |
|
| 44 |
**Task 4 - Schema Drift (Hard)**
|
| 45 |
-
A junior DBA ran a schema standardization script that renamed columns in the `inventory` and `categories` tables. The application code expects the original column names and is now broken. The `schema_changelog` table has a record of every rename. Reverse the column renames
|
| 46 |
|
| 47 |
**Task 5 - Referential Maze (Very Hard)**
|
| 48 |
A cross-database sync job crashed partway through, leaving orphaned FK references and missing records across four interrelated tables (`departments`, `projects`, `assignments`, `budgets`). The incident ticket blames a network timeout, but the real problem is scattered across all four tables. Not all missing records should be restored: some projects were deliberately decommissioned last quarter. The agent must check `decommission_log` before re-inserting anything and use `sync_audit` to find the data for legitimately missing records. Restoring a decommissioned project is penalized.
|
|
@@ -66,6 +66,47 @@ Each episode ends when the agent calls `submit_resolution` or runs out of budget
|
|
| 66 |
|
| 67 |
---
|
| 68 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 69 |
## Running Locally
|
| 70 |
|
| 71 |
```bash
|
|
|
|
| 42 |
A migration script died partway through. The incident log blames a timeout, but that is a red herring. The real problem is corrupted duplicate employee IDs in the source table that caused the crash. Clean the corruption, then finish the migration and preserve every salary mapping correctly.
|
| 43 |
|
| 44 |
**Task 4 - Schema Drift (Hard)**
|
| 45 |
+
A junior DBA ran a schema standardization script that renamed columns in the `inventory` and `categories` tables. The application code expects the original column names and is now broken. The `schema_changelog` table has a record of every rename. Reverse the column renames using `ALTER TABLE <table> RENAME COLUMN <current> TO <original>` for each entry.
|
| 46 |
|
| 47 |
**Task 5 - Referential Maze (Very Hard)**
|
| 48 |
A cross-database sync job crashed partway through, leaving orphaned FK references and missing records across four interrelated tables (`departments`, `projects`, `assignments`, `budgets`). The incident ticket blames a network timeout, but the real problem is scattered across all four tables. Not all missing records should be restored: some projects were deliberately decommissioned last quarter. The agent must check `decommission_log` before re-inserting anything and use `sync_audit` to find the data for legitimately missing records. Restoring a decommissioned project is penalized.
|
|
|
|
| 66 |
|
| 67 |
---
|
| 68 |
|
| 69 |
+
## Action Space
|
| 70 |
+
|
| 71 |
+
The agent has two actions:
|
| 72 |
+
|
| 73 |
+
| Action | Fields | Budget Cost |
|
| 74 |
+
|--------|--------|-------------|
|
| 75 |
+
| `execute_sql` | `query` (str) - any valid SQLite statement | -1 for SELECT/PRAGMA, -5 for INSERT/UPDATE/DELETE/ALTER/CREATE |
|
| 76 |
+
| `submit_resolution` | `notes` (str) - description of what was fixed | 0 (ends episode, triggers grader) |
|
| 77 |
+
|
| 78 |
+
## Observation Space
|
| 79 |
+
|
| 80 |
+
Every `reset()` and `step()` returns a `DBERObservation` with these fields:
|
| 81 |
+
|
| 82 |
+
| Field | Type | Description |
|
| 83 |
+
|-------|------|-------------|
|
| 84 |
+
| `incident_ticket` | str | The pager alert text describing the incident |
|
| 85 |
+
| `task_id` | int | Active task (1-5) |
|
| 86 |
+
| `last_action_type` | str | SELECT, MUTATION, META, ERROR, or NONE |
|
| 87 |
+
| `last_query_result` | list[dict] or str | Rows from the last SELECT, or a status string for mutations |
|
| 88 |
+
| `error_logs` | str | Exception message from the last action (empty if none) |
|
| 89 |
+
| `db_summary` | dict | Live snapshot: table names and current row counts |
|
| 90 |
+
| `violation_count` | int | Current broken constraint count; decreasing this gives +0.2 reward |
|
| 91 |
+
| `budget_remaining` | int | Remaining action budget (starts at 100) |
|
| 92 |
+
| `step_count` | int | Number of steps taken this episode |
|
| 93 |
+
|
| 94 |
+
## Baseline Scores
|
| 95 |
+
|
| 96 |
+
Running `inference.py` with a capable model (e.g. Llama-3.3-70B-Instruct) produces approximate scores:
|
| 97 |
+
|
| 98 |
+
| Task | Expected Range | Notes |
|
| 99 |
+
|------|---------------|-------|
|
| 100 |
+
| Task 1 - Phantom Duplicates | 0.85 - 1.0 | Straightforward once duplicates are found |
|
| 101 |
+
| Task 2 - Cascading Failure | 0.75 - 1.0 | Requires reading and parsing JSON audit logs |
|
| 102 |
+
| Task 3 - Payroll Black Hole | 0.50 - 0.85 | Misleading incident ticket; two-phase repair |
|
| 103 |
+
| Task 4 - Schema Drift | 0.60 - 1.0 | 10 column renames across 2 tables |
|
| 104 |
+
| Task 5 - Referential Maze | 0.40 - 0.85 | Decommission trap penalizes naive restore |
|
| 105 |
+
|
| 106 |
+
A perfect repair on every task is achievable. The `tests/test_dumb_agent.py` file contains verified perfect-score sequences for all five tasks.
|
| 107 |
+
|
| 108 |
+
---
|
| 109 |
+
|
| 110 |
## Running Locally
|
| 111 |
|
| 112 |
```bash
|