aadi-joshi commited on
Commit
c2a35d8
·
1 Parent(s): 3b7ad8b

submission fixes

Browse files
Files changed (1) hide show
  1. README.md +42 -1
README.md CHANGED
@@ -42,7 +42,7 @@ Someone ran a cleanup script that deleted three products still referenced by act
42
  A migration script died partway through. The incident log blames a timeout, but that is a red herring. The real problem is corrupted duplicate employee IDs in the source table that caused the crash. Clean the corruption, then finish the migration and preserve every salary mapping correctly.
43
 
44
  **Task 4 - Schema Drift (Hard)**
45
- A junior DBA ran a schema standardization script that renamed columns in the `inventory` and `categories` tables. The application code expects the original column names and is now broken. The `schema_changelog` table has a record of every rename. Reverse the column renames while preserving all data. Requires the SQLite CREATE-INSERT-DROP-RENAME pattern since ALTER TABLE column rename support is limited.
46
 
47
  **Task 5 - Referential Maze (Very Hard)**
48
  A cross-database sync job crashed partway through, leaving orphaned FK references and missing records across four interrelated tables (`departments`, `projects`, `assignments`, `budgets`). The incident ticket blames a network timeout, but the real problem is scattered across all four tables. Not all missing records should be restored: some projects were deliberately decommissioned last quarter. The agent must check `decommission_log` before re-inserting anything and use `sync_audit` to find the data for legitimately missing records. Restoring a decommissioned project is penalized.
@@ -66,6 +66,47 @@ Each episode ends when the agent calls `submit_resolution` or runs out of budget
66
 
67
  ---
68
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  ## Running Locally
70
 
71
  ```bash
 
42
  A migration script died partway through. The incident log blames a timeout, but that is a red herring. The real problem is corrupted duplicate employee IDs in the source table that caused the crash. Clean the corruption, then finish the migration and preserve every salary mapping correctly.
43
 
44
  **Task 4 - Schema Drift (Hard)**
45
+ A junior DBA ran a schema standardization script that renamed columns in the `inventory` and `categories` tables. The application code expects the original column names and is now broken. The `schema_changelog` table has a record of every rename. Reverse the column renames using `ALTER TABLE <table> RENAME COLUMN <current> TO <original>` for each entry.
46
 
47
  **Task 5 - Referential Maze (Very Hard)**
48
  A cross-database sync job crashed partway through, leaving orphaned FK references and missing records across four interrelated tables (`departments`, `projects`, `assignments`, `budgets`). The incident ticket blames a network timeout, but the real problem is scattered across all four tables. Not all missing records should be restored: some projects were deliberately decommissioned last quarter. The agent must check `decommission_log` before re-inserting anything and use `sync_audit` to find the data for legitimately missing records. Restoring a decommissioned project is penalized.
 
66
 
67
  ---
68
 
69
+ ## Action Space
70
+
71
+ The agent has two actions:
72
+
73
+ | Action | Fields | Budget Cost |
74
+ |--------|--------|-------------|
75
+ | `execute_sql` | `query` (str) - any valid SQLite statement | -1 for SELECT/PRAGMA, -5 for INSERT/UPDATE/DELETE/ALTER/CREATE |
76
+ | `submit_resolution` | `notes` (str) - description of what was fixed | 0 (ends episode, triggers grader) |
77
+
78
+ ## Observation Space
79
+
80
+ Every `reset()` and `step()` returns a `DBERObservation` with these fields:
81
+
82
+ | Field | Type | Description |
83
+ |-------|------|-------------|
84
+ | `incident_ticket` | str | The pager alert text describing the incident |
85
+ | `task_id` | int | Active task (1-5) |
86
+ | `last_action_type` | str | SELECT, MUTATION, META, ERROR, or NONE |
87
+ | `last_query_result` | list[dict] or str | Rows from the last SELECT, or a status string for mutations |
88
+ | `error_logs` | str | Exception message from the last action (empty if none) |
89
+ | `db_summary` | dict | Live snapshot: table names and current row counts |
90
+ | `violation_count` | int | Current broken constraint count; decreasing this gives +0.2 reward |
91
+ | `budget_remaining` | int | Remaining action budget (starts at 100) |
92
+ | `step_count` | int | Number of steps taken this episode |
93
+
94
+ ## Baseline Scores
95
+
96
+ Running `inference.py` with a capable model (e.g. Llama-3.3-70B-Instruct) produces approximate scores:
97
+
98
+ | Task | Expected Range | Notes |
99
+ |------|---------------|-------|
100
+ | Task 1 - Phantom Duplicates | 0.85 - 1.0 | Straightforward once duplicates are found |
101
+ | Task 2 - Cascading Failure | 0.75 - 1.0 | Requires reading and parsing JSON audit logs |
102
+ | Task 3 - Payroll Black Hole | 0.50 - 0.85 | Misleading incident ticket; two-phase repair |
103
+ | Task 4 - Schema Drift | 0.60 - 1.0 | 10 column renames across 2 tables |
104
+ | Task 5 - Referential Maze | 0.40 - 0.85 | Decommission trap penalizes naive restore |
105
+
106
+ A perfect repair on every task is achievable. The `tests/test_dumb_agent.py` file contains verified perfect-score sequences for all five tasks.
107
+
108
+ ---
109
+
110
  ## Running Locally
111
 
112
  ```bash