docs/implementation/COPILOT_EXECUTION_PLAYBOOK.md

# GitHub Copilot Execution Playbook
## How to Implement the KDD Data Agent Observatory Blueprint Phase by Phase

This document explains how to use the architecture blueprint safely and effectively with GitHub Copilot.

The governing architecture document is:

```text
KDD_Data_Agent_Observatory_Phased_Architecture_Blueprint.md
```

The blueprint defines the target architecture, phases, interfaces, acceptance criteria, and dependencies. This playbook defines the execution discipline: how to brief Copilot, how to review each phase, how to test, and when to commit.

---

# 1. Core Strategy

Do not ask Copilot to implement the full blueprint in one request.

Use this cycle for every phase:

```text
Phase selection
→ repository inspection
→ implementation plan
→ plan review
→ implementation
→ static checks
→ unit tests
→ integration tests
→ manual smoke test
→ artifact inspection
→ acceptance-criteria audit
→ commit
→ update implementation ledger
→ proceed to next phase
```

Each phase must be treated as an independently releasable change.

The blueprint is the architecture contract. The phase prompt is the execution contract.

---

# 2. Files to Add to the Repository Before Implementation

Place these files under the repository:

```text
docs/
  implementation/
    KDD_Data_Agent_Observatory_Phased_Architecture_Blueprint.md
    COPILOT_EXECUTION_PLAYBOOK.md
    IMPLEMENTATION_LEDGER.md
    DECISIONS.md
```

## 2.1 IMPLEMENTATION_LEDGER.md

Use this to track completion.

Recommended structure:

```markdown
# Implementation Ledger

## Current Baseline

- Branch:
- Base commit:
- Test command:
- Existing test result:
- Current known failures:
- Python version:
- Dependency manager:
- Streamlit entrypoint:
- CLI entrypoint:

## Phase Status

| Phase | Status | Branch | Commit | Tests | Acceptance | Notes |
|---|---|---|---|---|---|---|
| Phase 0 | Not started | | | | | |
| Phase 1 | Not started | | | | | |
...

## Phase Details

### Phase 0

- Started:
- Completed:
- Files changed:
- Tests added:
- Commands executed:
- Known limitations:
- Deferred work:
- Evidence:
```

## 2.2 DECISIONS.md

Record deviations and architecture decisions.

```markdown
# Architecture and Implementation Decisions

## DEC-001

- Date:
- Phase:
- Decision:
- Reason:
- Alternatives:
- Consequences:
- Blueprint impact:
```

No important blueprint deviation should remain only in a Copilot chat.

---

# 3. Branch and Commit Strategy

Use one branch per phase.

Recommended naming:

```text
feature/observatory-phase-00-baseline
feature/observatory-phase-01-run-service
feature/observatory-phase-02-evaluation-service
feature/observatory-phase-03-run-intelligence
...
```

For larger phases, use sub-branches only if necessary:

```text
feature/observatory-phase-07-plan-checkpoint
```

## Commit policy

Prefer one to three meaningful commits per phase:

```text
refactor(run): extract shared execution service
test(run): add execution service and CLI compatibility coverage
docs(run): record phase 1 implementation evidence
```

Do not make one commit for every tiny edit.

Do not commit until:

- tests pass,
- manual checks pass,
- artifacts are inspected,
- acceptance criteria are audited,
- unrelated file changes are removed.

## Commit message convention

```text
<type>(<scope>): <description>
```

Examples:

```text
test(cli): capture existing run and eval command contracts
refactor(run): extract reusable multi-task execution service
feat(ui): add predefined task run launcher
feat(steering): add persistent plan review checkpoint
feat(eval): compare guided and autonomous task cohorts
docs(observatory): update phase implementation ledger
```

---

# 4. Phase Execution Protocol

Each phase has four Copilot interactions.

## Interaction A — Repository Inspection and Plan

Copilot must inspect the repository and produce a plan only.

No code changes during this interaction.

It must identify:

- reusable code,
- current call paths,
- exact files to modify,
- exact files to create,
- tests affected,
- compatibility risks,
- ambiguities,
- planned validation commands.

Review this response before authorizing implementation.

## Interaction B — Implementation

After approving the plan, tell Copilot to implement the phase.

Copilot must:

- stay within phase scope,
- use existing conventions,
- update tests,
- preserve CLI compatibility,
- avoid placeholder implementations,
- avoid unrelated cleanup.

## Interaction C — Verification and Repair

Ask Copilot to run the complete validation matrix.

It must:

- run formatting/linting/type checks,
- run focused tests,
- run relevant broader tests,
- run CLI smoke tests,
- inspect generated artifacts,
- fix failures,
- rerun checks.

Do not accept “tests should pass.”

Require actual command output summaries.

## Interaction D — Completion Audit and Commit Preparation

Copilot must compare the implementation to every acceptance criterion.

It must provide:

- pass/fail for each criterion,
- files changed,
- test evidence,
- generated artifacts,
- known limitations,
- recommended commit message,
- implementation ledger update.

Only then commit.

---

# 5. Master Phase Prompt

Use this prompt for every phase, replacing the placeholders.

```text
You are implementing one phase of the KDD Data Agent Observatory architecture.

GOVERNING DOCUMENTS
1. docs/implementation/KDD_Data_Agent_Observatory_Phased_Architecture_Blueprint.md
2. docs/implementation/COPILOT_EXECUTION_PLAYBOOK.md
3. docs/implementation/IMPLEMENTATION_LEDGER.md
4. docs/implementation/DECISIONS.md

TARGET PHASE
Phase <PHASE_NUMBER>: <PHASE_NAME>

MODE FOR THIS REQUEST
Inspection and implementation planning only. Do not modify code yet.

OBJECTIVE
Read the complete target phase in the blueprint and inspect the current repository implementation. Produce an implementation plan grounded in the actual codebase.

MANDATORY REPOSITORY INSPECTION
Inspect at minimum:
- the current CLI entrypoint and relevant commands;
- all backend functions called by those commands;
- existing domain/config models;
- current artifact layout;
- current tests for the affected area;
- Streamlit pages/components if this phase affects UI;
- evaluator, runner, trace, replay, or checkpoint code relevant to this phase;
- current import and dependency conventions.

REQUIRED OUTPUT
Provide:

1. Current-state analysis
   - Existing relevant behavior.
   - Existing reusable functions/classes.
   - Current call graph.
   - Current artifact files and schemas.
   - Current tests.

2. Gap analysis against the blueprint
   - Requirement-by-requirement gap.
   - What is already complete.
   - What is partially complete.
   - What is missing.
   - What should not be rebuilt.

3. Proposed implementation
   - Exact files to create.
   - Exact files to modify.
   - Public interfaces and signatures.
   - Data model changes.
   - Persistence changes.
   - CLI changes.
   - Streamlit changes, if applicable.
   - Migration/backward-compatibility strategy.

4. Test plan
   - Unit tests.
   - Integration tests.
   - CLI smoke tests.
   - Streamlit smoke tests, if relevant.
   - Fixture requirements.
   - Failure and edge cases.
   - Exact commands to run.

5. Risks and decisions
   - Regression risks.
   - Concurrency/state risks.
   - Artifact compatibility risks.
   - Any ambiguity in the blueprint.
   - Recommended decision for each ambiguity.

6. Scope control
   - Explicit in-scope work.
   - Explicit out-of-scope work.
   - Unrelated cleanup that must not be performed.

7. Execution sequence
   - Ordered coding steps.
   - Validation after each major step.
   - Expected final artifacts.

RULES
- Do not change code in this planning request.
- Do not create a parallel execution or evaluation implementation.
- CLI and Streamlit must use shared services.
- Preserve existing CLI behavior unless the blueprint explicitly changes it.
- Do not place business logic in Streamlit.
- Do not use shell subprocess calls from Streamlit to invoke the CLI.
- Do not silently change artifact schemas.
- Do not claim something exists without opening and inspecting it.
- Call out conflicts between the blueprint and current repository.
- Prefer the smallest correct change that satisfies the complete phase.
```

---

# 6. Implementation Authorization Prompt

After reviewing Copilot's plan, use:

```text
Implement Phase <PHASE_NUMBER>: <PHASE_NAME> using the approved plan.

GOVERNING DOCUMENTS
- docs/implementation/KDD_Data_Agent_Observatory_Phased_Architecture_Blueprint.md
- docs/implementation/COPILOT_EXECUTION_PLAYBOOK.md
- docs/implementation/IMPLEMENTATION_LEDGER.md
- docs/implementation/DECISIONS.md

APPROVED PLAN
<PASTE OR REFERENCE THE APPROVED PLAN>

IMPLEMENTATION REQUIREMENTS
- Implement the complete phase, including production code, tests, documentation, and migrations required by the blueprint.
- Reuse existing runners, evaluators, repositories, and models where appropriate.
- Preserve existing CLI commands and artifact compatibility.
- Keep Streamlit as a thin adapter.
- Add type hints and public API docstrings.
- Use atomic writes for mutable manifest/state files.
- Handle failure and partial-artifact states explicitly.
- Do not add placeholders, fake metrics, hard-coded demo values, or TODO-only implementations.
- Do not perform unrelated refactoring.
- Record any necessary architecture deviation in docs/implementation/DECISIONS.md.
- Update docs/implementation/IMPLEMENTATION_LEDGER.md only after verification succeeds.

WORKING METHOD
1. Implement in small logical steps.
2. After each major step, run the narrowest relevant tests.
3. Fix failures before continuing.
4. At the end, run the complete phase validation matrix.
5. Do not commit automatically.

FINAL RESPONSE FORMAT
1. Summary of implementation.
2. Files created.
3. Files modified.
4. Interfaces introduced or changed.
5. Tests added.
6. Commands executed and results.
7. Generated artifacts inspected.
8. Acceptance criteria status.
9. Known limitations.
10. Suggested commit grouping and commit messages.
```

---

# 7. Verification and Repair Prompt

Use after implementation:

```text
Perform the verification and repair pass for Phase <PHASE_NUMBER>: <PHASE_NAME>.

Do not add new scope. Validate the implementation against the blueprint and repair only issues required to make this phase complete and regression-safe.

MANDATORY CHECKS

1. Repository hygiene
- Show git status.
- Identify unrelated modified or untracked files.
- Confirm no secrets, generated large artifacts, local paths, or temporary files will be committed.

2. Static validation
- Run the repository formatter check.
- Run linting.
- Run type checking if configured.
- Run import/compile checks.
- Fix all new issues introduced by this phase.

3. Focused tests
- Run all tests added for this phase.
- Run existing tests for affected modules.
- Run failure-path tests.
- Run backward-compatibility tests.

4. Integration tests
- Execute the relevant service through Python without CLI or Streamlit.
- Execute the relevant CLI command with fixture or safe test data.
- Verify generated artifacts and schemas.
- If Streamlit is affected, verify the app imports and the page loads without executing work on import.

5. Regression tests
- Run the broader practical test suite for the affected package.
- Verify existing commands remain available.
- Verify old artifact layouts remain readable where required.

6. Acceptance audit
For every acceptance criterion in the target phase:
- quote the criterion;
- mark PASS or FAIL;
- provide evidence;
- repair failures and rerun validation.

7. Artifact inspection
Open and inspect the generated JSON, JSONL, CSV, Markdown, or manifest files. Validate:
- required fields;
- valid values;
- no secrets;
- no fabricated values;
- consistent run/task IDs;
- correct state transitions;
- backward-compatible paths.

8. Final report
Return:
- commands run;
- test totals;
- failures found and fixed;
- final git diff summary;
- acceptance matrix;
- residual limitations;
- whether the phase is ready to commit.

Do not state that the phase is ready to commit unless all critical acceptance criteria pass.
Do not commit automatically.
```

---

# 8. Commit Preparation Prompt

Use only after verification passes:

```text
Prepare Phase <PHASE_NUMBER>: <PHASE_NAME> for commit.

Do not modify production behavior unless required to correct documentation or remove accidental files.

Tasks:
1. Review the final git diff.
2. Remove unrelated changes.
3. Confirm generated runtime artifacts are excluded unless they are intentional test fixtures.
4. Confirm no secrets or machine-specific paths are present.
5. Update docs/implementation/IMPLEMENTATION_LEDGER.md with:
   - phase status;
   - branch;
   - files changed;
   - tests and results;
   - acceptance-criteria status;
   - known limitations;
   - deferred work.
6. Update docs/implementation/DECISIONS.md for any architecture decisions made.
7. Recommend one to three commit groups.
8. Provide exact git add and git commit commands, but do not execute the commit.

Final response:
- final files to stage;
- files not to stage;
- commit group(s);
- exact commit message(s);
- final phase status;
- next phase dependency check.
```

---

# 9. Testing Ladder

Run tests in increasing scope.

## Level 1 — Syntax and imports

Examples:

```bash
python -m compileall src
python -c "import data_agent_baseline"
```

Use the project's actual environment command, such as `uv run`, if required.

## Level 2 — Focused unit tests

```bash
pytest tests/unit/application/test_run_execution_service.py -q
```

## Level 3 — Affected package tests

```bash
pytest tests/unit/application tests/unit/repositories -q
```

## Level 4 — CLI tests

```bash
pytest tests/integration/cli -q
```

Also inspect:

```bash
dabench --help
dabench run-lang-task --help
dabench eval-v2 --help
```

## Level 5 — Fixture integration

Execute against a reduced fixture run.

Examples:

```bash
dabench eval-v2 <fixture-run> --mode standard
dabench eval-v2 <fixture-run> --mode verbose
dabench eval-v2 <fixture-run> --mode research
```

## Level 6 — Streamlit startup

At minimum:

- import app,
- load affected page,
- confirm no execution on import,
- confirm service dependency can be mocked.

Where practical, run Streamlit headlessly and check startup logs.

## Level 7 — Real smoke test

Use one small real task before committing execution-related phases.

Do not run a full benchmark for every phase.

Use a full or 50-task run only at milestone gates.

---

# 10. Commit Gate Checklist

Before every commit, verify:

```text
[ ] Phase scope is complete.
[ ] All critical acceptance criteria pass.
[ ] Existing CLI commands remain functional.
[ ] New backend services can be called independently.
[ ] Streamlit contains no duplicated business logic.
[ ] Tests were actually run.
[ ] Manual smoke test passed where applicable.
[ ] Generated artifacts were inspected.
[ ] No secrets or machine-specific paths were added.
[ ] No unrelated changes are staged.
[ ] Documentation is updated.
[ ] Implementation ledger is updated.
[ ] Architecture decisions are recorded.
[ ] Known limitations are explicit.
[ ] Next phase dependencies are satisfied.
```

---

# 11. Phase-Specific Execution Guidance

## Phase 0 — Baseline Protection

### Test before coding

Capture:

```bash
git status
git rev-parse HEAD
pytest <current relevant suite>
dabench --help
dabench run-lang-task --help
dabench eval-v2 --help
```

### Commit after

- CLI contract tests,
- reduced fixtures,
- architecture inventory,
- current baseline results.

### Do not combine with UI work

Phase 0 should establish trustworthy regression protection.

---

## Phase 1 — Shared Run Execution Service

### Minimum manual test

Run one predefined task through:

1. the service directly,
2. the existing CLI.

Compare:

- run path,
- task path,
- success/failure status,
- prediction path,
- trace path,
- manifest,
- events.

### Commit gate

The execution loop must no longer live in the Typer command.

---

## Phase 2 — Evaluation Service

### Minimum manual test

For one fixture run:

- call service directly,
- run CLI in standard mode,
- run CLI in verbose mode,
- run CLI in research mode,
- compare artifact outputs,
- confirm warnings/errors semantics.

### Commit gate

No metric formula should accidentally change unless explicitly intended and tested.

---

## Phase 3 — Run Intelligence

### Minimum manual test

Open:

- valid evaluated run,
- run with warnings,
- partially evaluated run,
- missing evaluation,
- malformed optional artifact.

### Commit gate

Every high-level finding must link to task or metric evidence.

---

## Phase 4 — Run Launcher

### Minimum manual test

Launch:

- one task,
- multiple tasks,
- invalid selection,
- duplicate click scenario.

### Commit gate

Streamlit rerun must not create duplicate runs.

---

## Phase 5 — Live Execution

### Minimum manual test

- start task,
- refresh browser,
- observe restored state,
- observe task completion,
- observe failed task.

### Commit gate

The UI must read structured events, not console logs.

---

## Phase 6 — Custom Tasks

### Minimum manual test

- CSV task,
- multiple files,
- unsupported file,
- empty question,
- custom task without gold truth.

### Commit gate

Benchmark and custom tasks must enter the same canonical runner contract.

---

## Phase 7 — Plan Review Checkpoint

### Minimum manual test

- approve unchanged,
- revise plan,
- cancel,
- duplicate submit,
- refresh while paused,
- resume after process/UI restart if supported.

### Commit gate

The agent must resume from persisted state and not simply rerun the whole task.

---

## Phase 8 — Intervention Delta

### Minimum manual test

- no-change approval,
- one modified step,
- added/removed steps,
- unknown score before,
- cost delta available/unavailable.

### Commit gate

Unknown values must remain unknown, not be coerced to zero.

---

## Phase 9 — Guided vs Autonomous Evaluation

### Minimum manual test

- paired task,
- unpaired autonomous task,
- guided task without gold,
- improved case,
- degraded case.

### Commit gate

No correctness comparison for custom tasks without ground truth.

---

## Phase 10 — Ask This Run

### Minimum manual test

Ask:

- ranking question,
- filtering question,
- cohort comparison,
- failure question,
- intervention question,
- unsupported causal question.

### Commit gate

Queries must be read-only and evidence-linked.

---

# 12. Milestone Test Runs

Do not wait until the end for broad validation.

## Milestone A — after Phase 5

Run:

- one single task,
- one three-task run,
- one failed task case,
- eval-v2 all modes,
- Streamlit launch and live tracking,
- Run Intelligence drill-down.

## Milestone B — after Phase 6

Run:

- one benchmark task,
- one custom CSV task,
- one custom multi-source task,
- compare artifact completeness.

## Milestone C — after Phase 9

Run a small controlled experiment:

- 3–5 tasks autonomous,
- same 3–5 tasks guided,
- at least one useful intervention,
- at least one neutral or failed intervention,
- generate comparative run evaluation.

## Milestone D — final

Run the intended 50-task benchmark and selected guided/custom cases.

Generate:

- final evaluation,
- research report,
- run intelligence,
- intervention evidence,
- demo artifacts.

---

# 13. How to Handle Copilot Deviations

If Copilot proposes a different architecture, require a decision record before implementation.

Use:

```text
Your proposal differs from the governing blueprint.

Before changing code:
1. identify the exact blueprint requirement;
2. describe the proposed deviation;
3. explain why the current repository makes the blueprint approach unsuitable;
4. compare both options;
5. describe compatibility and migration consequences;
6. recommend one option;
7. draft the DECISIONS.md entry.

Do not implement until the deviation is approved.
```

---

# 14. How to Prevent Common Copilot Failure Modes

## Failure: UI-only implementation

Respond:

```text
This phase is not complete. The blueprint requires the capability in a reusable backend service. Move execution/evaluation/state logic out of Streamlit and add direct service tests.
```

## Failure: duplicate implementation

Respond:

```text
Do not create a second runner/evaluator. Identify the existing implementation and refactor it behind the shared service interface.
```

## Failure: placeholder data

Respond:

```text
Remove all hard-coded or fabricated metrics. The UI must load actual artifacts or show an explicit unavailable state.
```

## Failure: silent schema change

Respond:

```text
Document the schema change, add schema versioning or compatibility handling, update readers and validators, and add migration tests.
```

## Failure: tests not run

Respond:

```text
Run the exact validation commands now. Report actual results and repair failures before claiming completion.
```

## Failure: broad unrelated refactor

Respond:

```text
Revert unrelated changes. Keep the diff limited to the target phase and its required tests/documentation.
```

## Failure: full task rerun after steering

Respond:

```text
This does not satisfy the checkpoint requirement. Persist the checkpoint state and resume from the defined workflow boundary. Record preserved and invalidated work.
```

---

# 15. Recommended First Copilot Request

Start with Phase 0 only.

Use:

```text
You are implementing the KDD Data Agent Observatory architecture.

Read:
- docs/implementation/KDD_Data_Agent_Observatory_Phased_Architecture_Blueprint.md
- docs/implementation/COPILOT_EXECUTION_PLAYBOOK.md

Target:
Phase 0 — Baseline Protection and Architecture Inventory.

For this request, perform repository inspection and planning only. Do not modify code.

Inspect the actual repository and produce:
1. an inventory of the relevant CLI commands and backend call graph;
2. current artifact contracts;
3. current tests and coverage gaps;
4. representative fixture strategy;
5. exact files to create or modify;
6. exact tests to add;
7. exact commands to establish the baseline;
8. risks and scope boundaries;
9. a step-by-step implementation plan;
10. a mapping from every Phase 0 acceptance criterion to planned evidence.

Pay particular attention to:
- run-lang-task;
- run-lang-benchmark;
- eval-v2;
- view-task-v2;
- current runner functions;
- evaluator and hardening functions;
- current Streamlit entrypoint and task pages;
- artifact paths under runs;
- existing test conventions.

Rules:
- do not implement yet;
- do not redesign later phases;
- do not create a parallel runner or evaluator;
- preserve existing behavior;
- identify any hard-coded paths and testability issues without fixing unrelated items;
- state explicitly what should not be changed in Phase 0.
```

After approving the plan, use the implementation authorization prompt.

---

# 16. Recommended Working Rhythm

For each phase:

## Day or session start

```text
git checkout main
git pull
git checkout -b feature/observatory-phase-XX-name
```

## Baseline

Run the current relevant tests before edits.

Record failures as pre-existing.

## Copilot inspection

Use the planning prompt.

Review the plan yourself.

## Copilot implementation

Use the authorization prompt.

## Verification

Use the verification prompt.

## Manual inspection

Open generated artifacts and affected UI yourself.

## Commit

Use the commit-preparation prompt, then execute the commit manually.

## Merge

Merge only after:

- phase tests pass,
- milestone tests pass when applicable,
- ledger is updated.

---

# 17. Final Recommendation

Use the blueprint as a permanent repository document, not as a one-time prompt.

For every phase:

1. give Copilot the blueprint and playbook paths;
2. request inspection and planning only;
3. review and approve the plan;
4. request implementation;
5. request verification and repair;
6. manually inspect artifacts/UI;
7. request commit preparation;
8. commit manually;
9. update the ledger;
10. proceed only after the stop/go gate passes.

This creates strong control without requiring you to micromanage individual code edits. It also ensures that each phase is testable, reviewable, reversible, and independently committable.