File size: 3,316 Bytes
ee9fcd0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 | # Agent Canvas — Testing Matrix
**Priority key:** P0 = must pass before any release · P1 = must pass before GA · P2 = best-effort
---
## Install × OS × Agent
Each cell = smoke test: install → onboard → start conversation → agent replies.
| | **macOS** | **Linux** | **Windows** |
|---|:---:|:---:|:---:|
| **npm — OpenHands** | ☐ | ☐ | ☐ |
| **npm — Claude Code** | ☐ | ☐ | ☐ |
| **npm — Codex** | ☐ | ☐ | ☐ |
| **npm — Gemini CLI** | ☐ | ☐ | ☐ |
| **npm — Custom ACP** | ☐ | ☐ | ☐ |
| **Docker — OpenHands** | ☐ | ☐ | ☐ |
| **Docker — Claude Code** | ☐ | ☐ | ☐ |
| **Docker — Codex** | ☐ | ☐ | ☐ |
| **Docker — Gemini CLI** | ☐ | ☐ | ☐ |
| **Docker — Custom ACP** | ☐ | ☐ | ☐ |
---
## Automations × Install × Agent
Requires full stack (automation backend running).
| | **npm** | **Docker** |
|---|:---:|:---:|
| **OpenHands** | ✅ P0 | ✅ P0 |
| **Claude Code** | ✅ P1 | ✅ P1 |
| **Codex** | ✅ P1 | ✅ P1 |
| **Gemini CLI** | ✅ P2 | ✅ P2 |
Each cell = create automation → dispatch run → run reaches COMPLETED → conversation link works.
---
## Auth Modes
| | **npm** | **Docker** |
|---|:---:|:---:|
| **Local (auto-generated key)** | ✅ P0 | ✅ P0 |
| **Public (`--public` + user key)** | ✅ P1 | ✅ P1 |
---
## Feature Checklist
### npm
| Feature | OpenHands | Claude Code | Codex | Gemini CLI |
|---|:---:|:---:|:---:|:---:|
| Onboarding | ☐ | ☐ | ☐ | ☐ |
| Conversation — start, resume, history | ☐ | ☐ | ☐ | ☐ |
| Terminal tool | ☐ | ☐ | ☐ | ☐ |
| File editor tool | ☐ | ☐ | ☐ | ☐ |
| Browser tool | ☐ | ☐ | ☐ | ☐ |
| LLM profiles — create / switch | ☐ | — | — | — |
| Secrets — add / delete / forwarded | ☐ | ☐ | ☐ | ☐ |
| Automations — create, dispatch, COMPLETED | ☐ | ☐ | ☐ | ☐ |
| Files tab + Changes/diff tab | ☐ | ☐ | ☐ | ☐ |
| MCP server install | ☐ | ☐ | ☐ | ☐ |
| Image upload in chat | ☐ | ☐ | ☐ | ☐ |
| Key rotation | ☐ | ☐ | ☐ | ☐ |
### Docker
| Feature | OpenHands | Claude Code | Codex | Gemini CLI |
|---|:---:|:---:|:---:|:---:|
| Onboarding | ☐ | ☐ | ☐ | ☐ |
| Conversation — start, resume, history | ☐ | ☐ | ☐ | ☐ |
| Terminal tool | ☐ | ☐ | ☐ | ☐ |
| File editor tool | ☐ | ☐ | ☐ | ☐ |
| Browser tool | ☐ | ☐ | ☐ | ☐ |
| LLM profiles — create / switch | ☐ | — | — | — |
| Secrets — add / delete / forwarded | ☐ | ☐ | ☐ | ☐ |
| Automations — create, dispatch, COMPLETED | ☐ | ☐ | ☐ | ☐ |
| Files tab + Changes/diff tab | ☐ | ☐ | ☐ | ☐ |
| MCP server install | ☐ | ☐ | ☐ | ☐ |
| Image upload in chat | ☐ | ☐ | ☐ | ☐ |
| Key rotation | ☐ | ☐ | ☐ | ☐ |
---
## Automated Coverage
| Suite | Install | OS | Agents | Automations |
|---|---|---|---|---|
| `vitest` (unit) | — | Linux | — | partial |
| `test:e2e:mock-llm` | npm | Linux | OpenHands, ACP (mock) | ✅ full |
| `test:e2e:mock-llm:docker` | Docker | Linux | OpenHands, ACP (mock) | ✅ full |
| `test:e2e:live` | npm | Linux | OpenHands | ❌ |
**Not yet covered by CI:** real ACP credentials (Claude Code / Codex / Gemini), macOS, public auth mode, subscription login paths, Windows.
|