Spaces:
Paused
feat(round-12 S7): multi team workspace pool + auto failover
Browse files新增 src/autoteam/workspace_pool.py — WorkspacePool 持久化 K 个 team workspace
互为冷备(active=1, warm/cold N-1)。主 workspace 连续探测失败 ≥ MASTER_HEALTH_FAIL_THRESHOLD
(默认 3)秒切到 warm。register / set_active / mark_unhealthy / mark_healthy 全部
追加 transition_log,与 S1 default_machine 风格一致(原子写 + .bak 回滚 + lock-外
事件总线)。
admin_state.get_admin_email / get_chatgpt_account_id 改为路由 active workspace
的 admin_email / account_id;pool 空时回退原 state.json 读路径,单 workspace 模
式向后兼容。master_health 新增 apply_pool_health_signal helper:probe healthy
→ mark_healthy 重置 fail_count;subscription_cancelled / auth_invalid /
workspace_missing / role_not_owner → mark_unhealthy 累计触发故障切换;
network_error 视为瞬时不计入。cpa_sync 新增 get_active_sync_target helper +
sync_main_codex_to_cpa 上报当前 active workspace identity。
测试 35 case 覆盖:基础注册/激活/idempotent/重复 id 拒绝、set_active 切换/降级、
mark_unhealthy 阈值前不切换 + 阈值触发 failover + 无候选 → active=None +
force_failover、failover 候选选最旧 warm、mark_healthy 重置、transition_log
完整、原子写 .bak 回滚(模拟 os.replace 失败)、admin_state 路由 + 兜底、
master_health 集成 4 路径、subscriber 异常吞掉、5-thread 并发注册仅 1 active、
schema 不兼容 + JSON 损坏 + 字段非法值 normalize、cpa_sync get_active_sync_target。
向后兼容:workspaces.json 不存在时从 state.json (admin_email + UUID account_id)
自动 seed 为单一 active workspace;现有调用方零改动。
DoD:
- ruff check src/ tests/ 全绿
- pytest 699 passed (664 baseline + 35 新增,无退化)
- 所有改动文件已提交,严格 git add 文件名,未用 -A
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- .trellis/tasks/05-11-s7-multi-workspace-pool/check.jsonl +5 -0
- .trellis/tasks/05-11-s7-multi-workspace-pool/implement.jsonl +5 -0
- .trellis/tasks/05-11-s7-multi-workspace-pool/prd.md +123 -0
- .trellis/tasks/05-11-s7-multi-workspace-pool/task.json +26 -0
- src/autoteam/admin_state.py +28 -0
- src/autoteam/cpa_sync.py +41 -1
- src/autoteam/master_health.py +56 -0
- src/autoteam/workspace_pool.py +683 -0
- tests/unit/test_round12_s7_workspace_pool.py +483 -0
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"file": ".trellis/tasks/05-11-s7-multi-workspace-pool/prd.md", "reason": "DoD + 不变量 I1~I7 验收清单"}
|
| 2 |
+
{"file": "src/autoteam/workspace_pool.py", "reason": "新增核心模块 — 验证 register / set_active / mark_unhealthy / 自动 failover / 原子写 / 向后兼容 seed"}
|
| 3 |
+
{"file": "src/autoteam/admin_state.py", "reason": "路由改造 — 验证 pool 为空时回退 state.json,签名不变"}
|
| 4 |
+
{"file": "src/autoteam/master_health.py", "reason": "失败阈值累计 + 调 pool.mark_unhealthy 触发切换"}
|
| 5 |
+
{"file": "tests/unit/test_round12_s7_workspace_pool.py", "reason": "≥ 15 case:注册/切换/故障阈值/原子写回滚/向后兼容/边界"}
|
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"file": ".trellis/tasks/05-11-upstream-align-register-multimail-frontend-refresh/prd.md", "reason": "父 PRD — Q2 创新方向 5 多 workspace 池化 scope + 子任务依赖图 + round-12 波次范围"}
|
| 2 |
+
{"file": "src/autoteam/admin_state.py", "reason": "改造目标 1:get_admin_email / get_chatgpt_account_id 路由到 WorkspacePool.get_active(),向后兼容回退 state.json"}
|
| 3 |
+
{"file": "src/autoteam/master_health.py", "reason": "改造目标 2:失败路径(subscription_cancelled / auth_invalid / network_error)调 pool.mark_unhealthy 触发故障切换"}
|
| 4 |
+
{"file": "src/autoteam/cpa_sync.py", "reason": "改造目标 3:sync_to_cpa / sync_main_codex_to_cpa 同步目标跟随 active workspace(感知失败切换后立即新 workspace 推送)"}
|
| 5 |
+
{"file": "src/autoteam/account_state.py", "reason": "S1 状态机参考实现 — atomic 写盘 + .bak 回滚 + subscribe 事件总线 + IllegalTransitionError 风格对齐"}
|
|
@@ -0,0 +1,123 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# S7 — 多 team workspace 池化(冷备 + 秒级故障切换)
|
| 2 |
+
|
| 3 |
+
> **Round 12 子任务**,父任务 `05-11-upstream-align-register-multimail-frontend-refresh` 中
|
| 4 |
+
> 「Q2 创新方向 5 — 多 team workspace 池化」的实施单元。
|
| 5 |
+
|
| 6 |
+
## 1. 背景
|
| 7 |
+
|
| 8 |
+
当前 AutoTeam 仅维持一个 active team workspace(admin_state.json 单 admin email + account_id)。
|
| 9 |
+
单 workspace 故障(被封 / 订阅 cancel / Cloudflare 全域风控)整套自动化即停摆。
|
| 10 |
+
|
| 11 |
+
S7 引入 workspace 池:注册 K 个 workspace(active=1 + warm/cold N-1 互为冷备),主 workspace
|
| 12 |
+
连续探测失败超阈值时秒切到下一个 warm,cold 升 warm,整体不停摆。
|
| 13 |
+
|
| 14 |
+
## 2. 范围
|
| 15 |
+
|
| 16 |
+
### MVP(本任务)
|
| 17 |
+
|
| 18 |
+
1. 新建 `src/autoteam/workspace_pool.py`(持久化 `workspaces.json` + `WorkspacePool`)
|
| 19 |
+
2. `admin_state.get_admin_email/get_chatgpt_account_id` 路由到 `WorkspacePool.get_active()`
|
| 20 |
+
3. `master_health` 失败路径调 `pool.mark_unhealthy()` 触发自动切换(阈值 env `MASTER_HEALTH_FAIL_THRESHOLD=3`)
|
| 21 |
+
4. `cpa_sync` 同步目标改为 active workspace
|
| 22 |
+
5. 测试 ≥ 15 case(mock 故障注入;禁止真实 OpenAI 调用)
|
| 23 |
+
|
| 24 |
+
### 非范围
|
| 25 |
+
|
| 26 |
+
- 不实现真实 OpenAI workspace 创建 API(user 自建 workspace 后手动 register)
|
| 27 |
+
- 不动 mail provider / account_state / manager.py 主流程
|
| 28 |
+
- 不做 active workspace 内 accounts 数据隔离(accounts.json 仍是全局视图)
|
| 29 |
+
- 不动 web 前端(后续基于 SSE 流展示)
|
| 30 |
+
|
| 31 |
+
## 3. 数据 schema
|
| 32 |
+
|
| 33 |
+
### `workspaces.json`
|
| 34 |
+
|
| 35 |
+
```json
|
| 36 |
+
{
|
| 37 |
+
"schema_version": 1,
|
| 38 |
+
"active": "ws-aaa",
|
| 39 |
+
"workspaces": [
|
| 40 |
+
{
|
| 41 |
+
"id": "ws-aaa",
|
| 42 |
+
"admin_email": "ad1@example.com",
|
| 43 |
+
"account_id": "00000000-0000-0000-0000-000000000001",
|
| 44 |
+
"tier": "active",
|
| 45 |
+
"status": "healthy",
|
| 46 |
+
"fail_count": 0,
|
| 47 |
+
"last_check_ts": 1715300000.0,
|
| 48 |
+
"registered_at": 1715200000.0,
|
| 49 |
+
"transition_log": [
|
| 50 |
+
{"ts": 1715200000.0, "from": null, "to": "warm", "reason": "register"},
|
| 51 |
+
{"ts": 1715200100.0, "from": "warm", "to": "active", "reason": "promoted_no_active"}
|
| 52 |
+
]
|
| 53 |
+
}
|
| 54 |
+
]
|
| 55 |
+
}
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
### tier / status 取值
|
| 59 |
+
|
| 60 |
+
- **tier**: `active` | `warm` | `cold` (任一时刻 `active` 至多 1 个)
|
| 61 |
+
- **status**: `healthy` | `unhealthy` | `unknown`
|
| 62 |
+
|
| 63 |
+
## 4. 不变量(强制执行)
|
| 64 |
+
|
| 65 |
+
- **I1**: 任意时刻 workspaces[*].tier == "active" 至多 1 个
|
| 66 |
+
- **I2**: `mark_unhealthy` 命中 active 且 fail_count ≥ threshold → 自动 promote 一个 warm/cold 至 active;原 active 降为 cold + status=unhealthy
|
| 67 |
+
- **I3**: 写盘原子(snapshot `.bak` → write `.tmp` → `os.replace`),失败回滚
|
| 68 |
+
- **I4**: 单 workspace 模式向后兼容:`workspaces.json` 不存在时,从 `state.json` 读 admin_email/account_id 自动 seed 为单一 active workspace;外部 API(`get_admin_email` 等)签名不变
|
| 69 |
+
- **I5**: `register / set_active / mark_unhealthy` 全部追加 `transition_log` 条目
|
| 70 |
+
- **I6**: 与 S1 default_machine 风格一致(lock 内修改、lock 外发布事件)
|
| 71 |
+
- **I7**: 永不抛未捕获异常(与 master_health.py M-I1 看齐)
|
| 72 |
+
|
| 73 |
+
## 5. 关键 API
|
| 74 |
+
|
| 75 |
+
```python
|
| 76 |
+
class WorkspacePool:
|
| 77 |
+
def register(self, workspace_id: str, admin_email: str, account_id: str, tier: str = "warm") -> dict
|
| 78 |
+
def get_active(self) -> dict | None # 当前 active 单条;可能 None
|
| 79 |
+
def set_active(self, workspace_id: str) -> dict
|
| 80 |
+
def mark_unhealthy(self, workspace_id: str, reason: str) -> dict | None
|
| 81 |
+
def list_all(self) -> list[dict]
|
| 82 |
+
def get(self, workspace_id: str) -> dict | None
|
| 83 |
+
```
|
| 84 |
+
|
| 85 |
+
模块级单例 `default_pool: WorkspacePool` 用于生产路径。
|
| 86 |
+
|
| 87 |
+
## 6. 故障切换流程
|
| 88 |
+
|
| 89 |
+
```
|
| 90 |
+
master_health probe FAIL
|
| 91 |
+
│
|
| 92 |
+
▼
|
| 93 |
+
default_pool.mark_unhealthy(active_ws_id, reason)
|
| 94 |
+
│ fail_count += 1
|
| 95 |
+
▼
|
| 96 |
+
fail_count >= MASTER_HEALTH_FAIL_THRESHOLD ?
|
| 97 |
+
├── No → 仅累计,保持 active
|
| 98 |
+
└── Yes → 1) 当前 active.tier := cold, status := unhealthy
|
| 99 |
+
2) 选 warm(最早注册)→ tier := active, status := unknown
|
| 100 |
+
3) 若无 warm → 选 cold healthy → tier := active
|
| 101 |
+
4) 若无可升者 → active = None,记 transition_log {reason: no_failover_candidate}
|
| 102 |
+
5) 写盘 + 转移日志
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
切换写 `state_log.jsonl`(与 default_machine.subscribe 复用渠道,前端 SSE 流即可感知)。
|
| 106 |
+
|
| 107 |
+
## 7. 验收(DoD)
|
| 108 |
+
|
| 109 |
+
- ruff check 全绿
|
| 110 |
+
- pytest 不退化 + 新增 ≥ 15 case 全绿
|
| 111 |
+
- commit `feat(round-12 S7): multi team workspace pool + auto failover`
|
| 112 |
+
- 改动文件:
|
| 113 |
+
- 新增 `src/autoteam/workspace_pool.py`
|
| 114 |
+
- 新增 `tests/unit/test_round12_s7_workspace_pool.py`
|
| 115 |
+
- 改 `src/autoteam/admin_state.py`(路由)
|
| 116 |
+
- 改 `src/autoteam/master_health.py`(失败路径调 pool)
|
| 117 |
+
- 改 `src/autoteam/cpa_sync.py`(active workspace 切换感知)
|
| 118 |
+
|
| 119 |
+
## 8. 风险
|
| 120 |
+
|
| 121 |
+
- **R1**: 现有调用方 `get_admin_email/get_chatgpt_account_id` 极多,若 workspace_pool 退路不严密会破坏所有路径 → 强制兜底(pool 空时回退 state.json)
|
| 122 |
+
- **R2**: master_health 是热路径,增加 pool I/O 不能阻塞 → 写盘异步友好(已在 lock 内但写盘耗时 ms 级,可接受)
|
| 123 |
+
- **R3**: 用户当前无 team 母号子号,无法 e2e;MVP 只做单测 + mock 故障注入
|
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"id": "s7-multi-workspace-pool",
|
| 3 |
+
"name": "s7-multi-workspace-pool",
|
| 4 |
+
"title": "S7: 多 team workspace 池化",
|
| 5 |
+
"description": "",
|
| 6 |
+
"status": "in_progress",
|
| 7 |
+
"dev_type": null,
|
| 8 |
+
"scope": null,
|
| 9 |
+
"package": null,
|
| 10 |
+
"priority": "P2",
|
| 11 |
+
"creator": "ZRainbow",
|
| 12 |
+
"assignee": "ZRainbow",
|
| 13 |
+
"createdAt": "2026-05-11",
|
| 14 |
+
"completedAt": null,
|
| 15 |
+
"branch": null,
|
| 16 |
+
"base_branch": "main",
|
| 17 |
+
"worktree_path": null,
|
| 18 |
+
"commit": null,
|
| 19 |
+
"pr_url": null,
|
| 20 |
+
"subtasks": [],
|
| 21 |
+
"children": [],
|
| 22 |
+
"parent": "05-11-upstream-align-register-multimail-frontend-refresh",
|
| 23 |
+
"relatedFiles": [],
|
| 24 |
+
"notes": "",
|
| 25 |
+
"meta": {}
|
| 26 |
+
}
|
|
@@ -111,6 +111,19 @@ def clear_admin_state():
|
|
| 111 |
|
| 112 |
|
| 113 |
def get_admin_email():
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 114 |
return load_admin_state().get("email", "")
|
| 115 |
|
| 116 |
|
|
@@ -126,6 +139,21 @@ def _is_valid_uuid(value: str) -> bool:
|
|
| 126 |
|
| 127 |
|
| 128 |
def get_chatgpt_account_id():
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
state = load_admin_state()
|
| 130 |
state_id = state.get("account_id", "")
|
| 131 |
# state.json 里的值必须是 UUID 格式才有效(user-xxx 是 user ID 不是 account ID)
|
|
|
|
| 111 |
|
| 112 |
|
| 113 |
def get_admin_email():
|
| 114 |
+
"""Round 12 S7 — route through WorkspacePool active workspace.
|
| 115 |
+
|
| 116 |
+
Backwards-compat: if the pool is empty / module unavailable, fall back
|
| 117 |
+
to the legacy state.json read so single-workspace installs are unaffected.
|
| 118 |
+
"""
|
| 119 |
+
try:
|
| 120 |
+
from autoteam.workspace_pool import default_pool
|
| 121 |
+
|
| 122 |
+
active = default_pool.get_active()
|
| 123 |
+
if active and active.get("admin_email"):
|
| 124 |
+
return active["admin_email"]
|
| 125 |
+
except Exception:
|
| 126 |
+
pass
|
| 127 |
return load_admin_state().get("email", "")
|
| 128 |
|
| 129 |
|
|
|
|
| 139 |
|
| 140 |
|
| 141 |
def get_chatgpt_account_id():
|
| 142 |
+
"""Round 12 S7 — route through WorkspacePool active workspace.
|
| 143 |
+
|
| 144 |
+
Backwards-compat: pool empty → legacy state.json read → CHATGPT_ACCOUNT_ID env.
|
| 145 |
+
Pool ID still must pass UUID validation (filters legacy `user-xxx` ids).
|
| 146 |
+
"""
|
| 147 |
+
try:
|
| 148 |
+
from autoteam.workspace_pool import default_pool
|
| 149 |
+
|
| 150 |
+
active = default_pool.get_active()
|
| 151 |
+
if active:
|
| 152 |
+
pool_aid = (active.get("account_id") or "").strip()
|
| 153 |
+
if pool_aid and _is_valid_uuid(pool_aid):
|
| 154 |
+
return pool_aid
|
| 155 |
+
except Exception:
|
| 156 |
+
pass
|
| 157 |
state = load_admin_state()
|
| 158 |
state_id = state.get("account_id", "")
|
| 159 |
# state.json 里的值必须是 UUID 格式才有效(user-xxx 是 user ID 不是 account ID)
|
|
@@ -653,5 +653,45 @@ def sync_main_codex_to_cpa(filepath):
|
|
| 653 |
if not upload_to_cpa(filepath):
|
| 654 |
raise RuntimeError(f"上传主号认证文件失败: {name}")
|
| 655 |
|
| 656 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 657 |
return {"uploaded": name}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 653 |
if not upload_to_cpa(filepath):
|
| 654 |
raise RuntimeError(f"上传主号认证文件失败: {name}")
|
| 655 |
|
| 656 |
+
# Round 12 S7 — 主号同步成功后,记录当前 active workspace,便于上游观测
|
| 657 |
+
try:
|
| 658 |
+
active = _get_active_workspace_summary()
|
| 659 |
+
if active:
|
| 660 |
+
logger.info(
|
| 661 |
+
"[CPA] 主号 Codex 已同步 (active workspace: id=%s admin=%s account_id=%s)",
|
| 662 |
+
active.get("id"), active.get("admin_email"), active.get("account_id"),
|
| 663 |
+
)
|
| 664 |
+
else:
|
| 665 |
+
logger.info("[CPA] 主号 Codex 已同步: %s", name)
|
| 666 |
+
except Exception:
|
| 667 |
+
logger.info("[CPA] 主号 Codex 已同步: %s", name)
|
| 668 |
return {"uploaded": name}
|
| 669 |
+
|
| 670 |
+
|
| 671 |
+
def _get_active_workspace_summary():
|
| 672 |
+
"""Round 12 S7 — return the current active workspace (id/admin/account_id) or None.
|
| 673 |
+
|
| 674 |
+
Best-effort: never raises; pool unavailable returns None.
|
| 675 |
+
"""
|
| 676 |
+
try:
|
| 677 |
+
from autoteam.workspace_pool import default_pool
|
| 678 |
+
|
| 679 |
+
active = default_pool.get_active()
|
| 680 |
+
if not active:
|
| 681 |
+
return None
|
| 682 |
+
return {
|
| 683 |
+
"id": active.get("id"),
|
| 684 |
+
"admin_email": active.get("admin_email"),
|
| 685 |
+
"account_id": active.get("account_id"),
|
| 686 |
+
}
|
| 687 |
+
except Exception:
|
| 688 |
+
return None
|
| 689 |
+
|
| 690 |
+
|
| 691 |
+
def get_active_sync_target():
|
| 692 |
+
"""Round 12 S7 — public helper: which workspace identity should CPA sync to?
|
| 693 |
+
|
| 694 |
+
Returns dict({id, admin_email, account_id}) or None when single-workspace
|
| 695 |
+
mode (caller falls back to legacy admin_state).
|
| 696 |
+
"""
|
| 697 |
+
return _get_active_workspace_summary()
|
|
@@ -511,6 +511,62 @@ def is_master_subscription_healthy(
|
|
| 511 |
return healthy, reason, evidence
|
| 512 |
|
| 513 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 514 |
# ---------------------------------------------------------------------------
|
| 515 |
# Round 9 SPEC v1.1 §11~§12 — Retroactive helper + grace 期 JWT 解析
|
| 516 |
# ---------------------------------------------------------------------------
|
|
|
|
| 511 |
return healthy, reason, evidence
|
| 512 |
|
| 513 |
|
| 514 |
+
# ---------------------------------------------------------------------------
|
| 515 |
+
# Round 12 S7 — WorkspacePool failover hook
|
| 516 |
+
# ---------------------------------------------------------------------------
|
| 517 |
+
|
| 518 |
+
# reasons that should NOT trip workspace failover (transient / classification noise)
|
| 519 |
+
_TRANSIENT_FAIL_REASONS = ("network_error",)
|
| 520 |
+
# reasons that SHOULD trip failover (real master-side failure)
|
| 521 |
+
_HARD_FAIL_REASONS = ("auth_invalid", "subscription_cancelled", "workspace_missing", "role_not_owner")
|
| 522 |
+
|
| 523 |
+
|
| 524 |
+
def apply_pool_health_signal(
|
| 525 |
+
healthy: bool,
|
| 526 |
+
reason: str,
|
| 527 |
+
evidence: dict | None = None,
|
| 528 |
+
*,
|
| 529 |
+
pool=None,
|
| 530 |
+
) -> dict | None:
|
| 531 |
+
"""Round 12 S7 — feed a master_health probe result into the WorkspacePool.
|
| 532 |
+
|
| 533 |
+
Call this from any caller of :func:`is_master_subscription_healthy` to
|
| 534 |
+
drive auto failover. Behavior:
|
| 535 |
+
|
| 536 |
+
- healthy=True (active or subscription_grace) → pool.mark_healthy(active_id)
|
| 537 |
+
- healthy=False AND reason in _HARD_FAIL_REASONS → pool.mark_unhealthy(...)
|
| 538 |
+
which accumulates fail_count + (when ≥ threshold) auto-promotes a
|
| 539 |
+
warm/cold candidate.
|
| 540 |
+
- healthy=False AND reason in _TRANSIENT_FAIL_REASONS → no-op (don't
|
| 541 |
+
flap on intermittent network glitches).
|
| 542 |
+
|
| 543 |
+
Returns the **post-call** active workspace snapshot (None if pool empty
|
| 544 |
+
or failed). Never raises — M-I1 alignment.
|
| 545 |
+
|
| 546 |
+
``pool`` defaults to ``workspace_pool.default_pool``; tests can inject
|
| 547 |
+
their own.
|
| 548 |
+
"""
|
| 549 |
+
try:
|
| 550 |
+
if pool is None:
|
| 551 |
+
from autoteam.workspace_pool import default_pool as pool_obj
|
| 552 |
+
pool = pool_obj
|
| 553 |
+
active = pool.get_active()
|
| 554 |
+
if active is None:
|
| 555 |
+
return None
|
| 556 |
+
ws_id = active.get("id")
|
| 557 |
+
if not ws_id:
|
| 558 |
+
return None
|
| 559 |
+
if healthy and reason in ("active", "subscription_grace"):
|
| 560 |
+
return pool.mark_healthy(ws_id)
|
| 561 |
+
if (not healthy) and reason in _HARD_FAIL_REASONS:
|
| 562 |
+
return pool.mark_unhealthy(ws_id, reason)
|
| 563 |
+
# transient or unknown → leave state alone but bubble snapshot
|
| 564 |
+
return active
|
| 565 |
+
except Exception as exc:
|
| 566 |
+
logger.warning("[master_health] apply_pool_health_signal failed: %s", exc)
|
| 567 |
+
return None
|
| 568 |
+
|
| 569 |
+
|
| 570 |
# ---------------------------------------------------------------------------
|
| 571 |
# Round 9 SPEC v1.1 §11~§12 — Retroactive helper + grace 期 JWT 解析
|
| 572 |
# ---------------------------------------------------------------------------
|
|
@@ -0,0 +1,683 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Multi team workspace pool — Round 12 S7。
|
| 2 |
+
|
| 3 |
+
维持 K 个 team workspace 互为冷备:任意时刻仅 1 个 active(主热路径用),
|
| 4 |
+
其余作 warm/cold 备份;主 workspace 连续探测失败超阈值时秒切。
|
| 5 |
+
|
| 6 |
+
不变量(I1~I7,详见 .trellis/tasks/05-11-s7-multi-workspace-pool/prd.md):
|
| 7 |
+
|
| 8 |
+
- I1: workspaces[*].tier == "active" 至多 1 个
|
| 9 |
+
- I2: mark_unhealthy 命中 active 且 fail_count >= threshold → 自动 promote warm/cold
|
| 10 |
+
- I3: 写盘原子(snapshot .bak → tmp → os.replace),失败回滚
|
| 11 |
+
- I4: 单 workspace 模式向后兼容 — workspaces.json 不存在时 seed 自 state.json
|
| 12 |
+
- I5: register / set_active / mark_unhealthy / promote 均追加 transition_log
|
| 13 |
+
- I6: 与 default_machine 风格一致(lock 内修改、lock 外发布事件)
|
| 14 |
+
- I7: 永不抛未捕获异常(与 master_health.py M-I1 看齐)
|
| 15 |
+
"""
|
| 16 |
+
from __future__ import annotations
|
| 17 |
+
|
| 18 |
+
import json
|
| 19 |
+
import logging
|
| 20 |
+
import os
|
| 21 |
+
import shutil
|
| 22 |
+
import threading
|
| 23 |
+
import time
|
| 24 |
+
from collections.abc import Callable
|
| 25 |
+
from pathlib import Path
|
| 26 |
+
from typing import Any
|
| 27 |
+
|
| 28 |
+
logger = logging.getLogger(__name__)
|
| 29 |
+
|
| 30 |
+
PROJECT_ROOT = Path(__file__).parent.parent.parent
|
| 31 |
+
DEFAULT_POOL_FILE = PROJECT_ROOT / "workspaces.json"
|
| 32 |
+
POOL_FILE_MODE = 0o666
|
| 33 |
+
|
| 34 |
+
SCHEMA_VERSION = 1
|
| 35 |
+
|
| 36 |
+
TIER_ACTIVE = "active"
|
| 37 |
+
TIER_WARM = "warm"
|
| 38 |
+
TIER_COLD = "cold"
|
| 39 |
+
_VALID_TIERS = (TIER_ACTIVE, TIER_WARM, TIER_COLD)
|
| 40 |
+
|
| 41 |
+
STATUS_HEALTHY = "healthy"
|
| 42 |
+
STATUS_UNHEALTHY = "unhealthy"
|
| 43 |
+
STATUS_UNKNOWN = "unknown"
|
| 44 |
+
_VALID_STATUSES = (STATUS_HEALTHY, STATUS_UNHEALTHY, STATUS_UNKNOWN)
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def _default_fail_threshold() -> int:
|
| 48 |
+
raw = os.environ.get("MASTER_HEALTH_FAIL_THRESHOLD", "3")
|
| 49 |
+
try:
|
| 50 |
+
v = int(raw)
|
| 51 |
+
return v if v > 0 else 3
|
| 52 |
+
except Exception:
|
| 53 |
+
return 3
|
| 54 |
+
|
| 55 |
+
|
| 56 |
+
PoolSubscriber = Callable[[dict], None]
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
class WorkspacePool:
|
| 60 |
+
"""Persistent multi-workspace pool with auto failover.
|
| 61 |
+
|
| 62 |
+
The on-disk schema is documented in prd.md §3. The instance owns a
|
| 63 |
+
threading lock — all reads / writes serialize on it, mirroring
|
| 64 |
+
:class:`autoteam.account_state.StateMachine`.
|
| 65 |
+
"""
|
| 66 |
+
|
| 67 |
+
def __init__(
|
| 68 |
+
self,
|
| 69 |
+
path: Path | None = None,
|
| 70 |
+
*,
|
| 71 |
+
fail_threshold: int | None = None,
|
| 72 |
+
) -> None:
|
| 73 |
+
self._path: Path = Path(path) if path else DEFAULT_POOL_FILE
|
| 74 |
+
self._lock = threading.Lock()
|
| 75 |
+
self._subscribers: list[PoolSubscriber] = []
|
| 76 |
+
self._fail_threshold = fail_threshold if fail_threshold is not None else _default_fail_threshold()
|
| 77 |
+
|
| 78 |
+
# ---------------------------------------------------------------- config
|
| 79 |
+
@property
|
| 80 |
+
def path(self) -> Path:
|
| 81 |
+
return self._path
|
| 82 |
+
|
| 83 |
+
@property
|
| 84 |
+
def fail_threshold(self) -> int:
|
| 85 |
+
return self._fail_threshold
|
| 86 |
+
|
| 87 |
+
def set_fail_threshold(self, threshold: int) -> None:
|
| 88 |
+
if not isinstance(threshold, int) or threshold <= 0:
|
| 89 |
+
raise ValueError("fail_threshold must be a positive int")
|
| 90 |
+
self._fail_threshold = threshold
|
| 91 |
+
|
| 92 |
+
# ----------------------------------------------------------- event bus
|
| 93 |
+
def subscribe(self, callback: PoolSubscriber) -> PoolSubscriber:
|
| 94 |
+
if not callable(callback):
|
| 95 |
+
raise TypeError("subscribe expects a callable")
|
| 96 |
+
with self._lock:
|
| 97 |
+
if callback not in self._subscribers:
|
| 98 |
+
self._subscribers.append(callback)
|
| 99 |
+
return callback
|
| 100 |
+
|
| 101 |
+
def unsubscribe(self, callback: PoolSubscriber) -> bool:
|
| 102 |
+
with self._lock:
|
| 103 |
+
try:
|
| 104 |
+
self._subscribers.remove(callback)
|
| 105 |
+
return True
|
| 106 |
+
except ValueError:
|
| 107 |
+
return False
|
| 108 |
+
|
| 109 |
+
def _publish(self, event: dict) -> None:
|
| 110 |
+
with self._lock:
|
| 111 |
+
subs = list(self._subscribers)
|
| 112 |
+
for cb in subs:
|
| 113 |
+
try:
|
| 114 |
+
cb(event)
|
| 115 |
+
except Exception:
|
| 116 |
+
logger.exception("workspace_pool subscriber %r raised on event %s", cb, event)
|
| 117 |
+
|
| 118 |
+
# ---------------------------------------------------------------- I/O
|
| 119 |
+
def _load_raw(self) -> dict:
|
| 120 |
+
"""Load on-disk pool. Auto-seed from state.json on missing file (I4)."""
|
| 121 |
+
if not self._path.exists():
|
| 122 |
+
return self._seed_from_admin_state()
|
| 123 |
+
try:
|
| 124 |
+
text = self._path.read_text(encoding="utf-8").strip()
|
| 125 |
+
except Exception as exc:
|
| 126 |
+
logger.warning("[workspace_pool] read failed: %s — treating as empty", exc)
|
| 127 |
+
return _empty_doc()
|
| 128 |
+
if not text:
|
| 129 |
+
return _empty_doc()
|
| 130 |
+
try:
|
| 131 |
+
data = json.loads(text)
|
| 132 |
+
except Exception as exc:
|
| 133 |
+
logger.warning("[workspace_pool] json parse failed: %s — treating as empty", exc)
|
| 134 |
+
return _empty_doc()
|
| 135 |
+
if not isinstance(data, dict):
|
| 136 |
+
return _empty_doc()
|
| 137 |
+
if data.get("schema_version") != SCHEMA_VERSION:
|
| 138 |
+
# forward-incompatible / legacy → discard, will re-seed on next register
|
| 139 |
+
logger.warning(
|
| 140 |
+
"[workspace_pool] schema_version mismatch (got %r, expect %d) — discard",
|
| 141 |
+
data.get("schema_version"),
|
| 142 |
+
SCHEMA_VERSION,
|
| 143 |
+
)
|
| 144 |
+
return _empty_doc()
|
| 145 |
+
ws = data.get("workspaces")
|
| 146 |
+
if not isinstance(ws, list):
|
| 147 |
+
ws = []
|
| 148 |
+
# normalize each row
|
| 149 |
+
norm_ws: list[dict] = []
|
| 150 |
+
for row in ws:
|
| 151 |
+
n = _normalize_row(row)
|
| 152 |
+
if n is not None:
|
| 153 |
+
norm_ws.append(n)
|
| 154 |
+
active = data.get("active") or None
|
| 155 |
+
return {
|
| 156 |
+
"schema_version": SCHEMA_VERSION,
|
| 157 |
+
"active": active if isinstance(active, str) else None,
|
| 158 |
+
"workspaces": norm_ws,
|
| 159 |
+
}
|
| 160 |
+
|
| 161 |
+
def _seed_from_admin_state(self) -> dict:
|
| 162 |
+
"""I4 — single-workspace backwards compat.
|
| 163 |
+
|
| 164 |
+
On first read with no workspaces.json, attempt to seed a single
|
| 165 |
+
workspace from state.json (admin_email + account_id). Empty state
|
| 166 |
+
produces an empty doc (no workspaces). Never raises.
|
| 167 |
+
"""
|
| 168 |
+
try:
|
| 169 |
+
from autoteam.admin_state import load_admin_state # late import to break cycle
|
| 170 |
+
st = load_admin_state() or {}
|
| 171 |
+
except Exception as exc:
|
| 172 |
+
logger.warning("[workspace_pool] seed: cannot read admin_state: %s", exc)
|
| 173 |
+
return _empty_doc()
|
| 174 |
+
admin_email = (st.get("email") or "").strip()
|
| 175 |
+
account_id = (st.get("account_id") or "").strip()
|
| 176 |
+
if not admin_email or not account_id:
|
| 177 |
+
# nothing to seed — fresh install
|
| 178 |
+
return _empty_doc()
|
| 179 |
+
ws_id = f"ws-{account_id}"
|
| 180 |
+
now = time.time()
|
| 181 |
+
row = {
|
| 182 |
+
"id": ws_id,
|
| 183 |
+
"admin_email": admin_email,
|
| 184 |
+
"account_id": account_id,
|
| 185 |
+
"tier": TIER_ACTIVE,
|
| 186 |
+
"status": STATUS_UNKNOWN,
|
| 187 |
+
"fail_count": 0,
|
| 188 |
+
"last_check_ts": None,
|
| 189 |
+
"registered_at": now,
|
| 190 |
+
"transition_log": [
|
| 191 |
+
{"ts": now, "from": None, "to": TIER_ACTIVE, "reason": "seed_from_admin_state"},
|
| 192 |
+
],
|
| 193 |
+
}
|
| 194 |
+
return {
|
| 195 |
+
"schema_version": SCHEMA_VERSION,
|
| 196 |
+
"active": ws_id,
|
| 197 |
+
"workspaces": [row],
|
| 198 |
+
}
|
| 199 |
+
|
| 200 |
+
def _save_raw(self, doc: dict) -> None:
|
| 201 |
+
"""I3 — atomic write with .bak rollback."""
|
| 202 |
+
path = self._path
|
| 203 |
+
path.parent.mkdir(parents=True, exist_ok=True)
|
| 204 |
+
bak = path.with_suffix(path.suffix + ".bak")
|
| 205 |
+
tmp = path.with_suffix(path.suffix + ".tmp")
|
| 206 |
+
payload = json.dumps(doc, indent=2, ensure_ascii=False).encode("utf-8")
|
| 207 |
+
had_snapshot = False
|
| 208 |
+
if path.exists():
|
| 209 |
+
try:
|
| 210 |
+
shutil.copy2(path, bak)
|
| 211 |
+
had_snapshot = True
|
| 212 |
+
except OSError as exc:
|
| 213 |
+
logger.warning("[workspace_pool] snapshot failed %s -> %s: %s", path, bak, exc)
|
| 214 |
+
try:
|
| 215 |
+
with open(tmp, "wb") as fp:
|
| 216 |
+
fp.write(payload)
|
| 217 |
+
fp.flush()
|
| 218 |
+
try:
|
| 219 |
+
os.fsync(fp.fileno())
|
| 220 |
+
except OSError:
|
| 221 |
+
pass
|
| 222 |
+
os.replace(tmp, path)
|
| 223 |
+
except OSError:
|
| 224 |
+
logger.exception("[workspace_pool] write failed for %s, attempting rollback", path)
|
| 225 |
+
if had_snapshot and bak.exists():
|
| 226 |
+
try:
|
| 227 |
+
os.replace(bak, path)
|
| 228 |
+
except OSError:
|
| 229 |
+
logger.exception("[workspace_pool] rollback failed for %s", path)
|
| 230 |
+
if tmp.exists():
|
| 231 |
+
try:
|
| 232 |
+
tmp.unlink()
|
| 233 |
+
except OSError:
|
| 234 |
+
pass
|
| 235 |
+
raise
|
| 236 |
+
else:
|
| 237 |
+
try:
|
| 238 |
+
os.chmod(path, POOL_FILE_MODE)
|
| 239 |
+
except OSError:
|
| 240 |
+
pass
|
| 241 |
+
if had_snapshot and bak.exists():
|
| 242 |
+
try:
|
| 243 |
+
bak.unlink()
|
| 244 |
+
except OSError:
|
| 245 |
+
logger.warning("[workspace_pool] cannot delete .bak %s", bak)
|
| 246 |
+
|
| 247 |
+
# ---------------------------------------------------------- public API
|
| 248 |
+
|
| 249 |
+
def list_all(self) -> list[dict]:
|
| 250 |
+
"""Return all workspace rows (deep copy, safe to mutate)."""
|
| 251 |
+
with self._lock:
|
| 252 |
+
doc = self._load_raw()
|
| 253 |
+
return [dict(row) | {"transition_log": list(row.get("transition_log") or [])}
|
| 254 |
+
for row in doc["workspaces"]]
|
| 255 |
+
|
| 256 |
+
def get(self, workspace_id: str) -> dict | None:
|
| 257 |
+
"""Return one workspace by id (deep copy) or None."""
|
| 258 |
+
if not workspace_id:
|
| 259 |
+
return None
|
| 260 |
+
with self._lock:
|
| 261 |
+
doc = self._load_raw()
|
| 262 |
+
for row in doc["workspaces"]:
|
| 263 |
+
if row.get("id") == workspace_id:
|
| 264 |
+
return dict(row) | {"transition_log": list(row.get("transition_log") or [])}
|
| 265 |
+
return None
|
| 266 |
+
|
| 267 |
+
def get_active(self) -> dict | None:
|
| 268 |
+
"""Return the current active workspace (or None if pool empty)."""
|
| 269 |
+
with self._lock:
|
| 270 |
+
doc = self._load_raw()
|
| 271 |
+
active_id = doc.get("active")
|
| 272 |
+
if not active_id:
|
| 273 |
+
return None
|
| 274 |
+
for row in doc["workspaces"]:
|
| 275 |
+
if row.get("id") == active_id and row.get("tier") == TIER_ACTIVE:
|
| 276 |
+
return dict(row) | {"transition_log": list(row.get("transition_log") or [])}
|
| 277 |
+
return None
|
| 278 |
+
|
| 279 |
+
def register(
|
| 280 |
+
self,
|
| 281 |
+
workspace_id: str,
|
| 282 |
+
admin_email: str,
|
| 283 |
+
account_id: str,
|
| 284 |
+
tier: str = TIER_WARM,
|
| 285 |
+
) -> dict:
|
| 286 |
+
"""Register a new workspace. If id exists, raise ValueError.
|
| 287 |
+
|
| 288 |
+
- tier=active is allowed but will demote any prior active to cold
|
| 289 |
+
(I1 — at most one active).
|
| 290 |
+
- First registration with no active in pool auto-promotes (regardless
|
| 291 |
+
of supplied tier) so single-workspace fresh installs work.
|
| 292 |
+
"""
|
| 293 |
+
if not workspace_id or not isinstance(workspace_id, str):
|
| 294 |
+
raise ValueError("workspace_id required")
|
| 295 |
+
if not admin_email or not isinstance(admin_email, str):
|
| 296 |
+
raise ValueError("admin_email required")
|
| 297 |
+
if not account_id or not isinstance(account_id, str):
|
| 298 |
+
raise ValueError("account_id required")
|
| 299 |
+
if tier not in _VALID_TIERS:
|
| 300 |
+
raise ValueError(f"tier must be one of {_VALID_TIERS!r}")
|
| 301 |
+
|
| 302 |
+
events: list[dict] = []
|
| 303 |
+
with self._lock:
|
| 304 |
+
doc = self._load_raw()
|
| 305 |
+
for row in doc["workspaces"]:
|
| 306 |
+
if row.get("id") == workspace_id:
|
| 307 |
+
raise ValueError(f"workspace_id {workspace_id!r} already registered")
|
| 308 |
+
now = time.time()
|
| 309 |
+
target_tier = tier
|
| 310 |
+
# auto-promote if pool is empty / has no active
|
| 311 |
+
if doc.get("active") is None and not any(r.get("tier") == TIER_ACTIVE for r in doc["workspaces"]):
|
| 312 |
+
target_tier = TIER_ACTIVE
|
| 313 |
+
|
| 314 |
+
new_row = {
|
| 315 |
+
"id": workspace_id,
|
| 316 |
+
"admin_email": admin_email,
|
| 317 |
+
"account_id": account_id,
|
| 318 |
+
"tier": target_tier,
|
| 319 |
+
"status": STATUS_UNKNOWN,
|
| 320 |
+
"fail_count": 0,
|
| 321 |
+
"last_check_ts": None,
|
| 322 |
+
"registered_at": now,
|
| 323 |
+
"transition_log": [
|
| 324 |
+
{"ts": now, "from": None, "to": target_tier, "reason": "register"},
|
| 325 |
+
],
|
| 326 |
+
}
|
| 327 |
+
|
| 328 |
+
if target_tier == TIER_ACTIVE:
|
| 329 |
+
# I1 — demote any other active row
|
| 330 |
+
prior_active_id = doc.get("active")
|
| 331 |
+
for row in doc["workspaces"]:
|
| 332 |
+
if row.get("tier") == TIER_ACTIVE:
|
| 333 |
+
_append_transition(row, TIER_ACTIVE, TIER_COLD, "demote_on_register_active", now)
|
| 334 |
+
row["tier"] = TIER_COLD
|
| 335 |
+
doc["active"] = workspace_id
|
| 336 |
+
if prior_active_id and prior_active_id != workspace_id:
|
| 337 |
+
events.append({
|
| 338 |
+
"type": "demoted",
|
| 339 |
+
"workspace_id": prior_active_id,
|
| 340 |
+
"reason": "register_active",
|
| 341 |
+
"ts": now,
|
| 342 |
+
})
|
| 343 |
+
|
| 344 |
+
doc["workspaces"].append(new_row)
|
| 345 |
+
self._save_raw(doc)
|
| 346 |
+
events.append({
|
| 347 |
+
"type": "registered",
|
| 348 |
+
"workspace_id": workspace_id,
|
| 349 |
+
"tier": target_tier,
|
| 350 |
+
"ts": now,
|
| 351 |
+
})
|
| 352 |
+
snapshot = dict(new_row) | {"transition_log": list(new_row["transition_log"])}
|
| 353 |
+
|
| 354 |
+
for ev in events:
|
| 355 |
+
self._publish(ev)
|
| 356 |
+
return snapshot
|
| 357 |
+
|
| 358 |
+
def set_active(self, workspace_id: str) -> dict:
|
| 359 |
+
"""Promote ``workspace_id`` to active; demote any prior active to cold."""
|
| 360 |
+
if not workspace_id:
|
| 361 |
+
raise ValueError("workspace_id required")
|
| 362 |
+
events: list[dict] = []
|
| 363 |
+
with self._lock:
|
| 364 |
+
doc = self._load_raw()
|
| 365 |
+
target = None
|
| 366 |
+
for row in doc["workspaces"]:
|
| 367 |
+
if row.get("id") == workspace_id:
|
| 368 |
+
target = row
|
| 369 |
+
break
|
| 370 |
+
if target is None:
|
| 371 |
+
raise KeyError(f"workspace_id {workspace_id!r} not found")
|
| 372 |
+
now = time.time()
|
| 373 |
+
prior_active_id = doc.get("active")
|
| 374 |
+
if target.get("tier") == TIER_ACTIVE and prior_active_id == workspace_id:
|
| 375 |
+
# already active — short-circuit but still touch last_check_ts? no, idempotent no-op
|
| 376 |
+
snapshot = dict(target) | {"transition_log": list(target.get("transition_log") or [])}
|
| 377 |
+
return snapshot
|
| 378 |
+
for row in doc["workspaces"]:
|
| 379 |
+
if row is target:
|
| 380 |
+
continue
|
| 381 |
+
if row.get("tier") == TIER_ACTIVE:
|
| 382 |
+
_append_transition(row, TIER_ACTIVE, TIER_COLD, "demote_on_set_active", now)
|
| 383 |
+
row["tier"] = TIER_COLD
|
| 384 |
+
events.append({
|
| 385 |
+
"type": "demoted",
|
| 386 |
+
"workspace_id": row.get("id"),
|
| 387 |
+
"reason": "set_active",
|
| 388 |
+
"ts": now,
|
| 389 |
+
})
|
| 390 |
+
_append_transition(target, target.get("tier"), TIER_ACTIVE, "set_active", now)
|
| 391 |
+
target["tier"] = TIER_ACTIVE
|
| 392 |
+
target["status"] = STATUS_UNKNOWN # reset; next probe sets healthy/unhealthy
|
| 393 |
+
target["fail_count"] = 0
|
| 394 |
+
doc["active"] = workspace_id
|
| 395 |
+
self._save_raw(doc)
|
| 396 |
+
events.append({
|
| 397 |
+
"type": "promoted",
|
| 398 |
+
"workspace_id": workspace_id,
|
| 399 |
+
"reason": "set_active",
|
| 400 |
+
"ts": now,
|
| 401 |
+
})
|
| 402 |
+
snapshot = dict(target) | {"transition_log": list(target["transition_log"])}
|
| 403 |
+
|
| 404 |
+
for ev in events:
|
| 405 |
+
self._publish(ev)
|
| 406 |
+
return snapshot
|
| 407 |
+
|
| 408 |
+
def mark_unhealthy(
|
| 409 |
+
self,
|
| 410 |
+
workspace_id: str,
|
| 411 |
+
reason: str,
|
| 412 |
+
*,
|
| 413 |
+
force_failover: bool = False,
|
| 414 |
+
) -> dict | None:
|
| 415 |
+
"""Mark a workspace unhealthy. If active and fail_count >= threshold,
|
| 416 |
+
auto-promote a warm/cold healthy candidate (I2).
|
| 417 |
+
|
| 418 |
+
Returns the (possibly newly-promoted) active workspace snapshot, or
|
| 419 |
+
None if the pool ended up with no active candidate.
|
| 420 |
+
"""
|
| 421 |
+
if not workspace_id:
|
| 422 |
+
raise ValueError("workspace_id required")
|
| 423 |
+
events: list[dict] = []
|
| 424 |
+
with self._lock:
|
| 425 |
+
doc = self._load_raw()
|
| 426 |
+
target = None
|
| 427 |
+
for row in doc["workspaces"]:
|
| 428 |
+
if row.get("id") == workspace_id:
|
| 429 |
+
target = row
|
| 430 |
+
break
|
| 431 |
+
if target is None:
|
| 432 |
+
raise KeyError(f"workspace_id {workspace_id!r} not found")
|
| 433 |
+
|
| 434 |
+
now = time.time()
|
| 435 |
+
prev_status = target.get("status")
|
| 436 |
+
target["fail_count"] = int(target.get("fail_count") or 0) + 1
|
| 437 |
+
target["status"] = STATUS_UNHEALTHY
|
| 438 |
+
target["last_check_ts"] = now
|
| 439 |
+
_append_transition(
|
| 440 |
+
target,
|
| 441 |
+
prev_status,
|
| 442 |
+
STATUS_UNHEALTHY,
|
| 443 |
+
f"mark_unhealthy:{reason}",
|
| 444 |
+
now,
|
| 445 |
+
kind="status",
|
| 446 |
+
extra={"fail_count": target["fail_count"]},
|
| 447 |
+
)
|
| 448 |
+
events.append({
|
| 449 |
+
"type": "marked_unhealthy",
|
| 450 |
+
"workspace_id": workspace_id,
|
| 451 |
+
"reason": reason,
|
| 452 |
+
"fail_count": target["fail_count"],
|
| 453 |
+
"ts": now,
|
| 454 |
+
})
|
| 455 |
+
|
| 456 |
+
should_failover = (
|
| 457 |
+
target.get("tier") == TIER_ACTIVE
|
| 458 |
+
and (force_failover or target["fail_count"] >= self._fail_threshold)
|
| 459 |
+
)
|
| 460 |
+
promoted_id: str | None = None
|
| 461 |
+
if should_failover:
|
| 462 |
+
# demote current active → cold
|
| 463 |
+
_append_transition(target, TIER_ACTIVE, TIER_COLD, "failover_demote", now)
|
| 464 |
+
target["tier"] = TIER_COLD
|
| 465 |
+
if doc.get("active") == workspace_id:
|
| 466 |
+
doc["active"] = None
|
| 467 |
+
# pick promotion candidate
|
| 468 |
+
candidate = _pick_failover_candidate(doc["workspaces"])
|
| 469 |
+
if candidate is not None:
|
| 470 |
+
_append_transition(
|
| 471 |
+
candidate,
|
| 472 |
+
candidate.get("tier"),
|
| 473 |
+
TIER_ACTIVE,
|
| 474 |
+
"failover_promote",
|
| 475 |
+
now,
|
| 476 |
+
)
|
| 477 |
+
candidate["tier"] = TIER_ACTIVE
|
| 478 |
+
candidate["status"] = STATUS_UNKNOWN
|
| 479 |
+
candidate["fail_count"] = 0
|
| 480 |
+
doc["active"] = candidate.get("id")
|
| 481 |
+
promoted_id = candidate.get("id")
|
| 482 |
+
events.append({
|
| 483 |
+
"type": "promoted",
|
| 484 |
+
"workspace_id": promoted_id,
|
| 485 |
+
"reason": "failover",
|
| 486 |
+
"ts": now,
|
| 487 |
+
})
|
| 488 |
+
else:
|
| 489 |
+
events.append({
|
| 490 |
+
"type": "no_failover_candidate",
|
| 491 |
+
"workspace_id": workspace_id,
|
| 492 |
+
"ts": now,
|
| 493 |
+
})
|
| 494 |
+
|
| 495 |
+
self._save_raw(doc)
|
| 496 |
+
new_active = self._active_row(doc)
|
| 497 |
+
snapshot = (
|
| 498 |
+
dict(new_active) | {"transition_log": list(new_active.get("transition_log") or [])}
|
| 499 |
+
if new_active else None
|
| 500 |
+
)
|
| 501 |
+
|
| 502 |
+
for ev in events:
|
| 503 |
+
self._publish(ev)
|
| 504 |
+
return snapshot
|
| 505 |
+
|
| 506 |
+
def mark_healthy(self, workspace_id: str) -> dict | None:
|
| 507 |
+
"""Reset fail_count to 0 + status=healthy on a successful probe.
|
| 508 |
+
|
| 509 |
+
Idempotent. Useful for the master_health success path so that
|
| 510 |
+
intermittent failures don't accumulate to threshold.
|
| 511 |
+
"""
|
| 512 |
+
if not workspace_id:
|
| 513 |
+
raise ValueError("workspace_id required")
|
| 514 |
+
events: list[dict] = []
|
| 515 |
+
with self._lock:
|
| 516 |
+
doc = self._load_raw()
|
| 517 |
+
target = None
|
| 518 |
+
for row in doc["workspaces"]:
|
| 519 |
+
if row.get("id") == workspace_id:
|
| 520 |
+
target = row
|
| 521 |
+
break
|
| 522 |
+
if target is None:
|
| 523 |
+
raise KeyError(f"workspace_id {workspace_id!r} not found")
|
| 524 |
+
now = time.time()
|
| 525 |
+
prev_status = target.get("status")
|
| 526 |
+
prev_fail = int(target.get("fail_count") or 0)
|
| 527 |
+
target["fail_count"] = 0
|
| 528 |
+
target["status"] = STATUS_HEALTHY
|
| 529 |
+
target["last_check_ts"] = now
|
| 530 |
+
if prev_status != STATUS_HEALTHY or prev_fail != 0:
|
| 531 |
+
_append_transition(
|
| 532 |
+
target,
|
| 533 |
+
prev_status,
|
| 534 |
+
STATUS_HEALTHY,
|
| 535 |
+
"mark_healthy",
|
| 536 |
+
now,
|
| 537 |
+
kind="status",
|
| 538 |
+
extra={"prev_fail_count": prev_fail},
|
| 539 |
+
)
|
| 540 |
+
events.append({
|
| 541 |
+
"type": "marked_healthy",
|
| 542 |
+
"workspace_id": workspace_id,
|
| 543 |
+
"ts": now,
|
| 544 |
+
})
|
| 545 |
+
self._save_raw(doc)
|
| 546 |
+
snapshot = dict(target) | {"transition_log": list(target.get("transition_log") or [])}
|
| 547 |
+
for ev in events:
|
| 548 |
+
self._publish(ev)
|
| 549 |
+
return snapshot
|
| 550 |
+
|
| 551 |
+
# --------------------------------------------------------------- helpers
|
| 552 |
+
def _active_row(self, doc: dict) -> dict | None:
|
| 553 |
+
active_id = doc.get("active")
|
| 554 |
+
if not active_id:
|
| 555 |
+
return None
|
| 556 |
+
for row in doc["workspaces"]:
|
| 557 |
+
if row.get("id") == active_id:
|
| 558 |
+
return row
|
| 559 |
+
return None
|
| 560 |
+
|
| 561 |
+
def reset(self) -> None:
|
| 562 |
+
"""Test helper — wipe the pool file. Production code should not call."""
|
| 563 |
+
with self._lock:
|
| 564 |
+
try:
|
| 565 |
+
if self._path.exists():
|
| 566 |
+
self._path.unlink()
|
| 567 |
+
except OSError as exc:
|
| 568 |
+
logger.warning("[workspace_pool] reset failed: %s", exc)
|
| 569 |
+
|
| 570 |
+
|
| 571 |
+
# ---------------------------------------------------------------------------
|
| 572 |
+
# Internal helpers
|
| 573 |
+
# ---------------------------------------------------------------------------
|
| 574 |
+
def _empty_doc() -> dict:
|
| 575 |
+
return {"schema_version": SCHEMA_VERSION, "active": None, "workspaces": []}
|
| 576 |
+
|
| 577 |
+
|
| 578 |
+
def _normalize_row(row: Any) -> dict | None:
|
| 579 |
+
if not isinstance(row, dict):
|
| 580 |
+
return None
|
| 581 |
+
rid = row.get("id")
|
| 582 |
+
if not isinstance(rid, str) or not rid:
|
| 583 |
+
return None
|
| 584 |
+
tier = row.get("tier")
|
| 585 |
+
if tier not in _VALID_TIERS:
|
| 586 |
+
tier = TIER_WARM
|
| 587 |
+
status = row.get("status")
|
| 588 |
+
if status not in _VALID_STATUSES:
|
| 589 |
+
status = STATUS_UNKNOWN
|
| 590 |
+
log = row.get("transition_log")
|
| 591 |
+
if not isinstance(log, list):
|
| 592 |
+
log = []
|
| 593 |
+
fail_count = row.get("fail_count")
|
| 594 |
+
try:
|
| 595 |
+
fail_count = int(fail_count or 0)
|
| 596 |
+
except (TypeError, ValueError):
|
| 597 |
+
fail_count = 0
|
| 598 |
+
return {
|
| 599 |
+
"id": rid,
|
| 600 |
+
"admin_email": str(row.get("admin_email") or ""),
|
| 601 |
+
"account_id": str(row.get("account_id") or ""),
|
| 602 |
+
"tier": tier,
|
| 603 |
+
"status": status,
|
| 604 |
+
"fail_count": fail_count,
|
| 605 |
+
"last_check_ts": row.get("last_check_ts"),
|
| 606 |
+
"registered_at": row.get("registered_at"),
|
| 607 |
+
"transition_log": list(log),
|
| 608 |
+
}
|
| 609 |
+
|
| 610 |
+
|
| 611 |
+
def _append_transition(
|
| 612 |
+
row: dict,
|
| 613 |
+
from_value: Any,
|
| 614 |
+
to_value: Any,
|
| 615 |
+
reason: str,
|
| 616 |
+
ts: float,
|
| 617 |
+
*,
|
| 618 |
+
kind: str = "tier",
|
| 619 |
+
extra: dict | None = None,
|
| 620 |
+
) -> None:
|
| 621 |
+
entry: dict = {
|
| 622 |
+
"ts": ts,
|
| 623 |
+
"kind": kind,
|
| 624 |
+
"from": from_value,
|
| 625 |
+
"to": to_value,
|
| 626 |
+
"reason": reason,
|
| 627 |
+
}
|
| 628 |
+
if extra:
|
| 629 |
+
entry["extra"] = extra
|
| 630 |
+
log = row.get("transition_log")
|
| 631 |
+
if not isinstance(log, list):
|
| 632 |
+
log = []
|
| 633 |
+
row["transition_log"] = log
|
| 634 |
+
log.append(entry)
|
| 635 |
+
|
| 636 |
+
|
| 637 |
+
def _pick_failover_candidate(workspaces: list[dict]) -> dict | None:
|
| 638 |
+
"""Pick best workspace to promote to active.
|
| 639 |
+
|
| 640 |
+
Order:
|
| 641 |
+
1. tier=warm AND status != unhealthy, sorted by registered_at asc (oldest first)
|
| 642 |
+
2. tier=cold AND status == healthy, sorted by registered_at asc
|
| 643 |
+
3. tier=warm regardless of status (last resort), oldest first
|
| 644 |
+
4. None
|
| 645 |
+
"""
|
| 646 |
+
warm_healthy = [w for w in workspaces if w.get("tier") == TIER_WARM and w.get("status") != STATUS_UNHEALTHY]
|
| 647 |
+
if warm_healthy:
|
| 648 |
+
warm_healthy.sort(key=lambda w: w.get("registered_at") or 0)
|
| 649 |
+
return warm_healthy[0]
|
| 650 |
+
cold_healthy = [w for w in workspaces if w.get("tier") == TIER_COLD and w.get("status") == STATUS_HEALTHY]
|
| 651 |
+
if cold_healthy:
|
| 652 |
+
cold_healthy.sort(key=lambda w: w.get("registered_at") or 0)
|
| 653 |
+
return cold_healthy[0]
|
| 654 |
+
warm_any = [w for w in workspaces if w.get("tier") == TIER_WARM]
|
| 655 |
+
if warm_any:
|
| 656 |
+
warm_any.sort(key=lambda w: w.get("registered_at") or 0)
|
| 657 |
+
return warm_any[0]
|
| 658 |
+
return None
|
| 659 |
+
|
| 660 |
+
|
| 661 |
+
# ---------------------------------------------------------------------------
|
| 662 |
+
# Module-level default pool
|
| 663 |
+
# ---------------------------------------------------------------------------
|
| 664 |
+
default_pool = WorkspacePool()
|
| 665 |
+
|
| 666 |
+
|
| 667 |
+
def get_default_pool() -> WorkspacePool:
|
| 668 |
+
return default_pool
|
| 669 |
+
|
| 670 |
+
|
| 671 |
+
__all__ = [
|
| 672 |
+
"DEFAULT_POOL_FILE",
|
| 673 |
+
"SCHEMA_VERSION",
|
| 674 |
+
"STATUS_HEALTHY",
|
| 675 |
+
"STATUS_UNHEALTHY",
|
| 676 |
+
"STATUS_UNKNOWN",
|
| 677 |
+
"TIER_ACTIVE",
|
| 678 |
+
"TIER_COLD",
|
| 679 |
+
"TIER_WARM",
|
| 680 |
+
"WorkspacePool",
|
| 681 |
+
"default_pool",
|
| 682 |
+
"get_default_pool",
|
| 683 |
+
]
|
|
@@ -0,0 +1,483 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Round 12 S7 — multi team workspace pool unit tests.
|
| 2 |
+
|
| 3 |
+
Goals:
|
| 4 |
+
- 注册 / 切换 / 故障阈值 / 自动 promote
|
| 5 |
+
- 原子写 + .bak 回滚
|
| 6 |
+
- 单 workspace 模式 backwards-compat (admin_state.json seed)
|
| 7 |
+
- master_health 失败信号 → pool failover 集成
|
| 8 |
+
- admin_state 路由 (active workspace > state.json fallback)
|
| 9 |
+
|
| 10 |
+
Mock-only — 不触达真实 OpenAI workspace API。
|
| 11 |
+
"""
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
|
| 14 |
+
import json
|
| 15 |
+
import os
|
| 16 |
+
import threading
|
| 17 |
+
from pathlib import Path
|
| 18 |
+
from unittest.mock import patch
|
| 19 |
+
|
| 20 |
+
import pytest
|
| 21 |
+
|
| 22 |
+
from autoteam.workspace_pool import (
|
| 23 |
+
SCHEMA_VERSION,
|
| 24 |
+
STATUS_HEALTHY,
|
| 25 |
+
STATUS_UNHEALTHY,
|
| 26 |
+
STATUS_UNKNOWN,
|
| 27 |
+
TIER_ACTIVE,
|
| 28 |
+
TIER_COLD,
|
| 29 |
+
TIER_WARM,
|
| 30 |
+
WorkspacePool,
|
| 31 |
+
)
|
| 32 |
+
|
| 33 |
+
UUID_A = "00000000-0000-0000-0000-00000000aaaa"
|
| 34 |
+
UUID_B = "00000000-0000-0000-0000-00000000bbbb"
|
| 35 |
+
UUID_C = "00000000-0000-0000-0000-00000000cccc"
|
| 36 |
+
|
| 37 |
+
|
| 38 |
+
@pytest.fixture
|
| 39 |
+
def pool_path(tmp_path: Path) -> Path:
|
| 40 |
+
return tmp_path / "workspaces.json"
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
@pytest.fixture(autouse=True)
|
| 44 |
+
def _isolate_admin_state(monkeypatch):
|
| 45 |
+
"""Default: seed from admin_state returns empty so tests don't pick up the
|
| 46 |
+
real on-disk state.json. Tests that exercise the seed path override this
|
| 47 |
+
explicitly via their own monkeypatch."""
|
| 48 |
+
monkeypatch.setattr(
|
| 49 |
+
"autoteam.admin_state.load_admin_state",
|
| 50 |
+
lambda: {"email": "", "account_id": "", "session_token": ""},
|
| 51 |
+
)
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
@pytest.fixture
|
| 55 |
+
def pool(pool_path: Path) -> WorkspacePool:
|
| 56 |
+
return WorkspacePool(path=pool_path, fail_threshold=3)
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
# ---------------------------------------------------------------------------
|
| 60 |
+
# 1. register / get_active basics
|
| 61 |
+
# ---------------------------------------------------------------------------
|
| 62 |
+
def test_register_first_auto_promotes_to_active(pool: WorkspacePool):
|
| 63 |
+
snap = pool.register("ws-a", "ad1@example.com", UUID_A, tier=TIER_WARM)
|
| 64 |
+
# supplied tier was warm but pool was empty → auto-promote to active
|
| 65 |
+
assert snap["tier"] == TIER_ACTIVE
|
| 66 |
+
assert pool.get_active()["id"] == "ws-a"
|
| 67 |
+
assert len(pool.list_all()) == 1
|
| 68 |
+
|
| 69 |
+
|
| 70 |
+
def test_register_second_keeps_warm(pool: WorkspacePool):
|
| 71 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 72 |
+
snap = pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 73 |
+
assert snap["tier"] == TIER_WARM
|
| 74 |
+
assert pool.get_active()["id"] == "ws-a"
|
| 75 |
+
|
| 76 |
+
|
| 77 |
+
def test_register_active_demotes_prior_active(pool: WorkspacePool):
|
| 78 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 79 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_ACTIVE)
|
| 80 |
+
# only one active at a time
|
| 81 |
+
actives = [r for r in pool.list_all() if r["tier"] == TIER_ACTIVE]
|
| 82 |
+
assert len(actives) == 1
|
| 83 |
+
assert actives[0]["id"] == "ws-b"
|
| 84 |
+
# previous active is demoted to cold
|
| 85 |
+
a = pool.get("ws-a")
|
| 86 |
+
assert a["tier"] == TIER_COLD
|
| 87 |
+
|
| 88 |
+
|
| 89 |
+
def test_register_duplicate_id_raises(pool: WorkspacePool):
|
| 90 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 91 |
+
with pytest.raises(ValueError):
|
| 92 |
+
pool.register("ws-a", "ad1b@example.com", UUID_B)
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
def test_register_invalid_tier_raises(pool: WorkspacePool):
|
| 96 |
+
with pytest.raises(ValueError):
|
| 97 |
+
pool.register("ws-a", "ad1@example.com", UUID_A, tier="hot")
|
| 98 |
+
|
| 99 |
+
|
| 100 |
+
# ---------------------------------------------------------------------------
|
| 101 |
+
# 2. set_active / list_all
|
| 102 |
+
# ---------------------------------------------------------------------------
|
| 103 |
+
def test_set_active_swaps_and_demotes(pool: WorkspacePool):
|
| 104 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 105 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 106 |
+
pool.set_active("ws-b")
|
| 107 |
+
assert pool.get_active()["id"] == "ws-b"
|
| 108 |
+
assert pool.get("ws-a")["tier"] == TIER_COLD
|
| 109 |
+
assert pool.get("ws-b")["tier"] == TIER_ACTIVE
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
def test_set_active_unknown_id_raises(pool: WorkspacePool):
|
| 113 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 114 |
+
with pytest.raises(KeyError):
|
| 115 |
+
pool.set_active("ws-zzz")
|
| 116 |
+
|
| 117 |
+
|
| 118 |
+
def test_set_active_idempotent_when_already_active(pool: WorkspacePool):
|
| 119 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 120 |
+
snap = pool.set_active("ws-a")
|
| 121 |
+
assert snap["tier"] == TIER_ACTIVE
|
| 122 |
+
|
| 123 |
+
|
| 124 |
+
# ---------------------------------------------------------------------------
|
| 125 |
+
# 3. mark_unhealthy + failover
|
| 126 |
+
# ---------------------------------------------------------------------------
|
| 127 |
+
def test_mark_unhealthy_below_threshold_no_failover(pool: WorkspacePool):
|
| 128 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 129 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 130 |
+
snap = pool.mark_unhealthy("ws-a", "subscription_cancelled")
|
| 131 |
+
# only 1 fail (threshold=3) → no failover, ws-a still active
|
| 132 |
+
assert snap["id"] == "ws-a"
|
| 133 |
+
a = pool.get("ws-a")
|
| 134 |
+
assert a["fail_count"] == 1
|
| 135 |
+
assert a["status"] == STATUS_UNHEALTHY
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
def test_mark_unhealthy_at_threshold_triggers_failover(pool: WorkspacePool):
|
| 139 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 140 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 141 |
+
pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 142 |
+
pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 143 |
+
snap = pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 144 |
+
# 3rd fail at threshold=3 → failover to ws-b
|
| 145 |
+
assert snap is not None
|
| 146 |
+
assert snap["id"] == "ws-b"
|
| 147 |
+
assert pool.get_active()["id"] == "ws-b"
|
| 148 |
+
a = pool.get("ws-a")
|
| 149 |
+
assert a["tier"] == TIER_COLD
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def test_mark_unhealthy_no_warm_no_failover_candidate(pool: WorkspacePool):
|
| 153 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 154 |
+
# only ws-a (active), no warm/cold candidate
|
| 155 |
+
pool.set_fail_threshold(1)
|
| 156 |
+
snap = pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 157 |
+
# no candidate → active becomes None
|
| 158 |
+
assert snap is None
|
| 159 |
+
assert pool.get_active() is None
|
| 160 |
+
a = pool.get("ws-a")
|
| 161 |
+
assert a["tier"] == TIER_COLD
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def test_mark_unhealthy_force_failover_immediate(pool: WorkspacePool):
|
| 165 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 166 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 167 |
+
snap = pool.mark_unhealthy("ws-a", "manual_force", force_failover=True)
|
| 168 |
+
assert snap["id"] == "ws-b"
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
def test_failover_picks_oldest_warm_first(pool: WorkspacePool):
|
| 172 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 173 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 174 |
+
pool.register("ws-c", "ad3@example.com", UUID_C, tier=TIER_WARM)
|
| 175 |
+
# Set deterministic registered_at: ws-b oldest warm, ws-c younger
|
| 176 |
+
raw = json.loads(pool.path.read_text(encoding="utf-8"))
|
| 177 |
+
for row in raw["workspaces"]:
|
| 178 |
+
if row["id"] == "ws-b":
|
| 179 |
+
row["registered_at"] = 1000.0
|
| 180 |
+
elif row["id"] == "ws-c":
|
| 181 |
+
row["registered_at"] = 2000.0
|
| 182 |
+
pool.path.write_text(json.dumps(raw, indent=2), encoding="utf-8")
|
| 183 |
+
|
| 184 |
+
pool.set_fail_threshold(1)
|
| 185 |
+
snap = pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 186 |
+
assert snap["id"] == "ws-b" # oldest warm wins
|
| 187 |
+
|
| 188 |
+
|
| 189 |
+
def test_mark_healthy_resets_fail_count(pool: WorkspacePool):
|
| 190 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 191 |
+
pool.mark_unhealthy("ws-a", "subscription_cancelled")
|
| 192 |
+
pool.mark_unhealthy("ws-a", "subscription_cancelled")
|
| 193 |
+
pool.mark_healthy("ws-a")
|
| 194 |
+
a = pool.get("ws-a")
|
| 195 |
+
assert a["fail_count"] == 0
|
| 196 |
+
assert a["status"] == STATUS_HEALTHY
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
# ---------------------------------------------------------------------------
|
| 200 |
+
# 4. transition_log
|
| 201 |
+
# ---------------------------------------------------------------------------
|
| 202 |
+
def test_transition_log_register_promote_demote(pool: WorkspacePool):
|
| 203 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 204 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 205 |
+
pool.set_active("ws-b")
|
| 206 |
+
a = pool.get("ws-a")
|
| 207 |
+
reasons_a = [e["reason"] for e in a["transition_log"]]
|
| 208 |
+
assert "register" in reasons_a
|
| 209 |
+
assert "demote_on_set_active" in reasons_a
|
| 210 |
+
b = pool.get("ws-b")
|
| 211 |
+
reasons_b = [e["reason"] for e in b["transition_log"]]
|
| 212 |
+
assert "register" in reasons_b
|
| 213 |
+
assert "set_active" in reasons_b
|
| 214 |
+
|
| 215 |
+
|
| 216 |
+
def test_transition_log_failover_writes_two_rows(pool: WorkspacePool):
|
| 217 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 218 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 219 |
+
pool.set_fail_threshold(1)
|
| 220 |
+
pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 221 |
+
a = pool.get("ws-a")
|
| 222 |
+
b = pool.get("ws-b")
|
| 223 |
+
assert any(e["reason"] == "failover_demote" for e in a["transition_log"])
|
| 224 |
+
assert any(e["reason"] == "failover_promote" for e in b["transition_log"])
|
| 225 |
+
|
| 226 |
+
|
| 227 |
+
# ---------------------------------------------------------------------------
|
| 228 |
+
# 5. atomic write + .bak rollback (I3)
|
| 229 |
+
# ---------------------------------------------------------------------------
|
| 230 |
+
def test_atomic_write_replaces_file(pool: WorkspacePool):
|
| 231 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 232 |
+
raw = json.loads(pool.path.read_text(encoding="utf-8"))
|
| 233 |
+
assert raw["schema_version"] == SCHEMA_VERSION
|
| 234 |
+
assert raw["active"] == "ws-a"
|
| 235 |
+
# no leftover .tmp / .bak
|
| 236 |
+
assert not pool.path.with_suffix(pool.path.suffix + ".tmp").exists()
|
| 237 |
+
assert not pool.path.with_suffix(pool.path.suffix + ".bak").exists()
|
| 238 |
+
|
| 239 |
+
|
| 240 |
+
def test_atomic_write_rolls_back_on_os_replace_failure(pool_path: Path):
|
| 241 |
+
pool = WorkspacePool(path=pool_path)
|
| 242 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 243 |
+
original = pool_path.read_text(encoding="utf-8")
|
| 244 |
+
|
| 245 |
+
real_replace = os.replace
|
| 246 |
+
|
| 247 |
+
def boom(src, dst):
|
| 248 |
+
if str(dst) == str(pool_path):
|
| 249 |
+
raise OSError("simulated mid-write failure")
|
| 250 |
+
return real_replace(src, dst)
|
| 251 |
+
|
| 252 |
+
with patch("autoteam.workspace_pool.os.replace", side_effect=boom):
|
| 253 |
+
with pytest.raises(OSError):
|
| 254 |
+
pool.register("ws-b", "ad2@example.com", UUID_B)
|
| 255 |
+
|
| 256 |
+
# original content preserved (rollback restored .bak)
|
| 257 |
+
assert pool_path.read_text(encoding="utf-8") == original
|
| 258 |
+
|
| 259 |
+
|
| 260 |
+
# ---------------------------------------------------------------------------
|
| 261 |
+
# 6. backwards-compat seed from admin_state.json (I4)
|
| 262 |
+
# ---------------------------------------------------------------------------
|
| 263 |
+
def test_seed_from_admin_state_when_pool_missing(monkeypatch, pool_path: Path):
|
| 264 |
+
fake_state = {
|
| 265 |
+
"email": "legacy_admin@example.com",
|
| 266 |
+
"account_id": UUID_A,
|
| 267 |
+
"session_token": "tok",
|
| 268 |
+
}
|
| 269 |
+
monkeypatch.setattr("autoteam.admin_state.load_admin_state", lambda: fake_state)
|
| 270 |
+
pool = WorkspacePool(path=pool_path)
|
| 271 |
+
active = pool.get_active()
|
| 272 |
+
assert active is not None
|
| 273 |
+
assert active["admin_email"] == "legacy_admin@example.com"
|
| 274 |
+
assert active["account_id"] == UUID_A
|
| 275 |
+
assert active["tier"] == TIER_ACTIVE
|
| 276 |
+
|
| 277 |
+
|
| 278 |
+
def test_seed_skipped_when_state_empty(monkeypatch, pool_path: Path):
|
| 279 |
+
monkeypatch.setattr("autoteam.admin_state.load_admin_state", lambda: {"email": "", "account_id": ""})
|
| 280 |
+
pool = WorkspacePool(path=pool_path)
|
| 281 |
+
assert pool.get_active() is None
|
| 282 |
+
assert pool.list_all() == []
|
| 283 |
+
|
| 284 |
+
|
| 285 |
+
# ---------------------------------------------------------------------------
|
| 286 |
+
# 7. admin_state routing
|
| 287 |
+
# ---------------------------------------------------------------------------
|
| 288 |
+
def test_admin_state_get_admin_email_routes_to_pool(monkeypatch, pool_path: Path):
|
| 289 |
+
pool = WorkspacePool(path=pool_path)
|
| 290 |
+
pool.register("ws-x", "pool_admin@example.com", UUID_A)
|
| 291 |
+
monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
|
| 292 |
+
# state.json should NOT be consulted when pool active
|
| 293 |
+
monkeypatch.setattr("autoteam.admin_state.load_admin_state",
|
| 294 |
+
lambda: {"email": "stale@example.com", "account_id": ""})
|
| 295 |
+
from autoteam.admin_state import get_admin_email, get_chatgpt_account_id
|
| 296 |
+
assert get_admin_email() == "pool_admin@example.com"
|
| 297 |
+
assert get_chatgpt_account_id() == UUID_A
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
def test_admin_state_falls_back_to_state_json_when_pool_empty(monkeypatch, pool_path: Path):
|
| 301 |
+
pool = WorkspacePool(path=pool_path)
|
| 302 |
+
# empty pool (no register, and state seed returns empty)
|
| 303 |
+
monkeypatch.setattr("autoteam.admin_state.load_admin_state",
|
| 304 |
+
lambda: {"email": "fallback@example.com", "account_id": UUID_B})
|
| 305 |
+
monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
|
| 306 |
+
# also: pool.get_active will seed from state, so make seed return empty
|
| 307 |
+
monkeypatch.setattr(
|
| 308 |
+
WorkspacePool, "_seed_from_admin_state",
|
| 309 |
+
lambda self: {"schema_version": SCHEMA_VERSION, "active": None, "workspaces": []},
|
| 310 |
+
)
|
| 311 |
+
from autoteam.admin_state import get_admin_email, get_chatgpt_account_id
|
| 312 |
+
assert get_admin_email() == "fallback@example.com"
|
| 313 |
+
assert get_chatgpt_account_id() == UUID_B
|
| 314 |
+
|
| 315 |
+
|
| 316 |
+
# ---------------------------------------------------------------------------
|
| 317 |
+
# 8. master_health pool integration
|
| 318 |
+
# ---------------------------------------------------------------------------
|
| 319 |
+
def test_apply_pool_health_signal_healthy_marks_healthy(monkeypatch, pool_path: Path):
|
| 320 |
+
pool = WorkspacePool(path=pool_path)
|
| 321 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 322 |
+
pool.mark_unhealthy("ws-a", "subscription_cancelled") # fail_count=1
|
| 323 |
+
from autoteam.master_health import apply_pool_health_signal
|
| 324 |
+
snap = apply_pool_health_signal(True, "active", pool=pool)
|
| 325 |
+
assert snap is not None
|
| 326 |
+
a = pool.get("ws-a")
|
| 327 |
+
assert a["fail_count"] == 0
|
| 328 |
+
assert a["status"] == STATUS_HEALTHY
|
| 329 |
+
|
| 330 |
+
|
| 331 |
+
def test_apply_pool_health_signal_subscription_cancelled_drives_failover(pool_path: Path):
|
| 332 |
+
pool = WorkspacePool(path=pool_path, fail_threshold=2)
|
| 333 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 334 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 335 |
+
from autoteam.master_health import apply_pool_health_signal
|
| 336 |
+
apply_pool_health_signal(False, "subscription_cancelled", pool=pool)
|
| 337 |
+
snap = apply_pool_health_signal(False, "subscription_cancelled", pool=pool)
|
| 338 |
+
# threshold=2 reached → failover to ws-b
|
| 339 |
+
assert snap is not None
|
| 340 |
+
assert snap["id"] == "ws-b"
|
| 341 |
+
|
| 342 |
+
|
| 343 |
+
def test_apply_pool_health_signal_network_error_no_failover(pool_path: Path):
|
| 344 |
+
pool = WorkspacePool(path=pool_path, fail_threshold=1)
|
| 345 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 346 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 347 |
+
from autoteam.master_health import apply_pool_health_signal
|
| 348 |
+
snap = apply_pool_health_signal(False, "network_error", pool=pool)
|
| 349 |
+
# network_error is transient — don't trip
|
| 350 |
+
assert snap["id"] == "ws-a"
|
| 351 |
+
a = pool.get("ws-a")
|
| 352 |
+
assert a["fail_count"] == 0 # untouched
|
| 353 |
+
|
| 354 |
+
|
| 355 |
+
def test_apply_pool_health_signal_empty_pool_returns_none(pool_path: Path):
|
| 356 |
+
pool = WorkspacePool(path=pool_path)
|
| 357 |
+
from autoteam.master_health import apply_pool_health_signal
|
| 358 |
+
assert apply_pool_health_signal(False, "auth_invalid", pool=pool) is None
|
| 359 |
+
|
| 360 |
+
|
| 361 |
+
# ---------------------------------------------------------------------------
|
| 362 |
+
# 9. event subscribers
|
| 363 |
+
# ---------------------------------------------------------------------------
|
| 364 |
+
def test_subscriber_receives_register_event(pool: WorkspacePool):
|
| 365 |
+
events: list[dict] = []
|
| 366 |
+
pool.subscribe(events.append)
|
| 367 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 368 |
+
types = [e["type"] for e in events]
|
| 369 |
+
assert "registered" in types
|
| 370 |
+
|
| 371 |
+
|
| 372 |
+
def test_subscriber_receives_failover_promote_event(pool: WorkspacePool):
|
| 373 |
+
events: list[dict] = []
|
| 374 |
+
pool.subscribe(events.append)
|
| 375 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 376 |
+
pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
|
| 377 |
+
pool.set_fail_threshold(1)
|
| 378 |
+
pool.mark_unhealthy("ws-a", "auth_invalid")
|
| 379 |
+
types = [e["type"] for e in events]
|
| 380 |
+
assert "promoted" in types
|
| 381 |
+
assert "marked_unhealthy" in types
|
| 382 |
+
|
| 383 |
+
|
| 384 |
+
def test_subscriber_exceptions_swallowed(pool: WorkspacePool):
|
| 385 |
+
def bad(_e):
|
| 386 |
+
raise RuntimeError("boom")
|
| 387 |
+
pool.subscribe(bad)
|
| 388 |
+
# must not raise
|
| 389 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 390 |
+
|
| 391 |
+
|
| 392 |
+
# ---------------------------------------------------------------------------
|
| 393 |
+
# 10. concurrency safety (lock)
|
| 394 |
+
# ---------------------------------------------------------------------------
|
| 395 |
+
def test_concurrent_registers_no_duplicate_active(pool: WorkspacePool):
|
| 396 |
+
barrier = threading.Barrier(5)
|
| 397 |
+
errors: list[Exception] = []
|
| 398 |
+
|
| 399 |
+
def worker(i):
|
| 400 |
+
try:
|
| 401 |
+
barrier.wait(timeout=2)
|
| 402 |
+
pool.register(f"ws-{i}", f"ad{i}@example.com", f"00000000-0000-0000-0000-{i:012d}")
|
| 403 |
+
except Exception as exc:
|
| 404 |
+
errors.append(exc)
|
| 405 |
+
|
| 406 |
+
threads = [threading.Thread(target=worker, args=(i,)) for i in range(5)]
|
| 407 |
+
for t in threads:
|
| 408 |
+
t.start()
|
| 409 |
+
for t in threads:
|
| 410 |
+
t.join()
|
| 411 |
+
assert errors == []
|
| 412 |
+
actives = [r for r in pool.list_all() if r["tier"] == TIER_ACTIVE]
|
| 413 |
+
assert len(actives) == 1 # I1 — exactly one active
|
| 414 |
+
|
| 415 |
+
|
| 416 |
+
# ---------------------------------------------------------------------------
|
| 417 |
+
# 11. schema migration / corrupt-file resilience
|
| 418 |
+
# ---------------------------------------------------------------------------
|
| 419 |
+
def test_corrupt_json_treated_as_empty(pool_path: Path):
|
| 420 |
+
pool_path.write_text("{not valid json", encoding="utf-8")
|
| 421 |
+
pool = WorkspacePool(path=pool_path)
|
| 422 |
+
assert pool.list_all() == []
|
| 423 |
+
# subsequent register works (file is overwritten cleanly)
|
| 424 |
+
pool.register("ws-a", "ad1@example.com", UUID_A)
|
| 425 |
+
assert pool.get_active()["id"] == "ws-a"
|
| 426 |
+
|
| 427 |
+
|
| 428 |
+
def test_unknown_schema_version_discarded(pool_path: Path):
|
| 429 |
+
pool_path.write_text(
|
| 430 |
+
json.dumps({"schema_version": 999, "active": "ws-x", "workspaces": []}),
|
| 431 |
+
encoding="utf-8",
|
| 432 |
+
)
|
| 433 |
+
pool = WorkspacePool(path=pool_path)
|
| 434 |
+
assert pool.get_active() is None
|
| 435 |
+
|
| 436 |
+
|
| 437 |
+
def test_normalize_drops_invalid_tier(pool_path: Path):
|
| 438 |
+
pool_path.write_text(
|
| 439 |
+
json.dumps({
|
| 440 |
+
"schema_version": SCHEMA_VERSION,
|
| 441 |
+
"active": "ws-x",
|
| 442 |
+
"workspaces": [{
|
| 443 |
+
"id": "ws-x",
|
| 444 |
+
"admin_email": "a@e.com",
|
| 445 |
+
"account_id": UUID_A,
|
| 446 |
+
"tier": "hot", # invalid
|
| 447 |
+
"status": "weird", # invalid
|
| 448 |
+
"fail_count": "not-an-int",
|
| 449 |
+
"transition_log": "not-a-list",
|
| 450 |
+
}],
|
| 451 |
+
}),
|
| 452 |
+
encoding="utf-8",
|
| 453 |
+
)
|
| 454 |
+
pool = WorkspacePool(path=pool_path)
|
| 455 |
+
rows = pool.list_all()
|
| 456 |
+
assert len(rows) == 1
|
| 457 |
+
assert rows[0]["tier"] == TIER_WARM
|
| 458 |
+
assert rows[0]["status"] == STATUS_UNKNOWN
|
| 459 |
+
assert rows[0]["fail_count"] == 0
|
| 460 |
+
assert rows[0]["transition_log"] == []
|
| 461 |
+
|
| 462 |
+
|
| 463 |
+
# ---------------------------------------------------------------------------
|
| 464 |
+
# 12. cpa_sync helper
|
| 465 |
+
# ---------------------------------------------------------------------------
|
| 466 |
+
def test_cpa_sync_get_active_sync_target_returns_pool_summary(monkeypatch, pool_path: Path):
|
| 467 |
+
pool = WorkspacePool(path=pool_path)
|
| 468 |
+
pool.register("ws-a", "cpa_admin@example.com", UUID_A)
|
| 469 |
+
monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
|
| 470 |
+
from autoteam.cpa_sync import get_active_sync_target
|
| 471 |
+
target = get_active_sync_target()
|
| 472 |
+
assert target == {"id": "ws-a", "admin_email": "cpa_admin@example.com", "account_id": UUID_A}
|
| 473 |
+
|
| 474 |
+
|
| 475 |
+
def test_cpa_sync_get_active_sync_target_none_when_empty(monkeypatch, pool_path: Path):
|
| 476 |
+
pool = WorkspacePool(path=pool_path)
|
| 477 |
+
monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
|
| 478 |
+
monkeypatch.setattr(
|
| 479 |
+
WorkspacePool, "_seed_from_admin_state",
|
| 480 |
+
lambda self: {"schema_version": SCHEMA_VERSION, "active": None, "workspaces": []},
|
| 481 |
+
)
|
| 482 |
+
from autoteam.cpa_sync import get_active_sync_target
|
| 483 |
+
assert get_active_sync_target() is None
|