ZRainbow Claude Opus 4.7 (1M context) commited on
Commit
60afec4
·
1 Parent(s): 8577b73

feat(round-12 S7): multi team workspace pool + auto failover

Browse files

新增 src/autoteam/workspace_pool.py — WorkspacePool 持久化 K 个 team workspace
互为冷备(active=1, warm/cold N-1)。主 workspace 连续探测失败 ≥ MASTER_HEALTH_FAIL_THRESHOLD
(默认 3)秒切到 warm。register / set_active / mark_unhealthy / mark_healthy 全部
追加 transition_log,与 S1 default_machine 风格一致(原子写 + .bak 回滚 + lock-外
事件总线)。

admin_state.get_admin_email / get_chatgpt_account_id 改为路由 active workspace
的 admin_email / account_id;pool 空时回退原 state.json 读路径,单 workspace 模
式向后兼容。master_health 新增 apply_pool_health_signal helper:probe healthy
→ mark_healthy 重置 fail_count;subscription_cancelled / auth_invalid /
workspace_missing / role_not_owner → mark_unhealthy 累计触发故障切换;
network_error 视为瞬时不计入。cpa_sync 新增 get_active_sync_target helper +
sync_main_codex_to_cpa 上报当前 active workspace identity。

测试 35 case 覆盖:基础注册/激活/idempotent/重复 id 拒绝、set_active 切换/降级、
mark_unhealthy 阈值前不切换 + 阈值触发 failover + 无候选 → active=None +
force_failover、failover 候选选最旧 warm、mark_healthy 重置、transition_log
完整、原子写 .bak 回滚(模拟 os.replace 失败)、admin_state 路由 + 兜底、
master_health 集成 4 路径、subscriber 异常吞掉、5-thread 并发注册仅 1 active、
schema 不兼容 + JSON 损坏 + 字段非法值 normalize、cpa_sync get_active_sync_target。

向后兼容:workspaces.json 不存在时从 state.json (admin_email + UUID account_id)
自动 seed 为单一 active workspace;现有调用方零改动。

DoD:
- ruff check src/ tests/ 全绿
- pytest 699 passed (664 baseline + 35 新增,无退化)
- 所有改动文件已提交,严格 git add 文件名,未用 -A

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.trellis/tasks/05-11-s7-multi-workspace-pool/check.jsonl ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {"file": ".trellis/tasks/05-11-s7-multi-workspace-pool/prd.md", "reason": "DoD + 不变量 I1~I7 验收清单"}
2
+ {"file": "src/autoteam/workspace_pool.py", "reason": "新增核心模块 — 验证 register / set_active / mark_unhealthy / 自动 failover / 原子写 / 向后兼容 seed"}
3
+ {"file": "src/autoteam/admin_state.py", "reason": "路由改造 — 验证 pool 为空时回退 state.json,签名不变"}
4
+ {"file": "src/autoteam/master_health.py", "reason": "失败阈值累计 + 调 pool.mark_unhealthy 触发切换"}
5
+ {"file": "tests/unit/test_round12_s7_workspace_pool.py", "reason": "≥ 15 case:注册/切换/故障阈值/原子写回滚/向后兼容/边界"}
.trellis/tasks/05-11-s7-multi-workspace-pool/implement.jsonl ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {"file": ".trellis/tasks/05-11-upstream-align-register-multimail-frontend-refresh/prd.md", "reason": "父 PRD — Q2 创新方向 5 多 workspace 池化 scope + 子任务依赖图 + round-12 波次范围"}
2
+ {"file": "src/autoteam/admin_state.py", "reason": "改造目标 1:get_admin_email / get_chatgpt_account_id 路由到 WorkspacePool.get_active(),向后兼容回退 state.json"}
3
+ {"file": "src/autoteam/master_health.py", "reason": "改造目标 2:失败路径(subscription_cancelled / auth_invalid / network_error)调 pool.mark_unhealthy 触发故障切换"}
4
+ {"file": "src/autoteam/cpa_sync.py", "reason": "改造目标 3:sync_to_cpa / sync_main_codex_to_cpa 同步目标跟随 active workspace(感知失败切换后立即新 workspace 推送)"}
5
+ {"file": "src/autoteam/account_state.py", "reason": "S1 状态机参考实现 — atomic 写盘 + .bak 回滚 + subscribe 事件总线 + IllegalTransitionError 风格对齐"}
.trellis/tasks/05-11-s7-multi-workspace-pool/prd.md ADDED
@@ -0,0 +1,123 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # S7 — 多 team workspace 池化(冷备 + 秒级故障切换)
2
+
3
+ > **Round 12 子任务**,父任务 `05-11-upstream-align-register-multimail-frontend-refresh` 中
4
+ > 「Q2 创新方向 5 — 多 team workspace 池化」的实施单元。
5
+
6
+ ## 1. 背景
7
+
8
+ 当前 AutoTeam 仅维持一个 active team workspace(admin_state.json 单 admin email + account_id)。
9
+ 单 workspace 故障(被封 / 订阅 cancel / Cloudflare 全域风控)整套自动化即停摆。
10
+
11
+ S7 引入 workspace 池:注册 K 个 workspace(active=1 + warm/cold N-1 互为冷备),主 workspace
12
+ 连续探测失败超阈值时秒切到下一个 warm,cold 升 warm,整体不停摆。
13
+
14
+ ## 2. 范围
15
+
16
+ ### MVP(本任务)
17
+
18
+ 1. 新建 `src/autoteam/workspace_pool.py`(持久化 `workspaces.json` + `WorkspacePool`)
19
+ 2. `admin_state.get_admin_email/get_chatgpt_account_id` 路由到 `WorkspacePool.get_active()`
20
+ 3. `master_health` 失败路径调 `pool.mark_unhealthy()` 触发自动切换(阈值 env `MASTER_HEALTH_FAIL_THRESHOLD=3`)
21
+ 4. `cpa_sync` 同步目标改为 active workspace
22
+ 5. 测试 ≥ 15 case(mock 故障注入;禁止真实 OpenAI 调用)
23
+
24
+ ### 非范围
25
+
26
+ - 不实现真实 OpenAI workspace 创建 API(user 自建 workspace 后手动 register)
27
+ - 不动 mail provider / account_state / manager.py 主流程
28
+ - 不做 active workspace 内 accounts 数据隔离(accounts.json 仍是全局视图)
29
+ - 不动 web 前端(后续基于 SSE 流展示)
30
+
31
+ ## 3. 数据 schema
32
+
33
+ ### `workspaces.json`
34
+
35
+ ```json
36
+ {
37
+ "schema_version": 1,
38
+ "active": "ws-aaa",
39
+ "workspaces": [
40
+ {
41
+ "id": "ws-aaa",
42
+ "admin_email": "ad1@example.com",
43
+ "account_id": "00000000-0000-0000-0000-000000000001",
44
+ "tier": "active",
45
+ "status": "healthy",
46
+ "fail_count": 0,
47
+ "last_check_ts": 1715300000.0,
48
+ "registered_at": 1715200000.0,
49
+ "transition_log": [
50
+ {"ts": 1715200000.0, "from": null, "to": "warm", "reason": "register"},
51
+ {"ts": 1715200100.0, "from": "warm", "to": "active", "reason": "promoted_no_active"}
52
+ ]
53
+ }
54
+ ]
55
+ }
56
+ ```
57
+
58
+ ### tier / status 取值
59
+
60
+ - **tier**: `active` | `warm` | `cold` (任一时刻 `active` 至多 1 个)
61
+ - **status**: `healthy` | `unhealthy` | `unknown`
62
+
63
+ ## 4. 不变量(强制执行)
64
+
65
+ - **I1**: 任意时刻 workspaces[*].tier == "active" 至多 1 个
66
+ - **I2**: `mark_unhealthy` 命中 active 且 fail_count ≥ threshold → 自动 promote 一个 warm/cold 至 active;原 active 降为 cold + status=unhealthy
67
+ - **I3**: 写盘原子(snapshot `.bak` → write `.tmp` → `os.replace`),失败回滚
68
+ - **I4**: 单 workspace 模式向后兼容:`workspaces.json` 不存在时,从 `state.json` 读 admin_email/account_id 自动 seed 为单一 active workspace;外部 API(`get_admin_email` 等)签名不变
69
+ - **I5**: `register / set_active / mark_unhealthy` 全部追加 `transition_log` 条目
70
+ - **I6**: 与 S1 default_machine 风格一致(lock 内修改、lock 外发布事件)
71
+ - **I7**: 永不抛未捕获异常(与 master_health.py M-I1 看齐)
72
+
73
+ ## 5. 关键 API
74
+
75
+ ```python
76
+ class WorkspacePool:
77
+ def register(self, workspace_id: str, admin_email: str, account_id: str, tier: str = "warm") -> dict
78
+ def get_active(self) -> dict | None # 当前 active 单条;可能 None
79
+ def set_active(self, workspace_id: str) -> dict
80
+ def mark_unhealthy(self, workspace_id: str, reason: str) -> dict | None
81
+ def list_all(self) -> list[dict]
82
+ def get(self, workspace_id: str) -> dict | None
83
+ ```
84
+
85
+ 模块级单例 `default_pool: WorkspacePool` 用于生产路径。
86
+
87
+ ## 6. 故障切换流程
88
+
89
+ ```
90
+ master_health probe FAIL
91
+
92
+
93
+ default_pool.mark_unhealthy(active_ws_id, reason)
94
+ │ fail_count += 1
95
+
96
+ fail_count >= MASTER_HEALTH_FAIL_THRESHOLD ?
97
+ ├── No → 仅累计,保持 active
98
+ └── Yes → 1) 当前 active.tier := cold, status := unhealthy
99
+ 2) 选 warm(最早注册)→ tier := active, status := unknown
100
+ 3) 若无 warm → 选 cold healthy → tier := active
101
+ 4) 若无可升者 → active = None,记 transition_log {reason: no_failover_candidate}
102
+ 5) 写盘 + 转移日志
103
+ ```
104
+
105
+ 切换写 `state_log.jsonl`(与 default_machine.subscribe 复用渠道,前端 SSE 流即可感知)。
106
+
107
+ ## 7. 验收(DoD)
108
+
109
+ - ruff check 全绿
110
+ - pytest 不退化 + 新增 ≥ 15 case 全绿
111
+ - commit `feat(round-12 S7): multi team workspace pool + auto failover`
112
+ - 改动文件:
113
+ - 新增 `src/autoteam/workspace_pool.py`
114
+ - 新增 `tests/unit/test_round12_s7_workspace_pool.py`
115
+ - 改 `src/autoteam/admin_state.py`(路由)
116
+ - 改 `src/autoteam/master_health.py`(失败路径调 pool)
117
+ - 改 `src/autoteam/cpa_sync.py`(active workspace 切换感知)
118
+
119
+ ## 8. 风险
120
+
121
+ - **R1**: 现有调用方 `get_admin_email/get_chatgpt_account_id` 极多,若 workspace_pool 退路不严密会破坏所有路径 → 强制兜底(pool 空时回退 state.json)
122
+ - **R2**: master_health 是热路径,增加 pool I/O 不能阻塞 → 写盘异步友好(已在 lock 内但写盘耗时 ms 级,可接受)
123
+ - **R3**: 用户当前无 team 母号子号,无法 e2e;MVP 只做单测 + mock 故障注入
.trellis/tasks/05-11-s7-multi-workspace-pool/task.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "id": "s7-multi-workspace-pool",
3
+ "name": "s7-multi-workspace-pool",
4
+ "title": "S7: 多 team workspace 池化",
5
+ "description": "",
6
+ "status": "in_progress",
7
+ "dev_type": null,
8
+ "scope": null,
9
+ "package": null,
10
+ "priority": "P2",
11
+ "creator": "ZRainbow",
12
+ "assignee": "ZRainbow",
13
+ "createdAt": "2026-05-11",
14
+ "completedAt": null,
15
+ "branch": null,
16
+ "base_branch": "main",
17
+ "worktree_path": null,
18
+ "commit": null,
19
+ "pr_url": null,
20
+ "subtasks": [],
21
+ "children": [],
22
+ "parent": "05-11-upstream-align-register-multimail-frontend-refresh",
23
+ "relatedFiles": [],
24
+ "notes": "",
25
+ "meta": {}
26
+ }
src/autoteam/admin_state.py CHANGED
@@ -111,6 +111,19 @@ def clear_admin_state():
111
 
112
 
113
  def get_admin_email():
 
 
 
 
 
 
 
 
 
 
 
 
 
114
  return load_admin_state().get("email", "")
115
 
116
 
@@ -126,6 +139,21 @@ def _is_valid_uuid(value: str) -> bool:
126
 
127
 
128
  def get_chatgpt_account_id():
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
129
  state = load_admin_state()
130
  state_id = state.get("account_id", "")
131
  # state.json 里的值必须是 UUID 格式才有效(user-xxx 是 user ID 不是 account ID)
 
111
 
112
 
113
  def get_admin_email():
114
+ """Round 12 S7 — route through WorkspacePool active workspace.
115
+
116
+ Backwards-compat: if the pool is empty / module unavailable, fall back
117
+ to the legacy state.json read so single-workspace installs are unaffected.
118
+ """
119
+ try:
120
+ from autoteam.workspace_pool import default_pool
121
+
122
+ active = default_pool.get_active()
123
+ if active and active.get("admin_email"):
124
+ return active["admin_email"]
125
+ except Exception:
126
+ pass
127
  return load_admin_state().get("email", "")
128
 
129
 
 
139
 
140
 
141
  def get_chatgpt_account_id():
142
+ """Round 12 S7 — route through WorkspacePool active workspace.
143
+
144
+ Backwards-compat: pool empty → legacy state.json read → CHATGPT_ACCOUNT_ID env.
145
+ Pool ID still must pass UUID validation (filters legacy `user-xxx` ids).
146
+ """
147
+ try:
148
+ from autoteam.workspace_pool import default_pool
149
+
150
+ active = default_pool.get_active()
151
+ if active:
152
+ pool_aid = (active.get("account_id") or "").strip()
153
+ if pool_aid and _is_valid_uuid(pool_aid):
154
+ return pool_aid
155
+ except Exception:
156
+ pass
157
  state = load_admin_state()
158
  state_id = state.get("account_id", "")
159
  # state.json 里的值必须是 UUID 格式才有效(user-xxx 是 user ID 不是 account ID)
src/autoteam/cpa_sync.py CHANGED
@@ -653,5 +653,45 @@ def sync_main_codex_to_cpa(filepath):
653
  if not upload_to_cpa(filepath):
654
  raise RuntimeError(f"上传主号认证文件失败: {name}")
655
 
656
- logger.info("[CPA] 主号 Codex 同步: %s", name)
 
 
 
 
 
 
 
 
 
 
 
657
  return {"uploaded": name}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
653
  if not upload_to_cpa(filepath):
654
  raise RuntimeError(f"上传主号认证文件失败: {name}")
655
 
656
+ # Round 12 S7 — 主号同步成功后,记录当前 active workspace,便于上游观测
657
+ try:
658
+ active = _get_active_workspace_summary()
659
+ if active:
660
+ logger.info(
661
+ "[CPA] 主号 Codex 已同步 (active workspace: id=%s admin=%s account_id=%s)",
662
+ active.get("id"), active.get("admin_email"), active.get("account_id"),
663
+ )
664
+ else:
665
+ logger.info("[CPA] 主号 Codex 已同步: %s", name)
666
+ except Exception:
667
+ logger.info("[CPA] 主号 Codex 已同步: %s", name)
668
  return {"uploaded": name}
669
+
670
+
671
+ def _get_active_workspace_summary():
672
+ """Round 12 S7 — return the current active workspace (id/admin/account_id) or None.
673
+
674
+ Best-effort: never raises; pool unavailable returns None.
675
+ """
676
+ try:
677
+ from autoteam.workspace_pool import default_pool
678
+
679
+ active = default_pool.get_active()
680
+ if not active:
681
+ return None
682
+ return {
683
+ "id": active.get("id"),
684
+ "admin_email": active.get("admin_email"),
685
+ "account_id": active.get("account_id"),
686
+ }
687
+ except Exception:
688
+ return None
689
+
690
+
691
+ def get_active_sync_target():
692
+ """Round 12 S7 — public helper: which workspace identity should CPA sync to?
693
+
694
+ Returns dict({id, admin_email, account_id}) or None when single-workspace
695
+ mode (caller falls back to legacy admin_state).
696
+ """
697
+ return _get_active_workspace_summary()
src/autoteam/master_health.py CHANGED
@@ -511,6 +511,62 @@ def is_master_subscription_healthy(
511
  return healthy, reason, evidence
512
 
513
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
514
  # ---------------------------------------------------------------------------
515
  # Round 9 SPEC v1.1 §11~§12 — Retroactive helper + grace 期 JWT 解析
516
  # ---------------------------------------------------------------------------
 
511
  return healthy, reason, evidence
512
 
513
 
514
+ # ---------------------------------------------------------------------------
515
+ # Round 12 S7 — WorkspacePool failover hook
516
+ # ---------------------------------------------------------------------------
517
+
518
+ # reasons that should NOT trip workspace failover (transient / classification noise)
519
+ _TRANSIENT_FAIL_REASONS = ("network_error",)
520
+ # reasons that SHOULD trip failover (real master-side failure)
521
+ _HARD_FAIL_REASONS = ("auth_invalid", "subscription_cancelled", "workspace_missing", "role_not_owner")
522
+
523
+
524
+ def apply_pool_health_signal(
525
+ healthy: bool,
526
+ reason: str,
527
+ evidence: dict | None = None,
528
+ *,
529
+ pool=None,
530
+ ) -> dict | None:
531
+ """Round 12 S7 — feed a master_health probe result into the WorkspacePool.
532
+
533
+ Call this from any caller of :func:`is_master_subscription_healthy` to
534
+ drive auto failover. Behavior:
535
+
536
+ - healthy=True (active or subscription_grace) → pool.mark_healthy(active_id)
537
+ - healthy=False AND reason in _HARD_FAIL_REASONS → pool.mark_unhealthy(...)
538
+ which accumulates fail_count + (when ≥ threshold) auto-promotes a
539
+ warm/cold candidate.
540
+ - healthy=False AND reason in _TRANSIENT_FAIL_REASONS → no-op (don't
541
+ flap on intermittent network glitches).
542
+
543
+ Returns the **post-call** active workspace snapshot (None if pool empty
544
+ or failed). Never raises — M-I1 alignment.
545
+
546
+ ``pool`` defaults to ``workspace_pool.default_pool``; tests can inject
547
+ their own.
548
+ """
549
+ try:
550
+ if pool is None:
551
+ from autoteam.workspace_pool import default_pool as pool_obj
552
+ pool = pool_obj
553
+ active = pool.get_active()
554
+ if active is None:
555
+ return None
556
+ ws_id = active.get("id")
557
+ if not ws_id:
558
+ return None
559
+ if healthy and reason in ("active", "subscription_grace"):
560
+ return pool.mark_healthy(ws_id)
561
+ if (not healthy) and reason in _HARD_FAIL_REASONS:
562
+ return pool.mark_unhealthy(ws_id, reason)
563
+ # transient or unknown → leave state alone but bubble snapshot
564
+ return active
565
+ except Exception as exc:
566
+ logger.warning("[master_health] apply_pool_health_signal failed: %s", exc)
567
+ return None
568
+
569
+
570
  # ---------------------------------------------------------------------------
571
  # Round 9 SPEC v1.1 §11~§12 — Retroactive helper + grace 期 JWT 解析
572
  # ---------------------------------------------------------------------------
src/autoteam/workspace_pool.py ADDED
@@ -0,0 +1,683 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Multi team workspace pool — Round 12 S7。
2
+
3
+ 维持 K 个 team workspace 互为冷备:任意时刻仅 1 个 active(主热路径用),
4
+ 其余作 warm/cold 备份;主 workspace 连续探测失败超阈值时秒切。
5
+
6
+ 不变量(I1~I7,详见 .trellis/tasks/05-11-s7-multi-workspace-pool/prd.md):
7
+
8
+ - I1: workspaces[*].tier == "active" 至多 1 个
9
+ - I2: mark_unhealthy 命中 active 且 fail_count >= threshold → 自动 promote warm/cold
10
+ - I3: 写盘原子(snapshot .bak → tmp → os.replace),失败回滚
11
+ - I4: 单 workspace 模式向后兼容 — workspaces.json 不存在时 seed 自 state.json
12
+ - I5: register / set_active / mark_unhealthy / promote 均追加 transition_log
13
+ - I6: 与 default_machine 风格一致(lock 内修改、lock 外发布事件)
14
+ - I7: 永不抛未捕获异常(与 master_health.py M-I1 看齐)
15
+ """
16
+ from __future__ import annotations
17
+
18
+ import json
19
+ import logging
20
+ import os
21
+ import shutil
22
+ import threading
23
+ import time
24
+ from collections.abc import Callable
25
+ from pathlib import Path
26
+ from typing import Any
27
+
28
+ logger = logging.getLogger(__name__)
29
+
30
+ PROJECT_ROOT = Path(__file__).parent.parent.parent
31
+ DEFAULT_POOL_FILE = PROJECT_ROOT / "workspaces.json"
32
+ POOL_FILE_MODE = 0o666
33
+
34
+ SCHEMA_VERSION = 1
35
+
36
+ TIER_ACTIVE = "active"
37
+ TIER_WARM = "warm"
38
+ TIER_COLD = "cold"
39
+ _VALID_TIERS = (TIER_ACTIVE, TIER_WARM, TIER_COLD)
40
+
41
+ STATUS_HEALTHY = "healthy"
42
+ STATUS_UNHEALTHY = "unhealthy"
43
+ STATUS_UNKNOWN = "unknown"
44
+ _VALID_STATUSES = (STATUS_HEALTHY, STATUS_UNHEALTHY, STATUS_UNKNOWN)
45
+
46
+
47
+ def _default_fail_threshold() -> int:
48
+ raw = os.environ.get("MASTER_HEALTH_FAIL_THRESHOLD", "3")
49
+ try:
50
+ v = int(raw)
51
+ return v if v > 0 else 3
52
+ except Exception:
53
+ return 3
54
+
55
+
56
+ PoolSubscriber = Callable[[dict], None]
57
+
58
+
59
+ class WorkspacePool:
60
+ """Persistent multi-workspace pool with auto failover.
61
+
62
+ The on-disk schema is documented in prd.md §3. The instance owns a
63
+ threading lock — all reads / writes serialize on it, mirroring
64
+ :class:`autoteam.account_state.StateMachine`.
65
+ """
66
+
67
+ def __init__(
68
+ self,
69
+ path: Path | None = None,
70
+ *,
71
+ fail_threshold: int | None = None,
72
+ ) -> None:
73
+ self._path: Path = Path(path) if path else DEFAULT_POOL_FILE
74
+ self._lock = threading.Lock()
75
+ self._subscribers: list[PoolSubscriber] = []
76
+ self._fail_threshold = fail_threshold if fail_threshold is not None else _default_fail_threshold()
77
+
78
+ # ---------------------------------------------------------------- config
79
+ @property
80
+ def path(self) -> Path:
81
+ return self._path
82
+
83
+ @property
84
+ def fail_threshold(self) -> int:
85
+ return self._fail_threshold
86
+
87
+ def set_fail_threshold(self, threshold: int) -> None:
88
+ if not isinstance(threshold, int) or threshold <= 0:
89
+ raise ValueError("fail_threshold must be a positive int")
90
+ self._fail_threshold = threshold
91
+
92
+ # ----------------------------------------------------------- event bus
93
+ def subscribe(self, callback: PoolSubscriber) -> PoolSubscriber:
94
+ if not callable(callback):
95
+ raise TypeError("subscribe expects a callable")
96
+ with self._lock:
97
+ if callback not in self._subscribers:
98
+ self._subscribers.append(callback)
99
+ return callback
100
+
101
+ def unsubscribe(self, callback: PoolSubscriber) -> bool:
102
+ with self._lock:
103
+ try:
104
+ self._subscribers.remove(callback)
105
+ return True
106
+ except ValueError:
107
+ return False
108
+
109
+ def _publish(self, event: dict) -> None:
110
+ with self._lock:
111
+ subs = list(self._subscribers)
112
+ for cb in subs:
113
+ try:
114
+ cb(event)
115
+ except Exception:
116
+ logger.exception("workspace_pool subscriber %r raised on event %s", cb, event)
117
+
118
+ # ---------------------------------------------------------------- I/O
119
+ def _load_raw(self) -> dict:
120
+ """Load on-disk pool. Auto-seed from state.json on missing file (I4)."""
121
+ if not self._path.exists():
122
+ return self._seed_from_admin_state()
123
+ try:
124
+ text = self._path.read_text(encoding="utf-8").strip()
125
+ except Exception as exc:
126
+ logger.warning("[workspace_pool] read failed: %s — treating as empty", exc)
127
+ return _empty_doc()
128
+ if not text:
129
+ return _empty_doc()
130
+ try:
131
+ data = json.loads(text)
132
+ except Exception as exc:
133
+ logger.warning("[workspace_pool] json parse failed: %s — treating as empty", exc)
134
+ return _empty_doc()
135
+ if not isinstance(data, dict):
136
+ return _empty_doc()
137
+ if data.get("schema_version") != SCHEMA_VERSION:
138
+ # forward-incompatible / legacy → discard, will re-seed on next register
139
+ logger.warning(
140
+ "[workspace_pool] schema_version mismatch (got %r, expect %d) — discard",
141
+ data.get("schema_version"),
142
+ SCHEMA_VERSION,
143
+ )
144
+ return _empty_doc()
145
+ ws = data.get("workspaces")
146
+ if not isinstance(ws, list):
147
+ ws = []
148
+ # normalize each row
149
+ norm_ws: list[dict] = []
150
+ for row in ws:
151
+ n = _normalize_row(row)
152
+ if n is not None:
153
+ norm_ws.append(n)
154
+ active = data.get("active") or None
155
+ return {
156
+ "schema_version": SCHEMA_VERSION,
157
+ "active": active if isinstance(active, str) else None,
158
+ "workspaces": norm_ws,
159
+ }
160
+
161
+ def _seed_from_admin_state(self) -> dict:
162
+ """I4 — single-workspace backwards compat.
163
+
164
+ On first read with no workspaces.json, attempt to seed a single
165
+ workspace from state.json (admin_email + account_id). Empty state
166
+ produces an empty doc (no workspaces). Never raises.
167
+ """
168
+ try:
169
+ from autoteam.admin_state import load_admin_state # late import to break cycle
170
+ st = load_admin_state() or {}
171
+ except Exception as exc:
172
+ logger.warning("[workspace_pool] seed: cannot read admin_state: %s", exc)
173
+ return _empty_doc()
174
+ admin_email = (st.get("email") or "").strip()
175
+ account_id = (st.get("account_id") or "").strip()
176
+ if not admin_email or not account_id:
177
+ # nothing to seed — fresh install
178
+ return _empty_doc()
179
+ ws_id = f"ws-{account_id}"
180
+ now = time.time()
181
+ row = {
182
+ "id": ws_id,
183
+ "admin_email": admin_email,
184
+ "account_id": account_id,
185
+ "tier": TIER_ACTIVE,
186
+ "status": STATUS_UNKNOWN,
187
+ "fail_count": 0,
188
+ "last_check_ts": None,
189
+ "registered_at": now,
190
+ "transition_log": [
191
+ {"ts": now, "from": None, "to": TIER_ACTIVE, "reason": "seed_from_admin_state"},
192
+ ],
193
+ }
194
+ return {
195
+ "schema_version": SCHEMA_VERSION,
196
+ "active": ws_id,
197
+ "workspaces": [row],
198
+ }
199
+
200
+ def _save_raw(self, doc: dict) -> None:
201
+ """I3 — atomic write with .bak rollback."""
202
+ path = self._path
203
+ path.parent.mkdir(parents=True, exist_ok=True)
204
+ bak = path.with_suffix(path.suffix + ".bak")
205
+ tmp = path.with_suffix(path.suffix + ".tmp")
206
+ payload = json.dumps(doc, indent=2, ensure_ascii=False).encode("utf-8")
207
+ had_snapshot = False
208
+ if path.exists():
209
+ try:
210
+ shutil.copy2(path, bak)
211
+ had_snapshot = True
212
+ except OSError as exc:
213
+ logger.warning("[workspace_pool] snapshot failed %s -> %s: %s", path, bak, exc)
214
+ try:
215
+ with open(tmp, "wb") as fp:
216
+ fp.write(payload)
217
+ fp.flush()
218
+ try:
219
+ os.fsync(fp.fileno())
220
+ except OSError:
221
+ pass
222
+ os.replace(tmp, path)
223
+ except OSError:
224
+ logger.exception("[workspace_pool] write failed for %s, attempting rollback", path)
225
+ if had_snapshot and bak.exists():
226
+ try:
227
+ os.replace(bak, path)
228
+ except OSError:
229
+ logger.exception("[workspace_pool] rollback failed for %s", path)
230
+ if tmp.exists():
231
+ try:
232
+ tmp.unlink()
233
+ except OSError:
234
+ pass
235
+ raise
236
+ else:
237
+ try:
238
+ os.chmod(path, POOL_FILE_MODE)
239
+ except OSError:
240
+ pass
241
+ if had_snapshot and bak.exists():
242
+ try:
243
+ bak.unlink()
244
+ except OSError:
245
+ logger.warning("[workspace_pool] cannot delete .bak %s", bak)
246
+
247
+ # ---------------------------------------------------------- public API
248
+
249
+ def list_all(self) -> list[dict]:
250
+ """Return all workspace rows (deep copy, safe to mutate)."""
251
+ with self._lock:
252
+ doc = self._load_raw()
253
+ return [dict(row) | {"transition_log": list(row.get("transition_log") or [])}
254
+ for row in doc["workspaces"]]
255
+
256
+ def get(self, workspace_id: str) -> dict | None:
257
+ """Return one workspace by id (deep copy) or None."""
258
+ if not workspace_id:
259
+ return None
260
+ with self._lock:
261
+ doc = self._load_raw()
262
+ for row in doc["workspaces"]:
263
+ if row.get("id") == workspace_id:
264
+ return dict(row) | {"transition_log": list(row.get("transition_log") or [])}
265
+ return None
266
+
267
+ def get_active(self) -> dict | None:
268
+ """Return the current active workspace (or None if pool empty)."""
269
+ with self._lock:
270
+ doc = self._load_raw()
271
+ active_id = doc.get("active")
272
+ if not active_id:
273
+ return None
274
+ for row in doc["workspaces"]:
275
+ if row.get("id") == active_id and row.get("tier") == TIER_ACTIVE:
276
+ return dict(row) | {"transition_log": list(row.get("transition_log") or [])}
277
+ return None
278
+
279
+ def register(
280
+ self,
281
+ workspace_id: str,
282
+ admin_email: str,
283
+ account_id: str,
284
+ tier: str = TIER_WARM,
285
+ ) -> dict:
286
+ """Register a new workspace. If id exists, raise ValueError.
287
+
288
+ - tier=active is allowed but will demote any prior active to cold
289
+ (I1 — at most one active).
290
+ - First registration with no active in pool auto-promotes (regardless
291
+ of supplied tier) so single-workspace fresh installs work.
292
+ """
293
+ if not workspace_id or not isinstance(workspace_id, str):
294
+ raise ValueError("workspace_id required")
295
+ if not admin_email or not isinstance(admin_email, str):
296
+ raise ValueError("admin_email required")
297
+ if not account_id or not isinstance(account_id, str):
298
+ raise ValueError("account_id required")
299
+ if tier not in _VALID_TIERS:
300
+ raise ValueError(f"tier must be one of {_VALID_TIERS!r}")
301
+
302
+ events: list[dict] = []
303
+ with self._lock:
304
+ doc = self._load_raw()
305
+ for row in doc["workspaces"]:
306
+ if row.get("id") == workspace_id:
307
+ raise ValueError(f"workspace_id {workspace_id!r} already registered")
308
+ now = time.time()
309
+ target_tier = tier
310
+ # auto-promote if pool is empty / has no active
311
+ if doc.get("active") is None and not any(r.get("tier") == TIER_ACTIVE for r in doc["workspaces"]):
312
+ target_tier = TIER_ACTIVE
313
+
314
+ new_row = {
315
+ "id": workspace_id,
316
+ "admin_email": admin_email,
317
+ "account_id": account_id,
318
+ "tier": target_tier,
319
+ "status": STATUS_UNKNOWN,
320
+ "fail_count": 0,
321
+ "last_check_ts": None,
322
+ "registered_at": now,
323
+ "transition_log": [
324
+ {"ts": now, "from": None, "to": target_tier, "reason": "register"},
325
+ ],
326
+ }
327
+
328
+ if target_tier == TIER_ACTIVE:
329
+ # I1 — demote any other active row
330
+ prior_active_id = doc.get("active")
331
+ for row in doc["workspaces"]:
332
+ if row.get("tier") == TIER_ACTIVE:
333
+ _append_transition(row, TIER_ACTIVE, TIER_COLD, "demote_on_register_active", now)
334
+ row["tier"] = TIER_COLD
335
+ doc["active"] = workspace_id
336
+ if prior_active_id and prior_active_id != workspace_id:
337
+ events.append({
338
+ "type": "demoted",
339
+ "workspace_id": prior_active_id,
340
+ "reason": "register_active",
341
+ "ts": now,
342
+ })
343
+
344
+ doc["workspaces"].append(new_row)
345
+ self._save_raw(doc)
346
+ events.append({
347
+ "type": "registered",
348
+ "workspace_id": workspace_id,
349
+ "tier": target_tier,
350
+ "ts": now,
351
+ })
352
+ snapshot = dict(new_row) | {"transition_log": list(new_row["transition_log"])}
353
+
354
+ for ev in events:
355
+ self._publish(ev)
356
+ return snapshot
357
+
358
+ def set_active(self, workspace_id: str) -> dict:
359
+ """Promote ``workspace_id`` to active; demote any prior active to cold."""
360
+ if not workspace_id:
361
+ raise ValueError("workspace_id required")
362
+ events: list[dict] = []
363
+ with self._lock:
364
+ doc = self._load_raw()
365
+ target = None
366
+ for row in doc["workspaces"]:
367
+ if row.get("id") == workspace_id:
368
+ target = row
369
+ break
370
+ if target is None:
371
+ raise KeyError(f"workspace_id {workspace_id!r} not found")
372
+ now = time.time()
373
+ prior_active_id = doc.get("active")
374
+ if target.get("tier") == TIER_ACTIVE and prior_active_id == workspace_id:
375
+ # already active — short-circuit but still touch last_check_ts? no, idempotent no-op
376
+ snapshot = dict(target) | {"transition_log": list(target.get("transition_log") or [])}
377
+ return snapshot
378
+ for row in doc["workspaces"]:
379
+ if row is target:
380
+ continue
381
+ if row.get("tier") == TIER_ACTIVE:
382
+ _append_transition(row, TIER_ACTIVE, TIER_COLD, "demote_on_set_active", now)
383
+ row["tier"] = TIER_COLD
384
+ events.append({
385
+ "type": "demoted",
386
+ "workspace_id": row.get("id"),
387
+ "reason": "set_active",
388
+ "ts": now,
389
+ })
390
+ _append_transition(target, target.get("tier"), TIER_ACTIVE, "set_active", now)
391
+ target["tier"] = TIER_ACTIVE
392
+ target["status"] = STATUS_UNKNOWN # reset; next probe sets healthy/unhealthy
393
+ target["fail_count"] = 0
394
+ doc["active"] = workspace_id
395
+ self._save_raw(doc)
396
+ events.append({
397
+ "type": "promoted",
398
+ "workspace_id": workspace_id,
399
+ "reason": "set_active",
400
+ "ts": now,
401
+ })
402
+ snapshot = dict(target) | {"transition_log": list(target["transition_log"])}
403
+
404
+ for ev in events:
405
+ self._publish(ev)
406
+ return snapshot
407
+
408
+ def mark_unhealthy(
409
+ self,
410
+ workspace_id: str,
411
+ reason: str,
412
+ *,
413
+ force_failover: bool = False,
414
+ ) -> dict | None:
415
+ """Mark a workspace unhealthy. If active and fail_count >= threshold,
416
+ auto-promote a warm/cold healthy candidate (I2).
417
+
418
+ Returns the (possibly newly-promoted) active workspace snapshot, or
419
+ None if the pool ended up with no active candidate.
420
+ """
421
+ if not workspace_id:
422
+ raise ValueError("workspace_id required")
423
+ events: list[dict] = []
424
+ with self._lock:
425
+ doc = self._load_raw()
426
+ target = None
427
+ for row in doc["workspaces"]:
428
+ if row.get("id") == workspace_id:
429
+ target = row
430
+ break
431
+ if target is None:
432
+ raise KeyError(f"workspace_id {workspace_id!r} not found")
433
+
434
+ now = time.time()
435
+ prev_status = target.get("status")
436
+ target["fail_count"] = int(target.get("fail_count") or 0) + 1
437
+ target["status"] = STATUS_UNHEALTHY
438
+ target["last_check_ts"] = now
439
+ _append_transition(
440
+ target,
441
+ prev_status,
442
+ STATUS_UNHEALTHY,
443
+ f"mark_unhealthy:{reason}",
444
+ now,
445
+ kind="status",
446
+ extra={"fail_count": target["fail_count"]},
447
+ )
448
+ events.append({
449
+ "type": "marked_unhealthy",
450
+ "workspace_id": workspace_id,
451
+ "reason": reason,
452
+ "fail_count": target["fail_count"],
453
+ "ts": now,
454
+ })
455
+
456
+ should_failover = (
457
+ target.get("tier") == TIER_ACTIVE
458
+ and (force_failover or target["fail_count"] >= self._fail_threshold)
459
+ )
460
+ promoted_id: str | None = None
461
+ if should_failover:
462
+ # demote current active → cold
463
+ _append_transition(target, TIER_ACTIVE, TIER_COLD, "failover_demote", now)
464
+ target["tier"] = TIER_COLD
465
+ if doc.get("active") == workspace_id:
466
+ doc["active"] = None
467
+ # pick promotion candidate
468
+ candidate = _pick_failover_candidate(doc["workspaces"])
469
+ if candidate is not None:
470
+ _append_transition(
471
+ candidate,
472
+ candidate.get("tier"),
473
+ TIER_ACTIVE,
474
+ "failover_promote",
475
+ now,
476
+ )
477
+ candidate["tier"] = TIER_ACTIVE
478
+ candidate["status"] = STATUS_UNKNOWN
479
+ candidate["fail_count"] = 0
480
+ doc["active"] = candidate.get("id")
481
+ promoted_id = candidate.get("id")
482
+ events.append({
483
+ "type": "promoted",
484
+ "workspace_id": promoted_id,
485
+ "reason": "failover",
486
+ "ts": now,
487
+ })
488
+ else:
489
+ events.append({
490
+ "type": "no_failover_candidate",
491
+ "workspace_id": workspace_id,
492
+ "ts": now,
493
+ })
494
+
495
+ self._save_raw(doc)
496
+ new_active = self._active_row(doc)
497
+ snapshot = (
498
+ dict(new_active) | {"transition_log": list(new_active.get("transition_log") or [])}
499
+ if new_active else None
500
+ )
501
+
502
+ for ev in events:
503
+ self._publish(ev)
504
+ return snapshot
505
+
506
+ def mark_healthy(self, workspace_id: str) -> dict | None:
507
+ """Reset fail_count to 0 + status=healthy on a successful probe.
508
+
509
+ Idempotent. Useful for the master_health success path so that
510
+ intermittent failures don't accumulate to threshold.
511
+ """
512
+ if not workspace_id:
513
+ raise ValueError("workspace_id required")
514
+ events: list[dict] = []
515
+ with self._lock:
516
+ doc = self._load_raw()
517
+ target = None
518
+ for row in doc["workspaces"]:
519
+ if row.get("id") == workspace_id:
520
+ target = row
521
+ break
522
+ if target is None:
523
+ raise KeyError(f"workspace_id {workspace_id!r} not found")
524
+ now = time.time()
525
+ prev_status = target.get("status")
526
+ prev_fail = int(target.get("fail_count") or 0)
527
+ target["fail_count"] = 0
528
+ target["status"] = STATUS_HEALTHY
529
+ target["last_check_ts"] = now
530
+ if prev_status != STATUS_HEALTHY or prev_fail != 0:
531
+ _append_transition(
532
+ target,
533
+ prev_status,
534
+ STATUS_HEALTHY,
535
+ "mark_healthy",
536
+ now,
537
+ kind="status",
538
+ extra={"prev_fail_count": prev_fail},
539
+ )
540
+ events.append({
541
+ "type": "marked_healthy",
542
+ "workspace_id": workspace_id,
543
+ "ts": now,
544
+ })
545
+ self._save_raw(doc)
546
+ snapshot = dict(target) | {"transition_log": list(target.get("transition_log") or [])}
547
+ for ev in events:
548
+ self._publish(ev)
549
+ return snapshot
550
+
551
+ # --------------------------------------------------------------- helpers
552
+ def _active_row(self, doc: dict) -> dict | None:
553
+ active_id = doc.get("active")
554
+ if not active_id:
555
+ return None
556
+ for row in doc["workspaces"]:
557
+ if row.get("id") == active_id:
558
+ return row
559
+ return None
560
+
561
+ def reset(self) -> None:
562
+ """Test helper — wipe the pool file. Production code should not call."""
563
+ with self._lock:
564
+ try:
565
+ if self._path.exists():
566
+ self._path.unlink()
567
+ except OSError as exc:
568
+ logger.warning("[workspace_pool] reset failed: %s", exc)
569
+
570
+
571
+ # ---------------------------------------------------------------------------
572
+ # Internal helpers
573
+ # ---------------------------------------------------------------------------
574
+ def _empty_doc() -> dict:
575
+ return {"schema_version": SCHEMA_VERSION, "active": None, "workspaces": []}
576
+
577
+
578
+ def _normalize_row(row: Any) -> dict | None:
579
+ if not isinstance(row, dict):
580
+ return None
581
+ rid = row.get("id")
582
+ if not isinstance(rid, str) or not rid:
583
+ return None
584
+ tier = row.get("tier")
585
+ if tier not in _VALID_TIERS:
586
+ tier = TIER_WARM
587
+ status = row.get("status")
588
+ if status not in _VALID_STATUSES:
589
+ status = STATUS_UNKNOWN
590
+ log = row.get("transition_log")
591
+ if not isinstance(log, list):
592
+ log = []
593
+ fail_count = row.get("fail_count")
594
+ try:
595
+ fail_count = int(fail_count or 0)
596
+ except (TypeError, ValueError):
597
+ fail_count = 0
598
+ return {
599
+ "id": rid,
600
+ "admin_email": str(row.get("admin_email") or ""),
601
+ "account_id": str(row.get("account_id") or ""),
602
+ "tier": tier,
603
+ "status": status,
604
+ "fail_count": fail_count,
605
+ "last_check_ts": row.get("last_check_ts"),
606
+ "registered_at": row.get("registered_at"),
607
+ "transition_log": list(log),
608
+ }
609
+
610
+
611
+ def _append_transition(
612
+ row: dict,
613
+ from_value: Any,
614
+ to_value: Any,
615
+ reason: str,
616
+ ts: float,
617
+ *,
618
+ kind: str = "tier",
619
+ extra: dict | None = None,
620
+ ) -> None:
621
+ entry: dict = {
622
+ "ts": ts,
623
+ "kind": kind,
624
+ "from": from_value,
625
+ "to": to_value,
626
+ "reason": reason,
627
+ }
628
+ if extra:
629
+ entry["extra"] = extra
630
+ log = row.get("transition_log")
631
+ if not isinstance(log, list):
632
+ log = []
633
+ row["transition_log"] = log
634
+ log.append(entry)
635
+
636
+
637
+ def _pick_failover_candidate(workspaces: list[dict]) -> dict | None:
638
+ """Pick best workspace to promote to active.
639
+
640
+ Order:
641
+ 1. tier=warm AND status != unhealthy, sorted by registered_at asc (oldest first)
642
+ 2. tier=cold AND status == healthy, sorted by registered_at asc
643
+ 3. tier=warm regardless of status (last resort), oldest first
644
+ 4. None
645
+ """
646
+ warm_healthy = [w for w in workspaces if w.get("tier") == TIER_WARM and w.get("status") != STATUS_UNHEALTHY]
647
+ if warm_healthy:
648
+ warm_healthy.sort(key=lambda w: w.get("registered_at") or 0)
649
+ return warm_healthy[0]
650
+ cold_healthy = [w for w in workspaces if w.get("tier") == TIER_COLD and w.get("status") == STATUS_HEALTHY]
651
+ if cold_healthy:
652
+ cold_healthy.sort(key=lambda w: w.get("registered_at") or 0)
653
+ return cold_healthy[0]
654
+ warm_any = [w for w in workspaces if w.get("tier") == TIER_WARM]
655
+ if warm_any:
656
+ warm_any.sort(key=lambda w: w.get("registered_at") or 0)
657
+ return warm_any[0]
658
+ return None
659
+
660
+
661
+ # ---------------------------------------------------------------------------
662
+ # Module-level default pool
663
+ # ---------------------------------------------------------------------------
664
+ default_pool = WorkspacePool()
665
+
666
+
667
+ def get_default_pool() -> WorkspacePool:
668
+ return default_pool
669
+
670
+
671
+ __all__ = [
672
+ "DEFAULT_POOL_FILE",
673
+ "SCHEMA_VERSION",
674
+ "STATUS_HEALTHY",
675
+ "STATUS_UNHEALTHY",
676
+ "STATUS_UNKNOWN",
677
+ "TIER_ACTIVE",
678
+ "TIER_COLD",
679
+ "TIER_WARM",
680
+ "WorkspacePool",
681
+ "default_pool",
682
+ "get_default_pool",
683
+ ]
tests/unit/test_round12_s7_workspace_pool.py ADDED
@@ -0,0 +1,483 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Round 12 S7 — multi team workspace pool unit tests.
2
+
3
+ Goals:
4
+ - 注册 / 切换 / 故障阈值 / 自动 promote
5
+ - 原子写 + .bak 回滚
6
+ - 单 workspace 模式 backwards-compat (admin_state.json seed)
7
+ - master_health 失败信号 → pool failover 集成
8
+ - admin_state 路由 (active workspace > state.json fallback)
9
+
10
+ Mock-only — 不触达真实 OpenAI workspace API。
11
+ """
12
+ from __future__ import annotations
13
+
14
+ import json
15
+ import os
16
+ import threading
17
+ from pathlib import Path
18
+ from unittest.mock import patch
19
+
20
+ import pytest
21
+
22
+ from autoteam.workspace_pool import (
23
+ SCHEMA_VERSION,
24
+ STATUS_HEALTHY,
25
+ STATUS_UNHEALTHY,
26
+ STATUS_UNKNOWN,
27
+ TIER_ACTIVE,
28
+ TIER_COLD,
29
+ TIER_WARM,
30
+ WorkspacePool,
31
+ )
32
+
33
+ UUID_A = "00000000-0000-0000-0000-00000000aaaa"
34
+ UUID_B = "00000000-0000-0000-0000-00000000bbbb"
35
+ UUID_C = "00000000-0000-0000-0000-00000000cccc"
36
+
37
+
38
+ @pytest.fixture
39
+ def pool_path(tmp_path: Path) -> Path:
40
+ return tmp_path / "workspaces.json"
41
+
42
+
43
+ @pytest.fixture(autouse=True)
44
+ def _isolate_admin_state(monkeypatch):
45
+ """Default: seed from admin_state returns empty so tests don't pick up the
46
+ real on-disk state.json. Tests that exercise the seed path override this
47
+ explicitly via their own monkeypatch."""
48
+ monkeypatch.setattr(
49
+ "autoteam.admin_state.load_admin_state",
50
+ lambda: {"email": "", "account_id": "", "session_token": ""},
51
+ )
52
+
53
+
54
+ @pytest.fixture
55
+ def pool(pool_path: Path) -> WorkspacePool:
56
+ return WorkspacePool(path=pool_path, fail_threshold=3)
57
+
58
+
59
+ # ---------------------------------------------------------------------------
60
+ # 1. register / get_active basics
61
+ # ---------------------------------------------------------------------------
62
+ def test_register_first_auto_promotes_to_active(pool: WorkspacePool):
63
+ snap = pool.register("ws-a", "ad1@example.com", UUID_A, tier=TIER_WARM)
64
+ # supplied tier was warm but pool was empty → auto-promote to active
65
+ assert snap["tier"] == TIER_ACTIVE
66
+ assert pool.get_active()["id"] == "ws-a"
67
+ assert len(pool.list_all()) == 1
68
+
69
+
70
+ def test_register_second_keeps_warm(pool: WorkspacePool):
71
+ pool.register("ws-a", "ad1@example.com", UUID_A)
72
+ snap = pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
73
+ assert snap["tier"] == TIER_WARM
74
+ assert pool.get_active()["id"] == "ws-a"
75
+
76
+
77
+ def test_register_active_demotes_prior_active(pool: WorkspacePool):
78
+ pool.register("ws-a", "ad1@example.com", UUID_A)
79
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_ACTIVE)
80
+ # only one active at a time
81
+ actives = [r for r in pool.list_all() if r["tier"] == TIER_ACTIVE]
82
+ assert len(actives) == 1
83
+ assert actives[0]["id"] == "ws-b"
84
+ # previous active is demoted to cold
85
+ a = pool.get("ws-a")
86
+ assert a["tier"] == TIER_COLD
87
+
88
+
89
+ def test_register_duplicate_id_raises(pool: WorkspacePool):
90
+ pool.register("ws-a", "ad1@example.com", UUID_A)
91
+ with pytest.raises(ValueError):
92
+ pool.register("ws-a", "ad1b@example.com", UUID_B)
93
+
94
+
95
+ def test_register_invalid_tier_raises(pool: WorkspacePool):
96
+ with pytest.raises(ValueError):
97
+ pool.register("ws-a", "ad1@example.com", UUID_A, tier="hot")
98
+
99
+
100
+ # ---------------------------------------------------------------------------
101
+ # 2. set_active / list_all
102
+ # ---------------------------------------------------------------------------
103
+ def test_set_active_swaps_and_demotes(pool: WorkspacePool):
104
+ pool.register("ws-a", "ad1@example.com", UUID_A)
105
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
106
+ pool.set_active("ws-b")
107
+ assert pool.get_active()["id"] == "ws-b"
108
+ assert pool.get("ws-a")["tier"] == TIER_COLD
109
+ assert pool.get("ws-b")["tier"] == TIER_ACTIVE
110
+
111
+
112
+ def test_set_active_unknown_id_raises(pool: WorkspacePool):
113
+ pool.register("ws-a", "ad1@example.com", UUID_A)
114
+ with pytest.raises(KeyError):
115
+ pool.set_active("ws-zzz")
116
+
117
+
118
+ def test_set_active_idempotent_when_already_active(pool: WorkspacePool):
119
+ pool.register("ws-a", "ad1@example.com", UUID_A)
120
+ snap = pool.set_active("ws-a")
121
+ assert snap["tier"] == TIER_ACTIVE
122
+
123
+
124
+ # ---------------------------------------------------------------------------
125
+ # 3. mark_unhealthy + failover
126
+ # ---------------------------------------------------------------------------
127
+ def test_mark_unhealthy_below_threshold_no_failover(pool: WorkspacePool):
128
+ pool.register("ws-a", "ad1@example.com", UUID_A)
129
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
130
+ snap = pool.mark_unhealthy("ws-a", "subscription_cancelled")
131
+ # only 1 fail (threshold=3) → no failover, ws-a still active
132
+ assert snap["id"] == "ws-a"
133
+ a = pool.get("ws-a")
134
+ assert a["fail_count"] == 1
135
+ assert a["status"] == STATUS_UNHEALTHY
136
+
137
+
138
+ def test_mark_unhealthy_at_threshold_triggers_failover(pool: WorkspacePool):
139
+ pool.register("ws-a", "ad1@example.com", UUID_A)
140
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
141
+ pool.mark_unhealthy("ws-a", "auth_invalid")
142
+ pool.mark_unhealthy("ws-a", "auth_invalid")
143
+ snap = pool.mark_unhealthy("ws-a", "auth_invalid")
144
+ # 3rd fail at threshold=3 → failover to ws-b
145
+ assert snap is not None
146
+ assert snap["id"] == "ws-b"
147
+ assert pool.get_active()["id"] == "ws-b"
148
+ a = pool.get("ws-a")
149
+ assert a["tier"] == TIER_COLD
150
+
151
+
152
+ def test_mark_unhealthy_no_warm_no_failover_candidate(pool: WorkspacePool):
153
+ pool.register("ws-a", "ad1@example.com", UUID_A)
154
+ # only ws-a (active), no warm/cold candidate
155
+ pool.set_fail_threshold(1)
156
+ snap = pool.mark_unhealthy("ws-a", "auth_invalid")
157
+ # no candidate → active becomes None
158
+ assert snap is None
159
+ assert pool.get_active() is None
160
+ a = pool.get("ws-a")
161
+ assert a["tier"] == TIER_COLD
162
+
163
+
164
+ def test_mark_unhealthy_force_failover_immediate(pool: WorkspacePool):
165
+ pool.register("ws-a", "ad1@example.com", UUID_A)
166
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
167
+ snap = pool.mark_unhealthy("ws-a", "manual_force", force_failover=True)
168
+ assert snap["id"] == "ws-b"
169
+
170
+
171
+ def test_failover_picks_oldest_warm_first(pool: WorkspacePool):
172
+ pool.register("ws-a", "ad1@example.com", UUID_A)
173
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
174
+ pool.register("ws-c", "ad3@example.com", UUID_C, tier=TIER_WARM)
175
+ # Set deterministic registered_at: ws-b oldest warm, ws-c younger
176
+ raw = json.loads(pool.path.read_text(encoding="utf-8"))
177
+ for row in raw["workspaces"]:
178
+ if row["id"] == "ws-b":
179
+ row["registered_at"] = 1000.0
180
+ elif row["id"] == "ws-c":
181
+ row["registered_at"] = 2000.0
182
+ pool.path.write_text(json.dumps(raw, indent=2), encoding="utf-8")
183
+
184
+ pool.set_fail_threshold(1)
185
+ snap = pool.mark_unhealthy("ws-a", "auth_invalid")
186
+ assert snap["id"] == "ws-b" # oldest warm wins
187
+
188
+
189
+ def test_mark_healthy_resets_fail_count(pool: WorkspacePool):
190
+ pool.register("ws-a", "ad1@example.com", UUID_A)
191
+ pool.mark_unhealthy("ws-a", "subscription_cancelled")
192
+ pool.mark_unhealthy("ws-a", "subscription_cancelled")
193
+ pool.mark_healthy("ws-a")
194
+ a = pool.get("ws-a")
195
+ assert a["fail_count"] == 0
196
+ assert a["status"] == STATUS_HEALTHY
197
+
198
+
199
+ # ---------------------------------------------------------------------------
200
+ # 4. transition_log
201
+ # ---------------------------------------------------------------------------
202
+ def test_transition_log_register_promote_demote(pool: WorkspacePool):
203
+ pool.register("ws-a", "ad1@example.com", UUID_A)
204
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
205
+ pool.set_active("ws-b")
206
+ a = pool.get("ws-a")
207
+ reasons_a = [e["reason"] for e in a["transition_log"]]
208
+ assert "register" in reasons_a
209
+ assert "demote_on_set_active" in reasons_a
210
+ b = pool.get("ws-b")
211
+ reasons_b = [e["reason"] for e in b["transition_log"]]
212
+ assert "register" in reasons_b
213
+ assert "set_active" in reasons_b
214
+
215
+
216
+ def test_transition_log_failover_writes_two_rows(pool: WorkspacePool):
217
+ pool.register("ws-a", "ad1@example.com", UUID_A)
218
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
219
+ pool.set_fail_threshold(1)
220
+ pool.mark_unhealthy("ws-a", "auth_invalid")
221
+ a = pool.get("ws-a")
222
+ b = pool.get("ws-b")
223
+ assert any(e["reason"] == "failover_demote" for e in a["transition_log"])
224
+ assert any(e["reason"] == "failover_promote" for e in b["transition_log"])
225
+
226
+
227
+ # ---------------------------------------------------------------------------
228
+ # 5. atomic write + .bak rollback (I3)
229
+ # ---------------------------------------------------------------------------
230
+ def test_atomic_write_replaces_file(pool: WorkspacePool):
231
+ pool.register("ws-a", "ad1@example.com", UUID_A)
232
+ raw = json.loads(pool.path.read_text(encoding="utf-8"))
233
+ assert raw["schema_version"] == SCHEMA_VERSION
234
+ assert raw["active"] == "ws-a"
235
+ # no leftover .tmp / .bak
236
+ assert not pool.path.with_suffix(pool.path.suffix + ".tmp").exists()
237
+ assert not pool.path.with_suffix(pool.path.suffix + ".bak").exists()
238
+
239
+
240
+ def test_atomic_write_rolls_back_on_os_replace_failure(pool_path: Path):
241
+ pool = WorkspacePool(path=pool_path)
242
+ pool.register("ws-a", "ad1@example.com", UUID_A)
243
+ original = pool_path.read_text(encoding="utf-8")
244
+
245
+ real_replace = os.replace
246
+
247
+ def boom(src, dst):
248
+ if str(dst) == str(pool_path):
249
+ raise OSError("simulated mid-write failure")
250
+ return real_replace(src, dst)
251
+
252
+ with patch("autoteam.workspace_pool.os.replace", side_effect=boom):
253
+ with pytest.raises(OSError):
254
+ pool.register("ws-b", "ad2@example.com", UUID_B)
255
+
256
+ # original content preserved (rollback restored .bak)
257
+ assert pool_path.read_text(encoding="utf-8") == original
258
+
259
+
260
+ # ---------------------------------------------------------------------------
261
+ # 6. backwards-compat seed from admin_state.json (I4)
262
+ # ---------------------------------------------------------------------------
263
+ def test_seed_from_admin_state_when_pool_missing(monkeypatch, pool_path: Path):
264
+ fake_state = {
265
+ "email": "legacy_admin@example.com",
266
+ "account_id": UUID_A,
267
+ "session_token": "tok",
268
+ }
269
+ monkeypatch.setattr("autoteam.admin_state.load_admin_state", lambda: fake_state)
270
+ pool = WorkspacePool(path=pool_path)
271
+ active = pool.get_active()
272
+ assert active is not None
273
+ assert active["admin_email"] == "legacy_admin@example.com"
274
+ assert active["account_id"] == UUID_A
275
+ assert active["tier"] == TIER_ACTIVE
276
+
277
+
278
+ def test_seed_skipped_when_state_empty(monkeypatch, pool_path: Path):
279
+ monkeypatch.setattr("autoteam.admin_state.load_admin_state", lambda: {"email": "", "account_id": ""})
280
+ pool = WorkspacePool(path=pool_path)
281
+ assert pool.get_active() is None
282
+ assert pool.list_all() == []
283
+
284
+
285
+ # ---------------------------------------------------------------------------
286
+ # 7. admin_state routing
287
+ # ---------------------------------------------------------------------------
288
+ def test_admin_state_get_admin_email_routes_to_pool(monkeypatch, pool_path: Path):
289
+ pool = WorkspacePool(path=pool_path)
290
+ pool.register("ws-x", "pool_admin@example.com", UUID_A)
291
+ monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
292
+ # state.json should NOT be consulted when pool active
293
+ monkeypatch.setattr("autoteam.admin_state.load_admin_state",
294
+ lambda: {"email": "stale@example.com", "account_id": ""})
295
+ from autoteam.admin_state import get_admin_email, get_chatgpt_account_id
296
+ assert get_admin_email() == "pool_admin@example.com"
297
+ assert get_chatgpt_account_id() == UUID_A
298
+
299
+
300
+ def test_admin_state_falls_back_to_state_json_when_pool_empty(monkeypatch, pool_path: Path):
301
+ pool = WorkspacePool(path=pool_path)
302
+ # empty pool (no register, and state seed returns empty)
303
+ monkeypatch.setattr("autoteam.admin_state.load_admin_state",
304
+ lambda: {"email": "fallback@example.com", "account_id": UUID_B})
305
+ monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
306
+ # also: pool.get_active will seed from state, so make seed return empty
307
+ monkeypatch.setattr(
308
+ WorkspacePool, "_seed_from_admin_state",
309
+ lambda self: {"schema_version": SCHEMA_VERSION, "active": None, "workspaces": []},
310
+ )
311
+ from autoteam.admin_state import get_admin_email, get_chatgpt_account_id
312
+ assert get_admin_email() == "fallback@example.com"
313
+ assert get_chatgpt_account_id() == UUID_B
314
+
315
+
316
+ # ---------------------------------------------------------------------------
317
+ # 8. master_health pool integration
318
+ # ---------------------------------------------------------------------------
319
+ def test_apply_pool_health_signal_healthy_marks_healthy(monkeypatch, pool_path: Path):
320
+ pool = WorkspacePool(path=pool_path)
321
+ pool.register("ws-a", "ad1@example.com", UUID_A)
322
+ pool.mark_unhealthy("ws-a", "subscription_cancelled") # fail_count=1
323
+ from autoteam.master_health import apply_pool_health_signal
324
+ snap = apply_pool_health_signal(True, "active", pool=pool)
325
+ assert snap is not None
326
+ a = pool.get("ws-a")
327
+ assert a["fail_count"] == 0
328
+ assert a["status"] == STATUS_HEALTHY
329
+
330
+
331
+ def test_apply_pool_health_signal_subscription_cancelled_drives_failover(pool_path: Path):
332
+ pool = WorkspacePool(path=pool_path, fail_threshold=2)
333
+ pool.register("ws-a", "ad1@example.com", UUID_A)
334
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
335
+ from autoteam.master_health import apply_pool_health_signal
336
+ apply_pool_health_signal(False, "subscription_cancelled", pool=pool)
337
+ snap = apply_pool_health_signal(False, "subscription_cancelled", pool=pool)
338
+ # threshold=2 reached → failover to ws-b
339
+ assert snap is not None
340
+ assert snap["id"] == "ws-b"
341
+
342
+
343
+ def test_apply_pool_health_signal_network_error_no_failover(pool_path: Path):
344
+ pool = WorkspacePool(path=pool_path, fail_threshold=1)
345
+ pool.register("ws-a", "ad1@example.com", UUID_A)
346
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
347
+ from autoteam.master_health import apply_pool_health_signal
348
+ snap = apply_pool_health_signal(False, "network_error", pool=pool)
349
+ # network_error is transient — don't trip
350
+ assert snap["id"] == "ws-a"
351
+ a = pool.get("ws-a")
352
+ assert a["fail_count"] == 0 # untouched
353
+
354
+
355
+ def test_apply_pool_health_signal_empty_pool_returns_none(pool_path: Path):
356
+ pool = WorkspacePool(path=pool_path)
357
+ from autoteam.master_health import apply_pool_health_signal
358
+ assert apply_pool_health_signal(False, "auth_invalid", pool=pool) is None
359
+
360
+
361
+ # ---------------------------------------------------------------------------
362
+ # 9. event subscribers
363
+ # ---------------------------------------------------------------------------
364
+ def test_subscriber_receives_register_event(pool: WorkspacePool):
365
+ events: list[dict] = []
366
+ pool.subscribe(events.append)
367
+ pool.register("ws-a", "ad1@example.com", UUID_A)
368
+ types = [e["type"] for e in events]
369
+ assert "registered" in types
370
+
371
+
372
+ def test_subscriber_receives_failover_promote_event(pool: WorkspacePool):
373
+ events: list[dict] = []
374
+ pool.subscribe(events.append)
375
+ pool.register("ws-a", "ad1@example.com", UUID_A)
376
+ pool.register("ws-b", "ad2@example.com", UUID_B, tier=TIER_WARM)
377
+ pool.set_fail_threshold(1)
378
+ pool.mark_unhealthy("ws-a", "auth_invalid")
379
+ types = [e["type"] for e in events]
380
+ assert "promoted" in types
381
+ assert "marked_unhealthy" in types
382
+
383
+
384
+ def test_subscriber_exceptions_swallowed(pool: WorkspacePool):
385
+ def bad(_e):
386
+ raise RuntimeError("boom")
387
+ pool.subscribe(bad)
388
+ # must not raise
389
+ pool.register("ws-a", "ad1@example.com", UUID_A)
390
+
391
+
392
+ # ---------------------------------------------------------------------------
393
+ # 10. concurrency safety (lock)
394
+ # ---------------------------------------------------------------------------
395
+ def test_concurrent_registers_no_duplicate_active(pool: WorkspacePool):
396
+ barrier = threading.Barrier(5)
397
+ errors: list[Exception] = []
398
+
399
+ def worker(i):
400
+ try:
401
+ barrier.wait(timeout=2)
402
+ pool.register(f"ws-{i}", f"ad{i}@example.com", f"00000000-0000-0000-0000-{i:012d}")
403
+ except Exception as exc:
404
+ errors.append(exc)
405
+
406
+ threads = [threading.Thread(target=worker, args=(i,)) for i in range(5)]
407
+ for t in threads:
408
+ t.start()
409
+ for t in threads:
410
+ t.join()
411
+ assert errors == []
412
+ actives = [r for r in pool.list_all() if r["tier"] == TIER_ACTIVE]
413
+ assert len(actives) == 1 # I1 — exactly one active
414
+
415
+
416
+ # ---------------------------------------------------------------------------
417
+ # 11. schema migration / corrupt-file resilience
418
+ # ---------------------------------------------------------------------------
419
+ def test_corrupt_json_treated_as_empty(pool_path: Path):
420
+ pool_path.write_text("{not valid json", encoding="utf-8")
421
+ pool = WorkspacePool(path=pool_path)
422
+ assert pool.list_all() == []
423
+ # subsequent register works (file is overwritten cleanly)
424
+ pool.register("ws-a", "ad1@example.com", UUID_A)
425
+ assert pool.get_active()["id"] == "ws-a"
426
+
427
+
428
+ def test_unknown_schema_version_discarded(pool_path: Path):
429
+ pool_path.write_text(
430
+ json.dumps({"schema_version": 999, "active": "ws-x", "workspaces": []}),
431
+ encoding="utf-8",
432
+ )
433
+ pool = WorkspacePool(path=pool_path)
434
+ assert pool.get_active() is None
435
+
436
+
437
+ def test_normalize_drops_invalid_tier(pool_path: Path):
438
+ pool_path.write_text(
439
+ json.dumps({
440
+ "schema_version": SCHEMA_VERSION,
441
+ "active": "ws-x",
442
+ "workspaces": [{
443
+ "id": "ws-x",
444
+ "admin_email": "a@e.com",
445
+ "account_id": UUID_A,
446
+ "tier": "hot", # invalid
447
+ "status": "weird", # invalid
448
+ "fail_count": "not-an-int",
449
+ "transition_log": "not-a-list",
450
+ }],
451
+ }),
452
+ encoding="utf-8",
453
+ )
454
+ pool = WorkspacePool(path=pool_path)
455
+ rows = pool.list_all()
456
+ assert len(rows) == 1
457
+ assert rows[0]["tier"] == TIER_WARM
458
+ assert rows[0]["status"] == STATUS_UNKNOWN
459
+ assert rows[0]["fail_count"] == 0
460
+ assert rows[0]["transition_log"] == []
461
+
462
+
463
+ # ---------------------------------------------------------------------------
464
+ # 12. cpa_sync helper
465
+ # ---------------------------------------------------------------------------
466
+ def test_cpa_sync_get_active_sync_target_returns_pool_summary(monkeypatch, pool_path: Path):
467
+ pool = WorkspacePool(path=pool_path)
468
+ pool.register("ws-a", "cpa_admin@example.com", UUID_A)
469
+ monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
470
+ from autoteam.cpa_sync import get_active_sync_target
471
+ target = get_active_sync_target()
472
+ assert target == {"id": "ws-a", "admin_email": "cpa_admin@example.com", "account_id": UUID_A}
473
+
474
+
475
+ def test_cpa_sync_get_active_sync_target_none_when_empty(monkeypatch, pool_path: Path):
476
+ pool = WorkspacePool(path=pool_path)
477
+ monkeypatch.setattr("autoteam.workspace_pool.default_pool", pool)
478
+ monkeypatch.setattr(
479
+ WorkspacePool, "_seed_from_admin_state",
480
+ lambda self: {"schema_version": SCHEMA_VERSION, "active": None, "workspaces": []},
481
+ )
482
+ from autoteam.cpa_sync import get_active_sync_target
483
+ assert get_active_sync_target() is None