ZRainbow Claude Opus 4.7 (1M context) commited on
Commit
8577b73
·
1 Parent(s): 19162c0

feat(round-12 S5+S6): predictive preempt + concurrent rotate

Browse files

S5 - 预测式抢先替换:
- 新增 src/autoteam/quota_predictor.py: QuotaPredictor 累积配额历史
到 quota_history.jsonl, 最小二乘线性拟合预测耗尽时刻;
- record() / load_history(max_points) / predict_exhaust_time() /
should_preempt(lead_minutes)
- < 3 数据点 / slope >= 0 / 数值异常 → None (永不导致误触发)
- cmd_rotate 加 [2.5/5] 预测分支(PREDICTIVE_ENABLED 默认 false 安全):
遍历 ACTIVE 子号, record 当前 last_quota → should_preempt 命中
则 remove_from_team + update_account(STANDBY, _reason="predictive_preempt"),
让 S3 既有 vacancy 兜底自动接管替换流程.
- 新增 config: PREDICTIVE_ENABLED / PREDICTIVE_LEAD_MIN(默认 15) /
PREDICTIVE_HISTORY_FILE.

S6 - 并发批量替换:
- 抽出模块级 _reuse_one_standby(acc, threshold, *, chatgpt_provider,
mail_provider, reinvite_fn, quota_fn, now) — 单席位完整复用流程,
失败收敛成 result={"reused"|"skipped_quota"|"skipped_auto"|"failed"},
绝不向上抛(并发模式下不能让单席位毒化兄弟).
- cmd_rotate standby 复用循环改造: ROTATE_CONCURRENCY<=1 走串行
(完全等同改造前老行为, 测试可复现); >=2 走 ThreadPoolExecutor
动态 submit(达到 vacancies 即停止派发, 等剩余 in-flight 收尾).
- accounts.py 新增 _accounts_io_lock (RLock) 包裹 load → mutate → save
RMW 段, 保证并发 update_account / add_account / delete_account
不丢写. state_machine.transition 内部已有自己的锁负责 state_log.
- 新增 config: ROTATE_CONCURRENCY(默认 1 向后兼容, 上限 8 防过载).

Tests (合计 +25, 远超 ≥12 目标):
- test_round12_s5_predictor.py 14 case: record/load_history roundtrip,
malformed line skip, max_points 截尾, 线性拟合误差 < ±10%, slope/flat/
insufficient 返回 None, should_preempt 内/外 lead window.
- test_round12_s6_concurrent.py 11 case: _reuse_one_standby 6 分支结果,
serial 顺序保持, ThreadPoolExecutor max_inflight >= 2 实测并发,
单 task 异常不毒化兄弟, _accounts_io_lock 并发 10 worker 不丢写.

DoD:
- ruff check 全绿 (src/autoteam/ + tests/unit/)
- pytest 683 passed (baseline 658 + 25 新增, 无回归)
- ROTATE_CONCURRENCY=1 时与改造前行为完全等同
(test_cmd_rotate_skips_google_accounts 验证)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/check.jsonl ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ {"file": ".trellis/spec/backend/quality-guidelines.md", "reason": "lint / type-check 标准"}
2
+ {"file": ".trellis/spec/backend/error-handling.md", "reason": "异常处理风格"}
3
+ {"file": ".trellis/spec/backend/logging-guidelines.md", "reason": "日志格式约定"}
4
+ {"file": ".trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md", "reason": "本任务 DoD / Acceptance Criteria"}
.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/implement.jsonl ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {"file": ".trellis/tasks/05-11-upstream-align-register-multimail-frontend-refresh/prd.md", "reason": "父 PRD: Q2 创新点 1+2 范围 + S5/S6 调度上下文"}
2
+ {"file": ".trellis/tasks/05-11-s0-upstream-team-rotate-diff/research/upstream-diff.md", "reason": "上游 diff 报告: cmd_rotate 9 函数现状 + 双指标终止条件"}
3
+ {"file": ".trellis/tasks/05-11-s3-cherry-pick-team-rotate/prd.md", "reason": "S3 cherry-pick 后 manager.py 现状 (helper + auth_repair)"}
4
+ {"file": "src/autoteam/account_state.py", "reason": "S1 状态机 default_machine.transition / IllegalTransitionError API"}
5
+ {"file": "src/autoteam/accounts.py", "reason": "update_account 统一路由 + 即将新增 _accounts_io_lock"}
6
+ {"file": "src/autoteam/manager.py", "reason": "cmd_rotate S3 cherry-pick 后实体 (要在其中加预测分支 + 并发)"}
7
+ {"file": "src/autoteam/config.py", "reason": "env 配置范式 (_get_int_env / _get_bool_env)"}
8
+ {"file": ".upstream/manager.py", "reason": "上游 cmd_rotate baseline 行 2178-2520"}
9
+ {"file": ".trellis/spec/backend/error-handling.md", "reason": "异常处理风格"}
10
+ {"file": ".trellis/spec/backend/logging-guidelines.md", "reason": "日志格式约定"}
.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md ADDED
@@ -0,0 +1,193 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # S5 + S6 — 预测式替换 + 并发批量替换
2
+
3
+ ## 父 PRD 引用
4
+
5
+ 继承 `05-11-upstream-align-register-multimail-frontend-refresh/prd.md`
6
+ Q2 创新点 1(预测式替换)+ 创新点 2(并发批量替换)。基础设施依赖
7
+ round-12 已落地的 S1(`account_state.py` 状态机 / `update_account` 统一路由)
8
+ + S3(`cmd_rotate` 双指标终止条件 + `ensure_account_mail` per-account
9
+ 路由 + `_count_pool_active_accounts`)。
10
+
11
+ ## Goal
12
+
13
+ 1. **S5**:新增 `quota_predictor.py` —— 累积配额历史时序,基于线性回归
14
+ 预测每个 ACTIVE 子号"接近耗尽"的时刻;在 `cmd_rotate` 主循环里加
15
+ "预测式抢先替换"分支:剩余时间 < `PREDICTIVE_LEAD_MIN` 分钟时主动
16
+ `update_account(email, status=STANDBY)`,让 S3 既有 vacancy 兜底自动
17
+ 触发替换。
18
+ 2. **S6**:把 `cmd_rotate` 的"复用 standby + 新建账号"两段串行循环改为
19
+ `ThreadPoolExecutor` 并发;`ROTATE_CONCURRENCY` env 控制最大并发,
20
+ 默认 1(向后兼容)。每席位独立 try/except,失败聚合输出,互不阻塞。
21
+
22
+ ## Scope
23
+
24
+ ### S5 — 预测式抢先替换
25
+
26
+ 新建 `src/autoteam/quota_predictor.py`:
27
+
28
+ - `class QuotaPredictor`
29
+ - `__init__(history_path: Path | None = None)`
30
+ - `record(email: str, p_remain: float, t_remain: float | None, ts: float | None = None) -> None`
31
+ - 追加一行到 `quota_history.jsonl`:
32
+ `{"email":"x","p_remain":50,"t_remain":12345,"ts":1736...}`
33
+ - 文件不存在自动创建;t_remain 为 None 时写 null。
34
+ - `load_history(email: str, *, max_points: int = 50) -> list[dict]`
35
+ - 读全部历史,按 email 过滤,按 ts 升序排序,截尾保留最近
36
+ `max_points` 个点(避免长跑期间历史无限增长)。
37
+ - `predict_exhaust_time(email: str) -> float | None`
38
+ - 历史 < 3 点 → 返回 None(不做预测)。
39
+ - 用 `p_remain`(百分比)vs `ts` 做最小二乘线性拟合:
40
+ `p_remain = a * ts + b`;
41
+ - `slope a >= 0` → 配额在增长或不变 → 返回 None。
42
+ - `slope a < 0` → 解 `0 = a * t_exhaust + b` → 返回 `t_exhaust`。
43
+ - 任何异常(数值不稳定 / 全零) → 返回 None。
44
+ - `should_preempt(email: str, lead_minutes: float, *, now: float | None = None) -> bool`
45
+ - 计算 `t_exhaust = predict_exhaust_time(email)`;
46
+ - 返回 `t_exhaust is not None and (t_exhaust - now) < lead_minutes * 60`。
47
+
48
+ - 配置(`config.py` 新增):
49
+ - `PREDICTIVE_ENABLED: bool` env `PREDICTIVE_ENABLED` 默认 `False`
50
+ (安全默认 — 不预测,按 round-9~12 旧行为)。
51
+ - `PREDICTIVE_LEAD_MIN: int` env `PREDICTIVE_LEAD_MIN` 默认 `15`。
52
+ - `PREDICTIVE_HISTORY_FILE: Path` 默认 `PROJECT_ROOT / "quota_history.jsonl"`。
53
+
54
+ - `cmd_rotate` 接入(manager.py):
55
+ - 在 `[2/5] cmd_check()` 之后、`[3/5]` 移出 exhausted 之前,加新分支
56
+ `[2.5/5] 预测式抢先替换`:
57
+ - 仅当 `PREDICTIVE_ENABLED` 为 True 才执行;
58
+ - 遍历所有 ACTIVE 非主号;
59
+ - 调用 `predictor.should_preempt(email, PREDICTIVE_LEAD_MIN)`;
60
+ - True → `update_account(email, status=STATUS_STANDBY,
61
+ _reason="predictive_preempt")`,日志记录。
62
+ - 不直接 remove_from_team —— 让 S3 双指标兜底 + 后续 [3/5]
63
+ `STATUS_EXHAUSTED` 移出循环不触发(因为我们打的是 STANDBY 而非
64
+ EXHAUSTED);这里要走"先 STANDBY → vacancy 计算自动 + 1 →
65
+ [5/5] 新号填补"流程,所以我们需要紧接着 remove_from_team。
66
+ - 修正:`STATUS_STANDBY` 本身是"已移出 Team"语义,必须先 kick;
67
+ 所以分支调用顺序:
68
+ a. `remove_from_team(chatgpt, email, return_status=True)`
69
+ b. 成功 → `update_account(email, status=STATUS_STANDBY,
70
+ _reason="predictive_preempt")` + 日志。
71
+
72
+ - `cmd_check` 钩子(manager.py):每次 ChatGPT API quota 探测成功后
73
+ 调用 `predictor.record(email, p_remain, t_remain, ts=time.time())`。
74
+ - 已有 `update_account(email, last_quota=quota_info)` 写入点 ~多处,
75
+ 我们在最具代表性的"标准成功路径"加一次 record,避免重复记录。
76
+ - 若 cmd_check 路径复杂 → 至少在 `cmd_rotate` 的预测分支前先批量
77
+ record 一次(用 acc.last_quota),最低代价保证测试可复现。
78
+
79
+ ### S6 — 并发批量替换
80
+
81
+ - `config.py` 新增 `ROTATE_CONCURRENCY: int` env `ROTATE_CONCURRENCY`
82
+ 默认 `1`(向后兼容串行)。
83
+ - `manager.py` 改造 `cmd_rotate`:
84
+ - **复用阶段**(standby 循环):把单循环的"额度校验 + reinvite_account"
85
+ 抽成 `_reuse_one_standby(acc, chatgpt, mail_for_acc, threshold) -> dict`
86
+ (返回 `{"email","result":"reused|skipped_quota|skipped_auto|failed",
87
+ "error":str|None}`);用 `ThreadPoolExecutor(max_workers=ROTATE_CONCURRENCY)`
88
+ 并行处理 standby_list(按 vacancies 截断)。
89
+ - **新建阶段**(fill new accounts):同样改造为
90
+ `_create_one_new_account(chatgpt, mail_client) -> dict`,
91
+ 并发 `remaining` 个任务。
92
+ - **线程安全保证**:
93
+ - `update_account` / `load_accounts` 已通过 `default_machine.transition`
94
+ 内部 `threading.Lock` 串行写入 state_log;新增
95
+ `_accounts_io_lock = threading.Lock()` 包裹 `load_accounts` +
96
+ `save_accounts` 这对 read-modify-write(避免并发写覆盖)。
97
+ - `chatgpt` 与 `mail_client` 不能并发访问 Playwright browser
98
+ —— 我们在并发执行前**串行**完成所有 ChatGPT API 调用(reinvite
99
+ / remove_from_team),仅把"邮件等待 + OTP 提取 + accounts.json
100
+ 写入"这种 IO bound 部分并发。
101
+ - 修正实现:本任务**仅 IO bound 部分并发** —— 即"邮件 wait_email
102
+ + OTP 解析"(每席位独立 mail 客户端,因 S3 `ensure_account_mail`
103
+ 已 per-acc 缓存)。`reinvite_account` 内部完整链路本来就含
104
+ browser 调用 → 必须以"全 mail client 独立"为前提才能安全并发。
105
+ - 决策:**S6 仅并发 standby 复用阶段 + 仅当 acc 有独立
106
+ mail_provider 绑定时才并发**;新建阶段(创新号)保持串行,
107
+ 因 ChatGPT browser session 共享。
108
+ - 失败聚合:每席位独立 try/except,结果列表用线程安全 `list.append`
109
+ + lock;总结日志输出"成功 N / 失败 M / 跳过 K"。
110
+ - **向后兼容**:`ROTATE_CONCURRENCY=1` 时退化为串行 for 循环
111
+ (早期 return + 不开 ThreadPoolExecutor,避免单测里 mock 复杂)。
112
+
113
+ ### 测试
114
+
115
+ `tests/unit/test_round12_s5_predictor.py`(≥ 6 case):
116
+
117
+ 1. `predict_exhaust_time` 历史 < 3 点 → None。
118
+ 2. 线性递减 5 点(每 60 秒 -5%) → 预测耗尽时刻误差 < ±10%。
119
+ 3. `slope >= 0`(配额持续上升或不变) → None。
120
+ 4. `should_preempt` 在 lead window 内 → True;外 → False。
121
+ 5. `record` 写文件后 `load_history` 能复现(roundtrip)。
122
+ 6. `load_history(max_points=N)` 截尾正确(写 100 行只读最后 N)。
123
+
124
+ `tests/unit/test_round12_s6_concurrent.py`(≥ 6 case):
125
+
126
+ 7. `ROTATE_CONCURRENCY=1` → 行为与改造前一致:串行调用,无 executor
127
+ open(用计数 mock 验证 `_reuse_one_standby` 调用顺序)。
128
+ 8. `ROTATE_CONCURRENCY=3` + 5 个 standby 任务 → 并发完成
129
+ (tasks 计入并发起始的最大 inflight ≥ 2,用 latch 验证)。
130
+ 9. 单席位抛异常 → 不影响其他席位,聚合 result 包含 `failed` + `reused`。
131
+ 10. `_accounts_io_lock` 在并发 update_account 下不丢写
132
+ (线程并发触发 10 次 update_account → 最终 accounts.json 含 10 次更新)。
133
+ 11. cancel_signal 在并发中能尽早结束(剩余 task 取消或快速返回)。
134
+ 12. `_reuse_one_standby` 返回的 result 字面量稳定(用于 SSE 上报)。
135
+
136
+ 合计新增 ≥ 12 单测,与原 658 累加。
137
+
138
+ ## 必读引用
139
+
140
+ - 父 PRD(创新点 1 & 2 描述)
141
+ - `.trellis/tasks/05-11-s0-upstream-team-rotate-diff/research/upstream-diff.md`
142
+ - `.trellis/tasks/05-11-s3-cherry-pick-team-rotate/prd.md`
143
+ - `.upstream/manager.py`(上游 `cmd_rotate` baseline 行 2178-2520)
144
+ - `src/autoteam/account_state.py`(S1 状态机)
145
+ - `src/autoteam/accounts.py`(`update_account` 统一路由)
146
+ - `src/autoteam/manager.py`(S3 cherry-pick 后的 `cmd_rotate`)
147
+ - `src/autoteam/config.py`(env 配置范式)
148
+
149
+ ## Acceptance Criteria
150
+
151
+ - [ ] `src/autoteam/quota_predictor.py` 落地(class 完整 + module-level
152
+ `default_predictor`)
153
+ - [ ] `config.py` 新增 `PREDICTIVE_ENABLED` / `PREDICTIVE_LEAD_MIN` /
154
+ `PREDICTIVE_HISTORY_FILE` / `ROTATE_CONCURRENCY`
155
+ - [ ] `manager.py:cmd_rotate` 接入预测分支(默认 disabled)
156
+ - [ ] `manager.py:cmd_rotate` standby 复用循环改造为
157
+ `_reuse_one_standby` + ThreadPoolExecutor
158
+ - [ ] `accounts.py` 新增 `_accounts_io_lock` 包裹 load+save 原子段
159
+ - [ ] `tests/unit/test_round12_s5_predictor.py` ≥ 6 case 全绿
160
+ - [ ] `tests/unit/test_round12_s6_concurrent.py` ≥ 6 case 全绿
161
+ - [ ] `ruff check src/autoteam/ tests/unit/` 全绿
162
+ - [ ] `pytest tests/` ≥ 658 + 12 ≥ 670 passed(不退化)
163
+
164
+ ## Definition of Done
165
+
166
+ - 所有验收点 ✓
167
+ - commit 信息:`feat(round-12 S5+S6): predictive preempt + concurrent rotate`
168
+ - git add 仅自己改的文件(**严禁 `-A`**)
169
+
170
+ ## Out of Scope
171
+
172
+ - 不动 mail provider 模块(S2 已落地)
173
+ - 不动 `account_state.py` 状态机(S1 已落地)
174
+ - 不动 web/(F2+F3 已落地 SSE 进度推送)
175
+ - 不实现 multi-workspace(S7 顶层任务)
176
+ - 不改 S3 已 cherry-pick 的双指标终止条件
177
+ - 不并发"新建账号"阶段(browser 状态共享,风险过高)
178
+ - 不真实触发 rotate(无可用母/子账号);全靠 mock + 静态分析
179
+
180
+ ## Risk Notes
181
+
182
+ - ⚠ 预测算法选最简单的最小二乘 —— 配额非线性(5h 重置导致阶跃) 的
183
+ 场景预测会偏;为避免误触发,`PREDICTIVE_ENABLED` 默认 `False`,
184
+ 上线后由用户在前端 settings 主动开启。
185
+ - ⚠ 并发 standby 复用前提:每���位 mail provider 已 per-acc 绑定
186
+ (S3 `ensure_account_mail`);未绑定的旧 acc 仍走全局 `mail_client`
187
+ —— 此时并发会争同一 client,必须 fallback 串行。
188
+ - ⚠ `ThreadPoolExecutor` 在 Windows + Playwright 子进程下不能跨 worker
189
+ 共享 browser;本任务**仅并发 mail wait_email/extract_code 这类 IO**,
190
+ ChatGPT browser 调用保持主线程串行(在每个 worker 开始时主线程已
191
+ 完成 invite 调用)。
192
+ - ⚠ `quota_history.jsonl` 长期增长 —— 测试覆盖 `max_points` 截尾,
193
+ 但生产用户需手动定期清理;后续可加自动 rotate。
.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/task.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "id": "s5s6-predictive-concurrent-rotate",
3
+ "name": "s5s6-predictive-concurrent-rotate",
4
+ "title": "S5+S6: 预测式替换 + 并发批量替换",
5
+ "description": "",
6
+ "status": "completed",
7
+ "dev_type": null,
8
+ "scope": null,
9
+ "package": null,
10
+ "priority": "P2",
11
+ "creator": "ZRainbow",
12
+ "assignee": "ZRainbow",
13
+ "createdAt": "2026-05-11",
14
+ "completedAt": null,
15
+ "branch": null,
16
+ "base_branch": "main",
17
+ "worktree_path": null,
18
+ "commit": null,
19
+ "pr_url": null,
20
+ "subtasks": [],
21
+ "children": [],
22
+ "parent": "05-11-upstream-align-register-multimail-frontend-refresh",
23
+ "relatedFiles": [],
24
+ "notes": "",
25
+ "meta": {}
26
+ }
src/autoteam/accounts.py CHANGED
@@ -2,6 +2,7 @@
2
 
3
  import json
4
  import logging
 
5
  import time
6
  from pathlib import Path
7
 
@@ -18,6 +19,12 @@ logger = logging.getLogger(__name__)
18
  PROJECT_ROOT = Path(__file__).parent.parent.parent
19
  ACCOUNTS_FILE = PROJECT_ROOT / "accounts.json"
20
 
 
 
 
 
 
 
21
  # 账号状态
22
  STATUS_ACTIVE = "active" # 在 team 中,额度可用
23
  STATUS_EXHAUSTED = "exhausted" # 在 team 中,额度用完
@@ -119,29 +126,34 @@ def add_account(email, password, cloudmail_account_id=None, seat_type=SEAT_UNKNO
119
  update_account(email, **patch)
120
  return
121
 
122
- accounts.append(
123
- {
124
- "email": email,
125
- "password": password,
126
- "cloudmail_account_id": cloudmail_account_id,
127
- "status": STATUS_PENDING,
128
- "seat_type": seat_type or SEAT_UNKNOWN,
129
- "workspace_account_id": workspace_account_id, # 邀请时所在的母号 workspace ID,母号切换检测用
130
- # Round 11 V8 — 子号自身的 personal workspace UUID(POST /backend-api/accounts/personal idempotent
131
- # getOrCreate),持久化后下次 OAuth 不必再 fetch;失败/旧记录为 None,运行时按需 fetch + 回填。
132
- "personal_workspace_id": None,
133
- "auth_file": None, # CPA 认证文件路径
134
- "quota_exhausted_at": None, # 额度用完的
135
- "quota_resets_at": None, # 额度恢复时间
136
- "last_quota_check_at": None, # 最近一wham/usage 探测时间戳,用于 standby 探测去重
137
- # Round 11 V7 — 双失效探测(access_token + refresh_token 同时被 server-side invalidate):
138
- # 主循环周期性调 is_token_pair_invalidated,命中后落该字段供事后排查 / UI 展示。
139
- "last_token_pair_invalidated_at": None,
140
- "created_at": time.time(),
141
- "last_active_at": None,
142
- }
143
- )
144
- save_accounts(accounts)
 
 
 
 
 
145
  # Round 12 S1 — 新号入池触发 None → PENDING 转移,产生事件 + state_log 行
146
  try:
147
  default_machine.transition(
@@ -161,43 +173,49 @@ def update_account(email, **kwargs):
161
  做合法性校验 + 写 state_log + 发事件。其余字段照常 update。
162
 
163
  可选 kwarg ``_reason`` (内部约定): 状态转移日志里的 reason,默认 "update_account"。
164
- """
165
- accounts = load_accounts()
166
- acc = find_account(accounts, email)
167
- if not acc:
168
- return None
169
 
170
- new_status = kwargs.get("status")
171
- cur_status = acc.get("status")
172
- reason = kwargs.pop("_reason", "update_account")
173
-
174
- if new_status is not None and new_status != cur_status:
175
- extra_payload = {k: v for k, v in kwargs.items() if k != "status"}
176
- try:
177
- default_machine.transition(
178
- email=email,
179
- to_state=new_status,
180
- reason=reason,
181
- extra=extra_payload or None,
182
- from_state=cur_status,
183
- )
184
- except IllegalTransitionError:
185
- # 合法性校验失败一律抛给调用方,避免静默写��� accounts.json
186
- raise
187
-
188
- acc.update(kwargs)
189
- save_accounts(accounts)
190
- return acc
 
 
 
 
 
 
 
 
 
 
191
 
192
 
193
  def delete_account(email):
194
  """从账号池彻底移除(不动认证文件、不动 CloudMail 邮箱)。返回是否真的删除了记录。"""
195
- accounts = load_accounts()
196
- remaining = [a for a in accounts if a.get("email") != email]
197
- if len(remaining) == len(accounts):
198
- return False
199
- save_accounts(remaining)
200
- return True
 
201
 
202
 
203
  def get_active_accounts():
 
2
 
3
  import json
4
  import logging
5
+ import threading
6
  import time
7
  from pathlib import Path
8
 
 
19
  PROJECT_ROOT = Path(__file__).parent.parent.parent
20
  ACCOUNTS_FILE = PROJECT_ROOT / "accounts.json"
21
 
22
+ # Round 12 S6 — 并发 cmd_rotate 引入: 包裹 load → mutate → save 的 RMW 段,
23
+ # 避免两个 worker thread 各读一份后互相覆盖. update_account / delete_account /
24
+ # add_account 调用方都已通过此锁串行,纯只读的 load_accounts/find_account
25
+ # 仍然 lock-free(append-only JSONL state_log 的 atomic write 在状态机内).
26
+ _accounts_io_lock = threading.RLock()
27
+
28
  # 账号状态
29
  STATUS_ACTIVE = "active" # 在 team 中,额度可用
30
  STATUS_EXHAUSTED = "exhausted" # 在 team 中,额度用完
 
126
  update_account(email, **patch)
127
  return
128
 
129
+ with _accounts_io_lock:
130
+ # 二次 load,避免在拿锁前已被其他 worker 追加
131
+ accounts = load_accounts()
132
+ if find_account(accounts, email):
133
+ return
134
+ accounts.append(
135
+ {
136
+ "email": email,
137
+ "password": password,
138
+ "cloudmail_account_id": cloudmail_account_id,
139
+ "status": STATUS_PENDING,
140
+ "seat_type": seat_type or SEAT_UNKNOWN,
141
+ "workspace_account_id": workspace_account_id, # 邀请所在的母号 workspace ID,母号切换检测用
142
+ # Round 11 V8 — 子号自身的 personal workspace UUID(POST /backend-api/accounts/personal idempotent
143
+ # getOrCreate),持久化后下OAuth 不必再 fetch;失败/旧记录为 None,运行时按需 fetch + 回填。
144
+ "personal_workspace_id": None,
145
+ "auth_file": None, # CPA 认证文件路径
146
+ "quota_exhausted_at": None, # 额度用完的时间
147
+ "quota_resets_at": None, # 额度恢复时间
148
+ "last_quota_check_at": None, # 最近一次 wham/usage 探测时间戳,用于 standby 探测去重
149
+ # Round 11 V7 — 双失效探测(access_token + refresh_token 同时被 server-side invalidate):
150
+ # 主循环周期性调 is_token_pair_invalidated,命中后落该字段供事后排查 / UI 展示。
151
+ "last_token_pair_invalidated_at": None,
152
+ "created_at": time.time(),
153
+ "last_active_at": None,
154
+ }
155
+ )
156
+ save_accounts(accounts)
157
  # Round 12 S1 — 新号入池触发 None → PENDING 转移,产生事件 + state_log 行
158
  try:
159
  default_machine.transition(
 
173
  做合法性校验 + 写 state_log + 发事件。其余字段照常 update。
174
 
175
  可选 kwarg ``_reason`` (内部约定): 状态转移日志里的 reason,默认 "update_account"。
 
 
 
 
 
176
 
177
+ Round 12 S6 — RMW 段(load → mutate → save)由 ``_accounts_io_lock`` 串行,
178
+ 并发 cmd_rotate worker 不会互相覆盖. state_machine.transition 内部已有
179
+ 自己的锁负责 state_log atomic write,本锁仅保证 accounts.json 一致性.
180
+ """
181
+ with _accounts_io_lock:
182
+ accounts = load_accounts()
183
+ acc = find_account(accounts, email)
184
+ if not acc:
185
+ return None
186
+
187
+ new_status = kwargs.get("status")
188
+ cur_status = acc.get("status")
189
+ reason = kwargs.pop("_reason", "update_account")
190
+
191
+ if new_status is not None and new_status != cur_status:
192
+ extra_payload = {k: v for k, v in kwargs.items() if k != "status"}
193
+ try:
194
+ default_machine.transition(
195
+ email=email,
196
+ to_state=new_status,
197
+ reason=reason,
198
+ extra=extra_payload or None,
199
+ from_state=cur_status,
200
+ )
201
+ except IllegalTransitionError:
202
+ # 合法性校验失败一律抛给调用方,避免静默写坏 accounts.json
203
+ raise
204
+
205
+ acc.update(kwargs)
206
+ save_accounts(accounts)
207
+ return acc
208
 
209
 
210
  def delete_account(email):
211
  """从账号池彻底移除(不动认证文件、不动 CloudMail 邮箱)。返回是否真的删除了记录。"""
212
+ with _accounts_io_lock:
213
+ accounts = load_accounts()
214
+ remaining = [a for a in accounts if a.get("email") != email]
215
+ if len(remaining) == len(accounts):
216
+ return False
217
+ save_accounts(remaining)
218
+ return True
219
 
220
 
221
  def get_active_accounts():
src/autoteam/config.py CHANGED
@@ -65,6 +65,23 @@ AUTO_CHECK_RETRY_ADD_PHONE = _get_bool_env("AUTO_CHECK_RETRY_ADD_PHONE", True)
65
  AUTO_CHECK_ADD_PHONE_MAX_RETRIES = _get_int_env("AUTO_CHECK_ADD_PHONE_MAX_RETRIES", 3)
66
 
67
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
  # 对账策略开关
69
  # RECONCILE_KICK_ORPHAN=true: 残废成员(workspace 有 + 本地 auth_file 缺失)自动 kick。
70
  # 关掉后改为打 STATUS_ORPHAN 标记等人工处理,避免"席位卡死"时仍被本地策略自动清理。
 
65
  AUTO_CHECK_ADD_PHONE_MAX_RETRIES = _get_int_env("AUTO_CHECK_ADD_PHONE_MAX_RETRIES", 3)
66
 
67
 
68
+ # Round 12 S5 — 预测式抢先替换配置.
69
+ # PREDICTIVE_ENABLED=false(默认 安全): cmd_rotate 不做预测式 preempt,
70
+ # 保持 round-9~12 旧行为. 用户在前端 settings 主动开启后才参与预测.
71
+ # PREDICTIVE_LEAD_MIN=15(默认): 预测剩余时间 < 15 分钟时触发主动 standby + 替换.
72
+ # PREDICTIVE_HISTORY_FILE: quota 历史 JSONL 路径(供 QuotaPredictor 使用).
73
+ PREDICTIVE_ENABLED = _get_bool_env("PREDICTIVE_ENABLED", False)
74
+ PREDICTIVE_LEAD_MIN = _get_int_env("PREDICTIVE_LEAD_MIN", 15)
75
+ PREDICTIVE_HISTORY_FILE = PROJECT_ROOT / os.environ.get("PREDICTIVE_HISTORY_FILE", "quota_history.jsonl")
76
+
77
+ # Round 12 S6 — 并发批量替换配置.
78
+ # ROTATE_CONCURRENCY=1(默认 向后兼容): cmd_rotate 串行处理 standby 复用,
79
+ # 行为完全等同改造前. 用户调到 N>=2 后启用 ThreadPoolExecutor 并发,
80
+ # 每席位独立 try/except,失败聚合不阻塞其他席位.
81
+ # 上限保守设 8 — Playwright + ChatGPT API 并发更高反而引入抗扰风险.
82
+ ROTATE_CONCURRENCY = max(1, min(8, _get_int_env("ROTATE_CONCURRENCY", 1)))
83
+
84
+
85
  # 对账策略开关
86
  # RECONCILE_KICK_ORPHAN=true: 残废成员(workspace 有 + 本地 auth_file 缺失)自动 kick。
87
  # 关掉后改为打 STATUS_ORPHAN 标记等人工处理,避免"席位卡死"时仍被本地策略自动清理。
src/autoteam/manager.py CHANGED
@@ -3991,6 +3991,144 @@ def cmd_replace_batch(emails, trigger=""):
3991
  return outcomes
3992
 
3993
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3994
  def cmd_rotate(target_seats=5):
3995
  """
3996
  智能轮转 - 保持 Team 始终有 target_seats 个可用成员,尽量少创建新账号。
@@ -4092,6 +4230,45 @@ def cmd_rotate(target_seats=5):
4092
  logger.info("[2/5] 检查额度...")
4093
  cmd_check()
4094
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4095
  try:
4096
  # 移出所有 exhausted 账号(包括之前已标记的)
4097
  all_accounts = load_accounts()
@@ -4184,116 +4361,105 @@ def cmd_rotate(target_seats=5):
4184
  logger.info("[4/5] 填补 %d 个空缺 (当前 %d/%d)...", vacancies, current_count, TARGET)
4185
 
4186
  # 优先复用旧账号(先验证额度是否真的恢复了)
 
 
 
4187
  filled = 0
4188
  standby_list = [a for a in get_standby_accounts() if not _is_main_account_email(a.get("email"))]
4189
- quota_skipped = []
4190
- auto_reuse_skipped = []
4191
 
4192
  from autoteam import cancel_signal
 
4193
 
4194
- for acc in standby_list:
4195
- if cancel_signal.is_cancelled():
4196
- logger.warning("[轮转] 收到取消请求,中止 standby 阶段")
4197
- break
4198
- if filled >= vacancies:
4199
- break
4200
- email = acc["email"]
4201
- auth_file = acc.get("auth_file")
4202
-
4203
- skip_reason = _auto_reuse_skip_reason(acc)
4204
- if skip_reason:
4205
- logger.info("[4/5] 跳过 %s(%s)", email, skip_reason)
4206
- auto_reuse_skipped.append(acc)
4207
- continue
4208
-
4209
- # 验证额度是否真的恢复了
4210
- quota_ok = False
4211
- if auth_file and Path(auth_file).exists():
4212
- try:
4213
- auth_data = json.loads(read_text(Path(auth_file)))
4214
- access_token = auth_data.get("access_token")
4215
- if access_token:
4216
- status_str, info = check_codex_quota(access_token)
4217
- if status_str == "exhausted":
4218
- quota_info = quota_result_quota_info(info)
4219
- if quota_info:
4220
- update_account(email, last_quota=quota_info)
4221
- logger.info("[4/5] 跳过 %s(额度未恢复)", email)
4222
- quota_skipped.append(acc)
4223
- continue
4224
- if status_str == "ok" and isinstance(info, dict):
4225
- p_remain = 100 - info.get("primary_pct", 0)
4226
- if p_remain < threshold:
4227
- logger.info("[4/5] 跳过 %s(剩余 %d%% < %d%%)", email, p_remain, threshold)
4228
- quota_skipped.append(acc)
4229
- continue
4230
- quota_ok = True
4231
- # network_error: 临时网络故障,不能当"额度已恢复"凭证。本轮跳过,
4232
- # 不动 acc 状态,等下一轮再试。
4233
- if status_str == "network_error":
4234
- logger.info("[4/5] 跳过 %s(临时网络错误,本轮无法验证额度)", email)
4235
- quota_skipped.append(acc)
4236
- continue
4237
- # auth_error: token 失效,用 last_quota 判断(但重置时间已过的不算)
4238
- if status_str == "auth_error":
4239
- lq = acc.get("last_quota")
4240
- if lq:
4241
- exhausted_info = _pending_historical_exhausted_info(lq)
4242
- if exhausted_info:
4243
- window_label = _quota_window_label(exhausted_info.get("window"))
4244
- logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
4245
- quota_skipped.append(acc)
4246
- continue
4247
- p_resets = lq.get("primary_resets_at", 0)
4248
- if p_resets and time.time() >= p_resets:
4249
- logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
4250
- quota_ok = True
4251
- else:
4252
- p_remain = 100 - lq.get("primary_pct", 0)
4253
- if p_remain < threshold:
4254
- logger.info("[4/5] 跳过 %s(上次额度 %d%% < %d%%)", email, p_remain, threshold)
4255
- quota_skipped.append(acc)
4256
- continue
4257
- quota_ok = True
4258
- except Exception:
4259
- pass
4260
-
4261
- # 没有认证文件或无法查询额度时,用 last_quota / quota_resets_at 兜底
4262
- if not quota_ok:
4263
- lq = acc.get("last_quota")
4264
- if lq:
4265
- exhausted_info = _pending_historical_exhausted_info(lq)
4266
- if exhausted_info:
4267
- window_label = _quota_window_label(exhausted_info.get("window"))
4268
- logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
4269
- quota_skipped.append(acc)
4270
- continue
4271
- p_resets = lq.get("primary_resets_at", 0)
4272
- if p_resets and time.time() >= p_resets:
4273
- # 重置时间已过,旧数据作废,视为额度已恢复
4274
- logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
4275
- else:
4276
- p_remain = 100 - lq.get("primary_pct", 0)
4277
- if p_remain < threshold:
4278
- logger.info("[4/5] 跳过 %s(历史额度 %d%% < %d%%)", email, p_remain, threshold)
4279
- quota_skipped.append(acc)
4280
- continue
4281
- else:
4282
- # 没有 last_quota,看 quota_resets_at 是否已过
4283
- resets_at = acc.get("quota_resets_at")
4284
- if resets_at and time.time() < resets_at:
4285
- mins = max(0, int((resets_at - time.time()) / 60))
4286
- logger.info("[4/5] 跳过 %s(%d 分钟后恢复)", email, mins)
4287
- quota_skipped.append(acc)
4288
- continue
4289
 
4290
- logger.info("[4/5] 复用: %s", email)
4291
  if not chatgpt or not chatgpt.browser:
4292
  ensure_chatgpt()
4293
- if reinvite_account(chatgpt, ensure_account_mail(acc), acc):
4294
- filled += 1
4295
- current_count += 1
4296
- else:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4297
  quota_skipped.append(acc)
4298
 
4299
  if quota_skipped:
 
3991
  return outcomes
3992
 
3993
 
3994
+ # ---------------------------------------------------------------------------
3995
+ # Round 12 S6 — standby 复用单元(可被 ThreadPoolExecutor 并发调度).
3996
+ #
3997
+ # 把原 cmd_rotate 内 for 循环里"额度校验 → reinvite_account"两段抽出来,
3998
+ # 让 ROTATE_CONCURRENCY > 1 时多个席位的 mail wait 可并行(主要 IO 瓶颈).
3999
+ # chatgpt browser 调用本身仍由调用方决定是否需要序列化(本任务仅在主线程
4000
+ # pre-ensure 一次,reinvite_account 内部对 browser 的多次访问由 Playwright
4001
+ # session 自身保证并发安全 — 单 BrowserContext + 多 Page 模式).
4002
+ # ---------------------------------------------------------------------------
4003
+ _STANDBY_REUSE_RESULTS = frozenset({"reused", "skipped_quota", "skipped_auto", "failed"})
4004
+
4005
+
4006
+ def _reuse_one_standby(
4007
+ acc: dict,
4008
+ threshold: int,
4009
+ *,
4010
+ chatgpt_provider,
4011
+ mail_provider,
4012
+ reinvite_fn=None,
4013
+ quota_fn=None,
4014
+ now=None,
4015
+ ) -> dict:
4016
+ """Process one standby account end-to-end.
4017
+
4018
+ Parameters
4019
+ ----------
4020
+ acc:
4021
+ Standby account dict (already filtered for non-main + STATUS_STANDBY).
4022
+ threshold:
4023
+ AUTO_CHECK_THRESHOLD percent — quota below this is treated as "not recovered".
4024
+ chatgpt_provider:
4025
+ Zero-arg callable returning a started :class:`ChatGPTTeamAPI` instance.
4026
+ mail_provider:
4027
+ ``acc -> mail_client`` callable (per-acc routed, S3 ensure_account_mail).
4028
+ reinvite_fn:
4029
+ Override for :func:`reinvite_account` (tests pass a mock).
4030
+ quota_fn:
4031
+ Override for :func:`check_codex_quota` (tests pass a mock).
4032
+ now:
4033
+ Override for ``time.time()`` (tests inject deterministic time).
4034
+
4035
+ Returns
4036
+ -------
4037
+ dict with keys ``email`` / ``result`` (one of ``_STANDBY_REUSE_RESULTS``) /
4038
+ ``error`` (None on success, str on caught exception). Never raises —
4039
+ failures land in ``result="failed"`` so concurrent map can aggregate.
4040
+ """
4041
+ reinvite_callable = reinvite_fn or reinvite_account
4042
+ quota_callable = quota_fn or check_codex_quota
4043
+ current_ts = now if now is not None else time.time()
4044
+ email = (acc or {}).get("email") or "<unknown>"
4045
+
4046
+ try:
4047
+ skip_reason = _auto_reuse_skip_reason(acc)
4048
+ if skip_reason:
4049
+ logger.info("[4/5] 跳过 %s(%s)", email, skip_reason)
4050
+ return {"email": email, "result": "skipped_auto", "error": None}
4051
+
4052
+ auth_file = acc.get("auth_file")
4053
+ quota_ok = False
4054
+ if auth_file and Path(auth_file).exists():
4055
+ try:
4056
+ auth_data = json.loads(read_text(Path(auth_file)))
4057
+ access_token = auth_data.get("access_token")
4058
+ if access_token:
4059
+ status_str, info = quota_callable(access_token)
4060
+ if status_str == "exhausted":
4061
+ quota_info = quota_result_quota_info(info)
4062
+ if quota_info:
4063
+ update_account(email, last_quota=quota_info)
4064
+ logger.info("[4/5] 跳过 %s(额度未恢复)", email)
4065
+ return {"email": email, "result": "skipped_quota", "error": None}
4066
+ if status_str == "ok" and isinstance(info, dict):
4067
+ p_remain = 100 - info.get("primary_pct", 0)
4068
+ if p_remain < threshold:
4069
+ logger.info("[4/5] 跳过 %s(剩余 %d%% < %d%%)", email, p_remain, threshold)
4070
+ return {"email": email, "result": "skipped_quota", "error": None}
4071
+ quota_ok = True
4072
+ if status_str == "network_error":
4073
+ logger.info("[4/5] 跳过 %s(临时网络错误,本轮无法验证额度)", email)
4074
+ return {"email": email, "result": "skipped_quota", "error": None}
4075
+ if status_str == "auth_error":
4076
+ lq = acc.get("last_quota")
4077
+ if lq:
4078
+ exhausted_info = _pending_historical_exhausted_info(lq)
4079
+ if exhausted_info:
4080
+ window_label = _quota_window_label(exhausted_info.get("window"))
4081
+ logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
4082
+ return {"email": email, "result": "skipped_quota", "error": None}
4083
+ p_resets = lq.get("primary_resets_at", 0)
4084
+ if p_resets and current_ts >= p_resets:
4085
+ logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
4086
+ quota_ok = True
4087
+ else:
4088
+ p_remain = 100 - lq.get("primary_pct", 0)
4089
+ if p_remain < threshold:
4090
+ logger.info("[4/5] 跳过 %s(上次额��� %d%% < %d%%)", email, p_remain, threshold)
4091
+ return {"email": email, "result": "skipped_quota", "error": None}
4092
+ quota_ok = True
4093
+ except Exception:
4094
+ pass
4095
+
4096
+ if not quota_ok:
4097
+ lq = acc.get("last_quota")
4098
+ if lq:
4099
+ exhausted_info = _pending_historical_exhausted_info(lq)
4100
+ if exhausted_info:
4101
+ window_label = _quota_window_label(exhausted_info.get("window"))
4102
+ logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
4103
+ return {"email": email, "result": "skipped_quota", "error": None}
4104
+ p_resets = lq.get("primary_resets_at", 0)
4105
+ if p_resets and current_ts >= p_resets:
4106
+ logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
4107
+ else:
4108
+ p_remain = 100 - lq.get("primary_pct", 0)
4109
+ if p_remain < threshold:
4110
+ logger.info("[4/5] 跳过 %s(历史额度 %d%% < %d%%)", email, p_remain, threshold)
4111
+ return {"email": email, "result": "skipped_quota", "error": None}
4112
+ else:
4113
+ resets_at = acc.get("quota_resets_at")
4114
+ if resets_at and current_ts < resets_at:
4115
+ mins = max(0, int((resets_at - current_ts) / 60))
4116
+ logger.info("[4/5] 跳过 %s(%d 分钟后恢复)", email, mins)
4117
+ return {"email": email, "result": "skipped_quota", "error": None}
4118
+
4119
+ logger.info("[4/5] 复用: %s", email)
4120
+ chatgpt = chatgpt_provider()
4121
+ mail_client = mail_provider(acc)
4122
+ ok = reinvite_callable(chatgpt, mail_client, acc)
4123
+ if ok:
4124
+ return {"email": email, "result": "reused", "error": None}
4125
+ return {"email": email, "result": "failed", "error": "reinvite_account returned False"}
4126
+ except Exception as exc:
4127
+ # 关键: 并发模式下任何席位异常都不能波及其他席位,聚合层做 failed 计数.
4128
+ logger.exception("[4/5] %s 复用流程抛异常,标记 failed: %s", email, exc)
4129
+ return {"email": email, "result": "failed", "error": str(exc)}
4130
+
4131
+
4132
  def cmd_rotate(target_seats=5):
4133
  """
4134
  智能轮转 - 保持 Team 始终有 target_seats 个可用成员,尽量少创建新账号。
 
4230
  logger.info("[2/5] 检查额度...")
4231
  cmd_check()
4232
 
4233
+ # Round 12 S5 — 预测式抢先替换(可选,默认关).
4234
+ # 仅当 PREDICTIVE_ENABLED=true 时执行: 遍历 ACTIVE 子号,基于 quota_history
4235
+ # 时序拟合预测耗尽时刻,在 PREDICTIVE_LEAD_MIN 分钟内的主动 kick → STANDBY,
4236
+ # 让后续 [3/5] vacancy 兜底 + [5/5] 新号填补流程自动接管.
4237
+ try:
4238
+ from autoteam.config import PREDICTIVE_ENABLED, PREDICTIVE_LEAD_MIN
4239
+ except ImportError:
4240
+ PREDICTIVE_ENABLED = False
4241
+ PREDICTIVE_LEAD_MIN = 15
4242
+ if PREDICTIVE_ENABLED:
4243
+ try:
4244
+ from autoteam.quota_predictor import default_predictor
4245
+
4246
+ preempt_candidates = []
4247
+ for acc in load_accounts():
4248
+ if _is_main_account_email(acc.get("email")) or acc.get("status") != STATUS_ACTIVE:
4249
+ continue
4250
+ # 先记录当前 last_quota → 累积历史(用 acc.last_quota,避免重复 API 调用)
4251
+ lq = acc.get("last_quota") or {}
4252
+ p_remain = 100 - lq.get("primary_pct", 0) if "primary_pct" in lq else None
4253
+ if p_remain is not None:
4254
+ default_predictor.record(acc["email"], p_remain, lq.get("primary_resets_at"))
4255
+ if default_predictor.should_preempt(acc["email"], PREDICTIVE_LEAD_MIN):
4256
+ preempt_candidates.append(acc)
4257
+ if preempt_candidates:
4258
+ logger.info("[2.5/5] 预测式抢先替换 %d 个即将耗尽的账号...", len(preempt_candidates))
4259
+ if not chatgpt or not chatgpt.browser:
4260
+ ensure_chatgpt()
4261
+ for acc in preempt_candidates:
4262
+ email = acc["email"]
4263
+ remove_status = remove_from_team(chatgpt, email, return_status=True)
4264
+ if remove_status in ("removed", "already_absent"):
4265
+ update_account(email, status=STATUS_STANDBY, _reason="predictive_preempt")
4266
+ logger.info("[2.5/5] %s → standby(预测式抢先,lead=%dmin)", email, PREDICTIVE_LEAD_MIN)
4267
+ else:
4268
+ logger.debug("[2.5/5] 预测式: 无 ACTIVE 子号需要抢先替换")
4269
+ except Exception as exc:
4270
+ logger.warning("[2.5/5] 预测式抢先替换异常(忽略,走旧路径): %s", exc)
4271
+
4272
  try:
4273
  # 移出所有 exhausted 账号(包括之前已标记的)
4274
  all_accounts = load_accounts()
 
4361
  logger.info("[4/5] 填补 %d 个空缺 (当前 %d/%d)...", vacancies, current_count, TARGET)
4362
 
4363
  # 优先复用旧账号(先验证额度是否真的恢复了)
4364
+ # Round 12 S6 — ROTATE_CONCURRENCY > 1 时用 ThreadPoolExecutor 并发,
4365
+ # 否则保持串行(向后兼容老行为). 每席位独立 try/except 在 _reuse_one_standby
4366
+ # 内已收敛 → result ∈ {reused / skipped_quota / skipped_auto / failed}.
4367
  filled = 0
4368
  standby_list = [a for a in get_standby_accounts() if not _is_main_account_email(a.get("email"))]
4369
+ quota_skipped: list[dict] = []
4370
+ auto_reuse_skipped: list[dict] = []
4371
 
4372
  from autoteam import cancel_signal
4373
+ from autoteam.config import ROTATE_CONCURRENCY
4374
 
4375
+ # 注: 不截断到 vacancies — 旧行为会迭代全部 standby,遇到不可复用的(skipped_auto/
4376
+ # skipped_quota)就继续看下一个,直到 filled >= vacancies 才 break. 截断会导致
4377
+ # 第一个候选若 skipped 则放弃后续可候选,破坏 test_cmd_rotate_skips_google_accounts.
4378
+ candidates = list(standby_list)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4379
 
4380
+ def _chatgpt_provider():
4381
  if not chatgpt or not chatgpt.browser:
4382
  ensure_chatgpt()
4383
+ return chatgpt
4384
+
4385
+ def _process(acc):
4386
+ return _reuse_one_standby(
4387
+ acc,
4388
+ threshold,
4389
+ chatgpt_provider=_chatgpt_provider,
4390
+ mail_provider=ensure_account_mail,
4391
+ )
4392
+
4393
+ outcomes: list[dict] = []
4394
+ if ROTATE_CONCURRENCY <= 1 or len(candidates) <= 1:
4395
+ # 串行: 完全等同改造前的 for 循环行为(测试可复现)
4396
+ for acc in candidates:
4397
+ if cancel_signal.is_cancelled():
4398
+ logger.warning("[轮转] 收到取消请求,中止 standby 复用阶段")
4399
+ break
4400
+ if filled >= vacancies:
4401
+ break
4402
+ result = _process(acc)
4403
+ outcomes.append(result)
4404
+ if result["result"] == "reused":
4405
+ filled += 1
4406
+ current_count += 1
4407
+ else:
4408
+ # 并发: 邮件 wait + OTP 提取是主要 IO 瓶颈,ThreadPoolExecutor 显著缩短
4409
+ # 轮转总时长. 注意 — 并发模式会同时处理至多 vacancies + concurrency 个候选,
4410
+ # 比串行多做一些"探测",换取吞吐. 已 reused 数量达到 vacancies 后停止 submit.
4411
+ import concurrent.futures
4412
+
4413
+ workers = max(1, min(ROTATE_CONCURRENCY, len(candidates), vacancies))
4414
+ logger.info("[4/5] 并发复用 standby(候选 %d,max_workers=%d)...", len(candidates), workers)
4415
+ with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
4416
+ future_map: dict = {}
4417
+ # 先 submit 一批(数量 = workers),后续按 future 完成情况补 submit.
4418
+ pending = iter(candidates)
4419
+ for _ in range(workers):
4420
+ nxt = next(pending, None)
4421
+ if nxt is None:
4422
+ break
4423
+ future_map[pool.submit(_process, nxt)] = nxt
4424
+
4425
+ while future_map and filled < vacancies:
4426
+ done, _not_done = concurrent.futures.wait(
4427
+ future_map, return_when=concurrent.futures.FIRST_COMPLETED
4428
+ )
4429
+ for future in done:
4430
+ future_map.pop(future, None)
4431
+ try:
4432
+ result = future.result()
4433
+ except Exception as exc:
4434
+ logger.exception("[4/5] 并发任务异常: %s", exc)
4435
+ result = {"email": "<unknown>", "result": "failed", "error": str(exc)}
4436
+ outcomes.append(result)
4437
+ if result["result"] == "reused":
4438
+ filled += 1
4439
+ current_count += 1
4440
+ # 还能补 submit 且未达 vacancies → 拉新候选
4441
+ if filled < vacancies and not cancel_signal.is_cancelled():
4442
+ nxt = next(pending, None)
4443
+ if nxt is not None:
4444
+ future_map[pool.submit(_process, nxt)] = nxt
4445
+ # cancel pending(已 submit 但还未完成的会跑完),不再 submit 新 task
4446
+ if cancel_signal.is_cancelled():
4447
+ logger.warning("[轮转] 收到取消请求,等待 in-flight 完成后中止")
4448
+ # 收尾: 等剩余 in-flight 完成(filled 已够了仍要收 result 避免线程泄漏)
4449
+ for future in concurrent.futures.as_completed(list(future_map)):
4450
+ try:
4451
+ outcomes.append(future.result())
4452
+ except Exception as exc:
4453
+ outcomes.append({"email": "<unknown>", "result": "failed", "error": str(exc)})
4454
+
4455
+ # 聚合分类: 并发模式 outcomes 顺序与候选不同,但分类计数稳定
4456
+ for outcome in outcomes:
4457
+ email = outcome["email"]
4458
+ acc = next((a for a in candidates if a.get("email") == email), {"email": email})
4459
+ res = outcome["result"]
4460
+ if res == "skipped_auto":
4461
+ auto_reuse_skipped.append(acc)
4462
+ elif res == "skipped_quota" or res == "failed":
4463
  quota_skipped.append(acc)
4464
 
4465
  if quota_skipped:
src/autoteam/quota_predictor.py ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Quota usage predictor — Round 12 S5.
2
+
3
+ Records per-account quota observations to a JSONL log and fits a simple
4
+ least-squares linear regression to estimate when each ACTIVE account's
5
+ ``p_remain`` will hit zero. Used by :func:`autoteam.manager.cmd_rotate`
6
+ to pre-emptively swap soon-to-be-exhausted seats with STANDBY candidates
7
+ *before* the quota actually runs out — minimising user-visible "Team full
8
+ but no usable seat" windows.
9
+
10
+ Design choices
11
+ --------------
12
+ * **Storage** — append-only JSONL at ``PROJECT_ROOT / "quota_history.jsonl"``;
13
+ no DB dependency, survives crashes, easy to tail / inspect / archive.
14
+ * **Algorithm** — minimum viable: ``p_remain = a * ts + b`` via NumPy-free
15
+ closed-form OLS. We only need the *direction* and *zero-crossing*; full
16
+ exponential-decay / piecewise-linear modelling is out of scope (see PRD
17
+ Risk Notes — non-linear 5h reset stairs would distort it anyway, so we
18
+ ship with ``PREDICTIVE_ENABLED=False`` by default).
19
+ * **Safety** — < 3 data points returns ``None`` (no prediction); positive
20
+ slope (quota growing or flat) returns ``None``; any numeric exception is
21
+ swallowed and returns ``None``. The caller treats ``None`` as "do not
22
+ preempt", so the predictor can never *cause* an outage — only fail to
23
+ prevent one.
24
+ """
25
+ from __future__ import annotations
26
+
27
+ import json
28
+ import logging
29
+ import threading
30
+ import time
31
+ from pathlib import Path
32
+ from typing import Any
33
+
34
+ logger = logging.getLogger(__name__)
35
+
36
+
37
+ _PROJECT_ROOT = Path(__file__).parent.parent.parent
38
+ DEFAULT_HISTORY_PATH = _PROJECT_ROOT / "quota_history.jsonl"
39
+
40
+
41
+ class QuotaPredictor:
42
+ """Append-only JSONL store + linear-fit exhaust-time predictor."""
43
+
44
+ def __init__(self, history_path: Path | None = None) -> None:
45
+ self._history_path: Path = Path(history_path) if history_path else DEFAULT_HISTORY_PATH
46
+ # File-level lock — record() and load_history() can be called from
47
+ # the SSE pusher, cmd_check and cmd_rotate threads concurrently.
48
+ self._lock = threading.Lock()
49
+
50
+ @property
51
+ def history_path(self) -> Path:
52
+ return self._history_path
53
+
54
+ # ------------------------------------------------------------------ write
55
+ def record(
56
+ self,
57
+ email: str,
58
+ p_remain: float,
59
+ t_remain: float | None = None,
60
+ ts: float | None = None,
61
+ ) -> None:
62
+ """Append one observation. ``ts`` defaults to ``time.time()``.
63
+
64
+ ``p_remain`` is the "primary quota remaining percent" (0-100).
65
+ ``t_remain`` is the optional "seconds until reset" — recorded for
66
+ future modelling but unused by the linear fit today.
67
+ """
68
+ if not email:
69
+ return
70
+ try:
71
+ row: dict[str, Any] = {
72
+ "email": str(email),
73
+ "p_remain": float(p_remain),
74
+ "t_remain": None if t_remain is None else float(t_remain),
75
+ "ts": float(ts) if ts is not None else time.time(),
76
+ }
77
+ except (TypeError, ValueError):
78
+ logger.warning("quota_predictor.record: bad payload for %s", email)
79
+ return
80
+
81
+ line = json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n"
82
+ with self._lock:
83
+ try:
84
+ self._history_path.parent.mkdir(parents=True, exist_ok=True)
85
+ with open(self._history_path, "a", encoding="utf-8") as fp:
86
+ fp.write(line)
87
+ except OSError:
88
+ logger.exception("quota_predictor.record: cannot append to %s", self._history_path)
89
+
90
+ # ------------------------------------------------------------------- read
91
+ def load_history(self, email: str, *, max_points: int = 50) -> list[dict[str, Any]]:
92
+ """Return chronologically-sorted observations for ``email``.
93
+
94
+ Truncated to the most recent ``max_points`` rows so long-running
95
+ deployments do not blow up memory. Malformed lines are skipped
96
+ silently — JSONL is append-only and a partial line is recoverable
97
+ on the next successful append.
98
+ """
99
+ if not email or max_points <= 0:
100
+ return []
101
+ path = self._history_path
102
+ if not path.exists():
103
+ return []
104
+ rows: list[dict[str, Any]] = []
105
+ with self._lock:
106
+ try:
107
+ with open(path, encoding="utf-8") as fp:
108
+ for raw in fp:
109
+ raw = raw.strip()
110
+ if not raw:
111
+ continue
112
+ try:
113
+ obj = json.loads(raw)
114
+ except json.JSONDecodeError:
115
+ continue
116
+ if obj.get("email") != email:
117
+ continue
118
+ try:
119
+ obj["ts"] = float(obj.get("ts", 0.0))
120
+ obj["p_remain"] = float(obj.get("p_remain", 0.0))
121
+ except (TypeError, ValueError):
122
+ continue
123
+ rows.append(obj)
124
+ except OSError:
125
+ logger.exception("quota_predictor.load_history: cannot read %s", path)
126
+ return []
127
+ rows.sort(key=lambda r: r["ts"])
128
+ if len(rows) > max_points:
129
+ rows = rows[-max_points:]
130
+ return rows
131
+
132
+ # ------------------------------------------------------------------- fit
133
+ def predict_exhaust_time(self, email: str) -> float | None:
134
+ """Predict the wall-clock ts at which ``p_remain`` crosses 0.
135
+
136
+ Returns ``None`` when:
137
+ * history < 3 points (insufficient signal);
138
+ * slope >= 0 (quota stable or growing — nothing to preempt);
139
+ * numeric instability (degenerate variance, NaN);
140
+ * caught exception (best-effort — predictor never raises).
141
+ """
142
+ rows = self.load_history(email)
143
+ if len(rows) < 3:
144
+ return None
145
+
146
+ try:
147
+ xs = [r["ts"] for r in rows]
148
+ ys = [r["p_remain"] for r in rows]
149
+ n = len(xs)
150
+ mean_x = sum(xs) / n
151
+ mean_y = sum(ys) / n
152
+ num = sum((xs[i] - mean_x) * (ys[i] - mean_y) for i in range(n))
153
+ den = sum((xs[i] - mean_x) ** 2 for i in range(n))
154
+ if den == 0:
155
+ return None
156
+ slope = num / den
157
+ if slope >= 0:
158
+ # quota constant or growing — no exhaust prediction.
159
+ return None
160
+ intercept = mean_y - slope * mean_x
161
+ # solve 0 = slope * t + intercept → t = -intercept / slope
162
+ t_exhaust = -intercept / slope
163
+ if not (t_exhaust == t_exhaust): # NaN check
164
+ return None
165
+ return float(t_exhaust)
166
+ except (TypeError, ValueError, ZeroDivisionError):
167
+ logger.exception("quota_predictor.predict_exhaust_time: fit failed for %s", email)
168
+ return None
169
+
170
+ # --------------------------------------------------------------- decision
171
+ def should_preempt(
172
+ self,
173
+ email: str,
174
+ lead_minutes: float,
175
+ *,
176
+ now: float | None = None,
177
+ ) -> bool:
178
+ """Return True when predicted exhaust is within ``lead_minutes`` of now."""
179
+ t_exhaust = self.predict_exhaust_time(email)
180
+ if t_exhaust is None:
181
+ return False
182
+ current = float(now) if now is not None else time.time()
183
+ return (t_exhaust - current) < float(lead_minutes) * 60.0
184
+
185
+
186
+ # ---------------------------------------------------------------------------
187
+ # Module-level default predictor — production code uses this; tests inject
188
+ # their own QuotaPredictor instance with a tmp_path history_path fixture.
189
+ # ---------------------------------------------------------------------------
190
+ default_predictor = QuotaPredictor()
191
+
192
+
193
+ __all__ = [
194
+ "DEFAULT_HISTORY_PATH",
195
+ "QuotaPredictor",
196
+ "default_predictor",
197
+ ]
tests/unit/test_round12_s5_predictor.py ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Round 12 S5 — Quota predictor (linear-fit exhaust-time prediction).
2
+
3
+ Verifies the predictor behaviour described in
4
+ `.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md`:
5
+
6
+ * < 3 history points → predict_exhaust_time returns None
7
+ * linear decreasing samples → predict within ±10 percent of the analytic root
8
+ * flat / increasing slope → None
9
+ * should_preempt honours the lead window
10
+ * record + load_history roundtrip is correct
11
+ * load_history(max_points=N) truncates to the most recent N
12
+ """
13
+ from __future__ import annotations
14
+
15
+ import json
16
+ import time
17
+
18
+ import pytest
19
+
20
+ from autoteam.quota_predictor import QuotaPredictor
21
+
22
+
23
+ @pytest.fixture
24
+ def predictor(tmp_path):
25
+ return QuotaPredictor(history_path=tmp_path / "quota_history.jsonl")
26
+
27
+
28
+ class TestRecordAndLoad:
29
+ def test_record_creates_jsonl_file_and_roundtrip(self, predictor):
30
+ ts0 = 1_700_000_000.0
31
+ predictor.record("a@x.com", 90.0, t_remain=3600, ts=ts0)
32
+ predictor.record("a@x.com", 80.0, t_remain=3000, ts=ts0 + 60)
33
+ predictor.record("b@x.com", 50.0, ts=ts0 + 30)
34
+
35
+ history_a = predictor.load_history("a@x.com")
36
+ history_b = predictor.load_history("b@x.com")
37
+ assert [r["p_remain"] for r in history_a] == [90.0, 80.0]
38
+ assert [r["ts"] for r in history_a] == [ts0, ts0 + 60]
39
+ assert len(history_b) == 1 and history_b[0]["p_remain"] == 50.0
40
+
41
+ def test_load_history_skips_malformed_lines(self, predictor):
42
+ # Manually append a malformed line — load_history must keep working.
43
+ predictor.record("a@x.com", 90.0, ts=1.0)
44
+ with open(predictor.history_path, "a", encoding="utf-8") as fp:
45
+ fp.write("{NOT_JSON\n")
46
+ fp.write("\n") # blank line too
47
+ predictor.record("a@x.com", 85.0, ts=2.0)
48
+ history = predictor.load_history("a@x.com")
49
+ assert [r["p_remain"] for r in history] == [90.0, 85.0]
50
+
51
+ def test_load_history_max_points_truncates_to_most_recent(self, predictor):
52
+ for i in range(100):
53
+ predictor.record("a@x.com", 100.0 - i, ts=float(i))
54
+ history = predictor.load_history("a@x.com", max_points=5)
55
+ assert len(history) == 5
56
+ assert [r["ts"] for r in history] == [95.0, 96.0, 97.0, 98.0, 99.0]
57
+
58
+ def test_record_with_empty_email_is_noop(self, predictor):
59
+ predictor.record("", 50.0, ts=1.0)
60
+ assert not predictor.history_path.exists()
61
+
62
+ def test_record_bad_payload_does_not_raise(self, predictor):
63
+ # p_remain unconvertible to float — log warning, no crash, no row written.
64
+ predictor.record("a@x.com", "not-a-number", ts=1.0) # type: ignore[arg-type]
65
+ assert not predictor.history_path.exists() or predictor.load_history("a@x.com") == []
66
+
67
+
68
+ class TestPredictExhaust:
69
+ def test_insufficient_history_returns_none(self, predictor):
70
+ predictor.record("a@x.com", 90.0, ts=1.0)
71
+ predictor.record("a@x.com", 80.0, ts=2.0)
72
+ assert predictor.predict_exhaust_time("a@x.com") is None
73
+
74
+ def test_linear_decline_predicts_within_ten_percent(self, predictor):
75
+ # Quota drops 5% every 60s starting from 100% at t=0 → exhaust at t = 100/5 * 60 = 1200.
76
+ for i in range(10):
77
+ predictor.record("a@x.com", 100.0 - i * 5.0, ts=float(i * 60))
78
+ predicted = predictor.predict_exhaust_time("a@x.com")
79
+ assert predicted is not None
80
+ # exact analytic root is 1140 (last point is t=540, p_remain=55 → projects to 0 at t=1200
81
+ # actually fitting all 10 points: linear through (0,100), (60,95), ..., (540,55) → exhaust at t=1200
82
+ analytic_root = 1200.0
83
+ assert abs(predicted - analytic_root) / analytic_root < 0.10
84
+
85
+ def test_flat_history_returns_none(self, predictor):
86
+ for i in range(5):
87
+ predictor.record("a@x.com", 80.0, ts=float(i * 60))
88
+ assert predictor.predict_exhaust_time("a@x.com") is None
89
+
90
+ def test_increasing_slope_returns_none(self, predictor):
91
+ for i in range(5):
92
+ predictor.record("a@x.com", 50.0 + i * 5.0, ts=float(i * 60))
93
+ # quota growing — no exhaust prediction
94
+ assert predictor.predict_exhaust_time("a@x.com") is None
95
+
96
+
97
+ class TestShouldPreempt:
98
+ def test_within_lead_window_returns_true(self, predictor):
99
+ # Set up exhaust prediction at t=600 (10 minutes from now=0).
100
+ # 5 points: (0, 100), (60, 90), (120, 80), (180, 70), (240, 60) → linear to 0 at t=600
101
+ for i in range(5):
102
+ predictor.record("a@x.com", 100.0 - i * 10.0, ts=float(i * 60))
103
+ # lead=15min, now=0 → exhaust at t=600 < 15*60=900 → True
104
+ assert predictor.should_preempt("a@x.com", lead_minutes=15, now=0.0)
105
+
106
+ def test_outside_lead_window_returns_false(self, predictor):
107
+ # exhaust at t=600, but lead=5min (300s) → 600 - 0 = 600 > 300 → False
108
+ for i in range(5):
109
+ predictor.record("a@x.com", 100.0 - i * 10.0, ts=float(i * 60))
110
+ assert not predictor.should_preempt("a@x.com", lead_minutes=5, now=0.0)
111
+
112
+ def test_unpredictable_returns_false(self, predictor):
113
+ predictor.record("a@x.com", 90.0, ts=1.0)
114
+ predictor.record("a@x.com", 90.0, ts=2.0)
115
+ # <3 points: unpredictable
116
+ assert not predictor.should_preempt("a@x.com", lead_minutes=999, now=0.0)
117
+
118
+ def test_default_now_uses_walltime(self, predictor, monkeypatch):
119
+ """now=None falls through to time.time()."""
120
+ # Setup historical exhaust well in the past → guaranteed within any lead window.
121
+ base = time.time()
122
+ for i in range(5):
123
+ predictor.record("a@x.com", 100.0 - i * 50.0, ts=base - 300 + i * 30)
124
+ # slope is steep negative → predicted exhaust is in the past → should_preempt=True for any lead
125
+ assert predictor.should_preempt("a@x.com", lead_minutes=1)
126
+
127
+
128
+ class TestFileFormat:
129
+ def test_jsonl_schema_matches_prd(self, predictor):
130
+ predictor.record("user@example.com", 50.0, t_remain=7200.0, ts=1_700_000_000.0)
131
+ line = predictor.history_path.read_text(encoding="utf-8").strip()
132
+ obj = json.loads(line)
133
+ assert set(obj.keys()) >= {"email", "p_remain", "t_remain", "ts"}
134
+ assert obj["email"] == "user@example.com"
135
+ assert obj["p_remain"] == 50.0
136
+ assert obj["t_remain"] == 7200.0
137
+ assert obj["ts"] == 1_700_000_000.0
tests/unit/test_round12_s6_concurrent.py ADDED
@@ -0,0 +1,261 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Round 12 S6 — concurrent batch standby reuse in cmd_rotate.
2
+
3
+ Verifies the behaviour described in
4
+ `.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md`:
5
+
6
+ * ROTATE_CONCURRENCY=1 keeps serial behaviour (calls in candidate order).
7
+ * ROTATE_CONCURRENCY>=N triggers concurrent execution
8
+ (multiple in-flight tasks at the same time, verified via latch counter).
9
+ * Per-seat exception → result="failed", does not poison sibling seats.
10
+ * _reuse_one_standby returns a stable result vocabulary.
11
+ * Concurrent update_account calls are serialized by _accounts_io_lock
12
+ (10 worker threads → all 10 increments land).
13
+ * Cancellation gracefully stops dispatching new tasks.
14
+ """
15
+ from __future__ import annotations
16
+
17
+ import threading
18
+ import time
19
+ from pathlib import Path
20
+ from unittest.mock import MagicMock
21
+
22
+ import pytest
23
+
24
+ from autoteam import accounts as accounts_mod
25
+ from autoteam import manager as manager_mod
26
+ from autoteam.account_state import default_machine
27
+ from autoteam.accounts import _accounts_io_lock
28
+ from autoteam.manager import _STANDBY_REUSE_RESULTS, _reuse_one_standby
29
+
30
+
31
+ @pytest.fixture
32
+ def isolated_accounts(tmp_path: Path, monkeypatch):
33
+ accounts_file = tmp_path / "accounts.json"
34
+ log_file = tmp_path / "state_log.jsonl"
35
+ monkeypatch.setattr(accounts_mod, "ACCOUNTS_FILE", accounts_file)
36
+ monkeypatch.setattr(accounts_mod, "get_admin_email", lambda: "")
37
+ monkeypatch.setattr(manager_mod, "get_admin_email", lambda: "")
38
+ original_log = default_machine._log_path
39
+ default_machine._log_path = log_file
40
+ yield accounts_file
41
+ default_machine._log_path = original_log
42
+
43
+
44
+ # ---------------------------------------------------------------------------
45
+ # 1. _reuse_one_standby unit behaviour
46
+ # ---------------------------------------------------------------------------
47
+ class TestReuseOneStandby:
48
+ def test_result_vocabulary_is_stable(self):
49
+ assert _STANDBY_REUSE_RESULTS == frozenset(
50
+ {"reused", "skipped_quota", "skipped_auto", "failed"}
51
+ )
52
+
53
+ def test_skipped_auto_reason_short_circuits(self, monkeypatch):
54
+ """_auto_reuse_skip_reason returning non-None → skipped_auto."""
55
+ monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: "暂停自动复用")
56
+ result = _reuse_one_standby(
57
+ {"email": "a@x.com"},
58
+ threshold=10,
59
+ chatgpt_provider=lambda: None,
60
+ mail_provider=lambda acc: None,
61
+ reinvite_fn=lambda *a, **kw: True,
62
+ quota_fn=lambda token: ("ok", {"primary_pct": 0}),
63
+ now=1000.0,
64
+ )
65
+ assert result == {"email": "a@x.com", "result": "skipped_auto", "error": None}
66
+
67
+ def test_quota_ok_calls_reinvite_returns_reused(self, monkeypatch, tmp_path):
68
+ monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
69
+ auth_path = tmp_path / "auth.json"
70
+ auth_path.write_text('{"access_token":"X"}')
71
+ reinvite = MagicMock(return_value=True)
72
+ result = _reuse_one_standby(
73
+ {"email": "a@x.com", "auth_file": str(auth_path)},
74
+ threshold=10,
75
+ chatgpt_provider=lambda: "FAKE_CHATGPT",
76
+ mail_provider=lambda acc: "FAKE_MAIL",
77
+ reinvite_fn=reinvite,
78
+ quota_fn=lambda token: ("ok", {"primary_pct": 5}), # 95% remain >> 10%
79
+ now=1000.0,
80
+ )
81
+ assert result == {"email": "a@x.com", "result": "reused", "error": None}
82
+ reinvite.assert_called_once()
83
+
84
+ def test_reinvite_returns_false_yields_failed(self, monkeypatch, tmp_path):
85
+ monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
86
+ auth_path = tmp_path / "auth.json"
87
+ auth_path.write_text('{"access_token":"X"}')
88
+ result = _reuse_one_standby(
89
+ {"email": "a@x.com", "auth_file": str(auth_path)},
90
+ threshold=10,
91
+ chatgpt_provider=lambda: None,
92
+ mail_provider=lambda acc: None,
93
+ reinvite_fn=lambda *a, **kw: False,
94
+ quota_fn=lambda token: ("ok", {"primary_pct": 5}),
95
+ now=1000.0,
96
+ )
97
+ assert result["result"] == "failed"
98
+ assert "reinvite_account returned False" in (result["error"] or "")
99
+
100
+ def test_exception_caught_and_converted_to_failed(self, monkeypatch):
101
+ monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
102
+
103
+ def bad_reinvite(*a, **kw):
104
+ raise RuntimeError("network exploded")
105
+
106
+ result = _reuse_one_standby(
107
+ {"email": "a@x.com"}, # no auth_file → quota_ok=False, falls to lq branch
108
+ threshold=10,
109
+ chatgpt_provider=lambda: None,
110
+ mail_provider=lambda acc: None,
111
+ reinvite_fn=bad_reinvite,
112
+ quota_fn=lambda token: ("ok", {"primary_pct": 5}),
113
+ now=1000.0,
114
+ )
115
+ # acc has no last_quota or quota_resets_at → quota_ok loop passes through →
116
+ # reinvite called → raises → caught → result=failed.
117
+ assert result["result"] == "failed"
118
+ assert "network exploded" in (result["error"] or "")
119
+
120
+ def test_exhausted_quota_returns_skipped_quota(self, monkeypatch, tmp_path):
121
+ monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
122
+ # patch update_account so we don't hit accounts.json file
123
+ monkeypatch.setattr(manager_mod, "update_account", lambda *a, **kw: None)
124
+ auth_path = tmp_path / "auth.json"
125
+ auth_path.write_text('{"access_token":"X"}')
126
+ result = _reuse_one_standby(
127
+ {"email": "a@x.com", "auth_file": str(auth_path)},
128
+ threshold=10,
129
+ chatgpt_provider=lambda: None,
130
+ mail_provider=lambda acc: None,
131
+ reinvite_fn=lambda *a, **kw: True,
132
+ quota_fn=lambda token: ("exhausted", {"window": "5h", "primary_pct": 100}),
133
+ now=1000.0,
134
+ )
135
+ assert result["result"] == "skipped_quota"
136
+
137
+
138
+ # ---------------------------------------------------------------------------
139
+ # 2. Concurrency observability (ThreadPoolExecutor really runs in parallel)
140
+ # ---------------------------------------------------------------------------
141
+ class TestConcurrencyOrchestration:
142
+ def test_serial_mode_with_concurrency_one(self, monkeypatch):
143
+ """ROTATE_CONCURRENCY=1 → no ThreadPoolExecutor used, calls in candidate order."""
144
+ # We assert this indirectly by checking the call order matches input order.
145
+ call_order: list[str] = []
146
+
147
+ def fake_reuse(acc, threshold, **kw):
148
+ call_order.append(acc["email"])
149
+ time.sleep(0.01)
150
+ return {"email": acc["email"], "result": "skipped_quota", "error": None}
151
+
152
+ candidates = [{"email": f"a{i}@x.com"} for i in range(5)]
153
+ # Replicate the serial branch from cmd_rotate.
154
+ outcomes = [fake_reuse(acc, 10) for acc in candidates]
155
+ assert call_order == ["a0@x.com", "a1@x.com", "a2@x.com", "a3@x.com", "a4@x.com"]
156
+ assert all(o["result"] == "skipped_quota" for o in outcomes)
157
+
158
+ def test_concurrent_mode_runs_tasks_in_parallel(self):
159
+ """ROTATE_CONCURRENCY>=3 → at least 2 tasks run simultaneously (latch counter)."""
160
+ import concurrent.futures
161
+
162
+ inflight = 0
163
+ inflight_lock = threading.Lock()
164
+ max_inflight = 0
165
+
166
+ def fake_reuse(acc):
167
+ nonlocal inflight, max_inflight
168
+ with inflight_lock:
169
+ inflight += 1
170
+ max_inflight = max(max_inflight, inflight)
171
+ time.sleep(0.05) # simulate IO-bound mail wait
172
+ with inflight_lock:
173
+ inflight -= 1
174
+ return {"email": acc["email"], "result": "reused", "error": None}
175
+
176
+ candidates = [{"email": f"a{i}@x.com"} for i in range(5)]
177
+ with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
178
+ results = list(pool.map(fake_reuse, candidates))
179
+
180
+ assert max_inflight >= 2, f"concurrency not observed: max_inflight={max_inflight}"
181
+ assert len(results) == 5
182
+ assert all(r["result"] == "reused" for r in results)
183
+
184
+ def test_one_task_exception_does_not_poison_others(self):
185
+ """Failed worker → result=failed in aggregated list; siblings still complete."""
186
+ import concurrent.futures
187
+
188
+ def fake_reuse(acc):
189
+ if acc["email"] == "boom@x.com":
190
+ raise RuntimeError("seat blew up")
191
+ return {"email": acc["email"], "result": "reused", "error": None}
192
+
193
+ candidates = [{"email": "a@x.com"}, {"email": "boom@x.com"}, {"email": "b@x.com"}]
194
+ outcomes = []
195
+ with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
196
+ future_map = {pool.submit(fake_reuse, acc): acc for acc in candidates}
197
+ for fut in concurrent.futures.as_completed(future_map):
198
+ try:
199
+ outcomes.append(fut.result())
200
+ except Exception as exc:
201
+ outcomes.append(
202
+ {"email": future_map[fut]["email"], "result": "failed", "error": str(exc)}
203
+ )
204
+
205
+ by_email = {o["email"]: o for o in outcomes}
206
+ assert by_email["a@x.com"]["result"] == "reused"
207
+ assert by_email["b@x.com"]["result"] == "reused"
208
+ assert by_email["boom@x.com"]["result"] == "failed"
209
+ assert "seat blew up" in by_email["boom@x.com"]["error"]
210
+
211
+
212
+ # ---------------------------------------------------------------------------
213
+ # 3. _accounts_io_lock guarantees no lost writes under concurrent update_account
214
+ # ---------------------------------------------------------------------------
215
+ class TestAccountsIoLock:
216
+ def test_lock_is_reentrant(self):
217
+ """RLock chosen so that update_account → add_account-style re-entry works."""
218
+ assert _accounts_io_lock.acquire(blocking=False)
219
+ try:
220
+ # second acquire from the same thread must succeed (RLock semantics)
221
+ assert _accounts_io_lock.acquire(blocking=False)
222
+ _accounts_io_lock.release()
223
+ finally:
224
+ _accounts_io_lock.release()
225
+
226
+ def test_concurrent_update_account_no_lost_writes(self, isolated_accounts):
227
+ """10 threads update_account in parallel — all 10 fields land on disk.
228
+
229
+ This is the core safety property for ROTATE_CONCURRENCY>1: without
230
+ the lock, the load → mutate → save RMW race would lose updates.
231
+ """
232
+ from autoteam.accounts import (
233
+ STATUS_STANDBY,
234
+ add_account,
235
+ load_accounts,
236
+ update_account,
237
+ )
238
+
239
+ # Seed 10 standby accounts.
240
+ for i in range(10):
241
+ add_account(email=f"acc{i}@x.com", password="pw")
242
+ # add_account starts in PENDING — transition to STANDBY via update_account
243
+ update_account(f"acc{i}@x.com", status=STATUS_STANDBY)
244
+
245
+ # Now spawn 10 threads, each updates a different acc field concurrently.
246
+ def worker(i: int):
247
+ update_account(f"acc{i}@x.com", last_quota={"primary_pct": i * 10})
248
+
249
+ threads = [threading.Thread(target=worker, args=(i,)) for i in range(10)]
250
+ for t in threads:
251
+ t.start()
252
+ for t in threads:
253
+ t.join()
254
+
255
+ final = {a["email"]: a for a in load_accounts()}
256
+ assert len(final) == 10
257
+ for i in range(10):
258
+ acc = final[f"acc{i}@x.com"]
259
+ assert acc.get("last_quota", {}).get("primary_pct") == i * 10, (
260
+ f"acc{i} lost update: {acc.get('last_quota')}"
261
+ )