Spaces:
Paused
feat(round-12 S5+S6): predictive preempt + concurrent rotate
Browse filesS5 - 预测式抢先替换:
- 新增 src/autoteam/quota_predictor.py: QuotaPredictor 累积配额历史
到 quota_history.jsonl, 最小二乘线性拟合预测耗尽时刻;
- record() / load_history(max_points) / predict_exhaust_time() /
should_preempt(lead_minutes)
- < 3 数据点 / slope >= 0 / 数值异常 → None (永不导致误触发)
- cmd_rotate 加 [2.5/5] 预测分支(PREDICTIVE_ENABLED 默认 false 安全):
遍历 ACTIVE 子号, record 当前 last_quota → should_preempt 命中
则 remove_from_team + update_account(STANDBY, _reason="predictive_preempt"),
让 S3 既有 vacancy 兜底自动接管替换流程.
- 新增 config: PREDICTIVE_ENABLED / PREDICTIVE_LEAD_MIN(默认 15) /
PREDICTIVE_HISTORY_FILE.
S6 - 并发批量替换:
- 抽出模块级 _reuse_one_standby(acc, threshold, *, chatgpt_provider,
mail_provider, reinvite_fn, quota_fn, now) — 单席位完整复用流程,
失败收敛成 result={"reused"|"skipped_quota"|"skipped_auto"|"failed"},
绝不向上抛(并发模式下不能让单席位毒化兄弟).
- cmd_rotate standby 复用循环改造: ROTATE_CONCURRENCY<=1 走串行
(完全等同改造前老行为, 测试可复现); >=2 走 ThreadPoolExecutor
动态 submit(达到 vacancies 即停止派发, 等剩余 in-flight 收尾).
- accounts.py 新增 _accounts_io_lock (RLock) 包裹 load → mutate → save
RMW 段, 保证并发 update_account / add_account / delete_account
不丢写. state_machine.transition 内部已有自己的锁负责 state_log.
- 新增 config: ROTATE_CONCURRENCY(默认 1 向后兼容, 上限 8 防过载).
Tests (合计 +25, 远超 ≥12 目标):
- test_round12_s5_predictor.py 14 case: record/load_history roundtrip,
malformed line skip, max_points 截尾, 线性拟合误差 < ±10%, slope/flat/
insufficient 返回 None, should_preempt 内/外 lead window.
- test_round12_s6_concurrent.py 11 case: _reuse_one_standby 6 分支结果,
serial 顺序保持, ThreadPoolExecutor max_inflight >= 2 实测并发,
单 task 异常不毒化兄弟, _accounts_io_lock 并发 10 worker 不丢写.
DoD:
- ruff check 全绿 (src/autoteam/ + tests/unit/)
- pytest 683 passed (baseline 658 + 25 新增, 无回归)
- ROTATE_CONCURRENCY=1 时与改造前行为完全等同
(test_cmd_rotate_skips_google_accounts 验证)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- .trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/check.jsonl +4 -0
- .trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/implement.jsonl +10 -0
- .trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md +193 -0
- .trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/task.json +26 -0
- src/autoteam/accounts.py +73 -55
- src/autoteam/config.py +17 -0
- src/autoteam/manager.py +268 -102
- src/autoteam/quota_predictor.py +197 -0
- tests/unit/test_round12_s5_predictor.py +137 -0
- tests/unit/test_round12_s6_concurrent.py +261 -0
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"file": ".trellis/spec/backend/quality-guidelines.md", "reason": "lint / type-check 标准"}
|
| 2 |
+
{"file": ".trellis/spec/backend/error-handling.md", "reason": "异常处理风格"}
|
| 3 |
+
{"file": ".trellis/spec/backend/logging-guidelines.md", "reason": "日志格式约定"}
|
| 4 |
+
{"file": ".trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md", "reason": "本任务 DoD / Acceptance Criteria"}
|
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{"file": ".trellis/tasks/05-11-upstream-align-register-multimail-frontend-refresh/prd.md", "reason": "父 PRD: Q2 创新点 1+2 范围 + S5/S6 调度上下文"}
|
| 2 |
+
{"file": ".trellis/tasks/05-11-s0-upstream-team-rotate-diff/research/upstream-diff.md", "reason": "上游 diff 报告: cmd_rotate 9 函数现状 + 双指标终止条件"}
|
| 3 |
+
{"file": ".trellis/tasks/05-11-s3-cherry-pick-team-rotate/prd.md", "reason": "S3 cherry-pick 后 manager.py 现状 (helper + auth_repair)"}
|
| 4 |
+
{"file": "src/autoteam/account_state.py", "reason": "S1 状态机 default_machine.transition / IllegalTransitionError API"}
|
| 5 |
+
{"file": "src/autoteam/accounts.py", "reason": "update_account 统一路由 + 即将新增 _accounts_io_lock"}
|
| 6 |
+
{"file": "src/autoteam/manager.py", "reason": "cmd_rotate S3 cherry-pick 后实体 (要在其中加预测分支 + 并发)"}
|
| 7 |
+
{"file": "src/autoteam/config.py", "reason": "env 配置范式 (_get_int_env / _get_bool_env)"}
|
| 8 |
+
{"file": ".upstream/manager.py", "reason": "上游 cmd_rotate baseline 行 2178-2520"}
|
| 9 |
+
{"file": ".trellis/spec/backend/error-handling.md", "reason": "异常处理风格"}
|
| 10 |
+
{"file": ".trellis/spec/backend/logging-guidelines.md", "reason": "日志格式约定"}
|
|
@@ -0,0 +1,193 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# S5 + S6 — 预测式替换 + 并发批量替换
|
| 2 |
+
|
| 3 |
+
## 父 PRD 引用
|
| 4 |
+
|
| 5 |
+
继承 `05-11-upstream-align-register-multimail-frontend-refresh/prd.md`
|
| 6 |
+
Q2 创新点 1(预测式替换)+ 创新点 2(并发批量替换)。基础设施依赖
|
| 7 |
+
round-12 已落地的 S1(`account_state.py` 状态机 / `update_account` 统一路由)
|
| 8 |
+
+ S3(`cmd_rotate` 双指标终止条件 + `ensure_account_mail` per-account
|
| 9 |
+
路由 + `_count_pool_active_accounts`)。
|
| 10 |
+
|
| 11 |
+
## Goal
|
| 12 |
+
|
| 13 |
+
1. **S5**:新增 `quota_predictor.py` —— 累积配额历史时序,基于线性回归
|
| 14 |
+
预测每个 ACTIVE 子号"接近耗尽"的时刻;在 `cmd_rotate` 主循环里加
|
| 15 |
+
"预测式抢先替换"分支:剩余时间 < `PREDICTIVE_LEAD_MIN` 分钟时主动
|
| 16 |
+
`update_account(email, status=STANDBY)`,让 S3 既有 vacancy 兜底自动
|
| 17 |
+
触发替换。
|
| 18 |
+
2. **S6**:把 `cmd_rotate` 的"复用 standby + 新建账号"两段串行循环改为
|
| 19 |
+
`ThreadPoolExecutor` 并发;`ROTATE_CONCURRENCY` env 控制最大并发,
|
| 20 |
+
默认 1(向后兼容)。每席位独立 try/except,失败聚合输出,互不阻塞。
|
| 21 |
+
|
| 22 |
+
## Scope
|
| 23 |
+
|
| 24 |
+
### S5 — 预测式抢先替换
|
| 25 |
+
|
| 26 |
+
新建 `src/autoteam/quota_predictor.py`:
|
| 27 |
+
|
| 28 |
+
- `class QuotaPredictor`
|
| 29 |
+
- `__init__(history_path: Path | None = None)`
|
| 30 |
+
- `record(email: str, p_remain: float, t_remain: float | None, ts: float | None = None) -> None`
|
| 31 |
+
- 追加一行到 `quota_history.jsonl`:
|
| 32 |
+
`{"email":"x","p_remain":50,"t_remain":12345,"ts":1736...}`
|
| 33 |
+
- 文件不存在自动创建;t_remain 为 None 时写 null。
|
| 34 |
+
- `load_history(email: str, *, max_points: int = 50) -> list[dict]`
|
| 35 |
+
- 读全部历史,按 email 过滤,按 ts 升序排序,截尾保留最近
|
| 36 |
+
`max_points` 个点(避免长跑期间历史无限增长)。
|
| 37 |
+
- `predict_exhaust_time(email: str) -> float | None`
|
| 38 |
+
- 历史 < 3 点 → 返回 None(不做预测)。
|
| 39 |
+
- 用 `p_remain`(百分比)vs `ts` 做最小二乘线性拟合:
|
| 40 |
+
`p_remain = a * ts + b`;
|
| 41 |
+
- `slope a >= 0` → 配额在增长或不变 → 返回 None。
|
| 42 |
+
- `slope a < 0` → 解 `0 = a * t_exhaust + b` → 返回 `t_exhaust`。
|
| 43 |
+
- 任何异常(数值不稳定 / 全零) → 返回 None。
|
| 44 |
+
- `should_preempt(email: str, lead_minutes: float, *, now: float | None = None) -> bool`
|
| 45 |
+
- 计算 `t_exhaust = predict_exhaust_time(email)`;
|
| 46 |
+
- 返回 `t_exhaust is not None and (t_exhaust - now) < lead_minutes * 60`。
|
| 47 |
+
|
| 48 |
+
- 配置(`config.py` 新增):
|
| 49 |
+
- `PREDICTIVE_ENABLED: bool` env `PREDICTIVE_ENABLED` 默认 `False`
|
| 50 |
+
(安全默认 — 不预测,按 round-9~12 旧行为)。
|
| 51 |
+
- `PREDICTIVE_LEAD_MIN: int` env `PREDICTIVE_LEAD_MIN` 默认 `15`。
|
| 52 |
+
- `PREDICTIVE_HISTORY_FILE: Path` 默认 `PROJECT_ROOT / "quota_history.jsonl"`。
|
| 53 |
+
|
| 54 |
+
- `cmd_rotate` 接入(manager.py):
|
| 55 |
+
- 在 `[2/5] cmd_check()` 之后、`[3/5]` 移出 exhausted 之前,加新分支
|
| 56 |
+
`[2.5/5] 预测式抢先替换`:
|
| 57 |
+
- 仅当 `PREDICTIVE_ENABLED` 为 True 才执行;
|
| 58 |
+
- 遍历所有 ACTIVE 非主号;
|
| 59 |
+
- 调用 `predictor.should_preempt(email, PREDICTIVE_LEAD_MIN)`;
|
| 60 |
+
- True → `update_account(email, status=STATUS_STANDBY,
|
| 61 |
+
_reason="predictive_preempt")`,日志记录。
|
| 62 |
+
- 不直接 remove_from_team —— 让 S3 双指标兜底 + 后续 [3/5]
|
| 63 |
+
`STATUS_EXHAUSTED` 移出循环不触发(因为我们打的是 STANDBY 而非
|
| 64 |
+
EXHAUSTED);这里要走"先 STANDBY → vacancy 计算自动 + 1 →
|
| 65 |
+
[5/5] 新号填补"流程,所以我们需要紧接着 remove_from_team。
|
| 66 |
+
- 修正:`STATUS_STANDBY` 本身是"已移出 Team"语义,必须先 kick;
|
| 67 |
+
所以分支调用顺序:
|
| 68 |
+
a. `remove_from_team(chatgpt, email, return_status=True)`
|
| 69 |
+
b. 成功 → `update_account(email, status=STATUS_STANDBY,
|
| 70 |
+
_reason="predictive_preempt")` + 日志。
|
| 71 |
+
|
| 72 |
+
- `cmd_check` 钩子(manager.py):每次 ChatGPT API quota 探测成功后
|
| 73 |
+
调用 `predictor.record(email, p_remain, t_remain, ts=time.time())`。
|
| 74 |
+
- 已有 `update_account(email, last_quota=quota_info)` 写入点 ~多处,
|
| 75 |
+
我们在最具代表性的"标准成功路径"加一次 record,避免重复记录。
|
| 76 |
+
- 若 cmd_check 路径复杂 → 至少在 `cmd_rotate` 的预测分支前先批量
|
| 77 |
+
record 一次(用 acc.last_quota),最低代价保证测试可复现。
|
| 78 |
+
|
| 79 |
+
### S6 — 并发批量替换
|
| 80 |
+
|
| 81 |
+
- `config.py` 新增 `ROTATE_CONCURRENCY: int` env `ROTATE_CONCURRENCY`
|
| 82 |
+
默认 `1`(向后兼容串行)。
|
| 83 |
+
- `manager.py` 改造 `cmd_rotate`:
|
| 84 |
+
- **复用阶段**(standby 循环):把单循环的"额度校验 + reinvite_account"
|
| 85 |
+
抽成 `_reuse_one_standby(acc, chatgpt, mail_for_acc, threshold) -> dict`
|
| 86 |
+
(返回 `{"email","result":"reused|skipped_quota|skipped_auto|failed",
|
| 87 |
+
"error":str|None}`);用 `ThreadPoolExecutor(max_workers=ROTATE_CONCURRENCY)`
|
| 88 |
+
并行处理 standby_list(按 vacancies 截断)。
|
| 89 |
+
- **新建阶段**(fill new accounts):同样改造为
|
| 90 |
+
`_create_one_new_account(chatgpt, mail_client) -> dict`,
|
| 91 |
+
并发 `remaining` 个任务。
|
| 92 |
+
- **线程安全保证**:
|
| 93 |
+
- `update_account` / `load_accounts` 已通过 `default_machine.transition`
|
| 94 |
+
内部 `threading.Lock` 串行写入 state_log;新增
|
| 95 |
+
`_accounts_io_lock = threading.Lock()` 包裹 `load_accounts` +
|
| 96 |
+
`save_accounts` 这对 read-modify-write(避免并发写覆盖)。
|
| 97 |
+
- `chatgpt` 与 `mail_client` 不能并发访问 Playwright browser
|
| 98 |
+
—— 我们在并发执行前**串行**完成所有 ChatGPT API 调用(reinvite
|
| 99 |
+
/ remove_from_team),仅把"邮件等待 + OTP 提取 + accounts.json
|
| 100 |
+
写入"这种 IO bound 部分并发。
|
| 101 |
+
- 修正实现:本任务**仅 IO bound 部分并发** —— 即"邮件 wait_email
|
| 102 |
+
+ OTP 解析"(每席位独立 mail 客户端,因 S3 `ensure_account_mail`
|
| 103 |
+
已 per-acc 缓存)。`reinvite_account` 内部完整链路本来就含
|
| 104 |
+
browser 调用 → 必须以"全 mail client 独立"为前提才能安全并发。
|
| 105 |
+
- 决策:**S6 仅并发 standby 复用阶段 + 仅当 acc 有独立
|
| 106 |
+
mail_provider 绑定时才并发**;新建阶段(创新号)保持串行,
|
| 107 |
+
因 ChatGPT browser session 共享。
|
| 108 |
+
- 失败聚合:每席位独立 try/except,结果列表用线程安全 `list.append`
|
| 109 |
+
+ lock;总结日志输出"成功 N / 失败 M / 跳过 K"。
|
| 110 |
+
- **向后兼容**:`ROTATE_CONCURRENCY=1` 时退化为串行 for 循环
|
| 111 |
+
(早期 return + 不开 ThreadPoolExecutor,避免单测里 mock 复杂)。
|
| 112 |
+
|
| 113 |
+
### 测试
|
| 114 |
+
|
| 115 |
+
`tests/unit/test_round12_s5_predictor.py`(≥ 6 case):
|
| 116 |
+
|
| 117 |
+
1. `predict_exhaust_time` 历史 < 3 点 → None。
|
| 118 |
+
2. 线性递减 5 点(每 60 秒 -5%) → 预测耗尽时刻误差 < ±10%。
|
| 119 |
+
3. `slope >= 0`(配额持续上升或不变) → None。
|
| 120 |
+
4. `should_preempt` 在 lead window 内 → True;外 → False。
|
| 121 |
+
5. `record` 写文件后 `load_history` 能复现(roundtrip)。
|
| 122 |
+
6. `load_history(max_points=N)` 截尾正确(写 100 行只读最后 N)。
|
| 123 |
+
|
| 124 |
+
`tests/unit/test_round12_s6_concurrent.py`(≥ 6 case):
|
| 125 |
+
|
| 126 |
+
7. `ROTATE_CONCURRENCY=1` → 行为与改造前一致:串行调用,无 executor
|
| 127 |
+
open(用计数 mock 验证 `_reuse_one_standby` 调用顺序)。
|
| 128 |
+
8. `ROTATE_CONCURRENCY=3` + 5 个 standby 任务 → 并发完成
|
| 129 |
+
(tasks 计入并发起始的最大 inflight ≥ 2,用 latch 验证)。
|
| 130 |
+
9. 单席位抛异常 → 不影响其他席位,聚合 result 包含 `failed` + `reused`。
|
| 131 |
+
10. `_accounts_io_lock` 在并发 update_account 下不丢写
|
| 132 |
+
(线程并发触发 10 次 update_account → 最终 accounts.json 含 10 次更新)。
|
| 133 |
+
11. cancel_signal 在并发中能尽早结束(剩余 task 取消或快速返回)。
|
| 134 |
+
12. `_reuse_one_standby` 返回的 result 字面量稳定(用于 SSE 上报)。
|
| 135 |
+
|
| 136 |
+
合计新增 ≥ 12 单测,与原 658 累加。
|
| 137 |
+
|
| 138 |
+
## 必读引用
|
| 139 |
+
|
| 140 |
+
- 父 PRD(创新点 1 & 2 描述)
|
| 141 |
+
- `.trellis/tasks/05-11-s0-upstream-team-rotate-diff/research/upstream-diff.md`
|
| 142 |
+
- `.trellis/tasks/05-11-s3-cherry-pick-team-rotate/prd.md`
|
| 143 |
+
- `.upstream/manager.py`(上游 `cmd_rotate` baseline 行 2178-2520)
|
| 144 |
+
- `src/autoteam/account_state.py`(S1 状态机)
|
| 145 |
+
- `src/autoteam/accounts.py`(`update_account` 统一路由)
|
| 146 |
+
- `src/autoteam/manager.py`(S3 cherry-pick 后的 `cmd_rotate`)
|
| 147 |
+
- `src/autoteam/config.py`(env 配置范式)
|
| 148 |
+
|
| 149 |
+
## Acceptance Criteria
|
| 150 |
+
|
| 151 |
+
- [ ] `src/autoteam/quota_predictor.py` 落地(class 完整 + module-level
|
| 152 |
+
`default_predictor`)
|
| 153 |
+
- [ ] `config.py` 新增 `PREDICTIVE_ENABLED` / `PREDICTIVE_LEAD_MIN` /
|
| 154 |
+
`PREDICTIVE_HISTORY_FILE` / `ROTATE_CONCURRENCY`
|
| 155 |
+
- [ ] `manager.py:cmd_rotate` 接入预测分支(默认 disabled)
|
| 156 |
+
- [ ] `manager.py:cmd_rotate` standby 复用循环改造为
|
| 157 |
+
`_reuse_one_standby` + ThreadPoolExecutor
|
| 158 |
+
- [ ] `accounts.py` 新增 `_accounts_io_lock` 包裹 load+save 原子段
|
| 159 |
+
- [ ] `tests/unit/test_round12_s5_predictor.py` ≥ 6 case 全绿
|
| 160 |
+
- [ ] `tests/unit/test_round12_s6_concurrent.py` ≥ 6 case 全绿
|
| 161 |
+
- [ ] `ruff check src/autoteam/ tests/unit/` 全绿
|
| 162 |
+
- [ ] `pytest tests/` ≥ 658 + 12 ≥ 670 passed(不退化)
|
| 163 |
+
|
| 164 |
+
## Definition of Done
|
| 165 |
+
|
| 166 |
+
- 所有验收点 ✓
|
| 167 |
+
- commit 信息:`feat(round-12 S5+S6): predictive preempt + concurrent rotate`
|
| 168 |
+
- git add 仅自己改的文件(**严禁 `-A`**)
|
| 169 |
+
|
| 170 |
+
## Out of Scope
|
| 171 |
+
|
| 172 |
+
- 不动 mail provider 模块(S2 已落地)
|
| 173 |
+
- 不动 `account_state.py` 状态机(S1 已落地)
|
| 174 |
+
- 不动 web/(F2+F3 已落地 SSE 进度推送)
|
| 175 |
+
- 不实现 multi-workspace(S7 顶层任务)
|
| 176 |
+
- 不改 S3 已 cherry-pick 的双指标终止条件
|
| 177 |
+
- 不并发"新建账号"阶段(browser 状态共享,风险过高)
|
| 178 |
+
- 不真实触发 rotate(无可用母/子账号);全靠 mock + 静态分析
|
| 179 |
+
|
| 180 |
+
## Risk Notes
|
| 181 |
+
|
| 182 |
+
- ⚠ 预测算法选最简单的最小二乘 —— 配额非线性(5h 重置导致阶跃) 的
|
| 183 |
+
场景预测会偏;为避免误触发,`PREDICTIVE_ENABLED` 默认 `False`,
|
| 184 |
+
上线后由用户在前端 settings 主动开启。
|
| 185 |
+
- ⚠ 并发 standby 复用前提:每���位 mail provider 已 per-acc 绑定
|
| 186 |
+
(S3 `ensure_account_mail`);未绑定的旧 acc 仍走全局 `mail_client`
|
| 187 |
+
—— 此时并发会争同一 client,必须 fallback 串行。
|
| 188 |
+
- ⚠ `ThreadPoolExecutor` 在 Windows + Playwright 子进程下不能跨 worker
|
| 189 |
+
共享 browser;本任务**仅并发 mail wait_email/extract_code 这类 IO**,
|
| 190 |
+
ChatGPT browser 调用保持主线程串行(在每个 worker 开始时主线程已
|
| 191 |
+
完成 invite 调用)。
|
| 192 |
+
- ⚠ `quota_history.jsonl` 长期增长 —— 测试覆盖 `max_points` 截尾,
|
| 193 |
+
但生产用户需手动定期清理;后续可加自动 rotate。
|
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"id": "s5s6-predictive-concurrent-rotate",
|
| 3 |
+
"name": "s5s6-predictive-concurrent-rotate",
|
| 4 |
+
"title": "S5+S6: 预测式替换 + 并发批量替换",
|
| 5 |
+
"description": "",
|
| 6 |
+
"status": "completed",
|
| 7 |
+
"dev_type": null,
|
| 8 |
+
"scope": null,
|
| 9 |
+
"package": null,
|
| 10 |
+
"priority": "P2",
|
| 11 |
+
"creator": "ZRainbow",
|
| 12 |
+
"assignee": "ZRainbow",
|
| 13 |
+
"createdAt": "2026-05-11",
|
| 14 |
+
"completedAt": null,
|
| 15 |
+
"branch": null,
|
| 16 |
+
"base_branch": "main",
|
| 17 |
+
"worktree_path": null,
|
| 18 |
+
"commit": null,
|
| 19 |
+
"pr_url": null,
|
| 20 |
+
"subtasks": [],
|
| 21 |
+
"children": [],
|
| 22 |
+
"parent": "05-11-upstream-align-register-multimail-frontend-refresh",
|
| 23 |
+
"relatedFiles": [],
|
| 24 |
+
"notes": "",
|
| 25 |
+
"meta": {}
|
| 26 |
+
}
|
|
@@ -2,6 +2,7 @@
|
|
| 2 |
|
| 3 |
import json
|
| 4 |
import logging
|
|
|
|
| 5 |
import time
|
| 6 |
from pathlib import Path
|
| 7 |
|
|
@@ -18,6 +19,12 @@ logger = logging.getLogger(__name__)
|
|
| 18 |
PROJECT_ROOT = Path(__file__).parent.parent.parent
|
| 19 |
ACCOUNTS_FILE = PROJECT_ROOT / "accounts.json"
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
# 账号状态
|
| 22 |
STATUS_ACTIVE = "active" # 在 team 中,额度可用
|
| 23 |
STATUS_EXHAUSTED = "exhausted" # 在 team 中,额度用完
|
|
@@ -119,29 +126,34 @@ def add_account(email, password, cloudmail_account_id=None, seat_type=SEAT_UNKNO
|
|
| 119 |
update_account(email, **patch)
|
| 120 |
return
|
| 121 |
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
# Round 12 S1 — 新号入池触发 None → PENDING 转移,产生事件 + state_log 行
|
| 146 |
try:
|
| 147 |
default_machine.transition(
|
|
@@ -161,43 +173,49 @@ def update_account(email, **kwargs):
|
|
| 161 |
做合法性校验 + 写 state_log + 发事件。其余字段照常 update。
|
| 162 |
|
| 163 |
可选 kwarg ``_reason`` (内部约定): 状态转移日志里的 reason,默认 "update_account"。
|
| 164 |
-
"""
|
| 165 |
-
accounts = load_accounts()
|
| 166 |
-
acc = find_account(accounts, email)
|
| 167 |
-
if not acc:
|
| 168 |
-
return None
|
| 169 |
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 191 |
|
| 192 |
|
| 193 |
def delete_account(email):
|
| 194 |
"""从账号池彻底移除(不动认证文件、不动 CloudMail 邮箱)。返回是否真的删除了记录。"""
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
|
| 200 |
-
|
|
|
|
| 201 |
|
| 202 |
|
| 203 |
def get_active_accounts():
|
|
|
|
| 2 |
|
| 3 |
import json
|
| 4 |
import logging
|
| 5 |
+
import threading
|
| 6 |
import time
|
| 7 |
from pathlib import Path
|
| 8 |
|
|
|
|
| 19 |
PROJECT_ROOT = Path(__file__).parent.parent.parent
|
| 20 |
ACCOUNTS_FILE = PROJECT_ROOT / "accounts.json"
|
| 21 |
|
| 22 |
+
# Round 12 S6 — 并发 cmd_rotate 引入: 包裹 load → mutate → save 的 RMW 段,
|
| 23 |
+
# 避免两个 worker thread 各读一份后互相覆盖. update_account / delete_account /
|
| 24 |
+
# add_account 调用方都已通过此锁串行,纯只读的 load_accounts/find_account
|
| 25 |
+
# 仍然 lock-free(append-only JSONL state_log 的 atomic write 在状态机内).
|
| 26 |
+
_accounts_io_lock = threading.RLock()
|
| 27 |
+
|
| 28 |
# 账号状态
|
| 29 |
STATUS_ACTIVE = "active" # 在 team 中,额度可用
|
| 30 |
STATUS_EXHAUSTED = "exhausted" # 在 team 中,额度用完
|
|
|
|
| 126 |
update_account(email, **patch)
|
| 127 |
return
|
| 128 |
|
| 129 |
+
with _accounts_io_lock:
|
| 130 |
+
# 二次 load,避免在拿锁前已被其他 worker 追加
|
| 131 |
+
accounts = load_accounts()
|
| 132 |
+
if find_account(accounts, email):
|
| 133 |
+
return
|
| 134 |
+
accounts.append(
|
| 135 |
+
{
|
| 136 |
+
"email": email,
|
| 137 |
+
"password": password,
|
| 138 |
+
"cloudmail_account_id": cloudmail_account_id,
|
| 139 |
+
"status": STATUS_PENDING,
|
| 140 |
+
"seat_type": seat_type or SEAT_UNKNOWN,
|
| 141 |
+
"workspace_account_id": workspace_account_id, # 邀请时所在的母号 workspace ID,母号切换检测用
|
| 142 |
+
# Round 11 V8 — 子号自身的 personal workspace UUID(POST /backend-api/accounts/personal idempotent
|
| 143 |
+
# getOrCreate),持久化后下次 OAuth 不必再 fetch;失败/旧记录为 None,运行时按需 fetch + 回填。
|
| 144 |
+
"personal_workspace_id": None,
|
| 145 |
+
"auth_file": None, # CPA 认证文件路径
|
| 146 |
+
"quota_exhausted_at": None, # 额度用完的时间
|
| 147 |
+
"quota_resets_at": None, # 额度恢复时间
|
| 148 |
+
"last_quota_check_at": None, # 最近一次 wham/usage 探测时间戳,用于 standby 探测去重
|
| 149 |
+
# Round 11 V7 — 双失效探测(access_token + refresh_token 同时被 server-side invalidate):
|
| 150 |
+
# 主循环周期性调 is_token_pair_invalidated,命中后落该字段供事后排查 / UI 展示。
|
| 151 |
+
"last_token_pair_invalidated_at": None,
|
| 152 |
+
"created_at": time.time(),
|
| 153 |
+
"last_active_at": None,
|
| 154 |
+
}
|
| 155 |
+
)
|
| 156 |
+
save_accounts(accounts)
|
| 157 |
# Round 12 S1 — 新号入池触发 None → PENDING 转移,产生事件 + state_log 行
|
| 158 |
try:
|
| 159 |
default_machine.transition(
|
|
|
|
| 173 |
做合法性校验 + 写 state_log + 发事件。其余字段照常 update。
|
| 174 |
|
| 175 |
可选 kwarg ``_reason`` (内部约定): 状态转移日志里的 reason,默认 "update_account"。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
+
Round 12 S6 — RMW 段(load → mutate → save)由 ``_accounts_io_lock`` 串行,
|
| 178 |
+
并发 cmd_rotate worker 不会互相覆盖. state_machine.transition 内部已有
|
| 179 |
+
自己的锁负责 state_log atomic write,本锁仅保证 accounts.json 一致性.
|
| 180 |
+
"""
|
| 181 |
+
with _accounts_io_lock:
|
| 182 |
+
accounts = load_accounts()
|
| 183 |
+
acc = find_account(accounts, email)
|
| 184 |
+
if not acc:
|
| 185 |
+
return None
|
| 186 |
+
|
| 187 |
+
new_status = kwargs.get("status")
|
| 188 |
+
cur_status = acc.get("status")
|
| 189 |
+
reason = kwargs.pop("_reason", "update_account")
|
| 190 |
+
|
| 191 |
+
if new_status is not None and new_status != cur_status:
|
| 192 |
+
extra_payload = {k: v for k, v in kwargs.items() if k != "status"}
|
| 193 |
+
try:
|
| 194 |
+
default_machine.transition(
|
| 195 |
+
email=email,
|
| 196 |
+
to_state=new_status,
|
| 197 |
+
reason=reason,
|
| 198 |
+
extra=extra_payload or None,
|
| 199 |
+
from_state=cur_status,
|
| 200 |
+
)
|
| 201 |
+
except IllegalTransitionError:
|
| 202 |
+
# 合法性校验失败一律抛给调用方,避免静默写坏 accounts.json
|
| 203 |
+
raise
|
| 204 |
+
|
| 205 |
+
acc.update(kwargs)
|
| 206 |
+
save_accounts(accounts)
|
| 207 |
+
return acc
|
| 208 |
|
| 209 |
|
| 210 |
def delete_account(email):
|
| 211 |
"""从账号池彻底移除(不动认证文件、不动 CloudMail 邮箱)。返回是否真的删除了记录。"""
|
| 212 |
+
with _accounts_io_lock:
|
| 213 |
+
accounts = load_accounts()
|
| 214 |
+
remaining = [a for a in accounts if a.get("email") != email]
|
| 215 |
+
if len(remaining) == len(accounts):
|
| 216 |
+
return False
|
| 217 |
+
save_accounts(remaining)
|
| 218 |
+
return True
|
| 219 |
|
| 220 |
|
| 221 |
def get_active_accounts():
|
|
@@ -65,6 +65,23 @@ AUTO_CHECK_RETRY_ADD_PHONE = _get_bool_env("AUTO_CHECK_RETRY_ADD_PHONE", True)
|
|
| 65 |
AUTO_CHECK_ADD_PHONE_MAX_RETRIES = _get_int_env("AUTO_CHECK_ADD_PHONE_MAX_RETRIES", 3)
|
| 66 |
|
| 67 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
# 对账策略开关
|
| 69 |
# RECONCILE_KICK_ORPHAN=true: 残废成员(workspace 有 + 本地 auth_file 缺失)自动 kick。
|
| 70 |
# 关掉后改为打 STATUS_ORPHAN 标记等人工处理,避免"席位卡死"时仍被本地策略自动清理。
|
|
|
|
| 65 |
AUTO_CHECK_ADD_PHONE_MAX_RETRIES = _get_int_env("AUTO_CHECK_ADD_PHONE_MAX_RETRIES", 3)
|
| 66 |
|
| 67 |
|
| 68 |
+
# Round 12 S5 — 预测式抢先替换配置.
|
| 69 |
+
# PREDICTIVE_ENABLED=false(默认 安全): cmd_rotate 不做预测式 preempt,
|
| 70 |
+
# 保持 round-9~12 旧行为. 用户在前端 settings 主动开启后才参与预测.
|
| 71 |
+
# PREDICTIVE_LEAD_MIN=15(默认): 预测剩余时间 < 15 分钟时触发主动 standby + 替换.
|
| 72 |
+
# PREDICTIVE_HISTORY_FILE: quota 历史 JSONL 路径(供 QuotaPredictor 使用).
|
| 73 |
+
PREDICTIVE_ENABLED = _get_bool_env("PREDICTIVE_ENABLED", False)
|
| 74 |
+
PREDICTIVE_LEAD_MIN = _get_int_env("PREDICTIVE_LEAD_MIN", 15)
|
| 75 |
+
PREDICTIVE_HISTORY_FILE = PROJECT_ROOT / os.environ.get("PREDICTIVE_HISTORY_FILE", "quota_history.jsonl")
|
| 76 |
+
|
| 77 |
+
# Round 12 S6 — 并发批量替换配置.
|
| 78 |
+
# ROTATE_CONCURRENCY=1(默认 向后兼容): cmd_rotate 串行处理 standby 复用,
|
| 79 |
+
# 行为完全等同改造前. 用户调到 N>=2 后启用 ThreadPoolExecutor 并发,
|
| 80 |
+
# 每席位独立 try/except,失败聚合不阻塞其他席位.
|
| 81 |
+
# 上限保守设 8 — Playwright + ChatGPT API 并发更高反而引入抗扰风险.
|
| 82 |
+
ROTATE_CONCURRENCY = max(1, min(8, _get_int_env("ROTATE_CONCURRENCY", 1)))
|
| 83 |
+
|
| 84 |
+
|
| 85 |
# 对账策略开关
|
| 86 |
# RECONCILE_KICK_ORPHAN=true: 残废成员(workspace 有 + 本地 auth_file 缺失)自动 kick。
|
| 87 |
# 关掉后改为打 STATUS_ORPHAN 标记等人工处理,避免"席位卡死"时仍被本地策略自动清理。
|
|
@@ -3991,6 +3991,144 @@ def cmd_replace_batch(emails, trigger=""):
|
|
| 3991 |
return outcomes
|
| 3992 |
|
| 3993 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3994 |
def cmd_rotate(target_seats=5):
|
| 3995 |
"""
|
| 3996 |
智能轮转 - 保持 Team 始终有 target_seats 个可用成员,尽量少创建新账号。
|
|
@@ -4092,6 +4230,45 @@ def cmd_rotate(target_seats=5):
|
|
| 4092 |
logger.info("[2/5] 检查额度...")
|
| 4093 |
cmd_check()
|
| 4094 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4095 |
try:
|
| 4096 |
# 移出所有 exhausted 账号(包括之前已标记的)
|
| 4097 |
all_accounts = load_accounts()
|
|
@@ -4184,116 +4361,105 @@ def cmd_rotate(target_seats=5):
|
|
| 4184 |
logger.info("[4/5] 填补 %d 个空缺 (当前 %d/%d)...", vacancies, current_count, TARGET)
|
| 4185 |
|
| 4186 |
# 优先复用旧账号(先验证额度是否真的恢复了)
|
|
|
|
|
|
|
|
|
|
| 4187 |
filled = 0
|
| 4188 |
standby_list = [a for a in get_standby_accounts() if not _is_main_account_email(a.get("email"))]
|
| 4189 |
-
quota_skipped = []
|
| 4190 |
-
auto_reuse_skipped = []
|
| 4191 |
|
| 4192 |
from autoteam import cancel_signal
|
|
|
|
| 4193 |
|
| 4194 |
-
|
| 4195 |
-
|
| 4196 |
-
|
| 4197 |
-
|
| 4198 |
-
if filled >= vacancies:
|
| 4199 |
-
break
|
| 4200 |
-
email = acc["email"]
|
| 4201 |
-
auth_file = acc.get("auth_file")
|
| 4202 |
-
|
| 4203 |
-
skip_reason = _auto_reuse_skip_reason(acc)
|
| 4204 |
-
if skip_reason:
|
| 4205 |
-
logger.info("[4/5] 跳过 %s(%s)", email, skip_reason)
|
| 4206 |
-
auto_reuse_skipped.append(acc)
|
| 4207 |
-
continue
|
| 4208 |
-
|
| 4209 |
-
# 验证额度是否真的恢复了
|
| 4210 |
-
quota_ok = False
|
| 4211 |
-
if auth_file and Path(auth_file).exists():
|
| 4212 |
-
try:
|
| 4213 |
-
auth_data = json.loads(read_text(Path(auth_file)))
|
| 4214 |
-
access_token = auth_data.get("access_token")
|
| 4215 |
-
if access_token:
|
| 4216 |
-
status_str, info = check_codex_quota(access_token)
|
| 4217 |
-
if status_str == "exhausted":
|
| 4218 |
-
quota_info = quota_result_quota_info(info)
|
| 4219 |
-
if quota_info:
|
| 4220 |
-
update_account(email, last_quota=quota_info)
|
| 4221 |
-
logger.info("[4/5] 跳过 %s(额度未恢复)", email)
|
| 4222 |
-
quota_skipped.append(acc)
|
| 4223 |
-
continue
|
| 4224 |
-
if status_str == "ok" and isinstance(info, dict):
|
| 4225 |
-
p_remain = 100 - info.get("primary_pct", 0)
|
| 4226 |
-
if p_remain < threshold:
|
| 4227 |
-
logger.info("[4/5] 跳过 %s(剩余 %d%% < %d%%)", email, p_remain, threshold)
|
| 4228 |
-
quota_skipped.append(acc)
|
| 4229 |
-
continue
|
| 4230 |
-
quota_ok = True
|
| 4231 |
-
# network_error: 临时网络故障,不能当"额度已恢复"凭证。本轮跳过,
|
| 4232 |
-
# 不动 acc 状态,等下一轮再试。
|
| 4233 |
-
if status_str == "network_error":
|
| 4234 |
-
logger.info("[4/5] 跳过 %s(临时网络错误,本轮无法验证额度)", email)
|
| 4235 |
-
quota_skipped.append(acc)
|
| 4236 |
-
continue
|
| 4237 |
-
# auth_error: token 失效,用 last_quota 判断(但重置时间已过的不算)
|
| 4238 |
-
if status_str == "auth_error":
|
| 4239 |
-
lq = acc.get("last_quota")
|
| 4240 |
-
if lq:
|
| 4241 |
-
exhausted_info = _pending_historical_exhausted_info(lq)
|
| 4242 |
-
if exhausted_info:
|
| 4243 |
-
window_label = _quota_window_label(exhausted_info.get("window"))
|
| 4244 |
-
logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
|
| 4245 |
-
quota_skipped.append(acc)
|
| 4246 |
-
continue
|
| 4247 |
-
p_resets = lq.get("primary_resets_at", 0)
|
| 4248 |
-
if p_resets and time.time() >= p_resets:
|
| 4249 |
-
logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
|
| 4250 |
-
quota_ok = True
|
| 4251 |
-
else:
|
| 4252 |
-
p_remain = 100 - lq.get("primary_pct", 0)
|
| 4253 |
-
if p_remain < threshold:
|
| 4254 |
-
logger.info("[4/5] 跳过 %s(上次额度 %d%% < %d%%)", email, p_remain, threshold)
|
| 4255 |
-
quota_skipped.append(acc)
|
| 4256 |
-
continue
|
| 4257 |
-
quota_ok = True
|
| 4258 |
-
except Exception:
|
| 4259 |
-
pass
|
| 4260 |
-
|
| 4261 |
-
# 没有认证文件或无法查询额度时,用 last_quota / quota_resets_at 兜底
|
| 4262 |
-
if not quota_ok:
|
| 4263 |
-
lq = acc.get("last_quota")
|
| 4264 |
-
if lq:
|
| 4265 |
-
exhausted_info = _pending_historical_exhausted_info(lq)
|
| 4266 |
-
if exhausted_info:
|
| 4267 |
-
window_label = _quota_window_label(exhausted_info.get("window"))
|
| 4268 |
-
logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
|
| 4269 |
-
quota_skipped.append(acc)
|
| 4270 |
-
continue
|
| 4271 |
-
p_resets = lq.get("primary_resets_at", 0)
|
| 4272 |
-
if p_resets and time.time() >= p_resets:
|
| 4273 |
-
# 重置时间已过,旧数据作废,视为额度已恢复
|
| 4274 |
-
logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
|
| 4275 |
-
else:
|
| 4276 |
-
p_remain = 100 - lq.get("primary_pct", 0)
|
| 4277 |
-
if p_remain < threshold:
|
| 4278 |
-
logger.info("[4/5] 跳过 %s(历史额度 %d%% < %d%%)", email, p_remain, threshold)
|
| 4279 |
-
quota_skipped.append(acc)
|
| 4280 |
-
continue
|
| 4281 |
-
else:
|
| 4282 |
-
# 没有 last_quota,看 quota_resets_at 是否已过
|
| 4283 |
-
resets_at = acc.get("quota_resets_at")
|
| 4284 |
-
if resets_at and time.time() < resets_at:
|
| 4285 |
-
mins = max(0, int((resets_at - time.time()) / 60))
|
| 4286 |
-
logger.info("[4/5] 跳过 %s(%d 分钟后恢复)", email, mins)
|
| 4287 |
-
quota_skipped.append(acc)
|
| 4288 |
-
continue
|
| 4289 |
|
| 4290 |
-
|
| 4291 |
if not chatgpt or not chatgpt.browser:
|
| 4292 |
ensure_chatgpt()
|
| 4293 |
-
|
| 4294 |
-
|
| 4295 |
-
|
| 4296 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4297 |
quota_skipped.append(acc)
|
| 4298 |
|
| 4299 |
if quota_skipped:
|
|
|
|
| 3991 |
return outcomes
|
| 3992 |
|
| 3993 |
|
| 3994 |
+
# ---------------------------------------------------------------------------
|
| 3995 |
+
# Round 12 S6 — standby 复用单元(可被 ThreadPoolExecutor 并发调度).
|
| 3996 |
+
#
|
| 3997 |
+
# 把原 cmd_rotate 内 for 循环里"额度校验 → reinvite_account"两段抽出来,
|
| 3998 |
+
# 让 ROTATE_CONCURRENCY > 1 时多个席位的 mail wait 可并行(主要 IO 瓶颈).
|
| 3999 |
+
# chatgpt browser 调用本身仍由调用方决定是否需要序列化(本任务仅在主线程
|
| 4000 |
+
# pre-ensure 一次,reinvite_account 内部对 browser 的多次访问由 Playwright
|
| 4001 |
+
# session 自身保证并发安全 — 单 BrowserContext + 多 Page 模式).
|
| 4002 |
+
# ---------------------------------------------------------------------------
|
| 4003 |
+
_STANDBY_REUSE_RESULTS = frozenset({"reused", "skipped_quota", "skipped_auto", "failed"})
|
| 4004 |
+
|
| 4005 |
+
|
| 4006 |
+
def _reuse_one_standby(
|
| 4007 |
+
acc: dict,
|
| 4008 |
+
threshold: int,
|
| 4009 |
+
*,
|
| 4010 |
+
chatgpt_provider,
|
| 4011 |
+
mail_provider,
|
| 4012 |
+
reinvite_fn=None,
|
| 4013 |
+
quota_fn=None,
|
| 4014 |
+
now=None,
|
| 4015 |
+
) -> dict:
|
| 4016 |
+
"""Process one standby account end-to-end.
|
| 4017 |
+
|
| 4018 |
+
Parameters
|
| 4019 |
+
----------
|
| 4020 |
+
acc:
|
| 4021 |
+
Standby account dict (already filtered for non-main + STATUS_STANDBY).
|
| 4022 |
+
threshold:
|
| 4023 |
+
AUTO_CHECK_THRESHOLD percent — quota below this is treated as "not recovered".
|
| 4024 |
+
chatgpt_provider:
|
| 4025 |
+
Zero-arg callable returning a started :class:`ChatGPTTeamAPI` instance.
|
| 4026 |
+
mail_provider:
|
| 4027 |
+
``acc -> mail_client`` callable (per-acc routed, S3 ensure_account_mail).
|
| 4028 |
+
reinvite_fn:
|
| 4029 |
+
Override for :func:`reinvite_account` (tests pass a mock).
|
| 4030 |
+
quota_fn:
|
| 4031 |
+
Override for :func:`check_codex_quota` (tests pass a mock).
|
| 4032 |
+
now:
|
| 4033 |
+
Override for ``time.time()`` (tests inject deterministic time).
|
| 4034 |
+
|
| 4035 |
+
Returns
|
| 4036 |
+
-------
|
| 4037 |
+
dict with keys ``email`` / ``result`` (one of ``_STANDBY_REUSE_RESULTS``) /
|
| 4038 |
+
``error`` (None on success, str on caught exception). Never raises —
|
| 4039 |
+
failures land in ``result="failed"`` so concurrent map can aggregate.
|
| 4040 |
+
"""
|
| 4041 |
+
reinvite_callable = reinvite_fn or reinvite_account
|
| 4042 |
+
quota_callable = quota_fn or check_codex_quota
|
| 4043 |
+
current_ts = now if now is not None else time.time()
|
| 4044 |
+
email = (acc or {}).get("email") or "<unknown>"
|
| 4045 |
+
|
| 4046 |
+
try:
|
| 4047 |
+
skip_reason = _auto_reuse_skip_reason(acc)
|
| 4048 |
+
if skip_reason:
|
| 4049 |
+
logger.info("[4/5] 跳过 %s(%s)", email, skip_reason)
|
| 4050 |
+
return {"email": email, "result": "skipped_auto", "error": None}
|
| 4051 |
+
|
| 4052 |
+
auth_file = acc.get("auth_file")
|
| 4053 |
+
quota_ok = False
|
| 4054 |
+
if auth_file and Path(auth_file).exists():
|
| 4055 |
+
try:
|
| 4056 |
+
auth_data = json.loads(read_text(Path(auth_file)))
|
| 4057 |
+
access_token = auth_data.get("access_token")
|
| 4058 |
+
if access_token:
|
| 4059 |
+
status_str, info = quota_callable(access_token)
|
| 4060 |
+
if status_str == "exhausted":
|
| 4061 |
+
quota_info = quota_result_quota_info(info)
|
| 4062 |
+
if quota_info:
|
| 4063 |
+
update_account(email, last_quota=quota_info)
|
| 4064 |
+
logger.info("[4/5] 跳过 %s(额度未恢复)", email)
|
| 4065 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4066 |
+
if status_str == "ok" and isinstance(info, dict):
|
| 4067 |
+
p_remain = 100 - info.get("primary_pct", 0)
|
| 4068 |
+
if p_remain < threshold:
|
| 4069 |
+
logger.info("[4/5] 跳过 %s(剩余 %d%% < %d%%)", email, p_remain, threshold)
|
| 4070 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4071 |
+
quota_ok = True
|
| 4072 |
+
if status_str == "network_error":
|
| 4073 |
+
logger.info("[4/5] 跳过 %s(临时网络错误,本轮无法验证额度)", email)
|
| 4074 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4075 |
+
if status_str == "auth_error":
|
| 4076 |
+
lq = acc.get("last_quota")
|
| 4077 |
+
if lq:
|
| 4078 |
+
exhausted_info = _pending_historical_exhausted_info(lq)
|
| 4079 |
+
if exhausted_info:
|
| 4080 |
+
window_label = _quota_window_label(exhausted_info.get("window"))
|
| 4081 |
+
logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
|
| 4082 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4083 |
+
p_resets = lq.get("primary_resets_at", 0)
|
| 4084 |
+
if p_resets and current_ts >= p_resets:
|
| 4085 |
+
logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
|
| 4086 |
+
quota_ok = True
|
| 4087 |
+
else:
|
| 4088 |
+
p_remain = 100 - lq.get("primary_pct", 0)
|
| 4089 |
+
if p_remain < threshold:
|
| 4090 |
+
logger.info("[4/5] 跳过 %s(上次额��� %d%% < %d%%)", email, p_remain, threshold)
|
| 4091 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4092 |
+
quota_ok = True
|
| 4093 |
+
except Exception:
|
| 4094 |
+
pass
|
| 4095 |
+
|
| 4096 |
+
if not quota_ok:
|
| 4097 |
+
lq = acc.get("last_quota")
|
| 4098 |
+
if lq:
|
| 4099 |
+
exhausted_info = _pending_historical_exhausted_info(lq)
|
| 4100 |
+
if exhausted_info:
|
| 4101 |
+
window_label = _quota_window_label(exhausted_info.get("window"))
|
| 4102 |
+
logger.info("[4/5] 跳过 %s(%s额度未恢复)", email, window_label)
|
| 4103 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4104 |
+
p_resets = lq.get("primary_resets_at", 0)
|
| 4105 |
+
if p_resets and current_ts >= p_resets:
|
| 4106 |
+
logger.info("[4/5] %s 的 5h 重置时间已过,视为额度已恢复", email)
|
| 4107 |
+
else:
|
| 4108 |
+
p_remain = 100 - lq.get("primary_pct", 0)
|
| 4109 |
+
if p_remain < threshold:
|
| 4110 |
+
logger.info("[4/5] 跳过 %s(历史额度 %d%% < %d%%)", email, p_remain, threshold)
|
| 4111 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4112 |
+
else:
|
| 4113 |
+
resets_at = acc.get("quota_resets_at")
|
| 4114 |
+
if resets_at and current_ts < resets_at:
|
| 4115 |
+
mins = max(0, int((resets_at - current_ts) / 60))
|
| 4116 |
+
logger.info("[4/5] 跳过 %s(%d 分钟后恢复)", email, mins)
|
| 4117 |
+
return {"email": email, "result": "skipped_quota", "error": None}
|
| 4118 |
+
|
| 4119 |
+
logger.info("[4/5] 复用: %s", email)
|
| 4120 |
+
chatgpt = chatgpt_provider()
|
| 4121 |
+
mail_client = mail_provider(acc)
|
| 4122 |
+
ok = reinvite_callable(chatgpt, mail_client, acc)
|
| 4123 |
+
if ok:
|
| 4124 |
+
return {"email": email, "result": "reused", "error": None}
|
| 4125 |
+
return {"email": email, "result": "failed", "error": "reinvite_account returned False"}
|
| 4126 |
+
except Exception as exc:
|
| 4127 |
+
# 关键: 并发模式下任何席位异常都不能波及其他席位,聚合层做 failed 计数.
|
| 4128 |
+
logger.exception("[4/5] %s 复用流程抛异常,标记 failed: %s", email, exc)
|
| 4129 |
+
return {"email": email, "result": "failed", "error": str(exc)}
|
| 4130 |
+
|
| 4131 |
+
|
| 4132 |
def cmd_rotate(target_seats=5):
|
| 4133 |
"""
|
| 4134 |
智能轮转 - 保持 Team 始终有 target_seats 个可用成员,尽量少创建新账号。
|
|
|
|
| 4230 |
logger.info("[2/5] 检查额度...")
|
| 4231 |
cmd_check()
|
| 4232 |
|
| 4233 |
+
# Round 12 S5 — 预测式抢先替换(可选,默认关).
|
| 4234 |
+
# 仅当 PREDICTIVE_ENABLED=true 时执行: 遍历 ACTIVE 子号,基于 quota_history
|
| 4235 |
+
# 时序拟合预测耗尽时刻,在 PREDICTIVE_LEAD_MIN 分钟内的主动 kick → STANDBY,
|
| 4236 |
+
# 让后续 [3/5] vacancy 兜底 + [5/5] 新号填补流程自动接管.
|
| 4237 |
+
try:
|
| 4238 |
+
from autoteam.config import PREDICTIVE_ENABLED, PREDICTIVE_LEAD_MIN
|
| 4239 |
+
except ImportError:
|
| 4240 |
+
PREDICTIVE_ENABLED = False
|
| 4241 |
+
PREDICTIVE_LEAD_MIN = 15
|
| 4242 |
+
if PREDICTIVE_ENABLED:
|
| 4243 |
+
try:
|
| 4244 |
+
from autoteam.quota_predictor import default_predictor
|
| 4245 |
+
|
| 4246 |
+
preempt_candidates = []
|
| 4247 |
+
for acc in load_accounts():
|
| 4248 |
+
if _is_main_account_email(acc.get("email")) or acc.get("status") != STATUS_ACTIVE:
|
| 4249 |
+
continue
|
| 4250 |
+
# 先记录当前 last_quota → 累积历史(用 acc.last_quota,避免重复 API 调用)
|
| 4251 |
+
lq = acc.get("last_quota") or {}
|
| 4252 |
+
p_remain = 100 - lq.get("primary_pct", 0) if "primary_pct" in lq else None
|
| 4253 |
+
if p_remain is not None:
|
| 4254 |
+
default_predictor.record(acc["email"], p_remain, lq.get("primary_resets_at"))
|
| 4255 |
+
if default_predictor.should_preempt(acc["email"], PREDICTIVE_LEAD_MIN):
|
| 4256 |
+
preempt_candidates.append(acc)
|
| 4257 |
+
if preempt_candidates:
|
| 4258 |
+
logger.info("[2.5/5] 预测式抢先替换 %d 个即将耗尽的账号...", len(preempt_candidates))
|
| 4259 |
+
if not chatgpt or not chatgpt.browser:
|
| 4260 |
+
ensure_chatgpt()
|
| 4261 |
+
for acc in preempt_candidates:
|
| 4262 |
+
email = acc["email"]
|
| 4263 |
+
remove_status = remove_from_team(chatgpt, email, return_status=True)
|
| 4264 |
+
if remove_status in ("removed", "already_absent"):
|
| 4265 |
+
update_account(email, status=STATUS_STANDBY, _reason="predictive_preempt")
|
| 4266 |
+
logger.info("[2.5/5] %s → standby(预测式抢先,lead=%dmin)", email, PREDICTIVE_LEAD_MIN)
|
| 4267 |
+
else:
|
| 4268 |
+
logger.debug("[2.5/5] 预测式: 无 ACTIVE 子号需要抢先替换")
|
| 4269 |
+
except Exception as exc:
|
| 4270 |
+
logger.warning("[2.5/5] 预测式抢先替换异常(忽略,走旧路径): %s", exc)
|
| 4271 |
+
|
| 4272 |
try:
|
| 4273 |
# 移出所有 exhausted 账号(包括之前已标记的)
|
| 4274 |
all_accounts = load_accounts()
|
|
|
|
| 4361 |
logger.info("[4/5] 填补 %d 个空缺 (当前 %d/%d)...", vacancies, current_count, TARGET)
|
| 4362 |
|
| 4363 |
# 优先复用旧账号(先验证额度是否真的恢复了)
|
| 4364 |
+
# Round 12 S6 — ROTATE_CONCURRENCY > 1 时用 ThreadPoolExecutor 并发,
|
| 4365 |
+
# 否则保持串行(向后兼容老行为). 每席位独立 try/except 在 _reuse_one_standby
|
| 4366 |
+
# 内已收敛 → result ∈ {reused / skipped_quota / skipped_auto / failed}.
|
| 4367 |
filled = 0
|
| 4368 |
standby_list = [a for a in get_standby_accounts() if not _is_main_account_email(a.get("email"))]
|
| 4369 |
+
quota_skipped: list[dict] = []
|
| 4370 |
+
auto_reuse_skipped: list[dict] = []
|
| 4371 |
|
| 4372 |
from autoteam import cancel_signal
|
| 4373 |
+
from autoteam.config import ROTATE_CONCURRENCY
|
| 4374 |
|
| 4375 |
+
# 注: 不截断到 vacancies — 旧行为会迭代全部 standby,遇到不可复用的(skipped_auto/
|
| 4376 |
+
# skipped_quota)就继续看下一个,直到 filled >= vacancies 才 break. 截断会导致
|
| 4377 |
+
# 第一个候选若 skipped 则放弃后续可用候选,破坏 test_cmd_rotate_skips_google_accounts.
|
| 4378 |
+
candidates = list(standby_list)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4379 |
|
| 4380 |
+
def _chatgpt_provider():
|
| 4381 |
if not chatgpt or not chatgpt.browser:
|
| 4382 |
ensure_chatgpt()
|
| 4383 |
+
return chatgpt
|
| 4384 |
+
|
| 4385 |
+
def _process(acc):
|
| 4386 |
+
return _reuse_one_standby(
|
| 4387 |
+
acc,
|
| 4388 |
+
threshold,
|
| 4389 |
+
chatgpt_provider=_chatgpt_provider,
|
| 4390 |
+
mail_provider=ensure_account_mail,
|
| 4391 |
+
)
|
| 4392 |
+
|
| 4393 |
+
outcomes: list[dict] = []
|
| 4394 |
+
if ROTATE_CONCURRENCY <= 1 or len(candidates) <= 1:
|
| 4395 |
+
# 串行: 完全等同改造前的 for 循环行为(测试可复现)
|
| 4396 |
+
for acc in candidates:
|
| 4397 |
+
if cancel_signal.is_cancelled():
|
| 4398 |
+
logger.warning("[轮转] 收到取消请求,中止 standby 复用阶段")
|
| 4399 |
+
break
|
| 4400 |
+
if filled >= vacancies:
|
| 4401 |
+
break
|
| 4402 |
+
result = _process(acc)
|
| 4403 |
+
outcomes.append(result)
|
| 4404 |
+
if result["result"] == "reused":
|
| 4405 |
+
filled += 1
|
| 4406 |
+
current_count += 1
|
| 4407 |
+
else:
|
| 4408 |
+
# 并发: 邮件 wait + OTP 提取是主要 IO 瓶颈,ThreadPoolExecutor 显著缩短
|
| 4409 |
+
# 轮转总时长. 注意 — 并发模式会同时处理至多 vacancies + concurrency 个候选,
|
| 4410 |
+
# 比串行多做一些"探测",换取吞吐. 已 reused 数量达到 vacancies 后停止 submit.
|
| 4411 |
+
import concurrent.futures
|
| 4412 |
+
|
| 4413 |
+
workers = max(1, min(ROTATE_CONCURRENCY, len(candidates), vacancies))
|
| 4414 |
+
logger.info("[4/5] 并发复用 standby(候选 %d,max_workers=%d)...", len(candidates), workers)
|
| 4415 |
+
with concurrent.futures.ThreadPoolExecutor(max_workers=workers) as pool:
|
| 4416 |
+
future_map: dict = {}
|
| 4417 |
+
# 先 submit 一批(数量 = workers),后续按 future 完成情况补 submit.
|
| 4418 |
+
pending = iter(candidates)
|
| 4419 |
+
for _ in range(workers):
|
| 4420 |
+
nxt = next(pending, None)
|
| 4421 |
+
if nxt is None:
|
| 4422 |
+
break
|
| 4423 |
+
future_map[pool.submit(_process, nxt)] = nxt
|
| 4424 |
+
|
| 4425 |
+
while future_map and filled < vacancies:
|
| 4426 |
+
done, _not_done = concurrent.futures.wait(
|
| 4427 |
+
future_map, return_when=concurrent.futures.FIRST_COMPLETED
|
| 4428 |
+
)
|
| 4429 |
+
for future in done:
|
| 4430 |
+
future_map.pop(future, None)
|
| 4431 |
+
try:
|
| 4432 |
+
result = future.result()
|
| 4433 |
+
except Exception as exc:
|
| 4434 |
+
logger.exception("[4/5] 并发任务异常: %s", exc)
|
| 4435 |
+
result = {"email": "<unknown>", "result": "failed", "error": str(exc)}
|
| 4436 |
+
outcomes.append(result)
|
| 4437 |
+
if result["result"] == "reused":
|
| 4438 |
+
filled += 1
|
| 4439 |
+
current_count += 1
|
| 4440 |
+
# 还能补 submit 且未达 vacancies → 拉新候选
|
| 4441 |
+
if filled < vacancies and not cancel_signal.is_cancelled():
|
| 4442 |
+
nxt = next(pending, None)
|
| 4443 |
+
if nxt is not None:
|
| 4444 |
+
future_map[pool.submit(_process, nxt)] = nxt
|
| 4445 |
+
# cancel pending(已 submit 但还未完成的会跑完),不再 submit 新 task
|
| 4446 |
+
if cancel_signal.is_cancelled():
|
| 4447 |
+
logger.warning("[轮转] 收到取消请求,等待 in-flight 完成后中止")
|
| 4448 |
+
# 收尾: 等剩余 in-flight 完成(filled 已够了仍要收 result 避免线程泄漏)
|
| 4449 |
+
for future in concurrent.futures.as_completed(list(future_map)):
|
| 4450 |
+
try:
|
| 4451 |
+
outcomes.append(future.result())
|
| 4452 |
+
except Exception as exc:
|
| 4453 |
+
outcomes.append({"email": "<unknown>", "result": "failed", "error": str(exc)})
|
| 4454 |
+
|
| 4455 |
+
# 聚合分类: 并发模式 outcomes 顺序与候选不同,但分类计数稳定
|
| 4456 |
+
for outcome in outcomes:
|
| 4457 |
+
email = outcome["email"]
|
| 4458 |
+
acc = next((a for a in candidates if a.get("email") == email), {"email": email})
|
| 4459 |
+
res = outcome["result"]
|
| 4460 |
+
if res == "skipped_auto":
|
| 4461 |
+
auto_reuse_skipped.append(acc)
|
| 4462 |
+
elif res == "skipped_quota" or res == "failed":
|
| 4463 |
quota_skipped.append(acc)
|
| 4464 |
|
| 4465 |
if quota_skipped:
|
|
@@ -0,0 +1,197 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Quota usage predictor — Round 12 S5.
|
| 2 |
+
|
| 3 |
+
Records per-account quota observations to a JSONL log and fits a simple
|
| 4 |
+
least-squares linear regression to estimate when each ACTIVE account's
|
| 5 |
+
``p_remain`` will hit zero. Used by :func:`autoteam.manager.cmd_rotate`
|
| 6 |
+
to pre-emptively swap soon-to-be-exhausted seats with STANDBY candidates
|
| 7 |
+
*before* the quota actually runs out — minimising user-visible "Team full
|
| 8 |
+
but no usable seat" windows.
|
| 9 |
+
|
| 10 |
+
Design choices
|
| 11 |
+
--------------
|
| 12 |
+
* **Storage** — append-only JSONL at ``PROJECT_ROOT / "quota_history.jsonl"``;
|
| 13 |
+
no DB dependency, survives crashes, easy to tail / inspect / archive.
|
| 14 |
+
* **Algorithm** — minimum viable: ``p_remain = a * ts + b`` via NumPy-free
|
| 15 |
+
closed-form OLS. We only need the *direction* and *zero-crossing*; full
|
| 16 |
+
exponential-decay / piecewise-linear modelling is out of scope (see PRD
|
| 17 |
+
Risk Notes — non-linear 5h reset stairs would distort it anyway, so we
|
| 18 |
+
ship with ``PREDICTIVE_ENABLED=False`` by default).
|
| 19 |
+
* **Safety** — < 3 data points returns ``None`` (no prediction); positive
|
| 20 |
+
slope (quota growing or flat) returns ``None``; any numeric exception is
|
| 21 |
+
swallowed and returns ``None``. The caller treats ``None`` as "do not
|
| 22 |
+
preempt", so the predictor can never *cause* an outage — only fail to
|
| 23 |
+
prevent one.
|
| 24 |
+
"""
|
| 25 |
+
from __future__ import annotations
|
| 26 |
+
|
| 27 |
+
import json
|
| 28 |
+
import logging
|
| 29 |
+
import threading
|
| 30 |
+
import time
|
| 31 |
+
from pathlib import Path
|
| 32 |
+
from typing import Any
|
| 33 |
+
|
| 34 |
+
logger = logging.getLogger(__name__)
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
_PROJECT_ROOT = Path(__file__).parent.parent.parent
|
| 38 |
+
DEFAULT_HISTORY_PATH = _PROJECT_ROOT / "quota_history.jsonl"
|
| 39 |
+
|
| 40 |
+
|
| 41 |
+
class QuotaPredictor:
|
| 42 |
+
"""Append-only JSONL store + linear-fit exhaust-time predictor."""
|
| 43 |
+
|
| 44 |
+
def __init__(self, history_path: Path | None = None) -> None:
|
| 45 |
+
self._history_path: Path = Path(history_path) if history_path else DEFAULT_HISTORY_PATH
|
| 46 |
+
# File-level lock — record() and load_history() can be called from
|
| 47 |
+
# the SSE pusher, cmd_check and cmd_rotate threads concurrently.
|
| 48 |
+
self._lock = threading.Lock()
|
| 49 |
+
|
| 50 |
+
@property
|
| 51 |
+
def history_path(self) -> Path:
|
| 52 |
+
return self._history_path
|
| 53 |
+
|
| 54 |
+
# ------------------------------------------------------------------ write
|
| 55 |
+
def record(
|
| 56 |
+
self,
|
| 57 |
+
email: str,
|
| 58 |
+
p_remain: float,
|
| 59 |
+
t_remain: float | None = None,
|
| 60 |
+
ts: float | None = None,
|
| 61 |
+
) -> None:
|
| 62 |
+
"""Append one observation. ``ts`` defaults to ``time.time()``.
|
| 63 |
+
|
| 64 |
+
``p_remain`` is the "primary quota remaining percent" (0-100).
|
| 65 |
+
``t_remain`` is the optional "seconds until reset" — recorded for
|
| 66 |
+
future modelling but unused by the linear fit today.
|
| 67 |
+
"""
|
| 68 |
+
if not email:
|
| 69 |
+
return
|
| 70 |
+
try:
|
| 71 |
+
row: dict[str, Any] = {
|
| 72 |
+
"email": str(email),
|
| 73 |
+
"p_remain": float(p_remain),
|
| 74 |
+
"t_remain": None if t_remain is None else float(t_remain),
|
| 75 |
+
"ts": float(ts) if ts is not None else time.time(),
|
| 76 |
+
}
|
| 77 |
+
except (TypeError, ValueError):
|
| 78 |
+
logger.warning("quota_predictor.record: bad payload for %s", email)
|
| 79 |
+
return
|
| 80 |
+
|
| 81 |
+
line = json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n"
|
| 82 |
+
with self._lock:
|
| 83 |
+
try:
|
| 84 |
+
self._history_path.parent.mkdir(parents=True, exist_ok=True)
|
| 85 |
+
with open(self._history_path, "a", encoding="utf-8") as fp:
|
| 86 |
+
fp.write(line)
|
| 87 |
+
except OSError:
|
| 88 |
+
logger.exception("quota_predictor.record: cannot append to %s", self._history_path)
|
| 89 |
+
|
| 90 |
+
# ------------------------------------------------------------------- read
|
| 91 |
+
def load_history(self, email: str, *, max_points: int = 50) -> list[dict[str, Any]]:
|
| 92 |
+
"""Return chronologically-sorted observations for ``email``.
|
| 93 |
+
|
| 94 |
+
Truncated to the most recent ``max_points`` rows so long-running
|
| 95 |
+
deployments do not blow up memory. Malformed lines are skipped
|
| 96 |
+
silently — JSONL is append-only and a partial line is recoverable
|
| 97 |
+
on the next successful append.
|
| 98 |
+
"""
|
| 99 |
+
if not email or max_points <= 0:
|
| 100 |
+
return []
|
| 101 |
+
path = self._history_path
|
| 102 |
+
if not path.exists():
|
| 103 |
+
return []
|
| 104 |
+
rows: list[dict[str, Any]] = []
|
| 105 |
+
with self._lock:
|
| 106 |
+
try:
|
| 107 |
+
with open(path, encoding="utf-8") as fp:
|
| 108 |
+
for raw in fp:
|
| 109 |
+
raw = raw.strip()
|
| 110 |
+
if not raw:
|
| 111 |
+
continue
|
| 112 |
+
try:
|
| 113 |
+
obj = json.loads(raw)
|
| 114 |
+
except json.JSONDecodeError:
|
| 115 |
+
continue
|
| 116 |
+
if obj.get("email") != email:
|
| 117 |
+
continue
|
| 118 |
+
try:
|
| 119 |
+
obj["ts"] = float(obj.get("ts", 0.0))
|
| 120 |
+
obj["p_remain"] = float(obj.get("p_remain", 0.0))
|
| 121 |
+
except (TypeError, ValueError):
|
| 122 |
+
continue
|
| 123 |
+
rows.append(obj)
|
| 124 |
+
except OSError:
|
| 125 |
+
logger.exception("quota_predictor.load_history: cannot read %s", path)
|
| 126 |
+
return []
|
| 127 |
+
rows.sort(key=lambda r: r["ts"])
|
| 128 |
+
if len(rows) > max_points:
|
| 129 |
+
rows = rows[-max_points:]
|
| 130 |
+
return rows
|
| 131 |
+
|
| 132 |
+
# ------------------------------------------------------------------- fit
|
| 133 |
+
def predict_exhaust_time(self, email: str) -> float | None:
|
| 134 |
+
"""Predict the wall-clock ts at which ``p_remain`` crosses 0.
|
| 135 |
+
|
| 136 |
+
Returns ``None`` when:
|
| 137 |
+
* history < 3 points (insufficient signal);
|
| 138 |
+
* slope >= 0 (quota stable or growing — nothing to preempt);
|
| 139 |
+
* numeric instability (degenerate variance, NaN);
|
| 140 |
+
* caught exception (best-effort — predictor never raises).
|
| 141 |
+
"""
|
| 142 |
+
rows = self.load_history(email)
|
| 143 |
+
if len(rows) < 3:
|
| 144 |
+
return None
|
| 145 |
+
|
| 146 |
+
try:
|
| 147 |
+
xs = [r["ts"] for r in rows]
|
| 148 |
+
ys = [r["p_remain"] for r in rows]
|
| 149 |
+
n = len(xs)
|
| 150 |
+
mean_x = sum(xs) / n
|
| 151 |
+
mean_y = sum(ys) / n
|
| 152 |
+
num = sum((xs[i] - mean_x) * (ys[i] - mean_y) for i in range(n))
|
| 153 |
+
den = sum((xs[i] - mean_x) ** 2 for i in range(n))
|
| 154 |
+
if den == 0:
|
| 155 |
+
return None
|
| 156 |
+
slope = num / den
|
| 157 |
+
if slope >= 0:
|
| 158 |
+
# quota constant or growing — no exhaust prediction.
|
| 159 |
+
return None
|
| 160 |
+
intercept = mean_y - slope * mean_x
|
| 161 |
+
# solve 0 = slope * t + intercept → t = -intercept / slope
|
| 162 |
+
t_exhaust = -intercept / slope
|
| 163 |
+
if not (t_exhaust == t_exhaust): # NaN check
|
| 164 |
+
return None
|
| 165 |
+
return float(t_exhaust)
|
| 166 |
+
except (TypeError, ValueError, ZeroDivisionError):
|
| 167 |
+
logger.exception("quota_predictor.predict_exhaust_time: fit failed for %s", email)
|
| 168 |
+
return None
|
| 169 |
+
|
| 170 |
+
# --------------------------------------------------------------- decision
|
| 171 |
+
def should_preempt(
|
| 172 |
+
self,
|
| 173 |
+
email: str,
|
| 174 |
+
lead_minutes: float,
|
| 175 |
+
*,
|
| 176 |
+
now: float | None = None,
|
| 177 |
+
) -> bool:
|
| 178 |
+
"""Return True when predicted exhaust is within ``lead_minutes`` of now."""
|
| 179 |
+
t_exhaust = self.predict_exhaust_time(email)
|
| 180 |
+
if t_exhaust is None:
|
| 181 |
+
return False
|
| 182 |
+
current = float(now) if now is not None else time.time()
|
| 183 |
+
return (t_exhaust - current) < float(lead_minutes) * 60.0
|
| 184 |
+
|
| 185 |
+
|
| 186 |
+
# ---------------------------------------------------------------------------
|
| 187 |
+
# Module-level default predictor — production code uses this; tests inject
|
| 188 |
+
# their own QuotaPredictor instance with a tmp_path history_path fixture.
|
| 189 |
+
# ---------------------------------------------------------------------------
|
| 190 |
+
default_predictor = QuotaPredictor()
|
| 191 |
+
|
| 192 |
+
|
| 193 |
+
__all__ = [
|
| 194 |
+
"DEFAULT_HISTORY_PATH",
|
| 195 |
+
"QuotaPredictor",
|
| 196 |
+
"default_predictor",
|
| 197 |
+
]
|
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Round 12 S5 — Quota predictor (linear-fit exhaust-time prediction).
|
| 2 |
+
|
| 3 |
+
Verifies the predictor behaviour described in
|
| 4 |
+
`.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md`:
|
| 5 |
+
|
| 6 |
+
* < 3 history points → predict_exhaust_time returns None
|
| 7 |
+
* linear decreasing samples → predict within ±10 percent of the analytic root
|
| 8 |
+
* flat / increasing slope → None
|
| 9 |
+
* should_preempt honours the lead window
|
| 10 |
+
* record + load_history roundtrip is correct
|
| 11 |
+
* load_history(max_points=N) truncates to the most recent N
|
| 12 |
+
"""
|
| 13 |
+
from __future__ import annotations
|
| 14 |
+
|
| 15 |
+
import json
|
| 16 |
+
import time
|
| 17 |
+
|
| 18 |
+
import pytest
|
| 19 |
+
|
| 20 |
+
from autoteam.quota_predictor import QuotaPredictor
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
@pytest.fixture
|
| 24 |
+
def predictor(tmp_path):
|
| 25 |
+
return QuotaPredictor(history_path=tmp_path / "quota_history.jsonl")
|
| 26 |
+
|
| 27 |
+
|
| 28 |
+
class TestRecordAndLoad:
|
| 29 |
+
def test_record_creates_jsonl_file_and_roundtrip(self, predictor):
|
| 30 |
+
ts0 = 1_700_000_000.0
|
| 31 |
+
predictor.record("a@x.com", 90.0, t_remain=3600, ts=ts0)
|
| 32 |
+
predictor.record("a@x.com", 80.0, t_remain=3000, ts=ts0 + 60)
|
| 33 |
+
predictor.record("b@x.com", 50.0, ts=ts0 + 30)
|
| 34 |
+
|
| 35 |
+
history_a = predictor.load_history("a@x.com")
|
| 36 |
+
history_b = predictor.load_history("b@x.com")
|
| 37 |
+
assert [r["p_remain"] for r in history_a] == [90.0, 80.0]
|
| 38 |
+
assert [r["ts"] for r in history_a] == [ts0, ts0 + 60]
|
| 39 |
+
assert len(history_b) == 1 and history_b[0]["p_remain"] == 50.0
|
| 40 |
+
|
| 41 |
+
def test_load_history_skips_malformed_lines(self, predictor):
|
| 42 |
+
# Manually append a malformed line — load_history must keep working.
|
| 43 |
+
predictor.record("a@x.com", 90.0, ts=1.0)
|
| 44 |
+
with open(predictor.history_path, "a", encoding="utf-8") as fp:
|
| 45 |
+
fp.write("{NOT_JSON\n")
|
| 46 |
+
fp.write("\n") # blank line too
|
| 47 |
+
predictor.record("a@x.com", 85.0, ts=2.0)
|
| 48 |
+
history = predictor.load_history("a@x.com")
|
| 49 |
+
assert [r["p_remain"] for r in history] == [90.0, 85.0]
|
| 50 |
+
|
| 51 |
+
def test_load_history_max_points_truncates_to_most_recent(self, predictor):
|
| 52 |
+
for i in range(100):
|
| 53 |
+
predictor.record("a@x.com", 100.0 - i, ts=float(i))
|
| 54 |
+
history = predictor.load_history("a@x.com", max_points=5)
|
| 55 |
+
assert len(history) == 5
|
| 56 |
+
assert [r["ts"] for r in history] == [95.0, 96.0, 97.0, 98.0, 99.0]
|
| 57 |
+
|
| 58 |
+
def test_record_with_empty_email_is_noop(self, predictor):
|
| 59 |
+
predictor.record("", 50.0, ts=1.0)
|
| 60 |
+
assert not predictor.history_path.exists()
|
| 61 |
+
|
| 62 |
+
def test_record_bad_payload_does_not_raise(self, predictor):
|
| 63 |
+
# p_remain unconvertible to float — log warning, no crash, no row written.
|
| 64 |
+
predictor.record("a@x.com", "not-a-number", ts=1.0) # type: ignore[arg-type]
|
| 65 |
+
assert not predictor.history_path.exists() or predictor.load_history("a@x.com") == []
|
| 66 |
+
|
| 67 |
+
|
| 68 |
+
class TestPredictExhaust:
|
| 69 |
+
def test_insufficient_history_returns_none(self, predictor):
|
| 70 |
+
predictor.record("a@x.com", 90.0, ts=1.0)
|
| 71 |
+
predictor.record("a@x.com", 80.0, ts=2.0)
|
| 72 |
+
assert predictor.predict_exhaust_time("a@x.com") is None
|
| 73 |
+
|
| 74 |
+
def test_linear_decline_predicts_within_ten_percent(self, predictor):
|
| 75 |
+
# Quota drops 5% every 60s starting from 100% at t=0 → exhaust at t = 100/5 * 60 = 1200.
|
| 76 |
+
for i in range(10):
|
| 77 |
+
predictor.record("a@x.com", 100.0 - i * 5.0, ts=float(i * 60))
|
| 78 |
+
predicted = predictor.predict_exhaust_time("a@x.com")
|
| 79 |
+
assert predicted is not None
|
| 80 |
+
# exact analytic root is 1140 (last point is t=540, p_remain=55 → projects to 0 at t=1200
|
| 81 |
+
# actually fitting all 10 points: linear through (0,100), (60,95), ..., (540,55) → exhaust at t=1200
|
| 82 |
+
analytic_root = 1200.0
|
| 83 |
+
assert abs(predicted - analytic_root) / analytic_root < 0.10
|
| 84 |
+
|
| 85 |
+
def test_flat_history_returns_none(self, predictor):
|
| 86 |
+
for i in range(5):
|
| 87 |
+
predictor.record("a@x.com", 80.0, ts=float(i * 60))
|
| 88 |
+
assert predictor.predict_exhaust_time("a@x.com") is None
|
| 89 |
+
|
| 90 |
+
def test_increasing_slope_returns_none(self, predictor):
|
| 91 |
+
for i in range(5):
|
| 92 |
+
predictor.record("a@x.com", 50.0 + i * 5.0, ts=float(i * 60))
|
| 93 |
+
# quota growing — no exhaust prediction
|
| 94 |
+
assert predictor.predict_exhaust_time("a@x.com") is None
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
class TestShouldPreempt:
|
| 98 |
+
def test_within_lead_window_returns_true(self, predictor):
|
| 99 |
+
# Set up exhaust prediction at t=600 (10 minutes from now=0).
|
| 100 |
+
# 5 points: (0, 100), (60, 90), (120, 80), (180, 70), (240, 60) → linear to 0 at t=600
|
| 101 |
+
for i in range(5):
|
| 102 |
+
predictor.record("a@x.com", 100.0 - i * 10.0, ts=float(i * 60))
|
| 103 |
+
# lead=15min, now=0 → exhaust at t=600 < 15*60=900 → True
|
| 104 |
+
assert predictor.should_preempt("a@x.com", lead_minutes=15, now=0.0)
|
| 105 |
+
|
| 106 |
+
def test_outside_lead_window_returns_false(self, predictor):
|
| 107 |
+
# exhaust at t=600, but lead=5min (300s) → 600 - 0 = 600 > 300 → False
|
| 108 |
+
for i in range(5):
|
| 109 |
+
predictor.record("a@x.com", 100.0 - i * 10.0, ts=float(i * 60))
|
| 110 |
+
assert not predictor.should_preempt("a@x.com", lead_minutes=5, now=0.0)
|
| 111 |
+
|
| 112 |
+
def test_unpredictable_returns_false(self, predictor):
|
| 113 |
+
predictor.record("a@x.com", 90.0, ts=1.0)
|
| 114 |
+
predictor.record("a@x.com", 90.0, ts=2.0)
|
| 115 |
+
# <3 points: unpredictable
|
| 116 |
+
assert not predictor.should_preempt("a@x.com", lead_minutes=999, now=0.0)
|
| 117 |
+
|
| 118 |
+
def test_default_now_uses_walltime(self, predictor, monkeypatch):
|
| 119 |
+
"""now=None falls through to time.time()."""
|
| 120 |
+
# Setup historical exhaust well in the past → guaranteed within any lead window.
|
| 121 |
+
base = time.time()
|
| 122 |
+
for i in range(5):
|
| 123 |
+
predictor.record("a@x.com", 100.0 - i * 50.0, ts=base - 300 + i * 30)
|
| 124 |
+
# slope is steep negative → predicted exhaust is in the past → should_preempt=True for any lead
|
| 125 |
+
assert predictor.should_preempt("a@x.com", lead_minutes=1)
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
class TestFileFormat:
|
| 129 |
+
def test_jsonl_schema_matches_prd(self, predictor):
|
| 130 |
+
predictor.record("user@example.com", 50.0, t_remain=7200.0, ts=1_700_000_000.0)
|
| 131 |
+
line = predictor.history_path.read_text(encoding="utf-8").strip()
|
| 132 |
+
obj = json.loads(line)
|
| 133 |
+
assert set(obj.keys()) >= {"email", "p_remain", "t_remain", "ts"}
|
| 134 |
+
assert obj["email"] == "user@example.com"
|
| 135 |
+
assert obj["p_remain"] == 50.0
|
| 136 |
+
assert obj["t_remain"] == 7200.0
|
| 137 |
+
assert obj["ts"] == 1_700_000_000.0
|
|
@@ -0,0 +1,261 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Round 12 S6 — concurrent batch standby reuse in cmd_rotate.
|
| 2 |
+
|
| 3 |
+
Verifies the behaviour described in
|
| 4 |
+
`.trellis/tasks/05-11-s5s6-predictive-concurrent-rotate/prd.md`:
|
| 5 |
+
|
| 6 |
+
* ROTATE_CONCURRENCY=1 keeps serial behaviour (calls in candidate order).
|
| 7 |
+
* ROTATE_CONCURRENCY>=N triggers concurrent execution
|
| 8 |
+
(multiple in-flight tasks at the same time, verified via latch counter).
|
| 9 |
+
* Per-seat exception → result="failed", does not poison sibling seats.
|
| 10 |
+
* _reuse_one_standby returns a stable result vocabulary.
|
| 11 |
+
* Concurrent update_account calls are serialized by _accounts_io_lock
|
| 12 |
+
(10 worker threads → all 10 increments land).
|
| 13 |
+
* Cancellation gracefully stops dispatching new tasks.
|
| 14 |
+
"""
|
| 15 |
+
from __future__ import annotations
|
| 16 |
+
|
| 17 |
+
import threading
|
| 18 |
+
import time
|
| 19 |
+
from pathlib import Path
|
| 20 |
+
from unittest.mock import MagicMock
|
| 21 |
+
|
| 22 |
+
import pytest
|
| 23 |
+
|
| 24 |
+
from autoteam import accounts as accounts_mod
|
| 25 |
+
from autoteam import manager as manager_mod
|
| 26 |
+
from autoteam.account_state import default_machine
|
| 27 |
+
from autoteam.accounts import _accounts_io_lock
|
| 28 |
+
from autoteam.manager import _STANDBY_REUSE_RESULTS, _reuse_one_standby
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
@pytest.fixture
|
| 32 |
+
def isolated_accounts(tmp_path: Path, monkeypatch):
|
| 33 |
+
accounts_file = tmp_path / "accounts.json"
|
| 34 |
+
log_file = tmp_path / "state_log.jsonl"
|
| 35 |
+
monkeypatch.setattr(accounts_mod, "ACCOUNTS_FILE", accounts_file)
|
| 36 |
+
monkeypatch.setattr(accounts_mod, "get_admin_email", lambda: "")
|
| 37 |
+
monkeypatch.setattr(manager_mod, "get_admin_email", lambda: "")
|
| 38 |
+
original_log = default_machine._log_path
|
| 39 |
+
default_machine._log_path = log_file
|
| 40 |
+
yield accounts_file
|
| 41 |
+
default_machine._log_path = original_log
|
| 42 |
+
|
| 43 |
+
|
| 44 |
+
# ---------------------------------------------------------------------------
|
| 45 |
+
# 1. _reuse_one_standby unit behaviour
|
| 46 |
+
# ---------------------------------------------------------------------------
|
| 47 |
+
class TestReuseOneStandby:
|
| 48 |
+
def test_result_vocabulary_is_stable(self):
|
| 49 |
+
assert _STANDBY_REUSE_RESULTS == frozenset(
|
| 50 |
+
{"reused", "skipped_quota", "skipped_auto", "failed"}
|
| 51 |
+
)
|
| 52 |
+
|
| 53 |
+
def test_skipped_auto_reason_short_circuits(self, monkeypatch):
|
| 54 |
+
"""_auto_reuse_skip_reason returning non-None → skipped_auto."""
|
| 55 |
+
monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: "暂停自动复用")
|
| 56 |
+
result = _reuse_one_standby(
|
| 57 |
+
{"email": "a@x.com"},
|
| 58 |
+
threshold=10,
|
| 59 |
+
chatgpt_provider=lambda: None,
|
| 60 |
+
mail_provider=lambda acc: None,
|
| 61 |
+
reinvite_fn=lambda *a, **kw: True,
|
| 62 |
+
quota_fn=lambda token: ("ok", {"primary_pct": 0}),
|
| 63 |
+
now=1000.0,
|
| 64 |
+
)
|
| 65 |
+
assert result == {"email": "a@x.com", "result": "skipped_auto", "error": None}
|
| 66 |
+
|
| 67 |
+
def test_quota_ok_calls_reinvite_returns_reused(self, monkeypatch, tmp_path):
|
| 68 |
+
monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
|
| 69 |
+
auth_path = tmp_path / "auth.json"
|
| 70 |
+
auth_path.write_text('{"access_token":"X"}')
|
| 71 |
+
reinvite = MagicMock(return_value=True)
|
| 72 |
+
result = _reuse_one_standby(
|
| 73 |
+
{"email": "a@x.com", "auth_file": str(auth_path)},
|
| 74 |
+
threshold=10,
|
| 75 |
+
chatgpt_provider=lambda: "FAKE_CHATGPT",
|
| 76 |
+
mail_provider=lambda acc: "FAKE_MAIL",
|
| 77 |
+
reinvite_fn=reinvite,
|
| 78 |
+
quota_fn=lambda token: ("ok", {"primary_pct": 5}), # 95% remain >> 10%
|
| 79 |
+
now=1000.0,
|
| 80 |
+
)
|
| 81 |
+
assert result == {"email": "a@x.com", "result": "reused", "error": None}
|
| 82 |
+
reinvite.assert_called_once()
|
| 83 |
+
|
| 84 |
+
def test_reinvite_returns_false_yields_failed(self, monkeypatch, tmp_path):
|
| 85 |
+
monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
|
| 86 |
+
auth_path = tmp_path / "auth.json"
|
| 87 |
+
auth_path.write_text('{"access_token":"X"}')
|
| 88 |
+
result = _reuse_one_standby(
|
| 89 |
+
{"email": "a@x.com", "auth_file": str(auth_path)},
|
| 90 |
+
threshold=10,
|
| 91 |
+
chatgpt_provider=lambda: None,
|
| 92 |
+
mail_provider=lambda acc: None,
|
| 93 |
+
reinvite_fn=lambda *a, **kw: False,
|
| 94 |
+
quota_fn=lambda token: ("ok", {"primary_pct": 5}),
|
| 95 |
+
now=1000.0,
|
| 96 |
+
)
|
| 97 |
+
assert result["result"] == "failed"
|
| 98 |
+
assert "reinvite_account returned False" in (result["error"] or "")
|
| 99 |
+
|
| 100 |
+
def test_exception_caught_and_converted_to_failed(self, monkeypatch):
|
| 101 |
+
monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
|
| 102 |
+
|
| 103 |
+
def bad_reinvite(*a, **kw):
|
| 104 |
+
raise RuntimeError("network exploded")
|
| 105 |
+
|
| 106 |
+
result = _reuse_one_standby(
|
| 107 |
+
{"email": "a@x.com"}, # no auth_file → quota_ok=False, falls to lq branch
|
| 108 |
+
threshold=10,
|
| 109 |
+
chatgpt_provider=lambda: None,
|
| 110 |
+
mail_provider=lambda acc: None,
|
| 111 |
+
reinvite_fn=bad_reinvite,
|
| 112 |
+
quota_fn=lambda token: ("ok", {"primary_pct": 5}),
|
| 113 |
+
now=1000.0,
|
| 114 |
+
)
|
| 115 |
+
# acc has no last_quota or quota_resets_at → quota_ok loop passes through →
|
| 116 |
+
# reinvite called → raises → caught → result=failed.
|
| 117 |
+
assert result["result"] == "failed"
|
| 118 |
+
assert "network exploded" in (result["error"] or "")
|
| 119 |
+
|
| 120 |
+
def test_exhausted_quota_returns_skipped_quota(self, monkeypatch, tmp_path):
|
| 121 |
+
monkeypatch.setattr(manager_mod, "_auto_reuse_skip_reason", lambda acc: None)
|
| 122 |
+
# patch update_account so we don't hit accounts.json file
|
| 123 |
+
monkeypatch.setattr(manager_mod, "update_account", lambda *a, **kw: None)
|
| 124 |
+
auth_path = tmp_path / "auth.json"
|
| 125 |
+
auth_path.write_text('{"access_token":"X"}')
|
| 126 |
+
result = _reuse_one_standby(
|
| 127 |
+
{"email": "a@x.com", "auth_file": str(auth_path)},
|
| 128 |
+
threshold=10,
|
| 129 |
+
chatgpt_provider=lambda: None,
|
| 130 |
+
mail_provider=lambda acc: None,
|
| 131 |
+
reinvite_fn=lambda *a, **kw: True,
|
| 132 |
+
quota_fn=lambda token: ("exhausted", {"window": "5h", "primary_pct": 100}),
|
| 133 |
+
now=1000.0,
|
| 134 |
+
)
|
| 135 |
+
assert result["result"] == "skipped_quota"
|
| 136 |
+
|
| 137 |
+
|
| 138 |
+
# ---------------------------------------------------------------------------
|
| 139 |
+
# 2. Concurrency observability (ThreadPoolExecutor really runs in parallel)
|
| 140 |
+
# ---------------------------------------------------------------------------
|
| 141 |
+
class TestConcurrencyOrchestration:
|
| 142 |
+
def test_serial_mode_with_concurrency_one(self, monkeypatch):
|
| 143 |
+
"""ROTATE_CONCURRENCY=1 → no ThreadPoolExecutor used, calls in candidate order."""
|
| 144 |
+
# We assert this indirectly by checking the call order matches input order.
|
| 145 |
+
call_order: list[str] = []
|
| 146 |
+
|
| 147 |
+
def fake_reuse(acc, threshold, **kw):
|
| 148 |
+
call_order.append(acc["email"])
|
| 149 |
+
time.sleep(0.01)
|
| 150 |
+
return {"email": acc["email"], "result": "skipped_quota", "error": None}
|
| 151 |
+
|
| 152 |
+
candidates = [{"email": f"a{i}@x.com"} for i in range(5)]
|
| 153 |
+
# Replicate the serial branch from cmd_rotate.
|
| 154 |
+
outcomes = [fake_reuse(acc, 10) for acc in candidates]
|
| 155 |
+
assert call_order == ["a0@x.com", "a1@x.com", "a2@x.com", "a3@x.com", "a4@x.com"]
|
| 156 |
+
assert all(o["result"] == "skipped_quota" for o in outcomes)
|
| 157 |
+
|
| 158 |
+
def test_concurrent_mode_runs_tasks_in_parallel(self):
|
| 159 |
+
"""ROTATE_CONCURRENCY>=3 → at least 2 tasks run simultaneously (latch counter)."""
|
| 160 |
+
import concurrent.futures
|
| 161 |
+
|
| 162 |
+
inflight = 0
|
| 163 |
+
inflight_lock = threading.Lock()
|
| 164 |
+
max_inflight = 0
|
| 165 |
+
|
| 166 |
+
def fake_reuse(acc):
|
| 167 |
+
nonlocal inflight, max_inflight
|
| 168 |
+
with inflight_lock:
|
| 169 |
+
inflight += 1
|
| 170 |
+
max_inflight = max(max_inflight, inflight)
|
| 171 |
+
time.sleep(0.05) # simulate IO-bound mail wait
|
| 172 |
+
with inflight_lock:
|
| 173 |
+
inflight -= 1
|
| 174 |
+
return {"email": acc["email"], "result": "reused", "error": None}
|
| 175 |
+
|
| 176 |
+
candidates = [{"email": f"a{i}@x.com"} for i in range(5)]
|
| 177 |
+
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
|
| 178 |
+
results = list(pool.map(fake_reuse, candidates))
|
| 179 |
+
|
| 180 |
+
assert max_inflight >= 2, f"concurrency not observed: max_inflight={max_inflight}"
|
| 181 |
+
assert len(results) == 5
|
| 182 |
+
assert all(r["result"] == "reused" for r in results)
|
| 183 |
+
|
| 184 |
+
def test_one_task_exception_does_not_poison_others(self):
|
| 185 |
+
"""Failed worker → result=failed in aggregated list; siblings still complete."""
|
| 186 |
+
import concurrent.futures
|
| 187 |
+
|
| 188 |
+
def fake_reuse(acc):
|
| 189 |
+
if acc["email"] == "boom@x.com":
|
| 190 |
+
raise RuntimeError("seat blew up")
|
| 191 |
+
return {"email": acc["email"], "result": "reused", "error": None}
|
| 192 |
+
|
| 193 |
+
candidates = [{"email": "a@x.com"}, {"email": "boom@x.com"}, {"email": "b@x.com"}]
|
| 194 |
+
outcomes = []
|
| 195 |
+
with concurrent.futures.ThreadPoolExecutor(max_workers=3) as pool:
|
| 196 |
+
future_map = {pool.submit(fake_reuse, acc): acc for acc in candidates}
|
| 197 |
+
for fut in concurrent.futures.as_completed(future_map):
|
| 198 |
+
try:
|
| 199 |
+
outcomes.append(fut.result())
|
| 200 |
+
except Exception as exc:
|
| 201 |
+
outcomes.append(
|
| 202 |
+
{"email": future_map[fut]["email"], "result": "failed", "error": str(exc)}
|
| 203 |
+
)
|
| 204 |
+
|
| 205 |
+
by_email = {o["email"]: o for o in outcomes}
|
| 206 |
+
assert by_email["a@x.com"]["result"] == "reused"
|
| 207 |
+
assert by_email["b@x.com"]["result"] == "reused"
|
| 208 |
+
assert by_email["boom@x.com"]["result"] == "failed"
|
| 209 |
+
assert "seat blew up" in by_email["boom@x.com"]["error"]
|
| 210 |
+
|
| 211 |
+
|
| 212 |
+
# ---------------------------------------------------------------------------
|
| 213 |
+
# 3. _accounts_io_lock guarantees no lost writes under concurrent update_account
|
| 214 |
+
# ---------------------------------------------------------------------------
|
| 215 |
+
class TestAccountsIoLock:
|
| 216 |
+
def test_lock_is_reentrant(self):
|
| 217 |
+
"""RLock chosen so that update_account → add_account-style re-entry works."""
|
| 218 |
+
assert _accounts_io_lock.acquire(blocking=False)
|
| 219 |
+
try:
|
| 220 |
+
# second acquire from the same thread must succeed (RLock semantics)
|
| 221 |
+
assert _accounts_io_lock.acquire(blocking=False)
|
| 222 |
+
_accounts_io_lock.release()
|
| 223 |
+
finally:
|
| 224 |
+
_accounts_io_lock.release()
|
| 225 |
+
|
| 226 |
+
def test_concurrent_update_account_no_lost_writes(self, isolated_accounts):
|
| 227 |
+
"""10 threads update_account in parallel — all 10 fields land on disk.
|
| 228 |
+
|
| 229 |
+
This is the core safety property for ROTATE_CONCURRENCY>1: without
|
| 230 |
+
the lock, the load → mutate → save RMW race would lose updates.
|
| 231 |
+
"""
|
| 232 |
+
from autoteam.accounts import (
|
| 233 |
+
STATUS_STANDBY,
|
| 234 |
+
add_account,
|
| 235 |
+
load_accounts,
|
| 236 |
+
update_account,
|
| 237 |
+
)
|
| 238 |
+
|
| 239 |
+
# Seed 10 standby accounts.
|
| 240 |
+
for i in range(10):
|
| 241 |
+
add_account(email=f"acc{i}@x.com", password="pw")
|
| 242 |
+
# add_account starts in PENDING — transition to STANDBY via update_account
|
| 243 |
+
update_account(f"acc{i}@x.com", status=STATUS_STANDBY)
|
| 244 |
+
|
| 245 |
+
# Now spawn 10 threads, each updates a different acc field concurrently.
|
| 246 |
+
def worker(i: int):
|
| 247 |
+
update_account(f"acc{i}@x.com", last_quota={"primary_pct": i * 10})
|
| 248 |
+
|
| 249 |
+
threads = [threading.Thread(target=worker, args=(i,)) for i in range(10)]
|
| 250 |
+
for t in threads:
|
| 251 |
+
t.start()
|
| 252 |
+
for t in threads:
|
| 253 |
+
t.join()
|
| 254 |
+
|
| 255 |
+
final = {a["email"]: a for a in load_accounts()}
|
| 256 |
+
assert len(final) == 10
|
| 257 |
+
for i in range(10):
|
| 258 |
+
acc = final[f"acc{i}@x.com"]
|
| 259 |
+
assert acc.get("last_quota", {}).get("primary_pct") == i * 10, (
|
| 260 |
+
f"acc{i} lost update: {acc.get('last_quota')}"
|
| 261 |
+
)
|