doatlas-2 / docs /PROGRESS.md
Iostream-Li's picture
Add files using upload-large-folder tool
2a55985 verified
# DoAtlas 项目进度恢复文档
> 生成时间:2026-04-27 22:35 UTC
> 配套压缩包:`exports/doatlas-snapshot-20260427-2234.zip`(约 173 MB)
> 适用版本:`replit.md` 中所述「Wave B + #221 通用 6 通道 Reviewer」之后的状态
> 本文目的:换台机器、换个 Replit、或者隔几周回来,能照着这份文档把项目状态完整恢复,并接着没跑完的待办继续推进。
---
## 1. 项目当前定位
DoAtlas 是一个**生物医学研究协同工作台**,pnpm 单仓多包(monorepo)+ Node 24 + TypeScript 5.9 + Postgres + Python 研究引擎。当前阶段已经完成:
- 三波前端 + 后端基础(视觉打磨、核心产品形态、性能/可达性)
- 第四波 Python 研究引擎接入(HMAC 内部桥、6 个工具、DuckDB 快照)
- 实时聊天 + ReAct 工具循环 + 后台任务面板
- 长期跨会话记忆(opt-in)
- 因果网/药物网持久化进 Postgres
- Wave A 工具网注册中心 + Wave B 在线进化飞轮(影子 A/B、5 门 autoPromote、30 分钟 watchdog)
- Task #221 通用 6 通道 Reviewer 落地,drugclip 跑通 promote → rollback 闭环(详见 `docs/wave-b-acceptance-report.md`
仓库共 95 个项目任务:**55 已合并(MERGED)/ 13 已取消(CANCELLED)/ 27 待办(PROPOSED)**
---
## 2. 4 个 artifact 概览
| Artifact | 类型 | 端口 | 路径前缀 | 简介 |
|---|---|---|---|---|
| `@workspace/api-server` | api(Express 5 + Drizzle) | 8080 | `/api` | 业务 API、会话/聊天编排、工具路由、Wave B 飞轮、admin 面板 |
| `@workspace/doatlas-web` | web(Vite + React 19) | 23232 | `/` | 用户工作台 SPA:聊天、图谱、排行、审阅、快照、admin |
| `@workspace/research-engine` | api(Flask + DuckDB) | 8011 | `/research-engine` | Python 研究流水线 + 工具执行;只接受 HMAC 已签名内部请求 |
| `@workspace/mockup-sandbox` | design(Vite) | 8081 | `/__mockup` | 内部组件预览,不参与生产 |
共享库(`lib/`):
- `lib/db` — Drizzle 模型 + 迁移(PostgreSQL)
- `lib/api-spec` — OpenAPI v1.1(`docs/API_CONTRACT.md` 是中文契约)
- `lib/api-zod``lib/api-client-react` — 由 `pnpm --filter @workspace/api-spec run codegen` 自动生成
详细架构见 `docs/WORKFLOW.md``docs/REPLIT_SETUP.md``docs/LOCAL_SETUP.md`
---
## 3. 已完成(MERGED,55 个)— 按主题分组
> 排序按主题;每条带任务编号方便对照 `.local/tasks/` 下的源描述。
### 3.1 后端基础修复 + 核心 API(Wave 0–1)
- `#1` 修复后端崩溃导致工作台白屏与聊天报错
- `#2` 实现登录、聊天等核心后端接口
- `#3` 为前后端关键流程补充端到端测试
- `#13` Fix production login CORS error
- `#14` Show a clearer message when login is blocked by browser security
### 3.2 视觉打磨 / 工作台 / 性能(Wave 1–3)
- `#15` 第一波:视觉打磨(登录/工作台/设置全面升级到 Claude/ChatGPT 级别)
- `#16` 第二波:核心产品形态(因果图谱+证据溯源+审核中心+5 任务模式+P2 功能)
- `#17` 第三波:体验流畅与性能(首屏/虚拟化/SSE/键盘/移动端/a11y)
- `#71` Make the Causal Network browsable in the workbench
### 3.3 全栈生产化与部署
- `#9` DoAtlas 全栈生产化(双 profile + 真实 LLM + 数据源 + 部署)
### 3.4 Python 研究引擎接入(Wave 4)
- `#18` 第四波:接入 Python 研究引擎(在线跑 M2-M10 流水线产新边)
- `#26` Hook the research tools up to the real knowledge graph data
- `#27` Guard the research bridge against replayed requests before real pipelines go live
- `#28` Make replay protection survive running multiple research engine workers
- `#32` Run the real fulltext, validation, and codex pipelines instead of placeholders
- `#33` Mount the snapshot dataset into the research engine in production
### 3.5 聊天与工具循环(ReAct + 自主代理)
- `#25` Let the research assistant actually run and stream tool results in chat
- `#36` End-to-end user journey QA on the live workbench
- `#53` Make the chat assistant truly autonomous (plan, recover, self-check)
- `#54` Replay agent activity for past assistant messages on demand
- `#55` Apply the autonomous agent loop to background research pipelines
- `#56` Add automated tests for the agent loop's recovery and self-check paths
- `#57` Add a manual refresh option for the agent activity panel
- `#82` Show partial agent activity in the in-chat assistant message too
- `#83` Test the live progress panel updates as the engine streams snapshots
### 3.6 后台任务(pipeline supervisor / jobs panel)
- `#44` Show pending jobs in chat with progress and a follow-up when each finishes
- `#45` Let users cancel a long-running research job from the chat
- `#46` Show every past background job in the conversation timeline
- `#47` Add a regression test for the conversation jobs history panel
- `#48` Let users filter or search the background jobs history
- `#51` Remember the user's preferred history filters across visits
- `#52` Highlight matched search terms inside job summaries
- `#59` Show the new pipeline activity panel in the workbench UI
- `#60` Save background pipeline work so a server restart doesn't lose it
- `#61` Add automated tests for the pipeline supervisor
- `#69` Show partial pipeline progress in the agent activity panel
### 3.7 长期记忆(cross-session memory)
- `#58` Long-term cross-session memory (opt-in)
- `#62` Let users edit saved memories, not just delete them
- `#63` Group injected memories by category in the activity panel
- `#64` Add automated tests for the long-term memory pipeline
- `#84` Finalize long-term memory (#58 wiring + e2e)
- `#85` Lock down the long-term memory status response so the UI never breaks
- `#86` Type the research-job agent snapshot column so the build stops needing a manual cast
### 3.8 因果网与药物网持久化
- `#70` 冻结因果网与药物网接口设计 + 真实数据回环测试
- `#72` Persist the Causal Network into the database for fast queries
- `#73` Hook M3/M5 runners up to the new network format
- `#79` Wire the chat agent and graph browser to the new persistent causal store
- `#80` Pool causal-network estimates into a single number per bucket
- `#81` Persist the Drug Network the same way as the Causal Network
- `#87` Pool ratio measures (HR/OR/RR) on the log scale automatically
- `#88` Refresh the literature summary after a paper is removed
- `#89` Show the pooled literature estimate in the chat evidence drawer
- `#90` Wire the chat agent and graph browser to read drug data from the database
- `#91` Surface drug evidence by joining drugs to causal edges in one query
- `#95` Test that drug evidence shows up in the side panel and chat answers
> 备注:Wave A(工具网注册中心)/ Wave B(在线进化飞轮 + Reviewer 通用化)相关任务(#177、#181、#214、#215、#221 等)的实现细节与验收报告已合并到 `replit.md` 顶层 + `docs/wave-b-acceptance-report.md`,不在 95 个任务计数内(属于上层的能力框架阶段)。
---
## 4. 已取消(CANCELLED,13 个)— 含取消原因
| 任务 | 标题 | 取消原因(推断) |
|---|---|---|
| `#4` | 把账号、对话和任务保存到数据库, 重启不丢失 | 已被 `#2` 实现登录、聊天等核心后端接口覆盖(持久化随核心 API 一起做了) |
| `#5` | 让聊天接入真正的大模型, 给出有价值的回复 | 已被 `#9` 全栈生产化(含真实 LLM 接入)覆盖 |
| `#6` | 完善研究任务、运行记录和制品的真实数据 | 路径已经被 Wave 4 `#18` 重新规划成 Python 引擎接入 |
| `#7` | 在 CI 中自动运行端到端测试 | 改走 `Test Gate` workflow + `scripts/test-gate.sh`(参见 `docs/TESTING.md`) |
| `#8` | 补充会话切换、新建会话与登出的端到端测试 | 已并入 `#3` 的 e2e 套件与 Playwright 配置 |
| `#10` | Connect the chat UI to the live backend end-to-end | 已被 `#9` + `#25` 覆盖 |
| `#11` | Ship the local-deployment kit (Codex CLI + GLM + MiniMax) | 部分能力(Codex CLI adapter)改在 `docs/CODEX_CLI_ADAPTER.md` 单独维护,多 provider 走 Replit AI Integrations 反代 |
| `#12` | Publish a complete OpenAPI spec and wire it into CI | 改成 `lib/api-spec` + Orval 生成 zod/hooks,CI 不强卡 |
| `#39` | Plug in the real LLM full-text extractor when an API key is available | 已并入 `#32` 真实 pipeline 接入 |
| `#40` | Call the real Codex worker for analysis when its binary is mounted | 同上,CodexCLI runner 在适配器里完成 |
| `#76` | Show the Causal Network as an interactive node-link diagram | 改成表格 + bucket 浏览(`#71`),节点-链路图被搁置 |
| `#77` | Link drug trials to the literature evidence that mentions them | 已被 `#89`/`#91` 覆盖(药物-证据 join 在 SQL 一次完成) |
| `#78` | Load the full DrugNetwork from SQLite when the upstream DB ships | 改走 Postgres 持久化(`#81`/`#90`),SQLite 路径作废 |
---
## 5. 仍待办(PROPOSED,27 个)— 按主题 + 依赖分组
> 「依赖于」一栏标注的是该任务在创建时被记录的 dependsOn;当上游任务也是 MERGED 状态时表示当前已可启动。
### 5.1 持久化与审核数据闭环(Wave 4 收尾)
- `#19` 将审核决定持久化到 Postgres — 依赖 `#16`(已 MERGED,可起)
- `#20` 将证据抽屉里的 audit_checks 完整展示出来 — 依赖 `#16`(可起)
- `#21` 为工作台 4 个页面补充端到端回归测试 — 依赖 `#16`(可起)
### 5.2 体验流畅 / 可达性 / 性能(Wave 3 收尾)
- `#22` Let users resume interrupted chat replies when the network drops — 依赖 `#17`(可起)
- `#23` Verify the whole app is keyboard-and-screen-reader friendly — 依赖 `#17`(可起)
- `#24` Measure real-world page speed with automated performance checks — 依赖 `#17`(可起)
### 5.3 聊天与工具循环增强
- `#29` Show research-engine progress in the chat (status, partial results, errors) — 依赖 `#25`(可起)
- `#30` Make the research-vs-chat detector smarter than keyword matching — 依赖 `#25`(可起)
- `#31` Cover the chat tool loop with automated regression tests — 依赖 `#25`(可起)
- `#41` Make the assistant always answer the user's question, even when it runs research tools — 依赖 `#36`(可起)
- `#42` Stop the assistant from putting emojis and decorative ASCII boxes in answers — 依赖 `#36`(可起)
- `#43` Stop firing literature searches on every clinical-intake turn — 依赖 `#36`(可起)
### 5.4 HMAC / 引擎运维
- `#34` Set up shared replay protection automatically in production — 依赖 `#28`(可起)
- `#35` Stop unbounded growth of the nonce table when the engine sits idle — 依赖 `#28`(可起)
### 5.5 数据集 / 部署 pipeline
- `#37` Auto-update the snapshot pin when a newer dataset is downloaded — 依赖 `#33`(可起)
- `#38` Make the dataset download workflow run automatically before deployment — 依赖 `#33`(可起;目前 `Dataset Download` workflow 处于 failed 状态,先修后再做这两个)
### 5.6 后台任务(cancel + history)
- `#49` Cancel running tasks even after refreshing the page — 依赖 `#45`(可起)
- `#50` Cover the cancel-job flow with automated tests — 依赖 `#45`(可起)
### 5.7 长期记忆增强
- `#65` Let users hide entire memory categories from each turn's prompt — 依赖 `#63`(可起)
- `#66` Show injected memories on past assistant messages, not just fresh ones — 依赖 `#63`(可起)
### 5.8 后台 pipeline activity 流式
- `#67` Stream agent activity for background jobs while they're still running — 依赖 `#59`(可起)
- `#68` Test the background job activity panel end-to-end — 依赖 `#59`(可起)
### 5.9 因果网下游消费
- `#74` Wire downstream consumers (chat, graph, ranking) to the new local-causal-net files — 依赖 `#73`(可起)
- `#75` Expose the bucket aggregator as a chat tool so users can refresh the literature index — 依赖 `#73`(可起)
### 5.10 Postgres 化图谱后续
- `#92` Backfill production graph data into the new persistent store — 依赖 `#79`(可起)
- `#93` Show the richer evidence card from the persistent store on the edge drawer — 依赖 `#79`(可起)
- `#94` Cover the Postgres-backed graph routes with integration tests — 依赖 `#79`(可起)
> 推荐启动顺序:先把 5.1(审核持久化)+ 5.5(数据集 pipeline 修复)打通,让生产/部署链条不再依赖手工步骤;然后挑 5.10 + 5.4 收尾;体验类(5.2/5.3/5.6)可以并行。5.7/5.8/5.9 都是已有功能的延伸,可独立排期。
---
## 6. 关键运行入口与 workflow 状态
### 6.1 4 个 artifact 启动方式
```bash
# 一次安装
pnpm install
# Python 研究引擎
cd artifacts/research-engine && uv pip install --system -e . && cd -
# 单独启动(也可由 Replit workflows 自动起)
pnpm --filter @workspace/api-server run dev # 8080
pnpm --filter @workspace/doatlas-web run dev # 23232 (Vite)
pnpm --filter @workspace/research-engine run dev # 8011
pnpm --filter @workspace/mockup-sandbox run dev # 8081
```
### 6.2 当前 Replit workflow 状态(截至本文档生成时)
| Workflow | 用途 | 状态 |
|---|---|---|
| `artifacts/api-server: API Server` | 启动 Express api-server | running |
| `artifacts/doatlas-web: web` | 启动 Vite 前端 | running |
| `artifacts/research-engine: Research Engine` | 启动 Flask 引擎 | running |
| `artifacts/mockup-sandbox: Component Preview Server` | 内部组件预览 | running |
| `Test Gate` | `bash scripts/test-gate.sh` 全量测试门 | finished(最近一次完成) |
| `Provider Smoke Check` | `pnpm run smoke:providers` 探活 LLM 适配器 | **failed** ⚠️(informational 性质,不阻断生产;通常因第三方 LLM provider 当下不可达或 key 缺失,排查时先确认 `AI_INTEGRATIONS_*` 是否注入) |
| `Dataset Download` | `scripts/dataset-download.sh` 拉 808MB 快照 | **failed** ⚠️(建议在 ModelScope token 有效后重跑;不影响业务运行,但 `inspect_target` 在没有快照时会回退 `snapshot_unavailable`) |
| `agent-loop-tests` | `pnpm --filter @workspace/api-server test` | **failed** ⚠️(已知有 4 个 ephemeral schema 同步遗留 + 1 个 memory.test.mjs 长尾挂起,参见 `docs/wave-b-acceptance-report.md` CAVEAT-D;与生产路径无关) |
### 6.3 关键脚本
- `scripts/test-gate.sh` — 单一 CI 入口(hermetic 套件 ~25–60s);带 `--coverage` 出 c8 报告
- `scripts/start-api-server.sh` — Boot guard:先探活 healthz,命中就 sleep;没有就 pkill 再 exec
- `scripts/start-research-engine.sh` — 同上,避免 8011 端口竞争
- `scripts/dataset-download.sh` — DuckDB 快照下载,已存在则短路
- `scripts/post-merge.sh` — 合并后自动 `pnpm install` + `drizzle-kit push` + `seed:prod`
- `scripts/smoke-providers.ts` — LLM 多家供应商真流式探活
详细说明见 `docs/TESTING.md`、`docs/WORKFLOW.md`。
---
## 7. 换台机器 / 换环境后的恢复步骤
> 假设你拿到的是 `exports/doatlas-snapshot-YYYYMMDD-HHMM.zip`,目标机器是 Linux + Node 24 + Python 3.11(或 Replit)。
1. **解压**
```bash
unzip exports/doatlas-snapshot-20260427-2234.zip -d doatlas/
cd doatlas
```
2. **安装 Node 依赖**(必须 pnpm 9+)
```bash
corepack enable
pnpm install
```
3. **安装 Python 依赖**(推荐 uv)
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync # 用 uv.lock 还原
# 或者手动:
cd artifacts/research-engine && uv pip install --system -e . && cd -
```
4. **配置环境变量**
```bash
cp .env.replit.example .env
# 至少填:SESSION_SECRET, DOATLAS_BRIDGE_SECRET (openssl rand -hex 32),
# DATABASE_URL, INVITE_CODES (生产)
# AI_INTEGRATIONS_* 由 Replit 注入;本地需自填或用 Replit 反代
```
5. **数据库初始化**
```bash
pnpm --filter @workspace/db push
pnpm --filter @workspace/api-server seed:prod # 可选:写入 invite/admin
```
6. **数据集快照**(恢复 `inspect_target` 等工具)
```bash
bash scripts/dataset-download.sh
# 或手动:把 .duckdb 放到 artifacts/research-engine/data/snapshots/SNAP:*.duckdb
```
7. **启动 4 个 artifact**(本地用多个终端,Replit 用 workflows 自动起)
```bash
pnpm --filter @workspace/research-engine run dev &
pnpm --filter @workspace/api-server run dev &
pnpm --filter @workspace/doatlas-web run dev &
pnpm --filter @workspace/mockup-sandbox run dev &
```
8. **跑测试门**
```bash
pnpm run typecheck
bash scripts/test-gate.sh
pnpm --filter @workspace/research-engine run test
```
9. **冒烟(可选)**
```bash
pnpm run smoke:providers # LLM 适配器探活
curl -s http://127.0.0.1:8011/internal/healthz
curl -s http://127.0.0.1:8080/api/research-engine/healthz
```
10. **拉起来后第一时间检查**
- 工作台能登录(用 invite code 注册一个测试账号)
- 聊天能联通真实模型(看 SSE `content_delta`)
- `/admin/networks` 能看到 `drugclip_v1` 的 active 变体
- `Test Gate` workflow 能跑通(部分 ephemeral schema 失败可参考 `docs/wave-b-acceptance-report.md` CAVEAT-D 评估)
---
## 8. 已知风险 / 未完事项摘要
- **`Dataset Download` workflow 当前 failed** — `inspect_target` 在快照缺失时返回 `snapshot_unavailable` 是预期降级,但生产应尽快修复。
- **`agent-loop-tests` workflow 当前 failed** — 4 个 ephemeral schema 同步遗留(缺 `problem_classes.status` 列)+ 1 个 `memory.test.mjs` promise 长尾挂起。已在 `docs/wave-b-acceptance-report.md` CAVEAT-D 标注,不影响生产路径,跟踪在内部任务 #218。
- **Wave B drugclip runner 信号 expressiveness 受限** — `synthetic_ref` 模式下 v1/v2 SMILES top-K cosine 分布几乎一致,5 门控会走 `skipped_ci`。需要拉真 DUD-E `actives.smi` 入库(参考 `docs/wave-b-acceptance-report.md` CAVEAT-A 修订版 + 后续任务 #220)。
- **Wave B autoRollback watchdog cron** — 当前由 driver 单次手动调用证明路径正确;生产 cron 部署需后续做(CAVEAT-E)。
- **HMAC nonce 共享存储** — 单进程 OK,多 worker 需要 Postgres-backed nonce 表(待办 `#34`、`#35`)。
- **审核决定写回 Postgres** — 当前 `/review/edges/:id/decision` 只更新内存 + 审计 JSONL(待办 `#19`)。
- **`reference/` 大目录未打入快照** — 该目录是 2.5 GB 的 `doatlas-handoff` 原始 dump,可由 `Dataset Download` workflow 重新拉取,未纳入压缩包。
### 8.1 文档维护约定
- **`replit.md`** — 长期项目记忆 + 用户偏好;任何「架构级」变更(新增 artifact、新增依赖、改 schema、改进化策略)都要顺手更新顶部 Overview / System Architecture。
- **`docs/PROGRESS.md`(本文件)** — 每次「打包快照 / 离开项目较久」时重新生成,对应一份带时间戳的 zip。注意:本文件**不**是日常迭代日志,而是「换环境恢复指南」。
- **`docs/wave-b-acceptance-report.md`** — Wave B 飞轮的真路径验收证据,与代码变更同步更新。
- **`docs/WORKFLOW.md`** — 流程级地图;与 `docs/API_CONTRACT.md` 冲突时以契约为准并提交 PR 同步本文。
- **`.local/tasks/*.md`** — 项目任务原始描述(每个任务 1 个 markdown),与 `listProjectTasks()` 返回值一一对应。
---
## 9. 下次更新本文件前的检查清单
- [ ] `listProjectTasks({ includeDescription: false })` 返回的 MERGED / CANCELLED / PROPOSED 计数与本文 §3/§4/§5 是否一致?如不一致,按主题更新对应分组。
- [ ] `replit.md` 顶部 `Recent Changes`(如有)是否指向最新一个压缩包路径?
- [ ] 4 个业务 workflow + `Test Gate` + `Provider Smoke Check` + `Dataset Download` + `agent-loop-tests` 的状态是否还和 §6.2 一致?failed 的有没有新增/修复?
- [ ] `docs/wave-b-acceptance-report.md` 中的 CAVEAT-A/B/C/D/E 是否还成立?
- [ ] 新生成的快照里是否包含:`replit.md`、`docs/`(含本文件本身)、`pnpm-workspace.yaml`、`pnpm-lock.yaml`、`pyproject.toml`、`uv.lock`、`.replit`、`.env.replit.example`、`.local/tasks/*.md`、`artifacts/{api-server,doatlas-web,research-engine,mockup-sandbox}`、`lib/`、`scripts/`、`data/`?
- [ ] 是否仍正确排除:`node_modules/``.venv-doatlas/``.pythonlibs/``.cache/``.git/``dist-handoff/``.upm/``.config/``.agents/``reference/``exports/*`、各类 `__pycache__/``.pytest_cache/``.mypy_cache/`
- [ ] 压缩包体积是否仍 < 200 MB?若超出,在本文 §1 顶部明确标注。
---
> 末更:2026-04-27 22:35 UTC(与压缩包 `doatlas-snapshot-20260427-2234.zip` 同批)。