| # Decision Point Attention — 研究对话记录 |
|
|
| ## 项目起源 |
| - 日期: 2026-03-23 |
| - 目标会议: NeurIPS 2026 (DDL: May 6, 2026) |
|
|
| ## 核心 Idea |
| **Decision Point Attention (DPA)**: 在 agent trajectory 中，只有 ~10-15% 的 token 是关键决策点（tool call, plan revision, error recovery）。DPA 让这些 token 走 full softmax attention，其余走 linear attention，实现 Transformer 级推理能力 + 接近线性注意力的效率。 |
|
|
| ## 动机 |
| - MiniMax M2 放弃 linear attention 因为多轮推理失败 |
| - Jamba (1:7), Kimi Linear (1:3) 均匀混合，浪费算力 |
| - Agent trajectory 有明确结构，不是每步都同等重要 |
|
|
| ## 相关工作 |
| - Routing Transformer (ICLR 2020): content-based sparse routing |
| - NAtS-L (2026): token-level hybrid routing |
| - Kimi Linear KDA: 3:1 ratio for agentic workloads |
| - Based (ICML 2024): recall-throughput tradeoff |
| - Gated DeltaNet (ICLR 2025): delta rule improves recall |
| - Illusion of State (ICML 2024): SSM state-tracking limitations |
| - RNNs are not Transformers Yet (ICLR 2025): RAG/attention closes gap |
|
|
| ## 实验计划 |
| 1. Simulation: 模拟不同 decision ratio 下的 quality vs compute tradeoff (DONE) |
| 2. Trajectory Analysis: 分析真实 agent trajectories 中 decision point 比例 (DONE) |
| 3. Training: Fine-tune Qwen2.5-7B with DPA router on Merlin 8xH100 (TODO) |
| 4. Evaluation: HotpotQA, GSM8K, ToolBench benchmarks (TODO) |
| 5. Ablation: Router architecture, ratio sweeps, layer placement (TODO) |
|
|
| ## 文件结构 |
| - `src/models/router.py` — Decision Point Router (learned + fixed) |
| - `src/models/dpa_model.py` — DPA architecture (LinearAttn + FullAttn + Router) |
| - `src/models/baselines.py` — Full Transformer, Pure Linear, Uniform Hybrid |
| - `src/data/agent_trajectory.py` — Trajectory generator & labeling |
| - `src/data/datasets.py` — HotpotQA, GSM8K, ToolBench loaders |
| - `src/eval/benchmark.py` — Unified evaluation pipeline |
| - `src/eval/metrics.py` — FLOPs, latency, KV cache metrics |
| - `src/eval/visualize.py` — Publication figures |
| - `configs/` — Simulation & training configs |
| - `scripts/` — Run scripts for local & Merlin |
|
|
| ## Simulation 结果 (random init, 未训练) |
| | Model | FLOPs Ratio | PPL | |
| |-------|-------------|-----| |
| | Full Transformer | 100% | 38905 | |
| | Pure Linear | 12.5% | 38134 | |
| | Uniform Hybrid | 27.1% | 38856 | |
| | DPA (10%) | 22.5% | 36799 | |
| | DPA (15%) | 27.5% | 37978 | |
| | DPA (25%) | 37.5% | 38174 | |
|
|
| (PPL 差异来自随机初始化，训练后会有显著差别) |
|
|
| ## 下一步 |
| 1. 在 Merlin 上跑 `scripts/run_dpa.sh` 训练 7B 模型 |
| 2. 评估在 HotpotQA/GSM8K 上的多步推理准确率 |
| 3. 写 NeurIPS 论文（LaTeX 模板已准备） |
|
|
