Instructions to use opencsg/Agentic-27B-V2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use opencsg/Agentic-27B-V2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="opencsg/Agentic-27B-V2") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("opencsg/Agentic-27B-V2") model = AutoModelForMultimodalLM.from_pretrained("opencsg/Agentic-27B-V2", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use opencsg/Agentic-27B-V2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "opencsg/Agentic-27B-V2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "opencsg/Agentic-27B-V2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/opencsg/Agentic-27B-V2
- SGLang
How to use opencsg/Agentic-27B-V2 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "opencsg/Agentic-27B-V2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "opencsg/Agentic-27B-V2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "opencsg/Agentic-27B-V2" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "opencsg/Agentic-27B-V2", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use opencsg/Agentic-27B-V2 with Docker Model Runner:
docker model run hf.co/opencsg/Agentic-27B-V2
- OpenCSG Agentic-27B-V2
- 模型简介
- 核心亮点
- 技术路线选择
- 从真实失败到可验证的能力
- 基准测试结果
- Agentic 系列版本演进
- 行业贡献
- 模型规格
- 快速部署
- 已知限制
- 参考文献
- 使用协议
- Model Overview
- Highlights
- Choosing the Technical Approach
- From Real Failures to Verified Capability
- Benchmark Results
- Agentic Series Evolution
- Industry Contributions
- Model Specifications
- Quick Start
- Limitations
- References
- License
- 模型简介
OpenCSG Agentic-27B-V2
从真实使用数据出发、在可验证环境中补齐短板的智能体模型
An agent model that turns real-usage failures into verified training data
OpenCSG 模型仓库 / Models · 使用协议 / License Agreement · 商业授权 / Commercial License
| AutomationBench | SWE-bench Pro | vs. 教师模型 / vs. Teacher | 部署 / Deploy |
|---|---|---|---|
| 30.0% → 41.7% +38.9% rel. |
85.0% → 91.7% 失败 −44% / failures −44% |
AutomationBench 25 vs 20 学生超过老师 / student beats teacher |
1 × 80GB GPU 27B dense · BF16 |
模型简介
OpenCSG Agentic-27B-V2 是 OpenCSG Agentic 系列的第三个公开版本。模型以 Qwen/Qwen3.8-27B 为基础,使用 OpenCSG「真实使用数据 → 环境复现 → 针对性补数据」闭环产出的轨迹,完成 LoRA SFT 训练。
前两个版本解决的是「模型在 Skills 和工具调用上能不能用」。V2 面对的是更难的问题:知道模型在哪里失败,并不等于知道该造什么数据、造多少。 失败反馈和数据决策之间存在断层(feedback-to-decision gap)。V2 不再凭经验造题,而是把真实失败放回可验证的环境里重跑,只针对确实做错的地方补数据。
在两个独立的 held-out 评测上,Agentic-27B-V2 相比同底座 baseline 均有明显提升:
| Benchmark | Baseline(Qwen3.8-27B) | Agentic-27B-V2 | 提升 |
|---|---|---|---|
| AutomationBench(业务自动化工作流) | 18/60(30.00%) | 25/60(41.67%) | +11.67 pts,相对提升 +38.9% |
| SWE-bench Pro(真实软件工程) | 51/60(85.00%) | 55/60(91.67%) | +6.67 pts,失败任务减少 **44%**(9 → 5) |
核心亮点
- 学生超过老师:训练数据由 DeepSeek-V4.1-Flash-1017 生成,但 V2 在 AutomationBench 上拿到 25/60,超过教师模型本身的 20/60。原因在于 V2 只学习通过环境校验的轨迹,教师模型做错的轨迹不会进入训练集。
- 27B 超过同系列旗舰:在同一批 held-out 题目上,V2 在两个 benchmark 上都超过规模大得多的 Qwen3.8-Max-0902(AutomationBench 25 vs 20,SWE-bench Pro 55 vs 45)。
- 两条不同的 Agent 能力线同时提升:AutomationBench 考察跨 SaaS 系统的业务工作流,SWE-bench Pro 考察真实代码仓库中的工程修复。两者同时提升,说明收益来自通用的 Agent 执行能力,而不是只对单一题型过拟合。
- **业务自动化相对提升近 40%**:AutomationBench 通过数从 18 提升到 25,是 V2 提升最明显的方向,并且达到 Claude Opus 5.5 在同一题集上成绩的约 **76%**(25 vs 33)。
- 在高分区间继续减少失败:SWE-bench Pro 上 baseline 已达 85%,V2 把失败任务从 9 个降到 5 个,进入 90% 以上区间,与教师模型(56/60)只差 1 题。
- 数据来自真实失败,不是凭空出题:训练数据的起点是 OpenCSG 平台上的真实使用记录。每条补充数据都对应一个在环境中复现过的 bad case,并经环境校验器确认结果正确。
- 三道确定性闸门,全程可审计:失败必须在环境中复现、teacher 轨迹必须通过校验器、评测必须 held-out 隔离。每条训练样本都能追溯到具体的真实失败。
- 补救 + 自洽 + 通用三路数据配比:以补救成功的轨迹为主体,混入 27B 自身做对的轨迹和通用数据;数据预算向薄弱和覆盖不足的区域倾斜,补短板的同时锚住已有能力。
- 全链路自主可控:教师模型由 OpenCSG 自行私有化部署,数据生成全过程不依赖外部闭源 API,真实使用数据不出域。
- 单卡可部署的 27B dense 模型:BF16 权重可在单张 80GB GPU 上加载运行,附带 MTP 模块权重,适合企业私有化部署。
技术路线选择
Agent 后训练的瓶颈已经从「怎么优化」转移到「用什么数据优化」。V2 的方案不是从零设计的,而是对近一年 Agent 数据合成方向几项关键工作的系统性吸收:哪些做法被证明有效就直接采用,哪些做法被证明有坑就主动绕开。
研究依据
| 工作 | 核心发现 | V2 采纳了什么 |
|---|---|---|
| EvoSyn arXiv:2610.12593 |
找到模型薄弱区域 → 在薄弱区域批量造题 → teacher 作答 → 只保留做对的轨迹做 SFT → 重新评测、再找新弱点。Qwen3-8B 在 τ²-Bench + CAR-Bench 上的平均成功率从 **39.4% 提升到 77.8%**,比最强合成基线高 13.3 个点;第一轮就到 **70.9%**,3 轮累计仅约 2,000 条校验通过的轨迹。 | 弱点驱动的闭环;只保留校验通过的轨迹;数据预算向薄弱和覆盖不足的区域倾斜(而不是只做薄弱区域);并补上 EvoSyn 缺失的一环:混入保持数据,防止遗忘。 |
| AutoDataBench arXiv:2609.35025 |
给 AI 一道原题和目标模型的失败记录,让它照着弱点出一道难度合适的新题。最好的 AI 出题得分 不到 0.20(满分 1)。新题质量不差,70%~97% 的考点都能覆盖;问题出在难度,只有 15%~25% 的题难度合适,其余要么太简单,要么做不出来。 | 不让 AI 出新题。 直接复用 benchmark 原题及其现成的环境和校验器,把难度校准问题从源头上消除。 |
| Kimi K2 arXiv:2507.20534 |
工业级 Agent 数据管线:合成 2 万多个工具 → 为每个任务写清 rubric(成功标准和检查点)→ LLM 模拟用户、工具模拟器执行,有意制造失败和边界情况 → 按 rubric 给轨迹打分,挑出成功轨迹做 SFT。 | 「明确判定标准 + 按标准筛轨迹」。V2 把 rubric 换成环境自带的确定性校验器(系统终态比对 / 隐藏测试),判定更客观。 |
| EnvFactory / EnvScaler(2026) | 同一套思路:造环境 → 跑轨迹 → 按规则或 rubric 判对错 → SFT 或 RL。EnvFactory 只用 85 个环境、2,575 条轨迹,就让 Qwen3 在 τ²-Bench 上提升约 **6%**,在 BFCLv3 上最多提升 **15%**。 | 可执行环境是 Agent 数据的核心资产;少量高质量、可验证的轨迹即可带来显著提升。 |
| DeepSeek-V4.1-Flash 技术报告 arXiv:2609.19969 |
在优化方法很普通的情况下,几乎全部提升都来自数据和环境的规模、多样性和可验证性。 | 把投入集中在数据与环境,训练侧采用稳健、标准的 LoRA SFT。 |
三条造数据路线的对比
| 路线 | 代表工作 | 针对真实弱点 | 难度可控 | 结果可验证 | 数据是否出域 |
|---|---|---|---|---|---|
| 从现有数据里筛选 | 传统 SFT 数据清洗 | ✗ | ✗ | 部分 | 视情况 |
| 让 AI 照着弱点出新题 | AutoDataBench 评测的方法 | ✓ | ✗(仅 15%~25% 合适) | 部分 | 视情况 |
| 造环境 → 跑轨迹 → 按规则筛 | Kimi K2 / EnvFactory / EvoSyn | 部分 | ✓ | ✓ | EvoSyn 使用闭源 API teacher |
| 真实失败 → 原题环境重放 → 校验器筛选 | Agentic-27B-V2 | ✓(来自真实使用) | ✓(原题天然难度合适) | ✓(确定性校验器) | ✗(全链路私有化) |
AutoDataBench 实测:让 AI 照着弱点出新题,考点覆盖 70%~97%,但只有 15%~25% 的题难度合适,其余要么太简单、要么做不出来。
设计原则
- 失败驱动,而不是经验驱动。 数据决策的输入是真实使用中出现、并在环境中复现过的失败,而不是工程师对「模型可能哪里不行」的猜测。这直接回应了 feedback-to-decision gap:每一条补充数据都能追溯到一个具体的、可复现的失败。
- 重放原题,而不是生成新题。 AutoDataBench 表明,让 AI 出新题的瓶颈在难度校准而不在质量。benchmark 原题配有成熟的执行器和校验器,难度本身就落在模型的能力边界上,可以直接用来重放失败。
- 确定性校验器把关(rejection sampling)。 业务流程任务以最终系统状态是否正确判定,代码任务以隐藏测试是否通过判定,不依赖 LLM 打分。teacher 生成的轨迹只要没通过校验就会丢弃,所以训练集只包含 teacher 的正确行为。
- 定向补弱 + 保持数据,防止遗忘。 EvoSyn 没有混入通用数据,也没有测通用能力是否下降。V2 的训练集由三部分组成:补救成功的轨迹(主体)、27B 本来就做对的轨迹、V2 的通用数据;数据预算向薄弱和覆盖不足的区域倾斜,而不是只盯一个弱点。补短板的同时,把已有能力和通用能力一起锚住。
- teacher 私有化。 EvoSyn 的 teacher 是闭源 API(Claude Opus 4.5);V2 的 teacher DeepSeek-V4.1-Flash-1017 由 OpenCSG 自行部署,真实使用数据全程不出域。这让整套流程可以直接搬进企业客户的私有环境。
- 评测严格隔离。 EvoSyn 的提升属于领域内提升,训练和评测来自同一个 benchmark 家族。V2 的 120 道评测题全部为 held-out,与数据生成过程隔离,并且在业务自动化和软件工程两个完全不同的任务家族上同时验证。
与 EvoSyn 的异同
| 维度 | EvoSyn | Agentic-27B-V2 |
|---|---|---|
| 弱点从哪里来 | 在 benchmark 上评测得到 | 真实平台使用记录,脱敏后映射到环境中复现 |
| 训练题目 | 在薄弱区域批量生成新题 | 复用 benchmark 原题,不生成新题 |
| teacher | Claude Opus 4.5(闭源 API) | DeepSeek-V4.1-Flash-1017(OpenCSG 自部署) |
| 轨迹筛选 | 只保留做对的轨迹 | 只保留通过环境校验器的轨迹 |
| 数据预算 | 向薄弱和覆盖不足的区域倾斜 | 向薄弱和覆盖不足的区域倾斜,弱点来自真实使用 |
| 遗忘控制 | 未混入通用数据,未测通用能力 | 补救轨迹 + 27B 自身正确轨迹 + 通用数据三路混合 |
| 训练方法 | SFT | LoRA SFT |
| 评测 | 与训练同属一个 benchmark 家族 | held-out 隔离,业务 + 代码两个任务家族 |
从真实失败到可验证的能力
V2 的数据闭环分五步:
① 真实使用数据 ──▶ ② 匹配最贴合的 benchmark 环境 ──▶ ③ 在环境中复现 bad case
│
⑤ 在 held-out benchmark 上评测 ◀── LoRA SFT ◀── ④ 自部署强模型针对性补数据(校验通过才保留)
用伪代码表示:
failures = mine_failures(platform_logs, redact=True) # ① 真实失败,脱敏
budget = allocate(failures, favor=["weak", "under_covered"]) # 数据预算向薄弱和覆盖不足区域倾斜
for f in failures:
env = match_environment(f, benchmarks) # ② 业务流程 → SaaS 工作流环境;代码 → 真实仓库环境
if not env.replay(student, f.task).failed: # ③ 环境中复现不了的失败直接丢弃
continue
for traj in teacher.rollout(env, f.task, n=budget[f]): # ④ 自部署 teacher 按预算多次采样
if env.verify(traj): # 确定性校验器:系统终态 / 隐藏测试
rescued.append(traj) # 补救成功的轨迹
self_ok = [t for t in student.rollout(envs) if env.verify(t)] # 27B 本来就做对的轨迹
train_lora_sft(base, rescued + self_ok + general_data) # 补救(主体)+ 自身正确 + 通用数据
evaluate(model, heldout_sets) # ⑤ held-out 评测,与数据生成完全隔离
三道质量闸门
V2 的每一条训练数据和每一个评测结论,都要先通过三道闸门。闸门全部是确定性的:由环境执行结果判定,而不是靠人工或 LLM 主观打分。
| 闸门 | 判定方式 | 拦下的是什么 | 带来的保证 |
|---|---|---|---|
| ① 复现闸门 | 当前模型在可执行环境中重跑,必须真的失败 | 「以为是弱点、其实不是」的失败;环境无法承载的模糊场景 | 每条补充数据都对应一个真实、可复现的能力缺口 |
| ② 校验闸门 | teacher 轨迹必须让系统终态正确,或让隐藏测试全部通过 | teacher 做错、做一半、声称完成但没完成的轨迹 | 训练集里只有正确行为,学生能超过 teacher |
| ③ 隔离闸门 | 评测题与数据生成过程完全隔离(held-out) | 「在训练题上刷分」造成的虚高 | 提升反映的是能力迁移,而不是记忆 |
这三道闸门让整个流程可审计:任何一条训练样本都可以追溯到「哪个真实失败 → 哪个环境 → 哪次复现 → 哪条通过校验的 teacher 轨迹」。
①② 从真实失败到可执行环境
从 OpenCSG 平台的真实 Agent 使用记录出发,脱敏后定位高频失败模式,再把每类失败映射到公开 benchmark 中最接近的可执行环境:
| 失败模式 | 典型表现 | 映射到的环境 | 校验信号 |
|---|---|---|---|
| 工具选择错误 | 该查 CRM 却去查邮件;该改代码却只读文件 | AutomationBench / SWE-bench Pro | 系统终态 / 隐藏测试 |
| 参数构造错误 | 字段名、ID、过滤条件写错,写操作落到错误对象 | AutomationBench(SaaS 工作流) | 目标记录是否被正确修改 |
| 长链路中途放弃 | 多步任务做到一半停止,或丢失中间状态 | AutomationBench / SWE-bench Pro | 全部子目标是否完成 |
| 过早结束 | 未完成就声称完成 | SWE-bench Pro(真实仓库) | 隐藏测试是否通过 |
| 没有核验结果 | 写操作后不回读确认,修改后不跑测试 | AutomationBench / SWE-bench Pro | 终态比对 / 测试结果 |
业务流程类问题对应 AutomationBench 风格的 SaaS 工作流环境(CRM、邮件、表格、工单),代码与工程类问题对应 SWE-bench Pro 风格的真实仓库环境。这些环境自带执行器和校验器,可以客观判定对错。
③ 在环境中复现 bad case
让当前模型在这些环境中实际执行,收集真正失败的任务和轨迹。只有在环境中能复现的失败才进入下一步,避免「以为是弱点、其实不是」的无效补数据。这一步同时完成了难度校准:被选中的任务正好落在当前模型做不对、但 teacher 有机会做对的区间。
④ 自部署 teacher + 校验器筛选
使用 OpenCSG 自行部署的 DeepSeek-V4.1-Flash-1017 作为 teacher,针对失败任务多次采样生成新的执行轨迹。每条轨迹都要经过环境校验器:只有最终系统状态正确、或隐藏测试全部通过的轨迹才会保留。
这相当于对 teacher 做了一次拒绝采样(rejection sampling):teacher 的平均水平并不重要,重要的是它在这些任务上做对的那部分。这也是 V2 能在 AutomationBench 上超过 teacher 本身(25 vs 20)的原因:学生学到的是被校验器提纯过的行为,而不是 teacher 的平均行为。
训练数据配比:补救 + 自洽 + 通用
只用补救轨迹做 SFT,模型会学会新本事,但可能把原本会的东西学偏。V2 的训练集因此由三部分组成,并且在补救轨迹内部,数据预算向薄弱和覆盖不足的区域倾斜:失败越集中、现有数据覆盖越少的能力区域,分配越多的 teacher 采样和训练样本;但不会只盯一个弱点,已经覆盖较好的区域也保留一定份额,避免训练分布过度偏斜。
| 数据 | 来源 | 占比 | 作用 |
|---|---|---|---|
| 补救成功的轨迹 | 27B 在环境中做错、由自部署 teacher 重做并通过校验器的轨迹 | 主体 | 把真实失败点直接教会,是提升的主要来源 |
| 27B 本来就做对的轨迹 | 27B 自己在环境中跑通、并通过同一校验器的轨迹 | 辅助 | 用模型自己的正确行为做锚点:分布贴近模型本身(on-policy),强化已经掌握的执行习惯,防止补短板时把会做的题做坏 |
| V2 的通用数据 | Agentic 系列积累的通用 Agent 数据 | 辅助 | 保持通用指令遵循和工具调用能力,避免能力窄化 |
这套配比的目标是让 V2 在补短板的同时不丢已有能力:AutomationBench 和 SWE-bench Pro 两条能力线同时提升,没有出现此消彼长。
⑤ held-out 评测
评测使用的 60 道 AutomationBench 任务和 60 道 SWE-bench Pro 任务均为 held-out 集合,与训练数据生成过程隔离,用于检验能力是否真正迁移。baseline 与 V2 使用相同底座、相同题目、相同推理配置。
训练配置
| 项目 | 配置 |
|---|---|
| 基础模型 | Qwen/Qwen3.8-27B |
| 训练框架 | ms-swift 4.7(Megatron 后端) |
| 训练方式 | LoRA SFT |
| LoRA rank / alpha | 16 / 32 |
| LoRA 目标模块 | all-linear |
| 视觉塔 | 冻结 |
| 训练轮数 | 3 epochs |
| 最大训练长度 | 262,144 tokens(packing + padding-free) |
| 学习率 | 1e-5,cosine 衰减至 1e-6,warmup 5% |
| 优化器 | Adam(β1 = 0.9,β2 = 0.95),weight decay 0.1 |
| Loss | 仅在 assistant 输出上计算,忽略空的 think 段(default+ignore_empty_think) |
| 并行策略 | TP 4 × CP 2,sequence parallel,full recompute,FlashAttention |
| 精度 | BF16 |
| 训练卡数 | 8 |
| 教师模型 | DeepSeek-V4.1-Flash-1017(OpenCSG 自部署) |
为什么用 262K 训练长度:Agent 轨迹往往包含几十轮工具调用和大段工具返回(CRM 记录、代码文件、测试日志)。262K 的上下文窗口配合 packing,可以把长链路轨迹完整放进一个样本,不做截断,让模型学到从第一步到最终核验的完整执行过程,而不只是片段。
工程实现要点
- 整条轨迹不截断。 262,144 tokens 的训练长度配合 packing 和 padding-free,几十轮工具调用、大段 CRM 记录、源码和测试日志可以完整放进一个样本,模型学到的是从第一步到最终核验的完整执行过程。
- 超长序列的显存与吞吐。 在 8 张卡上训练 27B dense + 262K 序列,采用 TP 4 × CP 2 切分、sequence parallel、full recompute 和 FlashAttention,把超长上下文训练稳定跑通。
- 只在模型该学的地方计算 loss。 Loss 只作用于 assistant 的输出,工具返回和环境观测不参与梯度;空的 think 段被忽略(
default+ignore_empty_think),避免模型学会「跳过思考」。 - 轻量适配,部署零改动。 LoRA 覆盖全部线性层(all-linear),视觉塔冻结,MTP 模块随权重一起发布;合并后的权重与底座结构一致,可直接用 vLLM 的工具调用和推理解析器部署。
结果解读
- 严格的同口径对照:baseline 与 V2 底座相同、题目相同、推理配置相同,唯一变量是 V2 的数据闭环。
- 跨任务家族的一致提升:业务自动化和软件工程两类完全不同的任务同时提升,排除了单一题型过拟合。
- 学生超过老师:V2 在 AutomationBench 上超过 teacher(25 vs 20),这只有在校验闸门有效过滤掉 teacher 错误行为时才会发生。
- 有对照的版本差异:V1 没有针对 AutomationBench 做数据,成绩与 baseline 持平;V2 用环境复现的失败补数据后提升 7 题。提升来自数据闭环本身,而不是底座或评测波动。
基准测试结果
与 baseline 对比
| Benchmark | 评测集 | Baseline(Qwen3.8-27B) | Agentic-27B-V2 | 绝对提升 | 相对提升 |
|---|---|---|---|---|---|
| AutomationBench | 60 题 held-out | 18/60(30.00%) | 25/60(41.67%) | +11.67 pts | +38.9% |
| SWE-bench Pro | 60 题 held-out | 51/60(85.00%) | 55/60(91.67%) | +6.67 pts | +7.8% |
两组结果使用相同底座、相同评测题目、相同推理配置,唯一差别是是否经过 V2 的数据闭环训练。
- AutomationBench(Zapier 发布)评测 Agent 在真实业务工作流中的表现:跨 CRM、邮件、表格、工单等 SaaS 系统读取信息、做出决策并执行写操作,最终以系统状态是否正确判定通过。
- SWE-bench Pro(Scale AI 发布)评测 Agent 在真实、长链路的软件工程任务中的表现:在完整代码仓库中定位问题、修改代码,并以隐藏测试是否通过判定结果。
同一 held-out 集合上的横向对比
在同一批 60 + 60 道 held-out 题目上,Agentic-27B-V2 与主流开源、闭源模型的对比如下(各模型均使用最高推理强度):
| 模型 | 类型 | AutomationBench(60 题) | SWE-bench Pro(60 题) |
|---|---|---|---|
| GPT-6 Astra | 闭源 | 37/60(61.7%) | 59/60(98.3%) |
| Gemini 4 Carbon | 闭源 | 36/60(60.0%) | 58/60(96.7%) |
| Claude Opus 5.5 | 闭源 | 33/60(55.0%) | 58/60(96.7%) |
| Agentic-27B-V2 | 27B dense | 25/60(41.7%) | 55/60(91.7%) |
| DeepSeek-V4.1-Flash-1017(教师模型) | 开源 | 20/60(33.3%) | 56/60(93.3%) |
| Qwen3.8-Max-0902 | 开源 | 20/60(33.3%) | 45/60(75.0%) |
| Agentic-27B(V1) | 27B dense | 18/60(30.0%) | 52/60(86.7%) |
| Qwen3.8-27B(baseline) | 27B dense | 18/60(30.0%) | 51/60(85.0%) |
- AutomationBench 上超过教师模型和同系列旗舰:V2 以 27B 规模拿到 25/60,超过 DeepSeek-V4.1-Flash-1017(数据生成所用教师模型)和 Qwen3.8-Max-0902 的 20/60,在开源模型中排名第一。
- SWE-bench Pro 进入 90% 以上区间:V2 比 Qwen3.8-Max-0902 多做对 10 题,与教师模型 DeepSeek-V4.1-Flash-1017 只差 1 题。
- 针对性训练带来真实增量:V1 没有针对 AutomationBench 专门训练,成绩与 baseline 持平;V2 用环境复现的 bad case 针对性补数据后,AutomationBench 通过数提升 7 题,SWE-bench Pro 也比 V1 多做对 3 题。
公开榜单参考
下表列出部分模型在两个 benchmark 公开榜单上的成绩,便于了解所处的能力区间。
注意:公开榜单为全量题集,且不同来源的 harness、推理设置、是否厂商自报均不相同;Agentic-27B-V2 的成绩来自 60 题 held-out 子集。两者不能直接横向排名,仅作为量级参考。同口径对比请以上文的 baseline 对比和横向对比为准。
AutomationBench
| 模型 | 参数规模 / 类型 | 公开分数 | 来源 |
|---|---|---|---|
| Gemini 4 Carbon | 闭源 | 77.8% | Artificial Analysis 系榜单(partial credit) |
| MiniMax M3.1 Flash | 开源 | 62.2% | BenchLM |
| DeepSeek-V4.1-Flash-1017 | 开源 | 60.5% | BenchLM |
| Qwen3.8-Max 0902 | 开源 | 50.8% | 公开 Zapier 风格评测 |
| Claude Opus 5 (max) | 闭源 | 50.3% | 公开 Zapier 风格评测 |
| Qwen3.8-27B (x-high) | 27B dense | 48.2% | AutomationBench-AA(partial score) |
| Claude Opus 5.5 | 闭源 | 35.8%(独立复测) | 公开 Zapier 风格评测 |
SWE-bench Pro
| 模型 | 参数规模 / 类型 | 公开分数 | 来源 |
|---|---|---|---|
| Claude Fable 5.2 | 闭源 | 93.0% | BenchLM(Public set) |
| Claude Opus 5.5 | 闭源 | 89.9% | BenchLM(Public set) |
| Claude Sonnet 5.5 | 闭源 | 81.3% | BenchLM(Public set) |
| Qwen3.8-Max | 开源 | 67.7% | 公开榜单 |
| Qwen3.8-27B | 27B dense | 61.7% | Qwen 官方模型卡(自报) |
| GPT-5.5 | 闭源 | 58.6% | 公开榜单 |
| DeepSeek-V4-Pro (max) | 开源 | 55.4% | BenchLM |
Agentic 系列版本演进
| 版本 | 基础模型 | 训练方式 | 数据来源 | 主要评测 | 核心成绩 |
|---|---|---|---|---|---|
| Agentic-30B-A3B | Qwen/Qwen3-30B-A3B(MoE) |
LoRA DPO | 平台数据飞轮,约 3,000 条偏好样本 | Agentic Eval | 平均分 0.46,pass@3 37/119 |
| Agentic-27B | Qwen/Qwen3.8-27B(dense) |
LoRA DPO | 平台数据飞轮,5,857 条偏好样本 | Agentic Eval / Arena-Hard-v2.0 | 平均分 0.83,pass@3 96/103;hard_prompt 57.68 |
| Agentic-27B-V2 | Qwen/Qwen3.8-27B(dense) |
LoRA SFT | 真实使用数据 → 环境复现 bad case → 自部署教师模型补数据 → 校验器过滤 | AutomationBench / SWE-bench Pro(held-out) | **41.67% / 91.67%**,均明显超过 baseline |
升级方向
- 从「平台内评测」走向「公开可验证评测」:前两个版本主要在 OpenCSG 自建的 Agentic Eval 上评测;V2 开始在业界公开、可复现的 benchmark 上验证效果,结果可以被第三方对照。
- 从「偏好对齐」走向「能力补齐」:DPO 擅长在已有能力内部挑出更好的行为;V2 改用 SFT,直接把模型原本做不对的任务教会,更适合补短板。
- 从「经验造数据」走向「失败驱动造数据」:每条训练数据都对应一个真实出现、并在环境中复现过的失败,再由环境校验器确认结果正确,数据决策有据可依。
- 从「单一场景」走向「业务 + 代码双线」:同时覆盖业务流程自动化和软件工程两条最核心的 Agent 落地场景。
行业贡献
- 闭环可复制:「真实数据 → 环境匹配 → bad case 复现 → 针对性补数据 → held-out 评测」是一套可持续迭代的流程。换一个场景、换一批真实失败,就能再跑一轮,不依赖人工拍脑袋决定造什么数据。
- 能力可迁移:训练数据来自 OpenCSG 平台的真实失败,评测用的却是与数据生成完全隔离的公开 benchmark held-out 题目。V2 在这些从未见过的任务上依然提升,而且业务自动化和软件工程两个任务家族同时提升。这说明模型学到的是通用的执行能力(选对工具、构造正确参数、把长链路做完、核验结果),而不是对某几道题的记忆;同样的能力可以迁移到新的业务场景和新的挑战。
- 环境持续增长、反复复用:每一类真实失败一旦映射到可执行环境,这个环境连同它的执行器和校验器就进入环境库,之后每一轮都能直接复用:复现新的失败、校验新的 teacher 轨迹、对新版本模型做回归测试。遇到新的失败类型,只需补充新的环境。环境库越积越大,每一轮迭代的边际成本越来越低,能力提升也越来越快。
- 校验器把关,学生可以超过老师:只保留通过环境校验的轨迹,相当于把教师模型的正确行为提纯后再教给学生。V2 在 AutomationBench 上超过教师模型,说明这套流程的上限不受教师模型平均水平限制。
- 数据决策有据可依:只在环境中确认过的失败点上补数据。和「自动出新题」相比,避免了大量难度不合适、无法验证的无效数据。
- 全链路私有化:教师模型由 OpenCSG 自行部署,真实使用数据不出域,满足企业对数据安全和合规的要求;同一套流程可以直接在客户私有环境中复用。
- 小模型、真提升:在 27B dense、单卡可部署的规模上,两个公开 benchmark 同时提升,并在同口径对比中超过同系列旗舰 Qwen3.8-Max-0902,部署成本远低于百 B 级以上模型。
- 来自真实平台的数据飞轮:OpenCSG 平台上持续产生的 Agent 使用记录,是这套闭环源源不断的输入。平台被使用得越多,模型越贴近真实业务。
模型规格
| 项目 | 配置 |
|---|---|
| 基础模型 | Qwen/Qwen3.8-27B |
| 架构 | Qwen3_5ForConditionalGeneration |
| 权重精度 | BF16 |
| 总参数 | 约 27.36B |
| Transformer 层数 | 64 |
| Hidden size | 5,120 |
| Attention / KV heads | 24 / 4 |
| 词表大小 | 248,320 |
| 配置最大位置长度 | 262,144 |
| 附带权重 | MTP 模块(mtp.safetensors) |
快速部署
先下载权重:
hf download opencsg/Agentic-27B-V2 --local-dir ./Agentic-27B-V2
然后用 vLLM 启动 OpenAI 兼容服务(如下载目录不同,请把 ./Agentic-27B-V2 换成实际路径)。
实际显存占用会随 vLLM 版本、上下文长度、batch 配置和是否启用视觉组件而变化,请按目标 workload 调整。
vllm serve ./Agentic-27B-V2 \
--served-model-name OpenCSG/Agentic-27B-V2 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.88 \
--max-model-len 32768 \
--max-num-seqs 8 \
--max-num-batched-tokens 8192 \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--generation-config vllm
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="OpenCSG/Agentic-27B-V2",
messages=[
{"role": "system", "content": "You are a careful local agent."},
{"role": "user", "content": "Summarize today's action items."},
],
temperature=0.2,
max_tokens=512,
)
print(response.choices[0].message.content)
已知限制
- AutomationBench 和 SWE-bench Pro 的成绩基于各 60 题的 held-out 子集,与公开榜单的全量题集和 harness 不同,不应直接与榜单排名比较。
- 模型输出不应直接用于医疗、法律、金融决策或高权限系统操作。敏感写操作需要人工确认和最小权限控制。
参考文献
- EvoSyn:针对薄弱区域造数据的闭环。https://arxiv.org/html/2610.12593
- AutoDataBench:自动出题的难度问题。https://arxiv.org/html/2609.35025
- Kimi K2: Open Agentic Intelligence。https://arxiv.org/html/2507.20534
- EnvFactory、EnvScaler(2026):环境规模化与轨迹合成。
- DeepSeek-V4.1-Flash 技术报告。https://arxiv.org/pdf/2609.19969
- AutomationBench(Zapier);SWE-bench Pro(Scale AI)。
使用协议
使用本模型需要遵循 OpenCSG 模型与数据集许可协议。仓库 metadata 中的 license: other 表示本模型采用平台预设列表之外的许可协议,实际许可条款以该协议为准。
本模型可按 OpenCSG 模型与数据集许可协议申请商业用途。若计划将本模型,或基于本模型训练、增强的模型、系统、Agent、API 服务和商业产品用于商业场景,请发送邮件至 lorraineg@opencsg.com 获取许可。
当前公开快照中的 license_type: 商业授权 是记录级授权来源标记,不替代仓库级许可协议。
Model Overview
OpenCSG Agentic-27B-V2 is the third public release in the OpenCSG Agentic series. It starts from Qwen/Qwen3.8-27B and is trained with LoRA SFT on trajectories produced by OpenCSG's "real usage data → environment replay → targeted data" loop.
The first two releases answered the question "does the model work with Skills and tool calls?" V2 takes on a harder problem: knowing where a model fails does not tell you what data to build, or how much. There is a gap between failure feedback and data decisions (the feedback-to-decision gap). Instead of building tasks from intuition, V2 replays real failures inside verifiable environments and adds data only where the model actually gets things wrong.
On two independent held-out evaluations, Agentic-27B-V2 clearly improves over the same-base baseline:
| Benchmark | Baseline (Qwen3.8-27B) | Agentic-27B-V2 | Gain |
|---|---|---|---|
| AutomationBench (business automation workflows) | 18/60 (30.00%) | 25/60 (41.67%) | +11.67 pts, +38.9% relative |
| SWE-bench Pro (real-world software engineering) | 51/60 (85.00%) | 55/60 (91.67%) | +6.67 pts, failed tasks down 44% (9 → 5) |
Highlights
- The student beats the teacher: training trajectories come from DeepSeek-V4.1-Flash-1017, yet V2 scores 25/60 on AutomationBench, above the teacher's own 20/60. V2 only learns from trajectories that pass the environment verifier, so the teacher's mistakes never enter the training set.
- A 27B model ahead of its family flagship: on the same held-out tasks, V2 beats the much larger Qwen3.8-Max-0902 on both benchmarks (AutomationBench 25 vs 20, SWE-bench Pro 55 vs 45).
- Two very different agent skill lines improve together: AutomationBench tests business workflows across SaaS systems; SWE-bench Pro tests engineering fixes in real code repositories. Gains on both indicate better general agent execution, not overfitting to one task type.
- Nearly 40% relative gain on business automation: AutomationBench passes rise from 18 to 25, V2's largest improvement, reaching about 76% of Claude Opus 5.5's score on the same set (25 vs 33).
- Fewer failures even at the top of the range: the SWE-bench Pro baseline is already at 85%; V2 cuts failed tasks from 9 to 5, moving above 90% and landing one task behind the teacher (56/60).
- Data built from real failures, not invented tasks: the training data starts from real usage records on the OpenCSG platform. Every added sample maps to a bad case reproduced in an environment and confirmed correct by the environment verifier.
- Three deterministic gates, fully auditable: failures must reproduce in an environment, teacher trajectories must pass the verifier, and evaluation must be held out. Every training sample traces back to a specific real failure.
- A three-way data mix: rescued trajectories form the main part, mixed with the 27B model's own correct trajectories and general data; the data budget tilts toward weak and under-covered regions, closing gaps while anchoring existing skills.
- Fully self-controlled pipeline: the teacher model is self-hosted by OpenCSG. Data generation does not depend on any external closed-source API, and real usage data never leaves the domain.
- A 27B dense model that fits on one GPU: the BF16 weights load and run on a single 80GB GPU, and MTP module weights are included. Well suited to private enterprise deployment.
Choosing the Technical Approach
The bottleneck in agent post-training has shifted from "how to optimize" to "what data to optimize on". V2's recipe was not designed from scratch. It systematically absorbs several key works on agent data synthesis from the past year: practices shown to work are adopted directly, and practices shown to have pitfalls are deliberately avoided.
Research Basis
| Work | Key finding | What V2 adopts |
|---|---|---|
| EvoSyn arXiv:2610.12593 |
Find weak regions → generate tasks there in bulk → teacher solves them → keep only correct trajectories for SFT → re-evaluate and find new weaknesses. Qwen3-8B's average success on τ²-Bench + CAR-Bench rises from 39.4% to 77.8%, 13.3 points above the strongest synthetic baseline; round one alone reaches 70.9%, using only about 2,000 verified trajectories over 3 rounds. | A weakness-driven loop; keep only verified trajectories; tilt the data budget toward weak and under-covered regions (not weak regions alone); plus the piece EvoSyn leaves out: retention data to prevent forgetting. |
| AutoDataBench arXiv:2609.35025 |
Give an AI an original task and a target model's failure record, and ask it to write a new task of suitable difficulty targeting that weakness. The best AI scores below 0.20 (out of 1). Task quality is fine, covering 70%–97% of the intended skills, but only 15%–25% land at a suitable difficulty; the rest are too easy or unsolvable. | No AI-generated tasks. V2 reuses original benchmark tasks with their ready-made environments and verifiers, removing the difficulty-calibration problem at the source. |
| Kimi K2 arXiv:2507.20534 |
An industrial agent data pipeline: synthesize 20,000+ tools → write a rubric (success criteria and checkpoints) for every task → an LLM simulates users while tool simulators execute calls, deliberately creating failures and edge cases → score trajectories against the rubric and keep successful ones for SFT. | Explicit success criteria + filtering trajectories against them. V2 replaces the rubric with the environment's deterministic verifier (final-state checks / hidden tests) for more objective judgments. |
| EnvFactory / EnvScaler (2026) | The same idea: build environments → run trajectories → judge by rules or rubrics → SFT or RL. With only 85 environments and 2,575 trajectories, EnvFactory improves Qwen3 by about 6% on τ²-Bench and by up to 15% on BFCLv3. | Executable environments are the core asset for agent data; a small number of high-quality, verifiable trajectories brings large gains. |
| DeepSeek-V4.1-Flash technical report arXiv:2609.19969 |
With fairly ordinary optimization methods, nearly all gains come from the scale, diversity, and verifiability of data and environments. | Concentrate effort on data and environments, and keep the training side a robust, standard LoRA SFT. |
Three Routes to Agent Data
| Route | Representative work | Targets real weaknesses | Difficulty control | Verifiable outcomes | Data leaves the domain |
|---|---|---|---|---|---|
| Filter existing data | Classic SFT data cleaning | ✗ | ✗ | Partial | Depends |
| AI writes new tasks from weaknesses | Methods evaluated in AutoDataBench | ✓ | ✗ (only 15%–25% suitable) | Partial | Depends |
| Build environments → run trajectories → filter by rules | Kimi K2 / EnvFactory / EvoSyn | Partial | ✓ | ✓ | EvoSyn uses a closed-API teacher |
| Real failures → replay in original-task environments → verifier filtering | Agentic-27B-V2 | ✓ (from real usage) | ✓ (original tasks are naturally well-calibrated) | ✓ (deterministic verifiers) | ✗ (fully private pipeline) |
AutoDataBench: AI-written tasks cover 70%–97% of the intended skills, but only 15%–25% land at a suitable difficulty; the rest are too easy or unsolvable.
Design Principles
- Failure-driven, not intuition-driven. Data decisions start from failures that occurred in real usage and were reproduced in an environment, not from engineers' guesses about where the model might be weak. This directly addresses the feedback-to-decision gap: every added sample traces back to a specific, reproducible failure.
- Replay original tasks instead of generating new ones. AutoDataBench shows that the bottleneck in AI-written tasks is difficulty calibration, not quality. Original benchmark tasks come with mature executors and verifiers, and their difficulty already sits at the edge of the model's capability, so they can be used directly to replay failures.
- Deterministic verifiers as the gate (rejection sampling). Business workflow tasks are judged by whether the final system state is correct, and code tasks by whether hidden tests pass, with no LLM-based scoring. Any teacher trajectory that fails verification is discarded, so the training set contains only the teacher's correct behavior.
- Targeted fixes plus retention data to prevent forgetting. EvoSyn neither mixes in general data nor tests whether general capability drops. V2's training set has three parts: rescued trajectories (the main part), trajectories the 27B model already got right, and V2 general data; the data budget tilts toward weak and under-covered regions rather than a single weakness. Gaps are closed while existing and general skills stay anchored.
- A private teacher. EvoSyn's teacher is a closed API (Claude Opus 4.5). V2's teacher, DeepSeek-V4.1-Flash-1017, is self-hosted by OpenCSG, and real usage data never leaves the domain. The whole process can move directly into an enterprise customer's private environment.
- Strictly isolated evaluation. EvoSyn's gains are in-domain: training and evaluation come from the same benchmark family. All 120 of V2's evaluation tasks are held out and isolated from data generation, and validation spans two very different task families: business automation and software engineering.
V2 vs. EvoSyn
| Dimension | EvoSyn | Agentic-27B-V2 |
|---|---|---|
| Where weaknesses come from | Evaluation on benchmarks | Real platform usage, redacted and reproduced in environments |
| Training tasks | New tasks generated in weak regions | Original benchmark tasks; no new tasks generated |
| Teacher | Claude Opus 4.5 (closed API) | DeepSeek-V4.1-Flash-1017 (self-hosted by OpenCSG) |
| Trajectory filtering | Keep only correct trajectories | Keep only trajectories that pass the environment verifier |
| Data budget | Tilted toward weak and under-covered regions | Tilted toward weak and under-covered regions, with weaknesses drawn from real usage |
| Forgetting control | No general data mixed in; general capability not tested | Three-way mix: rescued + the 27B model's own correct trajectories + general data |
| Training method | SFT | LoRA SFT |
| Evaluation | Same benchmark family as training | Held-out isolation across two task families (business + code) |
From Real Failures to Verified Capability
The V2 data loop has five steps:
① Real usage data ──▶ ② Match the closest benchmark environment ──▶ ③ Replay bad cases in the environment
│
⑤ Evaluate on held-out benchmarks ◀── LoRA SFT ◀── ④ Self-hosted teacher fills targeted data (kept only if verified)
In pseudo-code:
failures = mine_failures(platform_logs, redact=True) # ① real failures, redacted
budget = allocate(failures, favor=["weak", "under_covered"]) # tilt the data budget toward weak / under-covered regions
for f in failures:
env = match_environment(f, benchmarks) # ② workflows → SaaS envs; code → real-repo envs
if not env.replay(student, f.task).failed: # ③ drop failures that don't reproduce
continue
for traj in teacher.rollout(env, f.task, n=budget[f]): # ④ self-hosted teacher, sampled per budget
if env.verify(traj): # deterministic verifier: final state / hidden tests
rescued.append(traj) # rescued trajectory
self_ok = [t for t in student.rollout(envs) if env.verify(t)] # tasks the 27B model already solves
train_lora_sft(base, rescued + self_ok + general_data) # rescued (main) + own-correct + general
evaluate(model, heldout_sets) # ⑤ held-out eval, isolated from data generation
Three Quality Gates
Every V2 training sample and every evaluation claim must pass three gates. All gates are deterministic: they are decided by environment execution results, not by human or LLM judgment.
| Gate | How it decides | What it blocks | What it guarantees |
|---|---|---|---|
| ① Reproduce | The current model reruns the task in an executable environment and must actually fail | "Weaknesses" that are not real; vague scenarios no environment can host | Every added sample maps to a real, reproducible capability gap |
| ② Verify | A teacher trajectory must produce the correct final system state, or pass all hidden tests | Teacher trajectories that are wrong, half-done, or claim completion falsely | The training set contains only correct behavior, so the student can beat the teacher |
| ③ Isolate | Evaluation tasks are fully isolated from data generation (held-out) | Inflated scores from training on the test | Gains reflect capability transfer, not memorization |
These gates make the whole process auditable: any training sample can be traced back through "which real failure → which environment → which replay → which verified teacher trajectory".
①② From Real Failures to Executable Environments
Starting from real agent usage records on the OpenCSG platform, the records are redacted, frequent failure modes are located, and each failure type is mapped to the closest executable environment in a public benchmark:
| Failure mode | Typical symptom | Mapped environment | Verification signal |
|---|---|---|---|
| Wrong tool selection | Querying email when the CRM is needed; only reading files when code must change | AutomationBench / SWE-bench Pro | Final state / hidden tests |
| Malformed arguments | Wrong field names, IDs, or filters; writes land on the wrong object | AutomationBench (SaaS workflows) | Whether the target record was changed correctly |
| Giving up mid-chain | Stopping halfway through a multi-step task, or losing intermediate state | AutomationBench / SWE-bench Pro | Whether all sub-goals are completed |
| Stopping too early | Claiming completion before the task is done | SWE-bench Pro (real repositories) | Whether hidden tests pass |
| No result verification | Not reading back after writes; not running tests after changes | AutomationBench / SWE-bench Pro | Final-state comparison / test results |
Business-process problems map to AutomationBench-style SaaS workflow environments (CRM, email, spreadsheets, ticketing); code and engineering problems map to SWE-bench Pro-style real repository environments. These environments ship with executors and verifiers that judge outcomes objectively.
③ Replaying Bad Cases in the Environment
The current model runs in these environments, and the tasks and trajectories that genuinely fail are collected. Only failures that reproduce in an environment move on, which avoids wasting data on "weaknesses" that are not real. This step also performs difficulty calibration: the selected tasks fall exactly where the current model fails but the teacher has a chance to succeed.
④ Self-hosted Teacher + Verifier Filtering
DeepSeek-V4.1-Flash-1017, self-hosted by OpenCSG, serves as the teacher and samples new trajectories for each failed task multiple times. Every trajectory goes through the environment verifier, and only those with a correct final system state, or with all hidden tests passing, are kept.
This amounts to rejection sampling on the teacher: its average performance does not matter, only the part it gets right on these tasks. It is also why V2 can beat the teacher itself on AutomationBench (25 vs 20): the student learns verifier-distilled behavior, not the teacher's average behavior.
Training Data Mix: Rescued + Self-consistent + General
SFT on rescued trajectories alone teaches new skills but can skew what the model already does well. V2's training set therefore has three parts, and within the rescued trajectories the data budget tilts toward weak and under-covered regions: capability areas with more concentrated failures and thinner existing coverage get more teacher samples and training examples. It never fixates on a single weakness, though; well-covered areas keep a share too, so the training distribution does not skew too far.
| Data | Source | Share | Role |
|---|---|---|---|
| Rescued trajectories | Tasks the 27B model failed in an environment, redone by the self-hosted teacher and passed by the verifier | Main part | Directly teaches the real failure points; the main source of gains |
| The 27B model's own correct trajectories | Tasks the 27B model already completes in the environment, passed by the same verifier | Supporting | Anchors training on the model's own correct behavior: on-policy, reinforces execution habits it already has, and prevents closing gaps from breaking tasks it could already do |
| V2 general data | General agent data accumulated across the Agentic series | Supporting | Preserves general instruction following and tool use, avoiding capability narrowing |
The goal of this mix is to close gaps without losing existing skills: AutomationBench and SWE-bench Pro improve together, with no trade-off between them.
⑤ Held-out Evaluation
The 60 AutomationBench tasks and 60 SWE-bench Pro tasks used for evaluation are held-out sets, isolated from data generation, to check whether the capability truly transfers. The baseline and V2 use the same base model, the same tasks, and the same inference configuration.
Training Configuration
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.8-27B |
| Framework | ms-swift 4.7 (Megatron backend) |
| Training method | LoRA SFT |
| LoRA rank / alpha | 16 / 32 |
| LoRA target modules | all-linear |
| Vision tower | Frozen |
| Epochs | 3 |
| Max training length | 262,144 tokens (packing + padding-free) |
| Learning rate | 1e-5, cosine decay to 1e-6, 5% warmup |
| Optimizer | Adam (β1 = 0.9, β2 = 0.95), weight decay 0.1 |
| Loss | Computed on assistant outputs only, ignoring empty think blocks (default+ignore_empty_think) |
| Parallelism | TP 4 × CP 2, sequence parallel, full recompute, FlashAttention |
| Precision | BF16 |
| GPUs | 8 |
| Teacher model | DeepSeek-V4.1-Flash-1017 (self-hosted by OpenCSG) |
Why a 262K training length: agent trajectories often contain dozens of tool-call turns and large tool outputs (CRM records, source files, test logs). A 262K window with packing fits a long trajectory into a single sample without truncation, so the model learns the full execution from the first step to the final check, not just fragments.
Engineering Highlights
- Whole trajectories, no truncation. A 262,144-token training length with packing and padding-free batching fits dozens of tool-call turns, large CRM records, source files, and test logs into one sample, so the model learns the full execution from the first step to the final check.
- Memory and throughput for ultra-long sequences. Training a 27B dense model on 262K sequences on 8 GPUs uses TP 4 × CP 2 sharding, sequence parallelism, full recompute, and FlashAttention to keep long-context training stable.
- Loss only where the model should learn. Loss applies only to assistant outputs; tool returns and environment observations carry no gradient. Empty think blocks are ignored (
default+ignore_empty_think), so the model does not learn to skip reasoning. - Lightweight adaptation, drop-in deployment. LoRA covers all linear layers (all-linear), the vision tower stays frozen, and the MTP module ships with the weights; the merged weights keep the base architecture and serve directly with vLLM's tool-call and reasoning parsers.
Interpreting the Results
- Strict like-for-like control: the baseline and V2 share the same base model, tasks, and inference configuration; the only variable is V2's data loop.
- Consistent gains across task families: business automation and software engineering, two very different task types, improve together, ruling out overfitting to one task type.
- The student beats the teacher: V2 outperforms its teacher on AutomationBench (25 vs 20), which can only happen if the verify gate filters out the teacher's mistakes.
- A controlled version difference: V1 had no AutomationBench-targeted data and matched the baseline; V2 gains 7 tasks after adding data from environment-replayed failures. The gain comes from the data loop itself, not from the base model or evaluation noise.
Benchmark Results
Comparison with the Baseline
| Benchmark | Eval set | Baseline (Qwen3.8-27B) | Agentic-27B-V2 | Absolute gain | Relative gain |
|---|---|---|---|---|---|
| AutomationBench | 60 held-out tasks | 18/60 (30.00%) | 25/60 (41.67%) | +11.67 pts | +38.9% |
| SWE-bench Pro | 60 held-out tasks | 51/60 (85.00%) | 55/60 (91.67%) | +6.67 pts | +7.8% |
Both runs use the same base model, the same evaluation tasks, and the same inference configuration. The only difference is whether the model went through V2's data-loop training.
- AutomationBench (released by Zapier) evaluates agents on real business workflows: reading information across SaaS systems such as CRM, email, spreadsheets, and ticketing, making decisions, and performing write operations. A task passes only if the final system state is correct.
- SWE-bench Pro (released by Scale AI) evaluates agents on real, long-horizon software engineering tasks: locating issues in a full code repository, modifying code, and passing hidden tests.
Comparison on the Same Held-out Sets
On the same 60 + 60 held-out tasks, Agentic-27B-V2 compares with leading open and closed models as follows (each model at its highest reasoning setting):
| Model | Type | AutomationBench (60 tasks) | SWE-bench Pro (60 tasks) |
|---|---|---|---|
| GPT-6 Astra | Closed | 37/60 (61.7%) | 59/60 (98.3%) |
| Gemini 4 Carbon | Closed | 36/60 (60.0%) | 58/60 (96.7%) |
| Claude Opus 5.5 | Closed | 33/60 (55.0%) | 58/60 (96.7%) |
| Agentic-27B-V2 | 27B dense | 25/60 (41.7%) | 55/60 (91.7%) |
| DeepSeek-V4.1-Flash-1017 (teacher) | Open | 20/60 (33.3%) | 56/60 (93.3%) |
| Qwen3.8-Max-0902 | Open | 20/60 (33.3%) | 45/60 (75.0%) |
| Agentic-27B (V1) | 27B dense | 18/60 (30.0%) | 52/60 (86.7%) |
| Qwen3.8-27B (baseline) | 27B dense | 18/60 (30.0%) | 51/60 (85.0%) |
- Ahead of its teacher and its family flagship on AutomationBench: at 27B, V2 scores 25/60, above both DeepSeek-V4.1-Flash-1017 (the teacher used for data generation) and Qwen3.8-Max-0902 at 20/60, and first among the open models listed.
- Above 90% on SWE-bench Pro: V2 solves 10 more tasks than Qwen3.8-Max-0902 and is one task behind the teacher, DeepSeek-V4.1-Flash-1017.
- Targeted training brings real gains: V1 was not trained specifically for AutomationBench and matched the baseline there. After V2 added targeted data from environment-replayed bad cases, AutomationBench passes rose by 7, and SWE-bench Pro passes rose by 3 over V1.
Public Leaderboard Reference
The tables below list public leaderboard scores for some models on the two benchmarks, to give a sense of the capability range.
Note: public leaderboards use the full task sets, and harnesses, inference settings, and whether scores are vendor-reported differ across sources. Agentic-27B-V2's scores come from 60-task held-out subsets. The two cannot be ranked against each other directly and are for order-of-magnitude reference only. For like-for-like comparisons, use the baseline and same-set comparisons above.
AutomationBench
| Model | Size / Type | Public score | Source |
|---|---|---|---|
| Gemini 4 Carbon | Closed | 77.8% | Artificial Analysis-style leaderboard (partial credit) |
| MiniMax M3.1 Flash | Open | 62.2% | BenchLM |
| DeepSeek-V4.1-Flash-1017 | Open | 60.5% | BenchLM |
| Qwen3.8-Max 0902 | Open | 50.8% | Public Zapier-style evaluation |
| Claude Opus 5 (max) | Closed | 50.3% | Public Zapier-style evaluation |
| Qwen3.8-27B (x-high) | 27B dense | 48.2% | AutomationBench-AA (partial score) |
| Claude Opus 5.5 | Closed | 35.8% (independent re-run) | Public Zapier-style evaluation |
SWE-bench Pro
| Model | Size / Type | Public score | Source |
|---|---|---|---|
| Claude Fable 5.2 | Closed | 93.0% | BenchLM (Public set) |
| Claude Opus 5.5 | Closed | 89.9% | BenchLM (Public set) |
| Claude Sonnet 5.5 | Closed | 81.3% | BenchLM (Public set) |
| Qwen3.8-Max | Open | 67.7% | Public leaderboard |
| Qwen3.8-27B | 27B dense | 61.7% | Official Qwen model card (self-reported) |
| GPT-5.5 | Closed | 58.6% | Public leaderboard |
| DeepSeek-V4-Pro (max) | Open | 55.4% | BenchLM |
Agentic Series Evolution
| Version | Base model | Training | Data source | Main evaluation | Key results |
|---|---|---|---|---|---|
| Agentic-30B-A3B | Qwen/Qwen3-30B-A3B (MoE) |
LoRA DPO | Platform data flywheel, ~3,000 preference samples | Agentic Eval | Mean 0.46, pass@3 37/119 |
| Agentic-27B | Qwen/Qwen3.8-27B (dense) |
LoRA DPO | Platform data flywheel, 5,857 preference samples | Agentic Eval / Arena-Hard-v2.0 | Mean 0.83, pass@3 96/103; hard_prompt 57.68 |
| Agentic-27B-V2 | Qwen/Qwen3.8-27B (dense) |
LoRA SFT | Real usage data → environment-replayed bad cases → self-hosted teacher fills data → verifier filtering | AutomationBench / SWE-bench Pro (held-out) | 41.67% / 91.67%, both clearly above baseline |
What Changed
- From in-platform evaluation to public, verifiable evaluation: the first two releases were mainly evaluated on OpenCSG's in-house Agentic Eval. V2 is validated on public, reproducible industry benchmarks, so third parties can check the results.
- From preference alignment to capability building: DPO is good at picking better behavior within existing capabilities. V2 switches to SFT to directly teach tasks the model used to get wrong, which suits closing gaps.
- From intuition-driven data to failure-driven data: every training sample maps to a real failure reproduced in an environment and confirmed correct by the environment verifier, so data decisions are grounded in evidence.
- From a single scenario to business + code: V2 covers both business process automation and software engineering, the two most important real-world agent scenarios.
Industry Contributions
- A repeatable loop: "real data → environment matching → bad-case replay → targeted data → held-out evaluation" is a process that can keep iterating. A new scenario or a new batch of real failures is enough to run another round, without guessing what data to build.
- Transferable capability: training data comes from real failures on the OpenCSG platform, while evaluation uses held-out tasks from public benchmarks fully isolated from data generation. V2 still improves on these unseen tasks, and on two task families (business automation and software engineering) at once. The model learns general execution skills (choosing the right tool, forming correct arguments, finishing long chains, verifying results) rather than memorizing specific tasks, so the same skills carry over to new business scenarios and new challenges.
- Environments keep growing and get reused: once a type of real failure is mapped to an executable environment, that environment, with its executor and verifier, joins the environment library and is reused every round after: reproducing new failures, verifying new teacher trajectories, and regression-testing new model versions. A new failure type only needs a new environment. As the library grows, the marginal cost of each iteration keeps falling and improvement keeps accelerating.
- Verifier-gated data lets the student beat the teacher: keeping only trajectories that pass the environment verifier distills the teacher's correct behavior before teaching it. V2 outperforming its teacher on AutomationBench shows the loop is not capped by the teacher's average performance.
- Evidence-based data decisions: data is added only at failure points confirmed in an environment. Compared with auto-generating new tasks, this avoids large amounts of unverifiable data with unsuitable difficulty.
- Fully private pipeline: the teacher model is self-hosted by OpenCSG and real usage data never leaves the domain, meeting enterprise data-security and compliance needs. The same process can be reused directly in a customer's private environment.
- Small model, real gains: at 27B dense and deployable on a single GPU, V2 improves on two public benchmarks at once and beats the family flagship Qwen3.8-Max-0902 on like-for-like comparisons, at a deployment cost far below 100B+ models.
- A data flywheel from a real platform: agent usage records continuously produced on the OpenCSG platform feed the loop. The more the platform is used, the closer the model gets to real business.
Model Specifications
| Item | Value |
|---|---|
| Base model | Qwen/Qwen3.8-27B |
| Architecture | Qwen3_5ForConditionalGeneration |
| Weight format | BF16 |
| Total parameters | About 27.36B |
| Transformer layers | 64 |
| Hidden size | 5,120 |
| Attention / KV heads | 24 / 4 |
| Vocabulary size | 248,320 |
| Configured maximum positions | 262,144 |
| Extra weights | MTP module (mtp.safetensors) |
Quick Start
Download the weights first:
hf download opencsg/Agentic-27B-V2 --local-dir ./Agentic-27B-V2
Then start an OpenAI-compatible server with vLLM (replace ./Agentic-27B-V2 with your actual download path if different).
Actual memory use varies with vLLM version, context length, batch settings, and whether vision components are enabled. Adjust these settings for your target workload.
vllm serve ./Agentic-27B-V2 \
--served-model-name OpenCSG/Agentic-27B-V2 \
--tensor-parallel-size 1 \
--gpu-memory-utilization 0.88 \
--max-model-len 32768 \
--max-num-seqs 8 \
--max-num-batched-tokens 8192 \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_xml \
--reasoning-parser qwen3 \
--generation-config vllm
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="OpenCSG/Agentic-27B-V2",
messages=[
{"role": "system", "content": "You are a careful local agent."},
{"role": "user", "content": "Summarize today's action items."},
],
temperature=0.2,
max_tokens=512,
)
print(response.choices[0].message.content)
Limitations
- AutomationBench and SWE-bench Pro scores are based on 60-task held-out subsets each. They differ from the full task sets and harnesses used by public leaderboards and should not be compared directly with leaderboard rankings.
- Do not use model outputs without review for medical, legal, financial, or high-privilege actions. Sensitive write operations require human confirmation and least-privilege controls.
References
- EvoSyn: a weakness-targeted data-synthesis loop.https://arxiv.org/html/2610.12593
- AutoDataBench: the difficulty problem in automatic task generation.https://arxiv.org/html/2609.35025
- Kimi K2: Open Agentic Intelligence.https://arxiv.org/html/2507.20534
- EnvFactory, EnvScaler (2026): environment scaling and trajectory synthesis.
- DeepSeek-V4.1-Flash technical report.https://arxiv.org/pdf/2609.19969
- AutomationBench (Zapier); SWE-bench Pro (Scale AI).
License
Use of this model is governed by the OpenCSG Model and Dataset License Agreement. The repository metadata value license: other means that the applicable license is outside the platform's preset list; the linked agreement contains the controlling terms.
Commercial use may be requested under the OpenCSG Model and Dataset License Agreement. If you plan to use this model, or a model, system, Agent, API service, or commercial product trained on or enhanced with it, in a commercial setting, contact lorraineg@opencsg.com for authorization.
The license_type: 商业授权 field in the current public snapshot is a record-level authorization-source label. It does not replace the repository-level license agreement.
- Downloads last month
- 75