[HER Hack-Astron #6] Spark-X2.5-4B 原生计算器配对评测:63 次工具调用均成功,为什么 pass@1 反而下降?

#19
by Forrest20231206 - opened

我实际运行了官方 XHToken/Spark-X2.5-4B 原始 BF16 权重,在 Apple
GSM-Symbolic 的 main / p1 / p2 三档上做了 60 道题、120 条轨迹的同题配对
评测。结果先说:

config 直接策略 计算器策略 样本内差值 McNemar p Holm p
main 17/20 = 85% 13/20 = 65% -20 pp 0.12500 0.37500
p1 16/20 = 80% 12/20 = 60% -20 pp 0.21875 0.43750
p2 12/20 = 60% 10/20 = 50% -10 pp 0.68750 0.68750
总计(描述性) 45/60 = 75.00% 35/60 = 58.33% -16.67 pp 不做汇总检验 不适用

三个 config 的 Holm 校正结果都不显著,所以这不是“计算器降低模型能力”的普遍
结论。更有价值的发现是:模型在真实 Agent 工具协议中,可能不是算错,而是无法在
有限调用预算内正确结束对话。

核心发现:执行器没错,答案却没有交付

计算器策略的 60 条轨迹中:

  • 40 条实际发出原生 function call,20 条没有遵循必调策略;
  • 共 63 次调用,安全执行器重放结果为 63/63 成功、0 executor error;
  • 23 条用满两次调用额度,其中 18 条在 schema 撤除后仍输出文字化第三个
    <tool_call>,没有 FINAL_ANSWER;
  • 直接策略可解析 55/60,计算器策略可解析 38/60;全部 27 条不可解析恰好由
    9 条 token 截断和上述 18 条工具协议失败组成。
计算器策略的实际调用数 轨迹数 严格正确 可解析末答
0 20 16 17
1 17 14 16
2 23 5 5

这个按调用数的表是后验描述,不是因果估计。13 个“仅直接策略答对”的 pair 中,
12 个对应两次调用后的末答交付失败;3 个“仅计算器策略答对”的 pair 中,只有
2 个真的调用了工具,另 1 个只是策略条件不同。因此本文始终把差异归因于
“策略前缀 + 工具访问”的联合条件,不把它包装成纯工具效应。

这也给部署留下了可操作建议:将“算式执行成功”“题意建模正确”“模型遵循工具
策略”和“最终答案已交付”设为四个独立观测项;多步题不要只给固定两次调用后突然
撤掉 schema,应提供明确的剩余预算/结束信号,或让模型一次提交组合表达式。

方法与严格口径

  • 模型:XHToken/Spark-X2.5-4B,revision
    5e10fcc0286756aebf7c41dc52c1e42d95c70281,官方 BF16,无量化、无训练;
  • 数据:apple/GSM-Symbolic,revision
    93b5b3758d9d9841ffe81d6cd2ae2b030685b078,test split;
  • 抽样:对 main/p1/p2 的公共 (id, instance) 做固定盐 SHA256 选择,20 个
    模板键 × 3 档 = 60 题;模型只收到 question,从未收到 answer/rationale/canary;
  • 配对:每题分别运行直接解题策略和要求原生 calculator 的策略;英文题面字节相同,
    两臂仅策略前缀和工具 schema 不同;
  • 解码:thinking off、temperature 0、top_p 1、top_k -1、
    repetition_penalty 1、n=1、base seed 20260913,episode seed 按题固定推导;
    累计 completion 上限 4096,最多两次工具调用;
  • 判分:只抽取终局唯一完整的 FINAL_ANSWER: <number>,以 Fraction 精确比较
    数据集末行 ####;不可解析、截断、协议失败均留在 120 条分母并计错;
  • 统计:显著性阈值 α=0.05;每 config 用精确双侧 McNemar,并对三次检验做
    Holm 校正。三档复用了模板块,所以总计只描述,不做伪精确的 pooled
    significance test。这里只报告 pass@1;没有 pass@k、self-consistency 或平局。

可复查的冻结输入与设置

正式评测绑定运行前协议 commit
cdfe83a112308380c6a4bae5adb9de14f38c7bc2。选择盐与模板键清单哈希分别为:

HER-Hack-Astron-6|Spark-X2.5-4B|apple/GSM-Symbolic|93b5b3758d9d9841ffe81d6cd2ae2b030685b078|paired-v1
73538354c63a79e00ee986cdac83cd748d14e256dc0015ee9dd4c231ca922d77

每条轨迹的 base seed 是 20260913,episode seed 为
20260913 + int(SHA256("seed|<eval_id>")[:8], 16) % 1_000_000;后续轮次依次
加 1。两臂的完整提示如下(随后拼接完全相同的 题目(原文): 与英文题面):

展开查看完整 prompt 与工具 schema

System prompt:

Solve the supplied math problem carefully. Follow the user policy. After any tool result, give a concise, checkable derivation. End with exactly one line: FINAL_ANSWER: <number>, where <number> has no unit. Do not write anything after that line.

直接策略 prefix:

请独立理解下面的英文数学题并手工计算,不要使用任何外部工具。给出简洁、可检查的推导,并以 FINAL_ANSWER: <number> 结尾。

计算器策略 prefix:

请先理解下面的英文数学题并确定所需算式,然后必须调用 calculator 工具精确计算。不要在工具返回前给最终答案。工具返回后再简要说明推导,并以 FINAL_ANSWER: <number> 结尾。

原生工具 schema:

{
  "type": "function",
  "function": {
    "name": "calculator",
    "description": "Exactly evaluate one arithmetic expression. When this tool is available, it must be called at least once. It does not interpret or solve the word problem.",
    "parameters": {
      "type": "object",
      "properties": {
        "expression": {
          "type": "string",
          "description": "Arithmetic using numeric literals, parentheses, and + - * / // % ** only."
        }
      },
      "required": ["expression"],
      "additionalProperties": false
    }
  }
}

正式运行命令(合成 pilot 通过后执行):

uv run python scripts/run_evaluation.py \
  --base-url http://127.0.0.1:50269/v1 \
  --model spark-x2.5-4b \
  --manifest data/selection_manifest.jsonl \
  --excerpt third_party/gsm_symbolic_selected.jsonl \
  --runtime-json evidence/environment.json \
  --protocol-commit cdfe83a112308380c6a4bae5adb9de14f38c7bc2 \
  --output-dir runs/live --log runs/run.log --workers 4

运行栈为 Python 3.12.7、OpenAI SDK 2.21.0、vLLM 0.23.0 镜像
vllm/vllm-openai:v0.23.0(repo digest
sha256:6d8429e38e3747723ca07ee1b17972e09bb9c51c4032b266f24fb1cc3b22ed8f),
Spark plugin commit ddef50f6f1eda417b6716b27e0aa35d4a6cd0cf3(wheel SHA256
a7728f6482fffd734ddbedd230a6e066be6263132c2b59c648f39bb28977c09b)。

脱敏日志片段:

Loading safetensors checkpoint shards: 100% Completed | 5/5
INFO: Application startup complete.
2026-09-13T08:14:05.220653+00:00 start jobs=120 workers=4
2026-09-13T08:16:56.733824+00:00 finish counts={"completed": 120}

正式生成前先用不重叠的合成题校准 vLLM spark25 工具接口;所有失败 pilot 都
保留。正式生成开始后没有改样本、prompt、工具、参数、评分器或分母。

推理完整性:不是只看末答

我按冻结规则审阅了 75/120 条轨迹,覆盖全部 40 条错误/不可解析、全部 40 条真实
工具调用、16 组 A/B 分歧的两臂,以及每个 config 的预选对照。主标签为:coherent
32、tool protocol 18、truncation 9、semantic/planning 7、relevant-clause
omission 4、dataset ambiguity 4、arithmetic 1。

代表案例包括:

  1. p1-id44-i39:直接策略把“男孩为女孩的 1/4”方向写反;计算器策略正确建模并
    通过两次原生调用得到 80,属于真实调用伴随的修复;
  2. main-id10-i44:两次调用分别正确得到 11 和 3,但模型还想发第三次调用,最终
    没有提交 14,是典型的答案交付失败;
  3. p1-id24-i08:模型把运动消耗写成错误符号,计算器只是精确执行了错误方程,
    说明 executor success 不等于 semantic success;
  4. p2-id26-i27:末答 1680 正确,但可见推导仍把连续减少 20% 和 50% 错称为
    总计减少 70%,是 1 条严格 RWRA;
  5. p2-id33-i27:两臂按“每周减少 5 个”的字面含义得到 14,而金标 10 依赖
    “降到每周 5 个”的读法,作为 dataset ambiguity 保留而不偷偷改分。

下面内嵌三条完整轨迹:模型内容未删改,仅规范化行尾排版空格,结构化字段仅做
排版;字节级记录可用 raw_record_sha256 对照仓库中的 runs/raw.jsonl。

1. 工具调用成功并交付正确答案:p1-id44-i39 / calculator
  • 严格金标:80
  • 严格预测:80,计为正确
  • episode seed:20318171;后续轮次依次加 1
  • raw_record_sha256:3e92c08c1a76c981e0303831fed5b126e596a9cfdd4d3ad938c884a0399a72cc

系统提示:

Solve the supplied math problem carefully. Follow the user policy. After any tool result, give a concise, checkable derivation. End with exactly one line: FINAL_ANSWER: <number>, where <number> has no unit. Do not write anything after that line.

用户消息:

请先理解下面的英文数学题并确定所需算式,然后必须调用 calculator 工具精确计算。不要在工具返回前给最终答案。工具返回后再简要说明推导,并以 FINAL_ANSWER: <number> 结尾。

题目(原文):
There are 8 girls in the garden. If there are 1/4 the number of boys in the garden, what percentage of the kids in the garden are girls?

第 1 轮 assistant(finish_reason=tool_calls):

Let me solve this step by step.

**Understanding the problem:**
- There are 8 girls in the garden.
- The number of boys is 1/4 the number of girls.
- We need to find what percentage of the total kids are girls.

**Step 1: Find the number of boys**
Boys = 1/4 × Girls = 1/4 × 8 = 2

**Step 2: Find the total number of kids**
Total = Girls + Boys = 8 + 2 = 10

**Step 3: Find the percentage of girls**
Percentage = (Girls / Total) × 100 = (8 / 10) × 100 = 80%

Let me verify with the calculator.

第 1 次原生函数调用与执行器返回:

{
  "call": {
    "function": {
      "arguments": "{\"expression\":\"(1/4) * 8\"}",
      "name": "calculator"
    },
    "id": "chatcmpl-tool-ac2ccbf6cd37f5eb",
    "type": "function"
  },
  "executor_error": null,
  "executor_result": {"decimal": "2", "exact": "2", "ok": true}
}

第 2 轮 assistant(finish_reason=tool_calls):原始 content 为空字符串。

第 2 次原生函数调用与执行器返回:

{
  "call": {
    "function": {
      "arguments": "{\"expression\":\"(8 / (8 + (1/4) * 8)) * 100\"}",
      "name": "calculator"
    },
    "id": "chatcmpl-tool-9ef0f5a7798f9a5a",
    "type": "function"
  },
  "executor_error": null,
  "executor_result": {"decimal": "80", "exact": "80", "ok": true}
}

第 3 轮 assistant(finish_reason=stop,tools_available=false):

**Derivation:**
- Number of boys = 1/4 × 8 = 2
- Total kids = 8 + 2 = 10
- Percentage of girls = (8 / 10) × 100 = 80%

FINAL_ANSWER: 80

这条轨迹在计算器联合条件中完成了题意建模、两次原生调用和末答交付;单条案例本身
不用于声称纯工具因果。

2. 两次调用均成功,但协议交付失败:main-id10-i44 / calculator
  • 严格金标:14
  • 严格预测:不可解析,计为错误
  • 解析错误:missing_or_multiple_final_answer_markers
  • episode seed:20610456;后续轮次依次加 1
  • raw_record_sha256:89d135e0b32ab1b5b5df980481898297eeb875fc1660535e63643cea61f68a4d

系统提示:

Solve the supplied math problem carefully. Follow the user policy. After any tool result, give a concise, checkable derivation. End with exactly one line: FINAL_ANSWER: <number>, where <number> has no unit. Do not write anything after that line.

用户消息:

请先理解下面的英文数学题并确定所需算式,然后必须调用 calculator 工具精确计算。不要在工具返回前给最终答案。工具返回后再简要说明推导,并以 FINAL_ANSWER: <number> 结尾。

题目(原文):
To make a call from a phone booth, you must pay €0.5 for each minute of your call. After 22 minutes, that price drops to €0.1 per minute. How much would a 52-minute call cost?

第 1 轮 assistant(finish_reason=tool_calls):

Let me break down this problem step by step.

The pricing structure is:
- First 22 minutes: €0.5 per minute
- After 22 minutes (remaining minutes): €0.1 per minute

For a 52-minute call:
- First 22 minutes cost: 22 × €0.5
- Remaining minutes: 52 - 22 = 30 minutes, costing: 30 × €0.1

Let me calculate each part.

第 1 次原生函数调用与执行器返回:

{
  "call": {
    "function": {"arguments": "{\"expression\":\"22 * 0.5\"}", "name": "calculator"},
    "id": "chatcmpl-tool-963101f8df8888d8",
    "type": "function"
  },
  "executor_error": null,
  "executor_result": {"decimal": "11", "exact": "11", "ok": true}
}

第 2 轮 assistant(finish_reason=tool_calls):原始 content 为空字符串。

第 2 次原生函数调用与执行器返回:

{
  "call": {
    "function": {
      "arguments": "{\"expression\":\"(52 - 22) * 0.1\"}",
      "name": "calculator"
    },
    "id": "chatcmpl-tool-baab61188a06015a",
    "type": "function"
  },
  "executor_error": null,
  "executor_result": {"decimal": "3", "exact": "3", "ok": true}
}

第 3 轮 assistant(finish_reason=stop,两次额度已用完,tools_available=false):

<tool_call>calculator<arg_key>expression</arg_key><arg_value>11 + 3</arg_value></tool_call>

两次执行结果 11 和 3 都正确,但最后一轮没有提交要求的 FINAL_ANSWER: 14,
而是生成了不可执行的文字化第三次调用。因此该轨迹保留在分母并计错,主标签为
tool_protocol。

3. 末答正确但推理保留矛盾:p2-id26-i27 / no_tool
  • 严格金标:1680
  • 严格预测:1680,末答计为正确
  • 推理审阅:right_answer_wrong_reasoning
  • episode seed:20789663
  • raw_record_sha256:927c4bd2d8ac5f6210b81a66b772906a50e41e220fae3f0ee56520c34ffb27ca

系统提示:

Solve the supplied math problem carefully. Follow the user policy. After any tool result, give a concise, checkable derivation. End with exactly one line: FINAL_ANSWER: <number>, where <number> has no unit. Do not write anything after that line.

用户消息:

请独立理解下面的英文数学题并手工计算,不要使用任何外部工具。给出简洁、可检查的推导,并以 FINAL_ANSWER: <number> 结尾。

题目(原文):
A grandfather eats 5 hamburgers per day, his partner eats 15 hamburgers per day, and their son eats three-fourths of the combined number of hamburgers they eat per day. After one month, the family decides to go on a diet in the second month, reducing their food consumption by 20% for the first half of the month, and then by an additional 50% for the second half. How many hamburgers does this family eat in two months?

第 1 轮 assistant(finish_reason=stop):

Step 1: Compute daily hamburger consumption.
Grandfather: 5
Partner: 15
Combined: 5 + 15 = 20

Step 2: Son eats three-fourths of the combined amount.
3/4 × 20 = 15

Step 3: Total daily consumption in first month (normal):
5 + 15 + 15 = 35 hamburgers per day

Step 4: First month total (assume 30 days):
35 × 30 = 1050

Step 5: Second month diet plan:
- First half (15 days): reduce by 20% → multiply by (1 – 0.20) = 0.80
  Daily consumption = 35 × 0.80 = 28
  Total for first half = 28 × 15 = 420

- Second half (15 days): reduce by an additional 50% from the already reduced level
  So total reduction = 20% + 50% = 70% reduction → multiply by (1 – 0.70) = 0.30
  Alternatively: 28 × (1 – 0.50) = 28 × 0.50 = 14
  Daily consumption = 14
  Total for second half = 14 × 15 = 210

Step 6: Second month total = 420 + 210 = 630

Step 7: Total for two months = First month + Second month = 1050 + 630 = 1680

FINAL_ANSWER: 1680

末答采用正确的连续折减路径 35×0.8×0.5=14,因此得到 1680;但同一段推导又
声称“20% + 50% = 总计减少 70%”,对应保留比例 0.3。两种说法相互矛盾,所以
这是严格的“答案正确、推理错误”,而不是普通的已纠正 false start。

七组双臂完整可读轨迹:
https://github.com/MittaPei/spark-x2-5-gsm-symbolic-tool-study/blob/f91d795c1b054b71142bdc32a6af22c8a3e31497/reasoning/cases.md

全部 75 条审阅与具体证据:
https://github.com/MittaPei/spark-x2-5-gsm-symbolic-tool-study/blob/f91d795c1b054b71142bdc32a6af22c8a3e31497/reasoning/review.jsonl

与现有 Issue / Discussion 投稿相比的互补优势

截至提交时,公开案例已经覆盖标准数学集、thinking 开关、多语言与数值扰动、答案
交付等方向,Issue 中也出现了受限计算器对照。本提交的互补价值在于把
Spark-X2.5-4B 原始 BF16、GSM-Symbolic main/p1/p2 三复杂度模板配平、完全
同题的直接策略/原生函数调用策略 A/B,以及工具采纳—执行器结果—语义建模—调用
预算—末答交付的分层分析
放进同一个运行前冻结、可离线复算的证据闭环。120 条
原始轨迹逐轮保留函数参数与返回,并结合逐 config 的 McNemar/Holm 校正和 75 条
推理审阅;差异化价值不在宣称更高单一分数,而在于把 Agent 工具链的实际失效位置
转化为可验证、可改进的部署证据。

完整证据与复现

固定版本仓库:
https://github.com/MittaPei/spark-x2-5-gsm-symbolic-tool-study/tree/f91d795c1b054b71142bdc32a6af22c8a3e31497

关键入口:

  • 完整报告与命令:README.md
  • 冻结协议与完整 prompt:PROTOCOL.md、src/spark_math_eval/runner.py
  • 120 条原始响应和工具转录:runs/raw.jsonl
  • 逐题评分与汇总:results/per_item.jsonl、results/summary.json
  • 环境、权重 LFS SHA256、数据源 SHA256、镜像 digest:evidence/environment.json
  • 正式运行和脱敏 vLLM 日志:runs/run.log、evidence/runtime/
  • 失败及通过的合成 pilot:evidence/pilot*
  • 证据完整性:SHA256SUMS
  • CI:https://github.com/MittaPei/spark-x2-5-gsm-symbolic-tool-study/actions/runs/34748779510

离线复算不需要 GPU:

uv sync --all-extras --locked
uv run python scripts/summarize.py
uv run python scripts/audit_reasoning.py
uv run python scripts/render_cases.py
uv run python scripts/verify_artifacts.py
uv run ruff check .
uv run ruff format --check .
uv run pytest -q

verify_artifacts.py 会重新评分、重放 63 次白名单算术调用、重建推理审阅、核验
SHA256,并阻止 secret、个人路径和模型权重进入公开材料。

运行环境为共享 A800-SXM4-80GB、vLLM 0.23.0、TP=1、4 workers;正式窗口
171.509 秒。直接/计算器策略的总 API token 分别为 52,444 / 107,596。本次复用
现有服务器和本地权重,增量费用为 0,整机历史成本未知。固定 seed 和 greedy
decoding 用于减少方差,但不承诺跨硬件 bit-identical generation。20 个模板只是
审计样本,数值替换也不能完全排除模板污染。

原创代码为 Apache-2.0;60 条必要且未修改的数据集摘录仍按 CC BY-NC-ND 4.0,
不受代码许可证覆盖。没有上传任何模型权重,也没有公开 token、个人 cache 路径、
内网地址或私有数据。

个人参赛。GitHub @MittaPei 与发布本 Discussion 的 Hugging Face
@Forrest20231206 是同一位参赛者。Codex 多代理辅助了实验设计、代码、真实执行、
逐条审阅和文稿;参赛者已审阅并理解结果、局限与材料,并对本提交负责。没有把 AI
角色列为真人参与者。

Sign up or log in to comment