XHToken

/

Spark-X2.5-4B

Like 1.43k

Follow

SparkLLM 908

Status: completed and locally verified
Publication status: approved by the user for official submission
Intended venue: XHToken/Spark-X2.5-4B Hugging Face Discussions
Intended entry confirmation: XHToken/Spark-X2.5 GitHub Issue #9

Abstract

This report presents a rapid diagnostic mathematics assessment of XHToken/Spark-X2.5-4B under constrained local-inference conditions

The evaluated artifact was a community-published Q4_K_M GGUF loaded through a locally compiled Spark-compatible llama.cpp and Ollama runtime on a Windows laptop with an NVIDIA GeForce RTX 4060 Laptop GPU

Thirty problems were selected before the run from the 100-row XHToken/gaokao_math_2026 test split using a pre-registered stratified pseudorandom procedure

The model received each raw dataset query exactly once, with no system prompt, answer-format instruction, tool, retry, continuation, or post-hoc rescue

Thinking mode and the upstream-recommended sampling parameters were retained, while every response was subject to an intentional hard generation budget of 8,192 tokens

The primary strict pass@1 result was 15/30, or 50.0%

All 15 submitted final answers were correct after manual review, while the other 15 responses exhausted the generation budget without emitting a final answer and were scored as failures

The main observed limitation was therefore timely answer submission under the fixed generation budget rather than an incorrect non-empty final answer

1. Evaluation Intent

This was designed as a rapid diagnostic assessment rather than an unlimited best-effort benchmark

The question was not whether the model could eventually solve every problem with unlimited reasoning time

The question was whether a local 4B model could independently solve and submit answers under a fixed one-attempt generation budget

The assessment followed four principles

1. Pre-register the sample before observing formal results
2. Send only the original dataset query
3. Allow one response per problem
4. Treat failure to submit within the budget as a strict failure

2. Model and Artifact Provenance

The local artifact matched the publisher-hosted file by filename, byte size, and SHA-256

The model was imported without local requantization

The publisher did not disclose a complete quantization command, importance matrix, calibration corpus, or conversion log

This report therefore does not claim bit-level equivalence with the official BF16 checkpoint

Sources

• Official Spark-X2.5-4B model card
• Community GGUF repository
• Pinned GGUF repository revision

3. Runtime and Hardware

The formal runtime log confirms CUDA selection, a 16,384-token context, full model-layer offload, and successful local HTTP responses

Sanitized excerpts from the retained formal runtime logs

inference compute library=CUDA compute=8.9 description="NVIDIA GeForce RTX 4060 Laptop GPU" total="8.0 GiB"
load_tensors: offloaded 37/37 layers to GPU
load_tensors: CUDA0 model buffer size = 2474.63 MiB
srv llama_server: model loaded
[GIN] 2026/09/12 - 15:06:41 | 200 | 7.6225413s | POST "/api/chat"
[GIN] 2026/09/12 - 15:06:48 | 200 | 6.714592s | POST "/api/chat"
[GIN] 2026/09/12 - 15:07:08 | 200 | 20.7428955s | POST "/api/chat"

Host paths, local ports, proxy values, and device identifiers were removed from this excerpt

4. Dataset and Pre-Registered Sample

Question-type quotas were fixed before the run

Seven problems seen during engineering pilots were excluded before formal selection

Eligible rows were ranked within paper and question-type strata by SHA256(seed|id) and selected according to the fixed quotas

The proof quota was applied before the run to approximate the source dataset's proof proportion

The frozen manifest is the definitive sample definition and was not changed after results were observed

The selection is therefore described as pre-registered, stratified, and pseudorandom rather than as unconstrained pure random sampling

Source: XHToken/gaokao_math_2026

Dataset byte hashes

The original downloader computed a hash over canonical LF text before Windows converted the written file to CRLF line endings

Both scopes are disclosed

The formal experiment contract records the physical file hash

No question content changed

5. Prompt and Generation Contract

The model received one user message containing the exact dataset query value

No system message or user wrapper was added

The following metadata was never included in a model request

• Target answer
• Scoring rule
• Proof flag
• Question type
• Paper score metadata
• Manual-review information

The upstream model card recommends thinking mode with temperature 1.0, top-p 0.95, and top-k -1

This assessment retained those sampling settings and intentionally constrained the generation budget

The 900-second value was a transport and service safety timeout, not a fixed wall-clock examination limit

The enforced assessment cutoff was the 8,192-token generation budget

6. No-Rescue and Scoring Policy

Each problem received exactly one formal response

No hint, correction, reflection request, continuation, answer-format repair, tool call, or target-derived feedback was sent

A problem was counted as correct only when the retained final-content field contained a correct submitted answer

If the model exhausted the generation budget without emitting final content, the problem was scored as a failure even when internal thinking appeared promising

Internal thinking was used only for qualitative failure analysis and was never converted into a submitted answer

The primary metric was

strict pass@1 = correct submitted final answers / all formal problems

Submitted-answer correctness was reported only as a secondary diagnostic metric

7. Results

Primary outcome

The strict pass@1 value of 50.0% is the primary result

The 100.0% submitted-answer correctness is diagnostic only and does not replace the primary score

The 72/223 point yield is an all-or-nothing score for the fixed subset and is not a full 750-point Gaokao score

Results by question type

All three proof problems exhausted the generation budget without final content

Results by paper

Runtime profile

All 30 formal requests returned HTTP 200

Earlier engineering-pilot HTTP 500 responses occurred before the formal window and were excluded from the formal result

An optional online recommendation-refresh warning occurred after the formal run and did not affect local inference

8. Manual Adjudication

All 15 non-empty final responses were manually checked against the retained dataset target and visible reasoning

Twelve were also recognized automatically after conservative parser normalization

Three required manual mathematical adjudication

The parser was also extended to recognize a common Chinese choice-conclusion phrase meaning therefore choose B

No incorrect non-empty final answer was observed in this sample

This does not establish 100% overall mathematical accuracy because half of the formal problems had no submitted final answer

9. Representative Reasoning Review

The sanitized publication package preserves all 30 raw queries, thinking traces, final-content fields, stop reasons, token counts, and timing records

The following evidence blocks are reproduced verbatim in the original source language rather than translated or reconstructed

Verbatim successful trace

Problem 20260820_math_5_10

Raw query

化简 \((3 + i)^2 =\) ______。

Complete raw thinking field

我们被要求化简 \((3 + i)^2\)，其中 \(i\) 是虚数单位，满足 \(i^2 = -1\)。

计算 \((3 + i)^2\)：

\[
(3 + i)^2 = (3)^2 + 2 \cdot 3 \cdot i + i^2 = 9 + 6i + i^2
\]

因为 \(i^2 = -1\)，所以：

\[
9 + 6i + (-1) = 9 - 1 + 6i = 8 + 6i
\]

因此，\((3 + i)^2 = 8 + 6i\)。

答案应该填入：\(8 + 6i\)。

Complete raw final-content field

\((3 + i)^2 = 9 + 6i + i^2 = 9 + 6i - 1 = 8 + 6i\)

\boxed{8 + 6i}

Recorded termination: done_reason=stop, eval_count=224

Verbatim budget-exhaustion evidence

Problem 20260820_math_1_7 produced an empty final-content field with done_reason=length and eval_count=8192

Beginning of the retained raw thinking field

我们被问到："一百零八塔位于宁夏回族自治区青铜峡市，以其独特的建筑格局和深远的历史文化闻名遐迩。该塔群共有$108$座塔，依山势自上而下排成$12$行，将第$i$行中塔的座数记为$a_i(i=1,2,\dots,12)$，其中$a_1=1$，$a_2=a_3=3$，$a_4=a_5=5$，且$a_6,a_7,\dots,a_{12}$是一个首项为$7$，公差为$2$的等差数列。

我们需要找出这12个数，将它们分成6组，每组两个数，它们的和构成一个公差为d的等差数列，求d。

End of the retained raw thinking field

3+3=6
3+5=8
3+9=12
3+11=14
3+13=16
3+15=18
3+17=20
3+19=22
5+5=10
5+9=14
5+11=16
5+13=18
5+15=20
5+17=22
5+19=24
9+9=18 (但只有一个9)
9+11=20
9+13=22
9+15=24
9+17

The final line ends mid-enumeration in the retained response, and no final answer was emitted

The middle of this 8,192-token trace is omitted from the Discussion for readability, not from the immutable local evidence

Correct concise response

20260820_math_5_10 expanded (3+i)^2 directly and submitted 8+6i

The response stopped after 224 generated tokens and matched the target exactly

Correct multi-field response

20260820_math_4_14 derived the base area as sqrt(3) and the tetrahedron volume as 2sqrt(3)/3

Both submitted fields matched the target after manual review

Repetitive enumeration under budget

20260820_math_1_7 repeatedly explored pairing configurations and exhausted the generation budget without final content

It remained a strict failure

Partial multi-part derivation under budget

20260820_math_5_19 solved substantial intermediate sequence structure and continued into the second part but did not emit final content before the cutoff

It remained a strict failure

Incomplete dataset input

20260820_math_2_15 refers to a frequency-distribution histogram, but the public text row does not contain the histogram values or an image payload

The model introduced hypothetical distributions and continued reasoning until budget exhaustion

The problem remained in the denominator and was scored as a strict failure because the pre-registered sample could not be replaced after observation

It is separately flagged as dataset input incompleteness

Wrong-reasoning-right-answer check

No clear wrong-reasoning-right-answer case was identified among the 15 submitted final answers

The report does not fabricate such a category when none was observed

10. Interpretation

The 50.0% strict pass@1 score measures end-to-end success under this assessment contract

It includes both mathematical solution quality and the ability to submit a final answer before the generation cutoff

The result must not be interpreted as a general claim that the model's unconstrained mathematical accuracy is 50.0%

It must also not be interpreted as a claim that the model would achieve 100.0% if allowed unlimited output

The observed evidence supports a narrower conclusion

Under a raw-query, thinking-enabled, one-attempt, 8,192-token hard-budget protocol, Spark-X2.5-4B Q4_K_M submitted 15 correct final answers and failed to submit 15 final answers

The dominant observed failure mode was generation-budget exhaustion before answer submission

No incorrect submitted final answer was observed

11. Limitations

• The sample covers 30% of one public dataset and is not the complete 100-problem benchmark
• The Q4_K_M artifact is community-published rather than the official BF16 checkpoint
• The full quantization recipe is unavailable
• Fixed seeded sampling improves replayability but does not guarantee bit-identical output across runtimes or hardware
• The assessment imposes a token budget rather than a strict per-problem wall-clock limit
• No second condition, pass@k, self-consistency, tool-assisted run, or larger-model comparison was executed
• Written and proof problems produced no final submissions, so no positive proof-quality conclusion can be drawn
• One selected problem has an incomplete text-only input
• Manual adjudication was used for mathematical equivalence and multi-field answers

12. Reproducibility Artifacts

The formal evaluation and scoring commands, executed from the project root, were

& <PYTHON-3.12> scripts/run_eval.py `
  --config config/official-raw-query.json `
  --run-id formal-raw-30pct-pass1

& <PYTHON-3.12> scripts/score_results.py `
  --run-dir runs/formal-raw-30pct-pass1 `
  --adjudication reports/formal-raw-30pct-pass1-manual-adjudication.json

& <PYTHON-3.12> scripts/build_publication_package.py `
  --run-dir runs/formal-raw-30pct-pass1 `
  --out-dir publication/formal-raw-30pct-pass1

<PYTHON-3.12> denotes the Python 3.12 interpreter used for the recorded run; the private absolute interpreter path is intentionally omitted

The local evidence set contains

• Frozen sample manifest and SHA-256
• Exact model and runtime revisions
• GGUF byte size and SHA-256
• Exact raw-query request construction
• Generation settings and seed
• Thirty raw model outputs
• Per-problem token and timing records
• Conservative automatic scoring
• Manual adjudication for every submitted final answer
• Sanitized publication package with its own file hashes

Internal source hashes

13. Privacy and Release Boundaries

• Usernames and personal cache paths are excluded
• Local endpoint details are redacted in the publication package
• Hugging Face tokens, cookies, authorization headers, and private data are excluded
• Model weights are not included
• Prompts and outputs are separated from target answers and grading metadata
• The raw internal evidence remains unchanged
• External publication requires explicit user review

14. AI-Assistance Disclosure

Codex assisted with evidence inspection, experimental-contract verification, scoring-tool development, manual result reconciliation, English technical editing, privacy review, and publication-package preparation

The evaluated mathematical responses were generated locally by Spark-X2.5-4B Q4_K_M

No AI persona is presented as a human participant or contributor

15. Submission References

• HER Hack-Astron #6 official challenge issue
• Spark-X2.5-4B Hugging Face Discussions
• Official Spark-X2.5 repository

This report is the complete evaluation prepared for the official challenge submission