Qwen3.5-0.8B-WSPForge-Math-v1

An experimental mathematical specialization of Qwen3.5-0.8B, produced through direct weight edits without gradient descent, backpropagation, or fine-tuning.

On a frozen, held-out practice cohort containing 64 GSM8K and 64 MATH problems, the original checkpoint scored 13/128 and this candidate scored 50/128. This is not a MATH-500 result or the official GSM8K test score. Independent coding evaluation did not confirm an improvement.

Attribution and license

The original model is by the Qwen team and retains its Apache 2.0 license. Weight-Space Programming is the method explored in the original research and its experiment logs. WSPForge does not claim authorship of that method.

Our experiment was informed by the published intervention family and layer findings. It computed its own intervention directions from separate discovery trajectories; it did not copy a published corrected checkpoint. The result should be described as an informed reproduction and further investigation, rather than a blind rediscovery of the affected layers.

What changed

The evaluated candidate, Sol014, edits only:

• model.language_model.layers.15.mlp.down_proj.weight
• model.language_model.layers.19.mlp.down_proj.weight

For each layer, an activation-derived direction is used to construct a masked rank-one correction over 64 selected MLP input columns, with intervention strength 6. The correction is serialized into BF16 weights. After rounding, 130,317 stored parameter values differ from the original: 65,064 in layer 15 and 65,253 in layer 19.

Directions were estimated from eight discovery trajectories, contrasting early states with repetitive late states. Activation measurements used matched 1,024-token surrogate contexts: a shared 512-token prefix and a differing 512-token segment, averaging the final 128 tokens. These surrogate contexts do not reconstruct the complete long-history KV state. “Early” does not mean that the associated solution was certified correct.

Columns were ranked by the mean MLP-activation contrast multiplied by their projection onto the measured direction. With normalized direction u, original FP32 working copy W, and a mask selecting those columns, the intervention is:

coefficients = u @ W
coefficients[unselected_columns] = 0
edited = BF16(W - 6 * outer(u, coefficients))

Saved directions and selected columns reproduce both edited matrices byte for byte. A separate sparse BF16-bit patch reconstructs the complete evaluated Safetensors file with the same SHA-256, avoiding differences from adding rounded floating-point deltas. This verifies reconstruction of the final artifact; re-estimating its directions from discovery trajectories is a distinct reproduction step.

This is 0.01491999% of 873,438,784 stored floating-point parameters, or 0.01732034% of 752,393,024 language-model parameters. These denominators are explicit; percentages based on a rounded “0.8B” size are not interchangeable. All 488 stored tensors were compared: the remaining 486 are unchanged. Tensor names, shapes, dtypes, architecture, embeddings, output head, final normalization, tokenizer, and original chat template are preserved. The release is a complete standalone checkpoint.

Held-out mathematical evaluation

Metric	Original	Sol014
Correct final answers	13/128 (10.16%)	50/128 (39.06%)
GSM8K practice subset	9/64	32/64
MATH practice subset	4/64	18/64
Responses reaching the token cap	116/128	71/128
Repetitive-ending heuristic	107/128	52/128
Mean generated tokens	30,163.55	24,922.02

There were 39 corrected answers and 2 lost previously correct answers. The paired gain was 28.91 percentage points; a problem-level paired bootstrap 95% interval was [20.31,37.50] percentage points. The candidate was frozen before evaluation and was not adjusted using these outcomes.

Both checkpoints used the original chat template/tokenizer, thinking enabled, temperature 0.6, top-p 0.95, top-k 20, presence penalty 1.5, up to 32,768 output tokens, matched problem-specific seeds, and the same fixed request queues. This cohort used one sample per problem. Answer equivalence was checked with math-verify; intermediate reasoning steps were not formally verified. One correct answer in each arm reached the token cap, so truncation and correctness were recorded independently.

These practice tasks are held out from intervention selection, but may have appeared in model pretraining. They are not a new secret benchmark. Published Astra results of 7.9%→48.1% on MATH-500 use a different evaluation scope; our current results do not establish a numerical reproduction of that endpoint.

Independent coding evaluation and regressions

On 48 held-out public HumanEval/MBPP tasks with three matched seeds per task, the original passed 55/144 generations (38.19%) and Sol014 passed 48/144 (33.33%). There were 5 improvements and 12 regressions across paired generations. The problem-clustered 95% interval for the difference was [−11.11,+1.39] percentage points.

A coding improvement is not supported. The point estimate is negative, while uncertainty includes zero. An earlier discovery-only increase from 18/72 to 25/72 did not transfer to this confirmation set. This release is not presented as a coding or agent improvement.

The code protocol was separately fixed: non-thinking, temperature 0.2, top-p 0.95, top-k 20, presence penalty 0, maximum 2,048 output tokens, and an isolated restricted-stdlib execution harness. These are compact public subsets with a custom chat prompt and sandbox, not official leaderboard runs. All 8 resource/process failures were re-executed under the same sandbox limits and reproduced; the original scores were retained.

Causal and language controls

These are discovery-only controls on 16 originally failing problems, with one sample per problem.

Intervention	Changed parameter values	Correct
Original	0	0/16
Sol014: both layers	130,317	8/16
Only column 19:11	1,024	2/16
Full edit without 19:11	129,293	8/16
Restore control column 19:1	129,294	6/16
Layer 15 only	Not promoted	Not completed; deferred
Layer 19 only	Not promoted	Not completed; deferred

The 19:11-only edit did not preserve the full effect. Removing that column from the full edit preserved the 8/16 point estimate but changed which problems succeeded. This does not support a single necessary-neuron explanation. These small discovery controls do not establish population-level equivalence or independent confirmation of the alternative checkpoints. The control-column intervention is not exactly matched in norm (approximately 0.7386 of the 19:11 delta norm). Optional layer-factorial tests were stopped/deferred when closing this stage; no layer-interaction conclusion is claimed from partial outputs.

A fixed 28-prompt screen used 16 public IFEval subset tasks plus 12 custom smoke tasks, one matched seed, and an 8,192-token cap. These are not full IFEval scores.

Category	Original	Sol014	Paired losses
English	2/4	3/4	0
IFEval subset	2/16	10/16	1
Basic reasoning	1/2	2/2	0
Russian (supplemental)	0/2	0/2	0
Tool-call format	0/2	1/2	0
Tool-result interpretation	2/2	2/2	0

English is the primary language check; Russian is supplemental. Shared failures are not counted as regressions. See wspforge/evidence/qwen_language_review.json for qualitative review and wspforge/evidence/qwen_closeout_regressions.json for task-level outcomes.

Language controls prioritize English and supplement it with Russian. A paired loss relative to the original is a regression; both checkpoints failing a task is a shared weakness. Language/keyword checks alone do not establish fluency. Text-only tool-call formatting and tool-result tasks do not constitute a complete multi-turn agent benchmark.

Intended use and limitations

This is a small experimental model for studying direct weight interventions and mathematical behavior. We cannot guarantee preservation of performance in every domain, language, task, or deployment configuration. It can produce incorrect answers and plausible but invalid reasoning. The measured coding regressions should be considered when choosing it for applications.

Available home-computer resources limited evaluation coverage. Full MATH-500, official GSM8K test, full IFEval, extensive language evaluations, vision evaluation, and comprehensive agent benchmarks were not run for this release. Reduced repetition and shorter outputs are not treated as mathematical gains without increased answer correctness. Findings from a single model family do not prove that the method generalizes to other architectures.

Community feedback is welcome, including negative results and independent benchmarks. Please report the checkpoint revision, benchmark release/subset, prompts, seeds, sampling settings, context/output limits, runtime, hardware, and any failures or regressions. Additional evaluations may help establish which abilities benefit and which deteriorate.

Reproducibility and local inference

The wspforge/ directory contains evaluation summaries, paired task IDs/seeds/outcomes, pinned protocols, research code, the original-model revision, an exact sparse BF16 patch, saved intervention vectors, and its replay checks. Local absolute paths are removed from exported metadata. Full private working logs and credentials are not included.

To serve the complete released checkpoint locally with a compatible vLLM installation:

vllm serve end000/Qwen3.5-0.8B-WSPForge-Math-v1 --host 127.0.0.1 --port 8000 \
  --dtype bfloat16 --max-model-len 36864 --max-num-seqs 8 \
  --max-num-batched-tokens 2048 --kv-cache-memory-bytes 4294967296 \
  --gpu-memory-utilization 0.65 --no-enable-prefix-caching

The standalone checkpoint requires no patch application. To reproduce its weight file independently, obtain the official base revision listed in wspforge/patch_manifest.json, install NumPy, and run:

python wspforge/apply_patch.py --original /path/to/original/WEIGHT_FILE \
  --manifest wspforge/patch_manifest.json --patch wspforge/sparse_patch.npz \
  --output /path/to/new/WEIGHT_FILE

Replace WEIGHT_FILE with original_weight_filename from the manifest. The tool verifies original, patch, and reconstructed-file checksums and refuses to overwrite an existing output. This exact artifact replay is separate from reproducing the discovery search and model evaluations. Public paired evidence omits full benchmark prompt text; obtain datasets from their original sources and use the recorded IDs/revisions. The included laboratory code requires its isolated dependencies and dataset preparation; it is not a one-command benchmark service.

Experiments ran on an RTX 4070 with 12 GB VRAM, a Ryzen 9 5950X, and 64 GB RAM. The mathematical runtime used vLLM 0.33.0, BF16, a 4 GiB KV cache, up to 8 active sequences, and 32-request queues. Memory and speed depend on context, batch size, runtime, and other GPU processes; weight file size alone is not total VRAM usage.