File size: 5,906 Bytes
5f847d0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6f9d714
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-generation
tags:
- code
- python
- from-scratch
- gated-deltanet
- hybrid
---

<p align="center"><img src="banner.png" alt="Spark 2.0-code" width="100%"></p>

# Spark 2.0-code

A small language model (**322M parameters**) that writes **Python code only**.
It was trained completely from scratch (pretraining, then two SFT stages) by one person on a single RTX 3090.

You describe a task in English, it answers with Python code.

> **Three versions.** *Spark 2.0-code* (this one) writes complete programs best. *Spark 2.0-code (stage 1)* has the
> highest HumanEval (52.4%) and is best for single functions. *Spark 2.0-code-base* is the pretrained base model that
> only continues code.

> **Honest summary.** Spark is good at functions, short scripts and console programs with a menu or a loop.
> It is weak at long programs where many parts must fit together (games like Snake, big multi-class apps).
> It often writes correct-looking code that fails on one detail. Always run and check what it writes.

## Quick start

Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from
[flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only.

```bash
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py "Write a Python function that checks whether a number is prime."
```

`generate.py` loads `model.safetensors` and uses the code in `spark/`. Generation is greedy (temperature 0).

Prompt format (the same as in training):

```
### User
<task in English>

### Assistant
```

The answer is plain code, followed by `<|endoftext|>`. Multi-turn works in the same format: after the answer add a new
`### User` block, for example with an error message ("The code above fails with this error: ... Fix the code.").

Ask "What model are you?" and it answers that it is Spark 2.0-code.

## Model

| | |
|---|---|
| Parameters | 322M (embeddings tied) |
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
| Context | 4096 tokens |
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
| Pretraining | about 18.4B tokens of Python, Muon + AdamW |
| SFT | stage 1 (functions) and stage 2 (complete programs, fixing code from an error message) |
| Weights | bf16, `model.safetensors`, 0.65 GB |

## Results

All numbers come from my own harness in this project, not from an external leaderboard.
Greedy decoding, code is run in a sandbox.

| | Pretrained base | SFT stage 1 | **SFT stage 2 (this model)** |
|---|---|---|---|
| HumanEval pass@1 (164 tasks) | 34.8% | 52.4% | **47.6%** |

HumanEval tests only single functions. Stage 2 moved the model towards complete programs, and its HumanEval score is
about 5 points lower than stage 1 (8 tasks out of 164; part of that can be noise).

My own small checks of the same model (few attempts each, so treat them as rough):

| Check | Stage 1 | Stage 2 |
|---|---|---|
| Console calculator with all 5 operations working (20 tries) | 1/20 | **12/20** |
| "Guess the number" game works (10 tries) | 10/10 | 8/10 |
| 10 short function tasks, greedy | 10/10 | 10/10 |
| Snake in pygame survives 300 frames (8 tries) | 0/8 | 1/8 |

HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution
were removed.

## Limitations

- Long programs (100-200 lines) usually contain several small bugs: a wrong argument count, a string used where an
  object is needed, a forgotten import, a method that is never called. Examples I tested: a library manager, a school
  journal, a pygame arcade game. The structure was fine, but most menu commands crashed on first run.
- It cannot reliably do geometry (positions and sizes of shapes) or game logic where many things depend on each other.
- It sometimes invents library functions and misses imports.
- It only knows Python. English prompts work best.
- A language model: it can write insecure or wrong code. Review before running.

## Training data and license

**License: CC BY-NC 4.0** (non-commercial, with attribution). Reason: the model was trained on data that includes
**KodCode (CC BY-NC 4.0)**, so I do not offer it for commercial use. If you want to use Spark in a commercial product,
contact the author first.

Data used (all text only Python-related; licenses as shown on their Hugging Face pages when I checked):

| Stage | Source | License |
|---|---|---|
| Pretraining | swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
| Pretraining | OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
| Pretraining | KodCode-V1 | CC BY-NC 4.0 |
| Pretraining | opc-annealing-corpus (synthetic Python snippets) | ODC-By |
| SFT | OpenCodeInstruct, KodCode-V1, OpenCoder opc-sft-stage1/2, Magicoder OSS-Instruct and Evol-Instruct, Code-Feedback, CodeFeedback-Filtered-Instruction, glaive-code-assistant-v3, self-oss-instruct-sc2-exec-filter-50k, LeetCodeDataset | MIT, Apache-2.0, CC BY 4.0, ODC-By, CC BY-NC 4.0 (KodCode) |
| SFT stage 2 | code_contests (Python 3 solutions, checked by running the tests) | CC BY 4.0 |
| SFT stage 2 | about 3 800 complete programs written by Qwen3.8-Flash-Next (local, quantized) and checked by running them in a sandbox | Qwen model license not verified by me |
| SFT stage 2 | about 1 200 "fix it from the error message" dialogues made from the real mistakes of stage 1 of this model | own |

Some of these datasets were themselves generated by other language models, so their own terms may apply.
This card is my best effort, not legal advice.

## Author

Trained from scratch as a personal project. Feedback and bug reports are welcome.

If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate