File size: 4,980 Bytes
7b4e7cd
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9139847
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-generation
tags:
- code
- python
- from-scratch
- gated-deltanet
- hybrid
---

<p align="center"><img src="banner.png" alt="Spark 2.0-code (stage 1)" width="100%"></p>

# Spark 2.0-code (SFT stage 1: functions)

A small language model (**322M parameters**) that writes **Python code only**, trained completely from scratch
by one person on a single RTX 3090.

This is the **stage 1** version: the pretrained base after the first instruction tuning, focused on **single
functions**. It has the **best HumanEval score** of the three Spark releases. The next version,
**Spark 2.0-code** (stage 2), was trained further on complete programs and writes console programs much better, but
scores about 5 points lower on HumanEval.

> **Which one should I use?** Functions and algorithm tasks: this model. Complete programs with input and menus:
> Spark 2.0-code (stage 2). Continuing existing code without chat: Spark 2.0-code-base.

> **Honest summary.** Good at functions. When asked for a complete program (with `input()`, a menu, a game loop) it
> often returns only helper functions without the part that runs them. It is weak at long programs and games.
> Always run and check what it writes.

## Quick start

Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from
[flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only.

```bash
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py "Write a Python function that checks whether a number is prime."
```

Prompt format:

```
### User
<task in English>

### Assistant
```

The answer is plain code followed by `<|endoftext|>`. Multi-turn works in the same format. Generation is greedy.
Ask "What model are you?" and it answers that it is Spark 2.0-code.

## Model

| | |
|---|---|
| Parameters | 322M (embeddings tied) |
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
| Context | 4096 tokens |
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
| Pretraining | about 18.4B tokens of Python, Muon + AdamW |
| SFT stage 1 | about 470 000 examples (mostly single functions, some multi-turn dialogues), 2 epochs, loss only on answers |
| Weights | bf16, `model.safetensors`, 0.65 GB |

## Results

All numbers come from my own harness in this project, not from an external leaderboard.
Greedy decoding, code is run in a sandbox.

| | Pretrained base | **SFT stage 1 (this model)** | SFT stage 2 |
|---|---|---|---|
| HumanEval pass@1 (164 tasks) | 34.8% | **52.4%** | 47.6% |

My own small checks (few attempts each, treat as rough):

| Check | Stage 1 (this model) | Stage 2 |
|---|---|---|
| 10 short function tasks, greedy | 10/10 | 10/10 |
| "Guess the number" game works (10 tries) | 10/10 | 8/10 |
| Console calculator with all 5 operations working (20 tries) | 1/20 | 12/20 |
| Snake in pygame survives 300 frames (8 tries) | 0/8 | 1/8 |

HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution
were removed.

## Limitations

- Complete programs are its weak spot: it tends to answer with functions and forget to call them, read input or print.
- Long programs and games (Snake, multi-class apps) usually do not work.
- It sometimes invents library functions and misses imports.
- It only knows Python. English prompts work best.
- A language model: it can write insecure or wrong code. Review before running.

## Training data and license

**License: CC BY-NC 4.0** (non-commercial, with attribution). The model was trained on data that includes
**KodCode (CC BY-NC 4.0)**, so I do not offer it for commercial use. For a commercial product, contact the author first.

Licenses are as shown on the Hugging Face dataset pages when I checked.

| Stage | Source | License |
|---|---|---|
| Pretraining | swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
| Pretraining | OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
| Pretraining | KodCode-V1 | CC BY-NC 4.0 |
| Pretraining | opc-annealing-corpus (synthetic Python snippets) | ODC-By |
| SFT stage 1 | OpenCodeInstruct, KodCode-V1, OpenCoder opc-sft-stage1/2, Magicoder OSS-Instruct and Evol-Instruct, Code-Feedback, CodeFeedback-Filtered-Instruction, glaive-code-assistant-v3, self-oss-instruct-sc2-exec-filter-50k, LeetCodeDataset, plus a few hundred hand-made "who are you" dialogues | MIT, Apache-2.0, CC BY 4.0, ODC-By, CC BY-NC 4.0 (KodCode) |

Some of these datasets were themselves generated by other language models, so their own terms may apply.
This card is my best effort, not legal advice.

## Author

Trained from scratch as a personal project. Feedback and bug reports are welcome.

If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate