File size: 3,806 Bytes
8620460
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
42471e3
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-generation
tags:
- code
- python
- from-scratch
- gated-deltanet
- hybrid
- base-model
---

<p align="center"><img src="banner.png" alt="Spark 2.0-code-base" width="100%"></p>

# Spark 2.0-code-base (pretrained base model)

The **base model** behind Spark 2.0-code: **322M parameters**, trained completely from scratch on Python code by one
person on a single RTX 3090. It was pretrained on about **18.4B tokens** and has **no instruction tuning**.

**This is not a chat model.** It does not follow tasks. You give it the **beginning of Python code** (a function
signature with a docstring, an import block, a class header) and it **continues** it. For instructions in plain
English use **Spark 2.0-code** or **Spark 2.0-code (stage 1)**.

Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions.

> **Honest summary.** It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after
> finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the
> next top-level `def`/`class`), or use a stop condition.

## Quick start

Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from
[flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only.

```bash
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py 'def is_prime(n):\n    # Return True if n is prime.\n'
```

`generate.py` prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text
you give is simply continued.

## Model

| | |
|---|---|
| Parameters | 322M (embeddings tied) |
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
| Context | 4096 tokens |
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
| Pretraining | about 18.4B tokens of Python (35 070 steps), Muon + AdamW |
| Validation loss | 0.50 |
| Weights | bf16, `model.safetensors`, 0.65 GB |

## Results

My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does.

| | **Base (this model)** | SFT stage 1 | SFT stage 2 |
|---|---|---|---|
| HumanEval pass@1 (164 tasks) | **34.8%** | 52.4% | 47.6% |

HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution
were removed.

## Limitations

- Not a chat or instruction model; it continues text.
- Does not stop after a function; trim the output.
- It sometimes invents library functions and misses imports.
- It only knows Python.
- A language model: it can write insecure or wrong code. Review before running.

## Training data and license

**License: CC BY-NC 4.0** (non-commercial, with attribution). The pretraining data includes
**KodCode (CC BY-NC 4.0)**, so I do not offer the model for commercial use. For a commercial product, contact the
author first.

Licenses are as shown on the Hugging Face dataset pages when I checked.

| Source | License |
|---|---|
| swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
| OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
| KodCode-V1 | CC BY-NC 4.0 |
| opc-annealing-corpus (synthetic Python snippets) | ODC-By |

Some of these datasets were themselves generated by other language models, so their own terms may apply.
This card is my best effort, not legal advice.

## Author

Trained from scratch as a personal project. Feedback and bug reports are welcome.

If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate