Spark-2.0-code-base / README.md
SparkCodeAI's picture
Add donation link
42471e3 verified
|
Raw History Blame Contribute Delete
3.81 kB
---
license: cc-by-nc-4.0
language:
- en
pipeline_tag: text-generation
tags:
- code
- python
- from-scratch
- gated-deltanet
- hybrid
- base-model
---
<p align="center"><img src="banner.png" alt="Spark 2.0-code-base" width="100%"></p>
# Spark 2.0-code-base (pretrained base model)
The **base model** behind Spark 2.0-code: **322M parameters**, trained completely from scratch on Python code by one
person on a single RTX 3090. It was pretrained on about **18.4B tokens** and has **no instruction tuning**.
**This is not a chat model.** It does not follow tasks. You give it the **beginning of Python code** (a function
signature with a docstring, an import block, a class header) and it **continues** it. For instructions in plain
English use **Spark 2.0-code** or **Spark 2.0-code (stage 1)**.
Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions.
> **Honest summary.** It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after
> finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the
> next top-level `def`/`class`), or use a stop condition.
## Quick start
Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from
[flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only.
```bash
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py 'def is_prime(n):\n # Return True if n is prime.\n'
```
`generate.py` prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text
you give is simply continued.
## Model
| | |
|---|---|
| Parameters | 322M (embeddings tied) |
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
| Context | 4096 tokens |
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
| Pretraining | about 18.4B tokens of Python (35 070 steps), Muon + AdamW |
| Validation loss | 0.50 |
| Weights | bf16, `model.safetensors`, 0.65 GB |
## Results
My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does.
| | **Base (this model)** | SFT stage 1 | SFT stage 2 |
|---|---|---|---|
| HumanEval pass@1 (164 tasks) | **34.8%** | 52.4% | 47.6% |
HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution
were removed.
## Limitations
- Not a chat or instruction model; it continues text.
- Does not stop after a function; trim the output.
- It sometimes invents library functions and misses imports.
- It only knows Python.
- A language model: it can write insecure or wrong code. Review before running.
## Training data and license
**License: CC BY-NC 4.0** (non-commercial, with attribution). The pretraining data includes
**KodCode (CC BY-NC 4.0)**, so I do not offer the model for commercial use. For a commercial product, contact the
author first.
Licenses are as shown on the Hugging Face dataset pages when I checked.
| Source | License |
|---|---|
| swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
| OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
| KodCode-V1 | CC BY-NC 4.0 |
| opc-annealing-corpus (synthetic Python snippets) | ODC-By |
Some of these datasets were themselves generated by other language models, so their own terms may apply.
This card is my best effort, not legal advice.
## Author
Trained from scratch as a personal project. Feedback and bug reports are welcome.
If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate