SparkCodeAI's picture
Add donation link
9139847 verified
|
Raw History Blame Contribute Delete
4.98 kB
metadata
license: cc-by-nc-4.0
language:
  - en
pipeline_tag: text-generation
tags:
  - code
  - python
  - from-scratch
  - gated-deltanet
  - hybrid

Spark 2.0-code (stage 1)

Spark 2.0-code (SFT stage 1: functions)

A small language model (322M parameters) that writes Python code only, trained completely from scratch by one person on a single RTX 3090.

This is the stage 1 version: the pretrained base after the first instruction tuning, focused on single functions. It has the best HumanEval score of the three Spark releases. The next version, Spark 2.0-code (stage 2), was trained further on complete programs and writes console programs much better, but scores about 5 points lower on HumanEval.

Which one should I use? Functions and algorithm tasks: this model. Complete programs with input and menus: Spark 2.0-code (stage 2). Continuing existing code without chat: Spark 2.0-code-base.

Honest summary. Good at functions. When asked for a complete program (with input(), a menu, a game loop) it often returns only helper functions without the part that runs them. It is weak at long programs and games. Always run and check what it writes.

Quick start

Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from flash-linear-attention, which run on GPU only.

pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py "Write a Python function that checks whether a number is prime."

Prompt format:

### User
<task in English>

### Assistant

The answer is plain code followed by <|endoftext|>. Multi-turn works in the same format. Generation is greedy. Ask "What model are you?" and it answers that it is Spark 2.0-code.

Model

Parameters 322M (embeddings tied)
Architecture hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024
Context 4096 tokens
Vocabulary 16 384 (own BPE tokenizer, Python-focused)
Pretraining about 18.4B tokens of Python, Muon + AdamW
SFT stage 1 about 470 000 examples (mostly single functions, some multi-turn dialogues), 2 epochs, loss only on answers
Weights bf16, model.safetensors, 0.65 GB

Results

All numbers come from my own harness in this project, not from an external leaderboard. Greedy decoding, code is run in a sandbox.

Pretrained base SFT stage 1 (this model) SFT stage 2
HumanEval pass@1 (164 tasks) 34.8% 52.4% 47.6%

My own small checks (few attempts each, treat as rough):

Check Stage 1 (this model) Stage 2
10 short function tasks, greedy 10/10 10/10
"Guess the number" game works (10 tries) 10/10 8/10
Console calculator with all 5 operations working (20 tries) 1/20 12/20
Snake in pygame survives 300 frames (8 tries) 0/8 1/8

HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution were removed.

Limitations

  • Complete programs are its weak spot: it tends to answer with functions and forget to call them, read input or print.
  • Long programs and games (Snake, multi-class apps) usually do not work.
  • It sometimes invents library functions and misses imports.
  • It only knows Python. English prompts work best.
  • A language model: it can write insecure or wrong code. Review before running.

Training data and license

License: CC BY-NC 4.0 (non-commercial, with attribution). The model was trained on data that includes KodCode (CC BY-NC 4.0), so I do not offer it for commercial use. For a commercial product, contact the author first.

Licenses are as shown on the Hugging Face dataset pages when I checked.

Stage Source License
Pretraining swallow-code-v2 (Python from The Stack v2, rewritten and filtered) Apache-2.0
Pretraining OpenCodeInstruct (only solutions that passed all unit tests) CC BY 4.0
Pretraining KodCode-V1 CC BY-NC 4.0
Pretraining opc-annealing-corpus (synthetic Python snippets) ODC-By
SFT stage 1 OpenCodeInstruct, KodCode-V1, OpenCoder opc-sft-stage1/2, Magicoder OSS-Instruct and Evol-Instruct, Code-Feedback, CodeFeedback-Filtered-Instruction, glaive-code-assistant-v3, self-oss-instruct-sc2-exec-filter-50k, LeetCodeDataset, plus a few hundred hand-made "who are you" dialogues MIT, Apache-2.0, CC BY 4.0, ODC-By, CC BY-NC 4.0 (KodCode)

Some of these datasets were themselves generated by other language models, so their own terms may apply. This card is my best effort, not legal advice.

Author

Trained from scratch as a personal project. Feedback and bug reports are welcome.

If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate