|
Download README.md from SparkCodeAI/Spark-2.0-code: direct link, hf CLI and curl.
- Browser
- Download file 5.91 kB
-
https://huggingface.co/SparkCodeAI/Spark-2.0-code/resolve/main/README.md
- Command line
-
hf download hf://SparkCodeAI/Spark-2.0-code/README.md
-
curl -L -o README.md https://huggingface.co/SparkCodeAI/Spark-2.0-code/resolve/main/README.md
5.91 kB
| license: cc-by-nc-4.0 | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| tags: | |
| - code | |
| - python | |
| - from-scratch | |
| - gated-deltanet | |
| - hybrid | |
| <p align="center"><img src="banner.png" alt="Spark 2.0-code" width="100%"></p> | |
| # Spark 2.0-code | |
| A small language model (**322M parameters**) that writes **Python code only**. | |
| It was trained completely from scratch (pretraining, then two SFT stages) by one person on a single RTX 3090. | |
| You describe a task in English, it answers with Python code. | |
| > **Three versions.** *Spark 2.0-code* (this one) writes complete programs best. *Spark 2.0-code (stage 1)* has the | |
| > highest HumanEval (52.4%) and is best for single functions. *Spark 2.0-code-base* is the pretrained base model that | |
| > only continues code. | |
| > **Honest summary.** Spark is good at functions, short scripts and console programs with a menu or a loop. | |
| > It is weak at long programs where many parts must fit together (games like Snake, big multi-class apps). | |
| > It often writes correct-looking code that fails on one detail. Always run and check what it writes. | |
| ## Quick start | |
| Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from | |
| [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only. | |
| ```bash | |
| pip install torch flash-linear-attention triton tokenizers safetensors | |
| python generate.py "Write a Python function that checks whether a number is prime." | |
| ``` | |
| `generate.py` loads `model.safetensors` and uses the code in `spark/`. Generation is greedy (temperature 0). | |
| Prompt format (the same as in training): | |
| ``` | |
| ### User | |
| <task in English> | |
| ### Assistant | |
| ``` | |
| The answer is plain code, followed by `<|endoftext|>`. Multi-turn works in the same format: after the answer add a new | |
| `### User` block, for example with an error message ("The code above fails with this error: ... Fix the code."). | |
| Ask "What model are you?" and it answers that it is Spark 2.0-code. | |
| ## Model | |
| | | | | |
| |---|---| | |
| | Parameters | 322M (embeddings tied) | | |
| | Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 | | |
| | Context | 4096 tokens | | |
| | Vocabulary | 16 384 (own BPE tokenizer, Python-focused) | | |
| | Pretraining | about 18.4B tokens of Python, Muon + AdamW | | |
| | SFT | stage 1 (functions) and stage 2 (complete programs, fixing code from an error message) | | |
| | Weights | bf16, `model.safetensors`, 0.65 GB | | |
| ## Results | |
| All numbers come from my own harness in this project, not from an external leaderboard. | |
| Greedy decoding, code is run in a sandbox. | |
| | | Pretrained base | SFT stage 1 | **SFT stage 2 (this model)** | | |
| |---|---|---|---| | |
| | HumanEval pass@1 (164 tasks) | 34.8% | 52.4% | **47.6%** | | |
| HumanEval tests only single functions. Stage 2 moved the model towards complete programs, and its HumanEval score is | |
| about 5 points lower than stage 1 (8 tasks out of 164; part of that can be noise). | |
| My own small checks of the same model (few attempts each, so treat them as rough): | |
| | Check | Stage 1 | Stage 2 | | |
| |---|---|---| | |
| | Console calculator with all 5 operations working (20 tries) | 1/20 | **12/20** | | |
| | "Guess the number" game works (10 tries) | 10/10 | 8/10 | | |
| | 10 short function tasks, greedy | 10/10 | 10/10 | | |
| | Snake in pygame survives 300 frames (8 tries) | 0/8 | 1/8 | | |
| HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution | |
| were removed. | |
| ## Limitations | |
| - Long programs (100-200 lines) usually contain several small bugs: a wrong argument count, a string used where an | |
| object is needed, a forgotten import, a method that is never called. Examples I tested: a library manager, a school | |
| journal, a pygame arcade game. The structure was fine, but most menu commands crashed on first run. | |
| - It cannot reliably do geometry (positions and sizes of shapes) or game logic where many things depend on each other. | |
| - It sometimes invents library functions and misses imports. | |
| - It only knows Python. English prompts work best. | |
| - A language model: it can write insecure or wrong code. Review before running. | |
| ## Training data and license | |
| **License: CC BY-NC 4.0** (non-commercial, with attribution). Reason: the model was trained on data that includes | |
| **KodCode (CC BY-NC 4.0)**, so I do not offer it for commercial use. If you want to use Spark in a commercial product, | |
| contact the author first. | |
| Data used (all text only Python-related; licenses as shown on their Hugging Face pages when I checked): | |
| | Stage | Source | License | | |
| |---|---|---| | |
| | Pretraining | swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 | | |
| | Pretraining | OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 | | |
| | Pretraining | KodCode-V1 | CC BY-NC 4.0 | | |
| | Pretraining | opc-annealing-corpus (synthetic Python snippets) | ODC-By | | |
| | SFT | OpenCodeInstruct, KodCode-V1, OpenCoder opc-sft-stage1/2, Magicoder OSS-Instruct and Evol-Instruct, Code-Feedback, CodeFeedback-Filtered-Instruction, glaive-code-assistant-v3, self-oss-instruct-sc2-exec-filter-50k, LeetCodeDataset | MIT, Apache-2.0, CC BY 4.0, ODC-By, CC BY-NC 4.0 (KodCode) | | |
| | SFT stage 2 | code_contests (Python 3 solutions, checked by running the tests) | CC BY 4.0 | | |
| | SFT stage 2 | about 3 800 complete programs written by Qwen3.8-Flash-Next (local, quantized) and checked by running them in a sandbox | Qwen model license not verified by me | | |
| | SFT stage 2 | about 1 200 "fix it from the error message" dialogues made from the real mistakes of stage 1 of this model | own | | |
| Some of these datasets were themselves generated by other language models, so their own terms may apply. | |
| This card is my best effort, not legal advice. | |
| ## Author | |
| Trained from scratch as a personal project. Feedback and bug reports are welcome. | |
| If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate | |