--- license: cc-by-nc-4.0 language: - en pipeline_tag: text-generation tags: - code - python - from-scratch - gated-deltanet - hybrid - base-model ---

Spark 2.0-code-base

# Spark 2.0-code-base (pretrained base model) The **base model** behind Spark 2.0-code: **322M parameters**, trained completely from scratch on Python code by one person on a single RTX 3090. It was pretrained on about **18.4B tokens** and has **no instruction tuning**. **This is not a chat model.** It does not follow tasks. You give it the **beginning of Python code** (a function signature with a docstring, an import block, a class header) and it **continues** it. For instructions in plain English use **Spark 2.0-code** or **Spark 2.0-code (stage 1)**. Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions. > **Honest summary.** It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after > finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the > next top-level `def`/`class`), or use a stop condition. ## Quick start Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only. ```bash pip install torch flash-linear-attention triton tokenizers safetensors python generate.py 'def is_prime(n):\n # Return True if n is prime.\n' ``` `generate.py` prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text you give is simply continued. ## Model | | | |---|---| | Parameters | 322M (embeddings tied) | | Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 | | Context | 4096 tokens | | Vocabulary | 16 384 (own BPE tokenizer, Python-focused) | | Pretraining | about 18.4B tokens of Python (35 070 steps), Muon + AdamW | | Validation loss | 0.50 | | Weights | bf16, `model.safetensors`, 0.65 GB | ## Results My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does. | | **Base (this model)** | SFT stage 1 | SFT stage 2 | |---|---|---|---| | HumanEval pass@1 (164 tasks) | **34.8%** | 52.4% | 47.6% | HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution were removed. ## Limitations - Not a chat or instruction model; it continues text. - Does not stop after a function; trim the output. - It sometimes invents library functions and misses imports. - It only knows Python. - A language model: it can write insecure or wrong code. Review before running. ## Training data and license **License: CC BY-NC 4.0** (non-commercial, with attribution). The pretraining data includes **KodCode (CC BY-NC 4.0)**, so I do not offer the model for commercial use. For a commercial product, contact the author first. Licenses are as shown on the Hugging Face dataset pages when I checked. | Source | License | |---|---| | swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 | | OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 | | KodCode-V1 | CC BY-NC 4.0 | | opc-annealing-corpus (synthetic Python snippets) | ODC-By | Some of these datasets were themselves generated by other language models, so their own terms may apply. This card is my best effort, not legal advice. ## Author Trained from scratch as a personal project. Feedback and bug reports are welcome. If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate