|
Download README.md from SparkCodeAI/Spark-2.0-code-base: direct link, hf CLI and curl.
- Browser
- Download file 3.81 kB
-
https://huggingface.co/SparkCodeAI/Spark-2.0-code-base/resolve/main/README.md
- Command line
-
hf download hf://SparkCodeAI/Spark-2.0-code-base/README.md
-
curl -L -o README.md https://huggingface.co/SparkCodeAI/Spark-2.0-code-base/resolve/main/README.md
3.81 kB
| license: cc-by-nc-4.0 | |
| language: | |
| - en | |
| pipeline_tag: text-generation | |
| tags: | |
| - code | |
| - python | |
| - from-scratch | |
| - gated-deltanet | |
| - hybrid | |
| - base-model | |
| <p align="center"><img src="banner.png" alt="Spark 2.0-code-base" width="100%"></p> | |
| # Spark 2.0-code-base (pretrained base model) | |
| The **base model** behind Spark 2.0-code: **322M parameters**, trained completely from scratch on Python code by one | |
| person on a single RTX 3090. It was pretrained on about **18.4B tokens** and has **no instruction tuning**. | |
| **This is not a chat model.** It does not follow tasks. You give it the **beginning of Python code** (a function | |
| signature with a docstring, an import block, a class header) and it **continues** it. For instructions in plain | |
| English use **Spark 2.0-code** or **Spark 2.0-code (stage 1)**. | |
| Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions. | |
| > **Honest summary.** It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after | |
| > finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the | |
| > next top-level `def`/`class`), or use a stop condition. | |
| ## Quick start | |
| Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from | |
| [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which run on GPU only. | |
| ```bash | |
| pip install torch flash-linear-attention triton tokenizers safetensors | |
| python generate.py 'def is_prime(n):\n # Return True if n is prime.\n' | |
| ``` | |
| `generate.py` prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text | |
| you give is simply continued. | |
| ## Model | |
| | | | | |
| |---|---| | |
| | Parameters | 322M (embeddings tied) | | |
| | Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 | | |
| | Context | 4096 tokens | | |
| | Vocabulary | 16 384 (own BPE tokenizer, Python-focused) | | |
| | Pretraining | about 18.4B tokens of Python (35 070 steps), Muon + AdamW | | |
| | Validation loss | 0.50 | | |
| | Weights | bf16, `model.safetensors`, 0.65 GB | | |
| ## Results | |
| My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does. | |
| | | **Base (this model)** | SFT stage 1 | SFT stage 2 | | |
| |---|---|---|---| | |
| | HumanEval pass@1 (164 tasks) | **34.8%** | 52.4% | 47.6% | | |
| HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution | |
| were removed. | |
| ## Limitations | |
| - Not a chat or instruction model; it continues text. | |
| - Does not stop after a function; trim the output. | |
| - It sometimes invents library functions and misses imports. | |
| - It only knows Python. | |
| - A language model: it can write insecure or wrong code. Review before running. | |
| ## Training data and license | |
| **License: CC BY-NC 4.0** (non-commercial, with attribution). The pretraining data includes | |
| **KodCode (CC BY-NC 4.0)**, so I do not offer the model for commercial use. For a commercial product, contact the | |
| author first. | |
| Licenses are as shown on the Hugging Face dataset pages when I checked. | |
| | Source | License | | |
| |---|---| | |
| | swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 | | |
| | OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 | | |
| | KodCode-V1 | CC BY-NC 4.0 | | |
| | opc-annealing-corpus (synthetic Python snippets) | ODC-By | | |
| Some of these datasets were themselves generated by other language models, so their own terms may apply. | |
| This card is my best effort, not legal advice. | |
| ## Author | |
| Trained from scratch as a personal project. Feedback and bug reports are welcome. | |
| If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate | |