| --- |
| license: apache-2.0 |
| library_name: vllm |
| inference: false |
| base_model: CohereLabs/North-Mini-Code-1.0-w4a16 |
| tags: |
| - north-mini-code |
| - speculative-decoding |
| - draft-model |
| - dspark |
| - code |
| --- |
| |
| # North Mini Code 1.0 DSpark |
|
|
| This is a research-preview **auxiliary draft checkpoint** for speculative decoding with `CohereLabs/North-Mini-Code-1.0-w4a16`. |
| It is **not** a standalone language model and it cannot generate text without the North verifier/target model. |
| It will likely also work with `CohereLabs/North-Mini-Code-1.0-fp8` with slightly lower acceptance rates. |
|
|
| ## Training |
|
|
| This DSpark draft model was trained on 10,000 Magicoder prompts with on-policy supervision from North Mini Code. |
|
|
| ## Important Notes |
|
|
| - not a base model |
| - not a fine-tuning target for standalone inference |
| - not a correctness guarantee |
| - not byte-parity with target-only greedy decoding |
|
|
| ## Architecture |
|
|
| - speculator type: `DSparkDraftModel` |
| - draft layers: 5 Qwen3 sliding-window layers |
| - hidden size: 2048 |
| - attention heads: 32 |
| - key/value heads: 4 |
| - head dim: 128 |
| - MLP intermediate size: 768 |
| - sliding window: 2048 |
| - draft vocabulary: 32,000 |
| - target vocabulary: 262,144 |
| - block size: 8 |
| - `sample_from_anchor`: `true` |
| - Markov head: vanilla rank 256 |
| - confidence head: trained and included, but skipped and unused by the validated North vLLM runtime |
| - auxiliary target hidden-state taps: layers `[2, 24, 46]` |
| - tensors: 66 |
| - serialized tensor elements: 808,577,793 |
| - BF16 parameters: 808,283,649 |
| - selected weight hash: `64fd7d60c1c5a91d030dae0b7300bf8593b536b0a5569f53db7209e4a5f5b7f6` |
|
|
| Largest tensors include the target embedding matrix, the Markov head matrices, and `lm_head.weight`. |
| The tensor inventory is recorded in `tensor_info.json`. |
|
|
| ## Training data and supervision |
|
|
| - prompt dataset: `ise-uiuc/Magicoder-Evol-Instruct-110K` |
| - license: Apache-2.0 |
| - training rows: 10,000 |
| - selected rows: 0-499, 600-5099, and 6100-11099 |
| - locked holdout: rows 500-599, never trained |
| - supervision: on-policy responses and target hidden states generated once from `CohereLabs/North-Mini-Code-1.0-w4a16` |
| - response cap: 2,048 tokens |
| - target layers used for supervision: `[2, 24, 46]` |
| - later acceptance-mined continuation data: not included in this release checkpoint |
|
|
| Training metadata is recorded in `training_metadata.json`. |
|
|
| ## Matched GB10 benchmarks |
|
|
| These numbers come from the same matched serving contract on a GB10 (DGX Spark) machine running Marlin (which is suboptimal but predictable). |
| Compare the rows **within the same host and contract only**. |
| These absolute token/s values are not representative of maximum performance, simply a demonstration that the draft model works. :) |
|
|
| | Contract | Target-only | DFlash K3 | DFlash / target | DSpark K4 | DSpark / target | |
| | --- | ---: | ---: | ---: | ---: | ---: | |
| | C1 | 39.9129 tok/s | 75.5106 tok/s | 1.8919x | 79.7341 tok/s | 1.9977x | |
| | C2 | 78.9486 tok/s | 121.5232 tok/s | 1.5393x | 127.6174 tok/s | 1.6165x | |
|
|
| | Model | C1 EAL | C1 draft acceptance | C2 EAL | C2 draft acceptance | |
| | --- | ---: | ---: | ---: | ---: | |
| | DFlash K3 | 2.2947 | 43.16% | 2.2856 | 42.85% | |
| | DSpark K4 | 2.4800 | 37.00% | 2.4779 | 36.95% | |
|
|
| Expected acceptance length (EAL) is the mean emitted tokens per speculative verification step, including the target bonus token. |
|
|
| ## Additional result: DSpark depth and K4-vs-DFlash |
|
|
| Matched DSpark runs showed better throughput than DFlash on the same contract. This serving margin is not a correctness claim. |
|
|
| - C1: DSpark vs DFlash = 1.0559x |
| - C2: DSpark vs DFlash = 1.0501x |
|
|
| Depth diagnostics on the 10K checkpoint: |
|
|
| - EAL 0-512: 2.4064 |
| - EAL 512-1024: 2.7832 |
| - EAL 1024-2048: 2.7322 |
|
|
| Those continuation segments restart from full prefill and are diagnostic only. |
|
|
| ## Default proposal depth |
|
|
| Default proposal depth is **K4**. |
| That is the selected deployment setting in `config.json` and the configuration used for the release evidence. |
| The validated K5 setting used the same weights but did not beat K4 end-to-end throughput, so K4 is where you'll see the best speed. |
| K5 only changes the proposal-depth setting; it is not a separate checkpoint. |
|
|
| ## Runtime boundary |
|
|
| This checkpoint is served as the **top-level model** because `config.json` already embeds the verifier reference. |
| Use it only with the companion North runtime, published at https://github.com/sdougbrown/north-mini-code-draft-runtime. Stock vLLM 0.25.1 is insufficient. The companion is a thin overlay on official vLLM 0.27.1, where DSpark support is present, carrying PRs #49819 and #50937 (Cohere2MoE auxiliary hidden states, and an expert-bias loading fix), published as `ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937`. |
|
|
| ```bash |
| export DRAFT_MODEL="${DRAFT_MODEL:?set this to the downloaded DSpark directory or Hub model ID}" |
| export VLLM_USE_V2_MODEL_RUNNER=1 |
| vllm serve "${DRAFT_MODEL}" \ |
| --tensor-parallel-size 1 \ |
| --max-model-len 320000 \ |
| --tool-call-parser cohere_command4 \ |
| --tokenizer-mode cohere \ |
| --cohere-format cmd4 \ |
| --reasoning-config '{"reasoning_start_str":"<|START_THINKING|>","reasoning_end_str":"<|END_THINKING|>"}' \ |
| --enable-auto-tool-choice |
| ``` |
|
|
| ## Limitations |
|
|
| - 32K draft vocabulary, not the 262,144-token target vocabulary |
| - target-dependent: cannot run standalone |
| - quantized greedy outputs are not guaranteed to match target-only greedy byte-for-byte |
| - throughput and acceptance do not establish correctness or response quality |
| - absolute throughput is host-specific |
| - this checkpoint has not been served on an NVIDIA RTX 3090; compatibility and performance on that GPU are unknown |
|
|
| ## References |
|
|
| - North base model: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-w4a16 |
| - Magicoder dataset: https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K |
| - Companion North runtime: https://github.com/sdougbrown/north-mini-code-draft-runtime |
| - Companion image: `ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937` |
| - Official Eagle card: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-eagle |
| - Speculators: pinned commit `f7ec34182826bc89934ce710283421778022b74d`, version `0.6.0` |
|
|
| ## Files in this repo |
|
|
| - `README.md` |
| - `config.json` |
| - `config.py` |
| - `model.safetensors` |
| - `tensor_info.json` |
| - `training_metadata.json` |
| - `SHA256SUMS` |
| - `LICENSE` |
| - `.gitattributes` |
|
|
