README.md

---
license: apache-2.0
library_name: vllm
inference: false
base_model: CohereLabs/North-Mini-Code-1.0-w4a16
tags:
- north-mini-code
- speculative-decoding
- draft-model
- dspark
- code
---

# North Mini Code 1.0 DSpark

This is a research-preview **auxiliary draft checkpoint** for speculative decoding with `CohereLabs/North-Mini-Code-1.0-w4a16`.
It is **not** a standalone language model and it cannot generate text without the North verifier/target model.
It will likely also work with `CohereLabs/North-Mini-Code-1.0-fp8` with slightly lower acceptance rates.

## Training

This DSpark draft model was trained on 10,000 Magicoder prompts with on-policy supervision from North Mini Code.

## Important Notes

- not a base model
- not a fine-tuning target for standalone inference
- not a correctness guarantee
- not byte-parity with target-only greedy decoding

## Architecture

- speculator type: `DSparkDraftModel`
- draft layers: 5 Qwen3 sliding-window layers
- hidden size: 2048
- attention heads: 32
- key/value heads: 4
- head dim: 128
- MLP intermediate size: 768
- sliding window: 2048
- draft vocabulary: 32,000
- target vocabulary: 262,144
- block size: 8
- `sample_from_anchor`: `true`
- Markov head: vanilla rank 256
- confidence head: trained and included, but skipped and unused by the validated North vLLM runtime
- auxiliary target hidden-state taps: layers `[2, 24, 46]`
- tensors: 66
- serialized tensor elements: 808,577,793
- BF16 parameters: 808,283,649
- selected weight hash: `64fd7d60c1c5a91d030dae0b7300bf8593b536b0a5569f53db7209e4a5f5b7f6`

Largest tensors include the target embedding matrix, the Markov head matrices, and `lm_head.weight`.
The tensor inventory is recorded in `tensor_info.json`.

## Training data and supervision

- prompt dataset: `ise-uiuc/Magicoder-Evol-Instruct-110K`
- license: Apache-2.0
- training rows: 10,000
- selected rows: 0-499, 600-5099, and 6100-11099
- locked holdout: rows 500-599, never trained
- supervision: on-policy responses and target hidden states generated once from `CohereLabs/North-Mini-Code-1.0-w4a16`
- response cap: 2,048 tokens
- target layers used for supervision: `[2, 24, 46]`
- later acceptance-mined continuation data: not included in this release checkpoint

Training metadata is recorded in `training_metadata.json`.

## Matched GB10 benchmarks

These numbers come from the same matched serving contract on a GB10 (DGX Spark) machine running Marlin (which is suboptimal but predictable).
Compare the rows **within the same host and contract only**.
These absolute token/s values are not representative of maximum performance, simply a demonstration that the draft model works. :)

| Contract | Target-only | DFlash K3 | DFlash / target | DSpark K4 | DSpark / target |
| --- | ---: | ---: | ---: | ---: | ---: |
| C1 | 39.9129 tok/s | 75.5106 tok/s | 1.8919x | 79.7341 tok/s | 1.9977x |
| C2 | 78.9486 tok/s | 121.5232 tok/s | 1.5393x | 127.6174 tok/s | 1.6165x |

| Model | C1 EAL | C1 draft acceptance | C2 EAL | C2 draft acceptance |
| --- | ---: | ---: | ---: | ---: |
| DFlash K3 | 2.2947 | 43.16% | 2.2856 | 42.85% |
| DSpark K4 | 2.4800 | 37.00% | 2.4779 | 36.95% |

Expected acceptance length (EAL) is the mean emitted tokens per speculative verification step, including the target bonus token.

## Additional result: DSpark depth and K4-vs-DFlash

Matched DSpark runs showed better throughput than DFlash on the same contract. This serving margin is not a correctness claim.

- C1: DSpark vs DFlash = 1.0559x
- C2: DSpark vs DFlash = 1.0501x

Depth diagnostics on the 10K checkpoint:

- EAL 0-512: 2.4064
- EAL 512-1024: 2.7832
- EAL 1024-2048: 2.7322

Those continuation segments restart from full prefill and are diagnostic only.

## Default proposal depth

Default proposal depth is **K4**.
That is the selected deployment setting in `config.json` and the configuration used for the release evidence.
The validated K5 setting used the same weights but did not beat K4 end-to-end throughput, so K4 is where you'll see the best speed.
K5 only changes the proposal-depth setting; it is not a separate checkpoint.

## Runtime boundary

This checkpoint is served as the **top-level model** because `config.json` already embeds the verifier reference.
Use it only with the companion North runtime, published at https://github.com/sdougbrown/north-mini-code-draft-runtime. Stock vLLM 0.25.1 is insufficient. The companion is a thin overlay on official vLLM 0.27.1, where DSpark support is present, carrying PRs #49819 and #50937 (Cohere2MoE auxiliary hidden states, and an expert-bias loading fix), published as `ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937`.

```bash
export DRAFT_MODEL="${DRAFT_MODEL:?set this to the downloaded DSpark directory or Hub model ID}"
export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve "${DRAFT_MODEL}" \
  --tensor-parallel-size 1 \
  --max-model-len 320000 \
  --tool-call-parser cohere_command4 \
  --tokenizer-mode cohere \
  --cohere-format cmd4 \
  --reasoning-config '{"reasoning_start_str":"<|START_THINKING|>","reasoning_end_str":"<|END_THINKING|>"}' \
  --enable-auto-tool-choice
```

## Limitations

- 32K draft vocabulary, not the 262,144-token target vocabulary
- target-dependent: cannot run standalone
- quantized greedy outputs are not guaranteed to match target-only greedy byte-for-byte
- throughput and acceptance do not establish correctness or response quality
- absolute throughput is host-specific
- this checkpoint has not been served on an NVIDIA RTX 3090; compatibility and performance on that GPU are unknown

## References

- North base model: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-w4a16
- Magicoder dataset: https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K
- Companion North runtime: https://github.com/sdougbrown/north-mini-code-draft-runtime
- Companion image: `ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937`
- Official Eagle card: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-eagle
- Speculators: pinned commit `f7ec34182826bc89934ce710283421778022b74d`, version `0.6.0`

## Files in this repo

- `README.md`
- `config.json`
- `config.py`
- `model.safetensors`
- `tensor_info.json`
- `training_metadata.json`
- `SHA256SUMS`
- `LICENSE`
- `.gitattributes`