README.md

---
license: apache-2.0
language:
  - code
tags:
  - code
  - fill-in-the-middle
  - multi-token-prediction
  - speculative-decoding
  - mlx
  - apple-silicon
library_name: transformers
pipeline_tag: text-generation
---

# Wisp Coder 110M

A small code model built for one job: completing code at the cursor. It targets
editor inline suggestions, but no production-runtime latency claim is made.

Two things make it unusual, and they are the reason it exists rather than a
feature list.

**It does fill in the middle natively.** Each tokenized source document was split
into chunks of at most 1024 tokens, and 70 percent of those chunks were
independently transformed so the model predicts a middle span given both the
prefix and the suffix. This is not a per-document or per-window rate. Cursor
completion is the primary format, not an inference-time prompt trick added to an
ordinary left-to-right checkpoint.

**It has multi-token-prediction heads trained in from step zero.** A shared MTP
module, applied recursively in the Qwen3-Next style, predicts two or more tokens
ahead from the trunk's own hidden state. The LM head is shared between trunk and
MTP module, which ties the two output distributions to a common projection. That
is a useful architectural choice, not a guarantee that draft and target
distributions agree; how close they actually are is what acceptance measures.

**What is and is not claimed here.** Small models with MTP drafters exist: Google
publishes `gemma-4-E2B-it-assistant` and related MTP assistant checkpoints. The
artifact contribution is the *combination* at this scale, a small code model
trained from step zero with both fill-in-the-middle on most tokenized chunks and a
recursively shared MTP module, plus a paired measurement of how suffix information
changes draft acceptance. A targeted search found no exact prior measurement, but
that is not proof of absence. No exclusivity or first-of-its-kind claim is made.

## What it is not

It is not a chat model, an instruction-following model, or an agent. It completes
text. Asking it to refactor a module produces nonsense.

It is also not competitive with Qwen2.5-Coder on raw completion quality, and the
arithmetic says it cannot be: Qwen2.5-Coder-0.5B saw roughly 5.5 trillion tokens
and Wisp saw 5 billion, three orders of magnitude fewer. If you want the best
completions available at small scale, use Qwen. Use Wisp to study FIM plus native
speculative drafting on Apple Silicon, or how those two interact.

## Architecture

| | |
|---|---|
| Parameters | 108.2M total, 100.7M trunk |
| Layers | 12, d_model 768, 12 heads / 4 KV heads, SwiGLU 2048 |
| Context | 2048, RoPE theta 100k |
| Vocab | 32,768 byte-level BPE, digits split, FIM sentinels |
| MTP | 1 shared module (1 transformer block), trained to depth 2, recursive at inference |
| Precision | bfloat16 weights, trained with float32 master weights |

The trunk is an ordinary Llama decoder. It loads in `transformers` and `mlx_lm`
with no custom modelling code and no `trust_remote_code`. The MTP module ships
alongside as `mtp.safetensors` and is ignorable by runtimes that cannot use it.
Generic Llama runtimes do not consume the sidecar automatically. The package
therefore also ships `wisp_mtp_model.py` and `wisp_mtp_reference.py`, a
manifest-bound MLX correctness decoder that requires the exact sidecar schema,
verifies every declared package hash, exposes the resolved runtime contract and
executed route, and fails unless its MTP path reproduces target greedy tokens.
It recomputes full prefixes and is not a production speed path.

The export was verified against `transformers` numerically, not assumed: relative
logit delta 2.16e-3 and argmax agreement 1.0000 against the MLX original on the
same tokens, which pins the RoPE convention, the RMSNorm epsilon, and the grouped
query head order.

## Training data

The configured target mixture was 92 percent code from
`bigcode/starcoderdata` and 8 percent `HuggingFaceFW/fineweb-edu`. This is a
source percentage, not a permissive-license percentage. StarCoderData declares
`license: other`, and its original repository terms and relevant attribution
clauses remain applicable. FineWeb-Edu is ODC-By 1.0 and remains subject to
Common Crawl terms. Run 1 retained no row-level source manifest, so no per-file
licensing or attribution guarantee is made.

Every run 1 document received structural size, line-length, and character
distribution filters. Because the loader expected `path` instead of
StarCoderData's `max_stars_repo_path`, Python AST and JSON extension filtering
did not activate for run 1. The schema-aware repair applies only to future
source streaming, not to these weights. Corpus content was never executed.
Full details and the machine-checked disclosure are in `DATA.md` and
`config/training_data_receipt.json`.

The 32K tokenizer was trained on 400,000 documents sampled round-robin across
the eleven source entries, not according to the later token-budget weights.
Current shard bytes are fully attested after the build, but the receipt is not a
raw-row manifest and cannot prove exact original example boundaries.

The Apache 2.0 metadata describes the released Wisp artifact. It does not
override training-source terms or licenses applicable to generated code.

## Usage

Plain completion:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("philipjohnbasile/wisp-coder-110m")
tok = AutoTokenizer.from_pretrained("philipjohnbasile/wisp-coder-110m")
print(tok.decode(model.generate(**tok("def quicksort(arr):", return_tensors="pt"),
                                max_new_tokens=64)[0]))
```

Fill in the middle. The model was trained with both orderings, evenly split:

```python
# PSM: prefix, suffix, then generate the middle
prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"
```

Exercise the native sidecar explicitly. This is a correctness check, not a
latency benchmark:

```python
import os
import sys
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

package = snapshot_download(
    "philipjohnbasile/wisp-coder-110m",
    local_dir=os.path.abspath("wisp-coder-110m-package"),
)
sys.path.insert(0, package)
from wisp_mtp_reference import WispMTPReferenceRuntime

tok = Tokenizer.from_file(os.path.join(package, "tokenizer.json"))
ids = tok.encode(prompt, add_special_tokens=False).ids
runtime = WispMTPReferenceRuntime.load(package)
result = runtime.verify_greedy_parity(ids, max_new_tokens=16, depth=2)
print(result["mtp_route"])
```

For evaluation, the trunk can be configured as Continue.dev's autocomplete model
while leaving a larger model as the chat and agent backend. Target-runtime batch-1
latency and editor user experience have not yet been measured, so this is not a
sub-100 ms claim.

## Evaluation

### Registered final results

| Measurement | Result |
|---|---|
| Final validation main NLL | 1.3873 [1.3428, 1.4335], perplexity 4.00 |
| Validation MTP depth 1 NLL | 1.5727 [1.5236, 1.6234] |
| Validation MTP depth 2 NLL | 1.6513 [1.6013, 1.7028] |
| FIM / shuffled-suffix acceptance, depth 2 | 1.0268 [1.0209, 1.0331], POSITIVE, 200 documents |
| Trained minus initialized acceptance-ratio lift | +0.0268 [+0.0210, +0.0331], CLEARS_CONTROL |
| Acceptance interpretation | POSITIVE_AND_CLEARS_CONTROL |
| FIM-training effect on shuffled-FIM minus L2R acceptance | +0.0814 [+0.0747, +0.0883], POSITIVE, 200 documents |
| FIM-training effect on true-suffix minus shuffled-suffix acceptance | +0.0049 [-0.0026, +0.0122], NULL: the interval includes zero |
| Adaptive minus fixed accepted drafts per verification | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents |
| Adaptive minus fixed output tokens per target forward | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents |
| Adaptive minus fixed drafts issued per output token | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents (issuance proxy) |
| Adaptive minus fixed draft recursions per output token | +0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents |
| Rollout policies selected on calibration | adaptive_h0.7 versus fixed_d4 |
| Branch-local greedy replay | Exact argmax for all 38400 emitted tokens across 600 scored policy-document rollouts |
| Cross-policy output identity | Identical realized output branches for all 100 calibration/test documents across compared policies |

Validation intervals measure Monte Carlo uncertainty from the frozen random-window sampler. Acceptance and rollout intervals resample paired target documents. They do not measure training-run or model uncertainty. Null and negative outcomes are retained rather than filtered from the release.

Rollout endpoint scope: The primary endpoint measures accepted drafts per verification. It does not establish verification-width cost or deployment latency. Draft recursions per output token is the registered drafter-work companion; issued drafts per output token is retained only as an issuance proxy. Target forwards exclude the added post-hoc branch-replay forward and independent verification pass.

Independent replay provenance scope: Unsigned local attestation bound to a pushed pre-execution receipt commit and the registered source, inputs, checkpoint, and argv; it is not a signed external or trusted-execution witness.

Format-ablation provenance limitation: This is repository-revision evidence, not a per-document raw-corpus manifest for run 1. Source drift between the run 1 build and the captured cache refs cannot be ruled out.

Format-ablation runtime limitation: Run 1 did not record source-file hashes or sampler state in its pre-fix checkpoints. A post-build receipt now hashes all 52 current shard files and binds deterministic visible-grammar normalization, but it does not prove that the bytes were unchanged since training began or recover exact original units. The step-300 process recovery reset the legacy sampler, causing 78,643,200 scheduled token positions, 1.57 percent of the training budget, to replay earlier random windows. sampler_reset_steps [300] matches that reset timing in both arms. Later checkpoints preserve exact sampler RNG state. Exact sampled-example equivalence still cannot be proven.

One control worth repeating here, because acceptance rate is easy to report
dishonestly: **an initialized model scored higher acceptance than a trained one
in the E0 instrument pilot** (0.482 against 0.331 for a model deliberately
overfit on 100k tokens), because two near-uniform distributions agree trivially.
Those are pilot diagnostics, not run 1 results. Any final acceptance number
without its exact initialized floor beside it is uninterpretable.

## Limitations

- 5B training tokens. Small, and it shows on unfamiliar APIs, which it will
  hallucinate confidently.
- 2048 token context.
- Ten languages, weighted toward Python, JavaScript, and TypeScript. Weaker
  everywhere else, and untested outside that set.
- No safety tuning of any kind. It is a base completion model trained on public
  GitHub code, and it will reproduce patterns from that data, including insecure
  ones. Review what it writes.
- No row-level training-source manifest, license mapping, or attribution index.
  Exact aggregate train-token totals survive in the recovered final build log,
  but accepted rows, rejection counts, and per-source validation overshoot
  cannot be reconstructed from the shards.
- The tokenizer document sampler was round-robin by source entry rather than
  weighted like the training-token mixture.
- Training windows are sampled with replacement. The registered schedule's
  uniform-interval approximation covers about 62.57 percent of corpus positions
  at least once, rather than exposing every position once.
- The source-stratified validation shard is not repository- or document-disjoint.
  A chunked source document can cross the validation-to-training boundary. The
  separate publication holdout is repository-disjoint.
- The MTP sidecar is not consumed automatically by generic Llama runtimes, and
  no production-runtime MTP latency or speedup is claimed.
- Trained and evaluated on one machine, one seed, one run.

## Citation

```bibtex
@software{wisp_coder,
  title  = {Wisp: fill-in-the-middle and native multi-token prediction in a small code model},
  year   = {2026},
  url    = {https://huggingface.co/philipjohnbasile/wisp-coder-110m}
}
```