philipjohnbasile/wisp-coder-110m

Wisp Coder 110M

A small code model built for one job: completing code at the cursor. It targets editor inline suggestions, but no production-runtime latency claim is made.

Two things make it unusual, and they are the reason it exists rather than a feature list.

It does fill in the middle natively. Each tokenized source document was split into chunks of at most 1024 tokens, and 70 percent of those chunks were independently transformed so the model predicts a middle span given both the prefix and the suffix. This is not a per-document or per-window rate. Cursor completion is the primary format, not an inference-time prompt trick added to an ordinary left-to-right checkpoint.

It has multi-token-prediction heads trained in from step zero. A shared MTP module, applied recursively in the Qwen3-Next style, predicts two or more tokens ahead from the trunk's own hidden state. The LM head is shared between trunk and MTP module, which ties the two output distributions to a common projection. That is a useful architectural choice, not a guarantee that draft and target distributions agree; how close they actually are is what acceptance measures.

What is and is not claimed here. Small models with MTP drafters exist: Google publishes gemma-4-E2B-it-assistant and related MTP assistant checkpoints. The artifact contribution is the combination at this scale, a small code model trained from step zero with both fill-in-the-middle on most tokenized chunks and a recursively shared MTP module, plus a paired measurement of how suffix information changes draft acceptance. A targeted search found no exact prior measurement, but that is not proof of absence. No exclusivity or first-of-its-kind claim is made.

What it is not

It is not a chat model, an instruction-following model, or an agent. It completes text. Asking it to refactor a module produces nonsense.

It is also not competitive with Qwen2.5-Coder on raw completion quality, and the arithmetic says it cannot be: Qwen2.5-Coder-0.5B saw roughly 5.5 trillion tokens and Wisp saw 5 billion, three orders of magnitude fewer. If you want the best completions available at small scale, use Qwen. Use Wisp to study FIM plus native speculative drafting on Apple Silicon, or how those two interact.

Architecture

Parameters	108.2M total, 100.7M trunk
Layers	12, d_model 768, 12 heads / 4 KV heads, SwiGLU 2048
Context	2048, RoPE theta 100k
Vocab	32,768 byte-level BPE, digits split, FIM sentinels
MTP	1 shared module (1 transformer block), trained to depth 2, recursive at inference
Precision	bfloat16 weights, trained with float32 master weights

The trunk is an ordinary Llama decoder. It loads in transformers and mlx_lm with no custom modelling code and no trust_remote_code. The MTP module ships alongside as mtp.safetensors and is ignorable by runtimes that cannot use it. Generic Llama runtimes do not consume the sidecar automatically. The package therefore also ships wisp_mtp_model.py and wisp_mtp_reference.py, a manifest-bound MLX correctness decoder that requires the exact sidecar schema, verifies every declared package hash, exposes the resolved runtime contract and executed route, and fails unless its MTP path reproduces target greedy tokens. It recomputes full prefixes and is not a production speed path.

The export was verified against transformers numerically, not assumed: relative logit delta 2.16e-3 and argmax agreement 1.0000 against the MLX original on the same tokens, which pins the RoPE convention, the RMSNorm epsilon, and the grouped query head order.

Training data

The configured target mixture was 92 percent code from bigcode/starcoderdata and 8 percent HuggingFaceFW/fineweb-edu. This is a source percentage, not a permissive-license percentage. StarCoderData declares license: other, and its original repository terms and relevant attribution clauses remain applicable. FineWeb-Edu is ODC-By 1.0 and remains subject to Common Crawl terms. Run 1 retained no row-level source manifest, so no per-file licensing or attribution guarantee is made.

Every run 1 document received structural size, line-length, and character distribution filters. Because the loader expected path instead of StarCoderData's max_stars_repo_path, Python AST and JSON extension filtering did not activate for run 1. The schema-aware repair applies only to future source streaming, not to these weights. Corpus content was never executed. Full details and the machine-checked disclosure are in DATA.md and config/training_data_receipt.json.

The 32K tokenizer was trained on 400,000 documents sampled round-robin across the eleven source entries, not according to the later token-budget weights. Current shard bytes are fully attested after the build, but the receipt is not a raw-row manifest and cannot prove exact original example boundaries.

The Apache 2.0 metadata describes the released Wisp artifact. It does not override training-source terms or licenses applicable to generated code.

Usage

Plain completion:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("philipjohnbasile/wisp-coder-110m")
tok = AutoTokenizer.from_pretrained("philipjohnbasile/wisp-coder-110m")
print(tok.decode(model.generate(**tok("def quicksort(arr):", return_tensors="pt"),
                                max_new_tokens=64)[0]))

Fill in the middle. The model was trained with both orderings, evenly split:

# PSM: prefix, suffix, then generate the middle
prompt = f"<|fim_prefix|>{prefix}<|fim_suffix|>{suffix}<|fim_middle|>"

Exercise the native sidecar explicitly. This is a correctness check, not a latency benchmark:

import os
import sys
from huggingface_hub import snapshot_download
from tokenizers import Tokenizer

package = snapshot_download(
    "philipjohnbasile/wisp-coder-110m",
    local_dir=os.path.abspath("wisp-coder-110m-package"),
)
sys.path.insert(0, package)
from wisp_mtp_reference import WispMTPReferenceRuntime

tok = Tokenizer.from_file(os.path.join(package, "tokenizer.json"))
ids = tok.encode(prompt, add_special_tokens=False).ids
runtime = WispMTPReferenceRuntime.load(package)
result = runtime.verify_greedy_parity(ids, max_new_tokens=16, depth=2)
print(result["mtp_route"])

For evaluation, the trunk can be configured as Continue.dev's autocomplete model while leaving a larger model as the chat and agent backend. Target-runtime batch-1 latency and editor user experience have not yet been measured, so this is not a sub-100 ms claim.

Evaluation

Registered final results

Measurement	Result
Final validation main NLL	1.3873 [1.3428, 1.4335], perplexity 4.00
Validation MTP depth 1 NLL	1.5727 [1.5236, 1.6234]
Validation MTP depth 2 NLL	1.6513 [1.6013, 1.7028]
FIM / shuffled-suffix acceptance, depth 2	1.0268 [1.0209, 1.0331], POSITIVE, 200 documents
Trained minus initialized acceptance-ratio lift	+0.0268 [+0.0210, +0.0331], CLEARS_CONTROL
Acceptance interpretation	POSITIVE_AND_CLEARS_CONTROL
FIM-training effect on shuffled-FIM minus L2R acceptance	+0.0814 [+0.0747, +0.0883], POSITIVE, 200 documents
FIM-training effect on true-suffix minus shuffled-suffix acceptance	+0.0049 [-0.0026, +0.0122], NULL: the interval includes zero
Adaptive minus fixed accepted drafts per verification	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents
Adaptive minus fixed output tokens per target forward	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents
Adaptive minus fixed drafts issued per output token	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents (issuance proxy)
Adaptive minus fixed draft recursions per output token	+0.0000 [+0.0000, +0.0000], NULL: the interval includes 0, 60 test documents
Rollout policies selected on calibration	adaptive_h0.7 versus fixed_d4
Branch-local greedy replay	Exact argmax for all 38400 emitted tokens across 600 scored policy-document rollouts
Cross-policy output identity	Identical realized output branches for all 100 calibration/test documents across compared policies

Validation intervals measure Monte Carlo uncertainty from the frozen random-window sampler. Acceptance and rollout intervals resample paired target documents. They do not measure training-run or model uncertainty. Null and negative outcomes are retained rather than filtered from the release.

Rollout endpoint scope: The primary endpoint measures accepted drafts per verification. It does not establish verification-width cost or deployment latency. Draft recursions per output token is the registered drafter-work companion; issued drafts per output token is retained only as an issuance proxy. Target forwards exclude the added post-hoc branch-replay forward and independent verification pass.

Independent replay provenance scope: Unsigned local attestation bound to a pushed pre-execution receipt commit and the registered source, inputs, checkpoint, and argv; it is not a signed external or trusted-execution witness.

Format-ablation provenance limitation: This is repository-revision evidence, not a per-document raw-corpus manifest for run 1. Source drift between the run 1 build and the captured cache refs cannot be ruled out.

Format-ablation runtime limitation: Run 1 did not record source-file hashes or sampler state in its pre-fix checkpoints. A post-build receipt now hashes all 52 current shard files and binds deterministic visible-grammar normalization, but it does not prove that the bytes were unchanged since training began or recover exact original units. The step-300 process recovery reset the legacy sampler, causing 78,643,200 scheduled token positions, 1.57 percent of the training budget, to replay earlier random windows. sampler_reset_steps [300] matches that reset timing in both arms. Later checkpoints preserve exact sampler RNG state. Exact sampled-example equivalence still cannot be proven.

One control worth repeating here, because acceptance rate is easy to report dishonestly: an initialized model scored higher acceptance than a trained one in the E0 instrument pilot (0.482 against 0.331 for a model deliberately overfit on 100k tokens), because two near-uniform distributions agree trivially. Those are pilot diagnostics, not run 1 results. Any final acceptance number without its exact initialized floor beside it is uninterpretable.

Limitations

• 5B training tokens. Small, and it shows on unfamiliar APIs, which it will hallucinate confidently.
• 2048 token context.
• Ten languages, weighted toward Python, JavaScript, and TypeScript. Weaker everywhere else, and untested outside that set.
• No safety tuning of any kind. It is a base completion model trained on public GitHub code, and it will reproduce patterns from that data, including insecure ones. Review what it writes.
• No row-level training-source manifest, license mapping, or attribution index. Exact aggregate train-token totals survive in the recovered final build log, but accepted rows, rejection counts, and per-source validation overshoot cannot be reconstructed from the shards.
• The tokenizer document sampler was round-robin by source entry rather than weighted like the training-token mixture.
• Training windows are sampled with replacement. The registered schedule's uniform-interval approximation covers about 62.57 percent of corpus positions at least once, rather than exposing every position once.
• The source-stratified validation shard is not repository- or document-disjoint. A chunked source document can cross the validation-to-training boundary. The separate publication holdout is repository-disjoint.
• The MTP sidecar is not consumed automatically by generic Llama runtimes, and no production-runtime MTP latency or speedup is claimed.
• Trained and evaluated on one machine, one seed, one run.

Citation

@software{wisp_coder,
  title  = {Wisp: fill-in-the-middle and native multi-token prediction in a small code model},
  year   = {2026},
  url    = {https://huggingface.co/philipjohnbasile/wisp-coder-110m}
}