Weevil-Recurrent-16M-Base

A 16,085,568-parameter recurrent decoder with post-decay pretraining weights. Weevil-Recurrent-16M-QA is this model fine-tuned for multiple-choice questions.

Checkpoints

Model	Description
ELBennett/Weevil-Recurrent-16M-Base	Post-decay pretraining checkpoint.
ELBennett/Weevil-Recurrent-16M-QA	The Base checkpoint fine-tuned for multiple-choice questions.

Model details

Property	Value
Parameters	16,085,568
Architecture	Causal Gated DeltaNet-2
Vocabulary	50,368
Factorized embedding	128
Hidden size	256
Stack	2-layer prelude + 4-layer core + 2-layer coda
Pretraining	9.75B first-mix tokens + the entire 750M decay stage = 10.5B content tokens
Precision	Float32 stored weights; bfloat16 linears
Pause token	[pause], token id 50285

How to use

pip install torch transformers

Generate

Pass unpadded rows: right-padding is consumed by the convolution.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ELBennett/Weevil-Recurrent-16M-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("ELBennett/Weevil-Recurrent-16M-Base")
model.eval()

prompt = "The capital of France is"
prompt_ids = torch.tensor([tokenizer.encode(prompt, add_special_tokens=False)])
output_ids = model.generate(prompt_ids, max_new_tokens=16)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Score choices

Continue with the model and tokenizer above.

choices = ["cat", "dog"]
prompt = """Which animal barks?

A. cat
B. dog
Answer:"""
pause_id = tokenizer.convert_tokens_to_ids("[pause]")
passes, n_pause = 16, 10

prompt_ids = torch.tensor(
    [tokenizer.encode(prompt, add_special_tokens=False) + [pause_id] * n_pause]
)
answers = [tokenizer.encode(" " + choice, add_special_tokens=False) for choice in choices]
with torch.no_grad():
    scores = model.score_continuations(prompt_ids, answers, core_passes=passes)
print(choices[int(scores.argmax())], [round(score, 3) for score in scores.tolist()])

The prompt and pauses are prefilled once. Each choice is scored from its own copy of the recurrent state.

Reuse the recurrent state

This continues the prompt_ids from the scoring example. generate prefills the GDN2 state and convolution windows, then advances them one token at a time. past_key_values and recurrent_state refer to the same object: one state matrix and one convolution window per mixer, with a separate copy for every core pass.

out = model(prompt_ids, use_cache=True)
next_id = out.logits[:, -1:].argmax(dim=-1)
nxt = model(next_id, past_key_values=out.recurrent_state, use_cache=True)
print(out.recurrent_state.seq_length, nxt.logits.shape)

Use recurrent_state.clone() or recurrent_state.repeat(n) to copy the state when computing token log-probabilities yourself.

Inference speed

On an H100 at 16 core passes, after the prompt is prefilled, one new token takes 2.52 ms at batch 1. Compiling each block and replaying the token step from a CUDA graph reaches 259,000 tokens/s at batch 16,384 (100% GPU utilization, about 532 W, 31 GB). Prefill of a long prompt still runs the chunk schedule; decode does not recompute it.

Pretraining mix

This checkpoint trained 9.75 billion tokens on the first mix. Ten percent of the positions in each pretraining sequence were [pause] tokens, and those positions are not part of the content count.

Weight is the probability of drawing the next document from a dataset. Tokens are that weight times 9.75 billion. Each packed document also contributes one end-of-text token toward the count.

Dataset	Subset	Weight	Tokens
FineWeb-Edu		25%	2.4375B
ClimbMix		30%	2.925B
TxT360-v2	web-high-medium	20%	1.95B
FineMath	finemath-4plus	10%	975M
Cosmopedia-v2	cosmopedia-v2	8%	780M
OpenMathInstruct-2	problem and generated solution	5%	487.5M
codeparrot-clean	Python source, autogenerated files skipped	2%	195M
First mix		100%	9.75B

Decay mix

The entire decay stage trained 750 million tokens, shifting weight toward ClimbMix, WikiHow-style tutorials, and math. Tokens are the weight times 750 million.

WikiHow and math rows are filters of Cosmopedia-v2: format = wikihow, and seed_data = auto_math_text.

Dataset	Subset	Weight	Tokens
ClimbMix		40%	300M
FineWeb-Edu		15%	112.5M
Cosmopedia-v2	WikiHow tutorials	15%	112.5M
OpenMathInstruct-2	problem and generated solution	12%	90M
FineMath	finemath-4plus	10%	75M
Cosmopedia-v2	math (auto_math_text)	8%	60M
Decay stage		100%	750M

Together that is 10.5 billion content tokens: 9.75 billion from the first mix and 750 million from the decay stage.

Architecture

core_passes repeats only the core. Each core pass adds input_scale * prelude_output back onto the residual. Each mixer and MLP branch has a 1-lane read/write gate. Next-token logits go back through the factorized embedding. Linears run in bfloat16.

[pause] is token id 50285. It is a special token, so the word "pause" in ordinary text is not rewritten to it. The model does not learn to emit pauses. Put them in the input.