---
library_name: transformers
tags:
  - weevil
  - recurrent
  - gated-deltanet
language:
  - en
pipeline_tag: text-generation
datasets:
  - HuggingFaceFW/fineweb-edu
  - OptimalScale/ClimbMix
  - IFM/TxT360-v2
  - HuggingFaceTB/finemath
  - HuggingFaceTB/smollm-corpus
  - nvidia/OpenMathInstruct-2
  - codeparrot/codeparrot-clean
---

# Weevil-Recurrent-16M-Base

A 16,085,568-parameter recurrent decoder with post-decay pretraining weights. [Weevil-Recurrent-16M-QA](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-QA) is this model fine-tuned for multiple-choice questions.

## Checkpoints

| Model | Description |
|---|---|
| [ELBennett/Weevil-Recurrent-16M-Base](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-Base) | Post-decay pretraining checkpoint. |
| [ELBennett/Weevil-Recurrent-16M-QA](https://huggingface.co/ELBennett/Weevil-Recurrent-16M-QA) | The Base checkpoint fine-tuned for multiple-choice questions. |

## Model details

| Property | Value |
|---|---|
| Parameters | 16,085,568 |
| Architecture | Causal Gated DeltaNet-2 |
| Vocabulary | 50,368 |
| Factorized embedding | 128 |
| Hidden size | 256 |
| Stack | 2-layer prelude + 4-layer core + 2-layer coda |
| Pretraining | 9.75B first-mix tokens + the entire 750M decay stage = 10.5B content tokens |
| Precision | Float32 stored weights; bfloat16 linears |
| Pause token | `[pause]`, token id 50285 |

## How to use

```bash
pip install torch transformers
```

### Generate

Pass unpadded rows: right-padding is consumed by the convolution.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ELBennett/Weevil-Recurrent-16M-Base", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("ELBennett/Weevil-Recurrent-16M-Base")
model.eval()

prompt = "The capital of France is"
prompt_ids = torch.tensor([tokenizer.encode(prompt, add_special_tokens=False)])
output_ids = model.generate(prompt_ids, max_new_tokens=16)
print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

### Score choices

Continue with the model and tokenizer above.

```python
choices = ["cat", "dog"]
prompt = """Which animal barks?

A. cat
B. dog
Answer:"""
pause_id = tokenizer.convert_tokens_to_ids("[pause]")
passes, n_pause = 16, 10

prompt_ids = torch.tensor(
    [tokenizer.encode(prompt, add_special_tokens=False) + [pause_id] * n_pause]
)
answers = [tokenizer.encode(" " + choice, add_special_tokens=False) for choice in choices]
with torch.no_grad():
    scores = model.score_continuations(prompt_ids, answers, core_passes=passes)
print(choices[int(scores.argmax())], [round(score, 3) for score in scores.tolist()])
```

The prompt and pauses are prefilled once. Each choice is scored from its own copy of the recurrent state.

### Reuse the recurrent state

This continues the `prompt_ids` from the scoring example. `generate` prefills the GDN2 state and convolution windows, then advances them one token at a time. `past_key_values` and `recurrent_state` refer to the same object: one state matrix and one convolution window per mixer, with a separate copy for every core pass.

```python
out = model(prompt_ids, use_cache=True)
next_id = out.logits[:, -1:].argmax(dim=-1)
nxt = model(next_id, past_key_values=out.recurrent_state, use_cache=True)
print(out.recurrent_state.seq_length, nxt.logits.shape)
```

Use `recurrent_state.clone()` or `recurrent_state.repeat(n)` to copy the state when computing token log-probabilities yourself.

### Inference speed

On an H100 at 16 core passes, after the prompt is prefilled, one new token takes 2.52 ms at batch 1. Compiling each block and replaying the token step from a CUDA graph reaches 259,000 tokens/s at batch 16,384 (100% GPU utilization, about 532 W, 31 GB). Prefill of a long prompt still runs the chunk schedule; decode does not recompute it.

## Pretraining mix

This checkpoint trained **9.75 billion tokens** on the first mix. Ten percent of the positions in each pretraining sequence were `[pause]` tokens, and those positions are not part of the content count.

Weight is the probability of drawing the next document from a dataset. Tokens are that weight times 9.75 billion. Each packed document also contributes one end-of-text token toward the count.

| Dataset | Subset | Weight | Tokens |
|---|---|---:|---:|
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | | 25% | 2.4375B |
| [ClimbMix](https://huggingface.co/datasets/OptimalScale/ClimbMix) | | 30% | 2.925B |
| [TxT360-v2](https://huggingface.co/datasets/IFM/TxT360-v2) | `web-high-medium` | 20% | 1.95B |
| [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | `finemath-4plus` | 10% | 975M |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | `cosmopedia-v2` | 8% | 780M |
| [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) | problem and generated solution | 5% | 487.5M |
| [codeparrot-clean](https://huggingface.co/datasets/codeparrot/codeparrot-clean) | Python source, autogenerated files skipped | 2% | 195M |
| **First mix** | | **100%** | **9.75B** |

## Decay mix

The entire decay stage trained **750 million tokens**, shifting weight toward ClimbMix, WikiHow-style tutorials, and math. Tokens are the weight times 750 million.

WikiHow and math rows are filters of [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus): `format = wikihow`, and `seed_data = auto_math_text`.

| Dataset | Subset | Weight | Tokens |
|---|---|---:|---:|
| [ClimbMix](https://huggingface.co/datasets/OptimalScale/ClimbMix) | | 40% | 300M |
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | | 15% | 112.5M |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | WikiHow tutorials | 15% | 112.5M |
| [OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) | problem and generated solution | 12% | 90M |
| [FineMath](https://huggingface.co/datasets/HuggingFaceTB/finemath) | `finemath-4plus` | 10% | 75M |
| [Cosmopedia-v2](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | math (`auto_math_text`) | 8% | 60M |
| **Decay stage** | | **100%** | **750M** |

Together that is **10.5 billion** content tokens: 9.75 billion from the first mix and 750 million from the decay stage.

## Architecture

`core_passes` repeats only the core. Each core pass adds `input_scale * prelude_output` back onto the residual. Each mixer and MLP branch has a 1-lane read/write gate. Next-token logits go back through the factorized embedding. Linears run in bfloat16.

`[pause]` is token id 50285. It is a special token, so the word "pause" in ordinary text is not rewritten to it. The model does not learn to emit pauses. Put them in the input.