Blame for README.md

---
license: cc-by-sa-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- oeis
- integer-sequences
- qwen3
- causal-lm
- mlx
datasets:
- N8Programs/oeis-ladder-10b
---

# NextTerm-2-440M

(Model Card Co-Written w/ Opus 5.5)

[![Benchmark radar](radar_chart.png)](radar_chart.png)

## Model Summary

NextTerm-2-440M is a 441M parameter causal transformer trained to continue integer
sequences. It is the largest model in the NextTerm-2 series (5M, 50M and 440M), three
models trained on the same 10B tokens with identical hyperparameters for controlled comparison. It uses the Qwen3 architecture with a byte-level
vocabulary: the 256 byte values plus BOS, EOS and PAD.

The training corpus, [N8Programs/oeis-ladder-10b](https://huggingface.co/datasets/N8Programs/oeis-ladder-10b),
contains every training-split OEIS sequence once (terms from its b-file, up to 4,096
bytes) plus 9.2B tokens of distinct, synthetically transformed copies of OEIS sequences.

Compared with NextTerm-440M, NextTerm-2-440M is better on every benchmark below while
training on about 30% fewer tokens (10B vs 14B). The largest gains are on Ryskina & Knight
(+14 points) and on polynomial continuation from short prompts, where it needs far fewer
terms to identify a cubic or quartic.

The model reads integer sequences formatted as comma-separated values, with a trailing
comma, for example:

```text
1,-2,3,-4,
```

Training data contained only digits, commas and `-`, with no spaces and no leading
zeros. Other characters are accepted by the byte-level tokenizer but are out of
distribution, so format prompts exactly as above.

## Training Details

| Field | Value |
| --- | --- |
| Parameters | 440,997,888 |
| Architecture | Qwen3-style causal LM |
| Layers | 28 |
| Hidden size | 1024 |
| FFN size | 3072 |
| Attention heads | 16 (head dim 128) |
| KV heads | 8 |
| Vocabulary size | 259 (256 bytes + BOS/EOS/PAD) |
| Training tokens | 9,999,999,999 (one epoch) |
| Sequence length cap | 4,096 bytes per sequence |
| Batch mode | Whole-document packing, 98,304 token positions per step (97% filled), no cross-document attention |
| Steps | 105,029 |
| Optimizer | Muon/AdamW hybrid with μP width scaling |
| LR schedule | Linear warmup over 2% of steps to `2e-3` for Muon, `5e-4` for AdamW, cosine decay to 0.1x |
| AdamW | β = (0.9, 0.99), weight decay 0.01 |
| Training hardware | 2× NVIDIA DGX Spark (GB10), data parallel, bf16 |
| Training time | 4 days 8 hours |
| Export dtype | bfloat16 |

Muon was used for 2D weight matrices and AdamW for embeddings, norm gains and the output
head. All three NextTerm-2 models share these settings.

## Evaluation Results

### Main Benchmarks

| Model | OEIS-Eval-Neo | Ryskina & Knight | M1 Competition 111 MAPE |
| --- | ---: | ---: | ---: |
| **NextTerm-2-440M** | **36.25%** | 66.67% | **17.1105** |
| NextTerm-2-50M | 31.56% | 66.67% | 18.3615 |
| NextTerm-2-5M | 23.38% | 42.11% | 20.8527 |
| NextTerm-440M | 34.43% | 52.63% | 17.6239 |
| NextTerm-47M | 29.49% | **70.18%** | 18.7621 |
| Qwen3-0.6B | 18.44% | 33.33% | 22.7984 |
| Qwen3-1.7B | 20.77% | 49.12% | 22.2411 |
| Qwen3-4B | 23.74% | 63.16% | 19.1731 |
| Qwen3-8B | 24.62% | 57.89% | 18.4027 |
| Qwen3-14B | 26.00% | 59.65% | 17.9837 |

OEIS-Eval-Neo is a held-out OEIS next-term evaluation (19,034 sequences). M1 Competition
111 reports macro MAPE, where lower is better; the Naive2 forecasting baseline scores
17.7987. Ryskina & Knight (2021) is a 57-sequence next-term benchmark based on
psychometrics and puzzles. NextTerm-47M remains ahead there, consistent with its strength
on very short prompts.


### Polynomial Continuation

The polynomial continuation evaluation samples integer sequences from
polynomials of degree 1 through 4 and asks for the next term. Accuracy is exact
match across 200 samples for each prompt length `k`.

| Model | Arithmetic | Quadratic | Cubic | Quartic |
| --- | ---: | ---: | ---: | ---: |
| **NextTerm-2-440M** | 96.15% | **94.39%** | **88.70%** | **83.74%** |
| NextTerm-2-50M | **96.31%** | 84.11% | 63.95% | 47.86% |
| NextTerm-2-5M | 78.21% | 23.87% | 1.89% | 0.50% |
| NextTerm-440M | 94.38% | 86.39% | 75.20% | 67.83% |
| NextTerm-47M | 94.15% | 81.07% | 37.43% | 15.17% |
| Qwen3-0.6B | 90.31% | 8.72% | 0.30% | 0.02% |
| Qwen3-1.7B | 93.10% | 41.57% | 5.36% | 0.71% |
| Qwen3-4B | 93.90% | 77.26% | 28.18% | 5.98% |
| Qwen3-8B | 96.10% | 80.59% | 32.93% | 7.95% |
| Qwen3-14B | 95.60% | 84.61% | 49.16% | 14.98% |

## Usage

### MLX

```bash
mlx_lm.generate --model N8Programs/NextTerm-2-440M --prompt "1,2,3,"
```

### Hugging Face Transformers

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "N8Programs/NextTerm-2-440M"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "1,2,3,4,5,"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=64,
    do_sample=False,
    eos_token_id=[tokenizer.convert_tokens_to_ids(","), tokenizer.eos_token_id],
    pad_token_id=tokenizer.pad_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

For strict next-term evaluation, stop generation on *comma or EOS* and parse the
text before the first comma as the predicted integer. The tokenizer adds BOS
automatically.

## Reproducibility

This repository contains the evaluation scripts and data used for the results above:

- `eval_neo.py` for OEIS-Eval-Neo (and the polynomial sets).
- `gen_poly_evals.py` to regenerate the arithmetic/quadratic/cubic/quartic sets.
- `eval_m1.py` for M1 Competition 111 MAPE.
- `eval_ryskina.py` for Ryskina & Knight.
- `oeis_val_neo.jsonl`, `m1_competition_111.jsonl` and `ryskina_knight_57.jsonl`.
- `eval_results.txt` for the compact result table.

The model was trained with [spark-gpt](https://github.com/N8python/spark-gpt) on the
[N8Programs/oeis-ladder-10b](https://huggingface.co/datasets/N8Programs/oeis-ladder-10b)
dataset, whose card documents the data recipe in full.

## Citation

```bibtex
@misc{nextterm2_2026,
  author       = {Nathan Breslow and Opus 5.5},
  title        = {NextTerm-2: A Scaling Ladder of Pretrained Transformers for Integer Sequence Prediction},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/N8Programs/NextTerm-2-440M}},
  note         = {441M parameter model trained on 10B tokens of augmented OEIS data}
}
```

## Attribution

This model was trained on data from the
**On-Line Encyclopedia of Integer Sequences (OEIS)**.

- Source: https://oeis.org/
- License: Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)
- OEIS End-User License Agreement: https://oeis.org/wiki/The_OEIS_End-User_License_Agreement