NextTerm-2-440M: bf16 weights, model card, eval scripts and data

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+radar_chart.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,192 @@
+---
+license: cc-by-sa-4.0
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- oeis
+- integer-sequences
+- qwen3
+- causal-lm
+- mlx
+datasets:
+- N8Programs/oeis-ladder-10b
+---
+
+# NextTerm-2-440M
+
+(Model Card Co-Written w/ Opus 5.5)
+
+[![Benchmark radar](radar_chart.png)](radar_chart.png)
+
+## Model Summary
+
+NextTerm-2-440M is a 441M parameter causal transformer trained to continue integer
+sequences. It is the largest model in the NextTerm-2 series (5M, 50M and 440M), three
+models trained on the same 10B tokens with identical hyperparameters for controlled comparison. It uses the Qwen3 architecture with a byte-level
+vocabulary: the 256 byte values plus BOS, EOS and PAD.
+
+The training corpus, [N8Programs/oeis-ladder-10b](https://huggingface.co/datasets/N8Programs/oeis-ladder-10b),
+contains every training-split OEIS sequence once (terms from its b-file, up to 4,096
+bytes) plus 9.2B tokens of distinct, synthetically transformed copies of OEIS sequences.
+
+Compared with NextTerm-440M, NextTerm-2-440M is better on every benchmark below while
+training on about 30% fewer tokens (10B vs 14B). The largest gains are on Ryskina & Knight
+(+14 points) and on polynomial continuation from short prompts, where it needs far fewer
+terms to identify a cubic or quartic.
+
+The model reads integer sequences formatted as comma-separated values, with a trailing
+comma, for example:
+
+```text
+1,-2,3,-4,
+```
+
+Training data contained only digits, commas and `-`, with no spaces and no leading
+zeros. Other characters are accepted by the byte-level tokenizer but are out of
+distribution, so format prompts exactly as above.
+
+## Training Details
+
+| Field | Value |
+| --- | --- |
+| Parameters | 440,997,888 |
+| Architecture | Qwen3-style causal LM |
+| Layers | 28 |
+| Hidden size | 1024 |
+| FFN size | 3072 |
+| Attention heads | 16 (head dim 128) |
+| KV heads | 8 |
+| Vocabulary size | 259 (256 bytes + BOS/EOS/PAD) |
+| Training tokens | 9,999,999,999 (one epoch) |
+| Sequence length cap | 4,096 bytes per sequence |
+| Batch mode | Whole-document packing, 98,304 token positions per step (97% filled), no cross-document attention |
+| Steps | 105,029 |
+| Optimizer | Muon/AdamW hybrid with μP width scaling |
+| LR schedule | Linear warmup over 2% of steps to `2e-3` for Muon, `5e-4` for AdamW, cosine decay to 0.1x |
+| AdamW | β = (0.9, 0.99), weight decay 0.01 |
+| Training hardware | 2× NVIDIA DGX Spark (GB10), data parallel, bf16 |
+| Training time | 4 days 8 hours |
+| Export dtype | bfloat16 |
+
+Muon was used for 2D weight matrices and AdamW for embeddings, norm gains and the output
+head. All three NextTerm-2 models share these settings.
+
+## Evaluation Results
+
+### Main Benchmarks
+
+| Model | OEIS-Eval-Neo | Ryskina & Knight | M1 Competition 111 MAPE |
+| --- | ---: | ---: | ---: |
+| **NextTerm-2-440M** | **36.25%** | 66.67% | **17.1105** |
+| NextTerm-2-50M | 31.56% | 66.67% | 18.3615 |
+| NextTerm-2-5M | 23.38% | 42.11% | 20.8527 |
+| NextTerm-440M | 34.43% | 52.63% | 17.6239 |
+| NextTerm-47M | 29.49% | **70.18%** | 18.7621 |
+| Qwen3-0.6B | 18.44% | 33.33% | 22.7984 |
+| Qwen3-1.7B | 20.77% | 49.12% | 22.2411 |
+| Qwen3-4B | 23.74% | 63.16% | 19.1731 |
+| Qwen3-8B | 24.62% | 57.89% | 18.4027 |
+| Qwen3-14B | 26.00% | 59.65% | 17.9837 |
+
+OEIS-Eval-Neo is a held-out OEIS next-term evaluation (19,034 sequences). M1 Competition
+111 reports macro MAPE, where lower is better; the Naive2 forecasting baseline scores
+17.7987. Ryskina & Knight (2021) is a 57-sequence next-term benchmark based on
+psychometrics and puzzles. NextTerm-47M remains ahead there, consistent with its strength
+on very short prompts.
+
+
+### Polynomial Continuation
+
+The polynomial continuation evaluation samples integer sequences from
+polynomials of degree 1 through 4 and asks for the next term. Accuracy is exact
+match across 200 samples for each prompt length `k`.
+
+| Model | Arithmetic | Quadratic | Cubic | Quartic |
+| --- | ---: | ---: | ---: | ---: |
+| **NextTerm-2-440M** | 96.15% | **94.39%** | **88.70%** | **83.74%** |
+| NextTerm-2-50M | **96.31%** | 84.11% | 63.95% | 47.86% |
+| NextTerm-2-5M | 78.21% | 23.87% | 1.89% | 0.50% |
+| NextTerm-440M | 94.38% | 86.39% | 75.20% | 67.83% |
+| NextTerm-47M | 94.15% | 81.07% | 37.43% | 15.17% |
+| Qwen3-0.6B | 90.31% | 8.72% | 0.30% | 0.02% |
+| Qwen3-1.7B | 93.10% | 41.57% | 5.36% | 0.71% |
+| Qwen3-4B | 93.90% | 77.26% | 28.18% | 5.98% |
+| Qwen3-8B | 96.10% | 80.59% | 32.93% | 7.95% |
+| Qwen3-14B | 95.60% | 84.61% | 49.16% | 14.98% |
+
+## Usage
+
+### MLX
+
+```bash
+mlx_lm.generate --model N8Programs/NextTerm-2-440M --prompt "1,2,3,"
+```
+
+### Hugging Face Transformers
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_name = "N8Programs/NextTerm-2-440M"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    dtype=torch.bfloat16,
+    device_map="auto",
+)
+
+prompt = "1,2,3,4,5,"
+inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=64,
+    do_sample=False,
+    eos_token_id=[tokenizer.convert_tokens_to_ids(","), tokenizer.eos_token_id],
+    pad_token_id=tokenizer.pad_token_id,
+)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+For strict next-term evaluation, stop generation on *comma or EOS* and parse the
+text before the first comma as the predicted integer. The tokenizer adds BOS
+automatically.
+
+## Reproducibility
+
+This repository contains the evaluation scripts and data used for the results above:
+
+- `eval_neo.py` for OEIS-Eval-Neo (and the polynomial sets).
+- `gen_poly_evals.py` to regenerate the arithmetic/quadratic/cubic/quartic sets.
+- `eval_m1.py` for M1 Competition 111 MAPE.
+- `eval_ryskina.py` for Ryskina & Knight.
+- `oeis_val_neo.jsonl`, `m1_competition_111.jsonl` and `ryskina_knight_57.jsonl`.
+- `eval_results.txt` for the compact result table.
+
+The model was trained with [spark-gpt](https://github.com/N8python/spark-gpt) on the
+[N8Programs/oeis-ladder-10b](https://huggingface.co/datasets/N8Programs/oeis-ladder-10b)
+dataset, whose card documents the data recipe in full.
+
+## Citation
+
+```bibtex
+@misc{nextterm2_2026,
+  author       = {Nathan Breslow and Opus 5.5},
+  title        = {NextTerm-2: A Scaling Ladder of Pretrained Transformers for Integer Sequence Prediction},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/N8Programs/NextTerm-2-440M}},
+  note         = {441M parameter model trained on 10B tokens of augmented OEIS data}
+}
+```
+
+## Attribution
+
+This model was trained on data from the
+**On-Line Encyclopedia of Integer Sequences (OEIS)**.
+
+- Source: https://oeis.org/
+- License: Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)
+- OEIS End-User License Agreement: https://oeis.org/wiki/The_OEIS_End-User_License_Agreement

config.json

@@ -0,0 +1,26 @@
+{
+  "architectures": [
+    "Qwen3ForCausalLM"
+  ],
+  "model_type": "qwen3",
+  "vocab_size": 259,
+  "hidden_size": 1024,
+  "num_hidden_layers": 28,
+  "intermediate_size": 3072,
+  "num_attention_heads": 16,
+  "num_key_value_heads": 8,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "rms_norm_eps": 1e-06,
+  "max_position_embeddings": 4096,
+  "rope_theta": 1000000.0,
+  "rope_scaling": null,
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "tie_word_embeddings": false,
+  "use_cache": true,
+  "bos_token_id": 256,
+  "eos_token_id": 257,
+  "pad_token_id": 258,
+  "torch_dtype": "bfloat16"
+}

eval_m1.py

@@ -0,0 +1,251 @@
+"""M1 Competition 111 forecasting eval (macro MAPE) for byte-level spark-gpt OEIS checkpoints.
+
+A PyTorch port of eval_m1_competition_mape_mlx.py from the NextTerm-440M repository (the protocol
+behind the NextTerm model cards: Naive2 17.7987, NextTerm-440M 17.6239, NextTerm-47M 18.7621),
+unchanged in substance:
+  * each series' context is scaled to integers by the largest number of decimal places visible
+    in the context, joined by commas, with a trailing comma
+  * greedy decoding with EOS and PAD suppressed, stopping once `horizon` commas are generated
+    (or after --max-new-tokens); series are decoded together in left-padded batches with
+    per-row stopping (same greedy result as one at a time, up to batching numerics)
+  * the first `horizon` generated integers, divided back by the scale, are the forecast;
+    missing terms are scored as 0 (missing_policy "zero")
+  * series MAPE = mean APE over the horizon (zero actuals skipped); macro MAPE = mean over series
+    (the number on the model cards); median MAPE = median over series is also reported, since
+    one runaway forecast (e.g. a parabola on a flat series, APE ~700) can swing the macro mean
+    by several points
+  * prompts of --max-context-tokens or more tokens (BOS included) are skipped
+
+Usage: python eval_m1.py <hf_dir> <m1_competition_111.jsonl> <out_prefix>
+Writes <out_prefix>.series.jsonl and <out_prefix>.summary.json.
+"""
+import argparse
+import json
+import math
+from decimal import Decimal, InvalidOperation, localcontext
+from statistics import mean, median
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+
+def parse_decimal(raw):
+    try:
+        value = Decimal(str(raw).strip())
+    except InvalidOperation as exc:
+        raise ValueError(f"Could not parse decimal value {raw!r}") from exc
+    if not value.is_finite():
+        raise ValueError(f"Non-finite decimal value {raw!r}")
+    return value
+
+
+def context_scale(raw_context):
+    places = 0
+    for raw in raw_context:
+        exponent = parse_decimal(raw).as_tuple().exponent
+        if exponent < 0:
+            places = max(places, -exponent)
+    return 10 ** places
+
+
+def scale_decimal_to_int(value, scale):
+    scaled = value * Decimal(scale)
+    with localcontext() as ctx:
+        ctx.prec = max(50, len(scaled.as_tuple().digits) + 10)
+        rounded = scaled.to_integral_value()
+    if rounded != scaled:
+        raise ValueError(f"Context scale {scale} does not make {value} integral")
+    return int(rounded)
+
+
+def parse_generated_terms(text, limit):
+    terms, current, malformed = [], [], False
+
+    def flush():
+        nonlocal malformed
+        if not current:
+            return
+        token = "".join(current)
+        current.clear()
+        if token == "-":
+            malformed = True
+            return
+        try:
+            terms.append(int(token))
+        except ValueError:
+            malformed = True
+
+    for ch in text:
+        if len(terms) >= limit:
+            break
+        if ch.isdigit() or (ch == "-" and not current):
+            current.append(ch)
+        elif ch == ",":
+            flush()
+        elif ch.isspace():
+            continue
+        else:
+            malformed = True
+            flush()
+            break
+    if len(terms) < limit:
+        flush()
+    return terms[:limit], malformed
+
+
+def ape(actual, prediction):
+    if actual == 0:
+        return None
+    return float(abs(actual - prediction) / abs(actual) * Decimal(100))
+
+
+def mape_for_predictions(actuals, predictions, missing_policy, fallback):
+    apes, missing = [], 0
+    for actual, pred in zip(actuals, predictions):
+        if pred is None:
+            missing += 1
+            if missing_policy == "skip":
+                apes.append(None)
+                continue
+            pred = Decimal(0) if missing_policy == "zero" else fallback
+        apes.append(ape(actual, pred))
+    valid = [x for x in apes if x is not None and math.isfinite(x)]
+    return (mean(valid) if valid else None), apes, missing
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("hf_dir")
+    p.add_argument("data")
+    p.add_argument("out_prefix")
+    p.add_argument("--max-new-tokens", type=int, default=1024)
+    p.add_argument("--max-context-tokens", type=int, default=2048)
+    p.add_argument("--missing-policy", choices=["zero", "last_context", "skip"], default="zero")
+    p.add_argument("--batch-size", type=int, default=128,
+                   help="series decoded together (greedy, left-padded, per-row stopping)")
+    args = p.parse_args()
+
+    tok = AutoTokenizer.from_pretrained(args.hf_dir)
+    model = AutoModelForCausalLM.from_pretrained(args.hf_dir, dtype=torch.float32).cuda().eval()
+    bos, eos, pad = tok.bos_token_id, tok.eos_token_id, tok.pad_token_id
+    comma = ord(",")
+    suppress = torch.tensor([t for t in (eos, pad) if t is not None], device="cuda")
+
+    rows = [json.loads(line) for line in open(args.data) if line.strip()]
+    series = []
+    skipped_long = 0
+    for row in rows:
+        raw_context = [str(x) for x in row["context"]]
+        context = [parse_decimal(x) for x in raw_context]
+        scale = context_scale(raw_context)
+        prompt_text = ",".join(str(scale_decimal_to_int(v, scale)) for v in context) + ","
+        prompt = [bos] + list(prompt_text.encode("utf-8"))
+        assert tok(prompt_text).input_ids == prompt, "tokenizer disagrees with BOS + bytes"
+        if args.max_context_tokens > 0 and len(prompt) >= args.max_context_tokens:
+            skipped_long += 1
+            continue
+        series.append((row, raw_context, context, scale, prompt, int(row["horizon"])))
+
+    @torch.no_grad()
+    def greedy_batch(batch):
+        """Greedy decode a left-padded batch with a KV cache, EOS/PAD suppressed; each row
+        stops once it has generated `horizon` commas (or after --max-new-tokens)."""
+        width = max(len(s[4]) for s in batch)
+        ids = torch.full((len(batch), width), pad, dtype=torch.long, device="cuda")
+        mask = torch.zeros((len(batch), width), dtype=torch.long, device="cuda")
+        for r, s in enumerate(batch):
+            ids[r, width - len(s[4]):] = torch.tensor(s[4], device="cuda")
+            mask[r, width - len(s[4]):] = 1
+        pos = (mask.cumsum(-1) - 1).clamp(min=0)
+        out = model(ids, attention_mask=mask, position_ids=pos, use_cache=True)
+        past, logits = out.past_key_values, out.logits[:, -1]
+        horizon = torch.tensor([s[5] for s in batch], device="cuda")
+        commas = torch.zeros(len(batch), dtype=torch.long, device="cuda")
+        done = torch.zeros(len(batch), dtype=torch.bool, device="cuda")
+        last_pos = pos[:, -1]
+        generated = []
+        for _ in range(args.max_new_tokens):
+            logits = logits.float()
+            logits[:, suppress] = float("-inf")
+            nxt = logits.argmax(-1)
+            generated.append(torch.where(done, torch.full_like(nxt, pad), nxt))
+            commas += (nxt == comma) & ~done
+            done |= commas >= horizon
+            if bool(done.all()):
+                break
+            mask = torch.cat([mask, torch.ones_like(mask[:, :1])], dim=1)
+            last_pos = last_pos + 1
+            out = model(nxt[:, None], attention_mask=mask, position_ids=last_pos[:, None],
+                        past_key_values=past, use_cache=True)
+            past, logits = out.past_key_values, out.logits[:, -1]
+        gen = torch.stack(generated, dim=1).tolist()
+        return [bytes(t for t in g if t < 256).decode("utf-8", "replace") for g in gen]
+
+    texts = []
+    for b in range(0, len(series), args.batch_size):
+        texts.extend(greedy_batch(series[b:b + args.batch_size]))
+
+    records = []
+    for (row, raw_context, context, scale, prompt, horizon), text in zip(series, texts):
+        target = [parse_decimal(str(x)) for x in row["target"]]
+        naive2 = [parse_decimal(str(x)) for x in row["naive2_forecast"]]
+        terms, malformed = parse_generated_terms(text, horizon)
+        predictions = [Decimal(t) / Decimal(scale) for t in terms] + [None] * (horizon - len(terms))
+        fallback = context[-1] if context else Decimal(0)
+        mape, apes, missing = mape_for_predictions(target, predictions, args.missing_policy, fallback)
+        naive2_mape, naive2_apes, _ = mape_for_predictions(target, naive2, "skip", fallback)
+        records.append({
+            "row_index": row["row_index"], "series_name": row["series_name"], "period": row["period"],
+            "horizon": horizon, "scale": scale, "prompt_tokens": len(prompt),
+            "target": [str(x) for x in target],
+            "prediction": [str(x) if x is not None else None for x in predictions],
+            "parsed_terms": len(terms), "missing_terms": missing, "malformed": malformed,
+            "mape": mape, "apes": apes, "naive2_mape": naive2_mape, "naive2_apes": naive2_apes,
+            "generated_text": text,
+        })
+
+    def finite(rs, key):
+        return [float(r[key]) for r in rs if r[key] is not None and math.isfinite(float(r[key]))]
+
+    def macro(rs, key):
+        vals = finite(rs, key)
+        return mean(vals) if vals else None
+
+    def med(rs, key):
+        vals = finite(rs, key)
+        return median(vals) if vals else None
+
+    by_period = {}
+    for period in sorted({r["period"] for r in records}):
+        rs = [r for r in records if r["period"] == period]
+        by_period[period] = {"series": len(rs),
+                             "model_median_series_mape": med(rs, "mape"),
+                             "model_macro_mape": macro(rs, "mape"),
+                             "naive2_median_series_mape": med(rs, "naive2_mape"),
+                             "naive2_macro_mape": macro(rs, "naive2_mape")}
+    summary = {
+        "checkpoint": args.hf_dir,
+        "series_evaluated": len(records),
+        "skipped_long": skipped_long,
+        "model_median_series_mape": med(records, "mape"),
+        "model_macro_mape": macro(records, "mape"),
+        "naive2_median_series_mape": med(records, "naive2_mape"),
+        "naive2_macro_mape": macro(records, "naive2_mape"),
+        "parsed_full_series": sum(1 for r in records if r["parsed_terms"] >= r["horizon"]),
+        "total_missing_terms": sum(r["missing_terms"] for r in records),
+        "malformed_series": sum(1 for r in records if r["malformed"]),
+        "by_period": by_period,
+    }
+    with open(args.out_prefix + ".series.jsonl", "w") as f:
+        for r in records:
+            f.write(json.dumps(r) + "\n")
+    with open(args.out_prefix + ".summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+    print(json.dumps({k: summary[k] for k in ["series_evaluated", "skipped_long",
+                                              "model_median_series_mape", "naive2_median_series_mape",
+                                              "model_macro_mape", "naive2_macro_mape", "parsed_full_series",
+                                              "total_missing_terms", "malformed_series"]}))
+
+
+if __name__ == "__main__":
+    main()

eval_neo.py

@@ -0,0 +1,165 @@
+"""OEIS-Eval-Neo: last-term exact-match accuracy on the decontaminated held-out OEIS set.
+
+A batched PyTorch/transformers port of oeis_eval_mlx_neo.py from the NextTerm-440M repository
+(the protocol behind that card's 34.43%), unchanged in substance:
+  * rows with fewer than 2 terms are skipped; prompt = every term but the last, comma-joined, with
+    a trailing comma; answer = the last term (compared as an exact integer)
+  * prompts of --max-context-tokens or more tokens (BOS included) are skipped
+  * greedy decoding, stopping at the comma token or EOS, up to --max-new-tokens (196)
+  * prediction = the generated text before the first comma, parsed as an integer
+    (unparseable, e.g. an immediate EOS, counts as wrong)
+Works with any tokenizer that has a single comma token (the byte-level spark-gpt checkpoints, or
+the NextTerm digit tokenizer).
+
+Two modes, same accuracy: --mode teacher (default) checks greedy correctness in one forward pass
+over prompt + answer + "," (correct iff the argmax reproduces every answer token and is then a
+comma or EOS, which is exactly when greedy decoding would produce the answer); --mode generate
+decodes greedily.
+
+Usage: python eval_neo.py <hf_dir> <oeis_val_neo.jsonl> <out_prefix> [--batch-size 128]
+Writes <out_prefix>.rows.jsonl and <out_prefix>.summary.json.
+"""
+import argparse
+import json
+import time
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+
+def parse_generated(text):
+    text = text.split(",")[0]
+    try:
+        return int(text)
+    except ValueError:
+        return None
+
+
+def main():
+    p = argparse.ArgumentParser()
+    p.add_argument("hf_dir")
+    p.add_argument("data")
+    p.add_argument("out_prefix")
+    p.add_argument("--batch-size", type=int, default=128)
+    p.add_argument("--max-new-tokens", type=int, default=196)
+    p.add_argument("--first-pass-tokens", type=int, default=24,
+                   help="decode budget for the first pass; unfinished rows are redone at --max-new-tokens")
+    p.add_argument("--max-context-tokens", type=int, default=4096)
+    p.add_argument("--max-examples", type=int, default=0)
+    p.add_argument("--dtype", choices=["float32", "bfloat16"], default="bfloat16")
+    p.add_argument("--mode", choices=["teacher", "generate"], default="teacher",
+                   help="teacher: one forward pass per row (exact for greedy); generate: decode")
+    args = p.parse_args()
+
+    started = time.perf_counter()
+    tok = AutoTokenizer.from_pretrained(args.hf_dir)
+    model = AutoModelForCausalLM.from_pretrained(
+        args.hf_dir, dtype=getattr(torch, args.dtype)).cuda().eval()
+    comma = tok.encode(",", add_special_tokens=False)
+    if not comma:  # the NextTerm tokenizer strips a leading comma
+        comma = tok.encode("1,", add_special_tokens=False)[-1:]
+    assert len(comma) == 1, f"comma is not a single token: {comma}"
+    comma = comma[0]
+    stops = [comma] + ([tok.eos_token_id] if tok.eos_token_id is not None else [])
+    pad = tok.pad_token_id if tok.pad_token_id is not None else tok.eos_token_id
+
+    prompts, answers, rows = [], [], []
+    for i, line in enumerate(open(args.data)):
+        seq = json.loads(line, parse_int=str).get("seq", [])
+        if len(seq) < 2:
+            continue
+        prompts.append(tok(",".join(str(x) for x in seq[:-1]) + ",").input_ids)
+        answers.append(int(seq[-1]))
+        rows.append(i)
+    if args.max_examples > 0:
+        prompts, answers, rows = prompts[:args.max_examples], answers[:args.max_examples], rows[:args.max_examples]
+    keep = [k for k, pr in enumerate(prompts)
+            if args.max_context_tokens <= 0 or len(pr) < args.max_context_tokens]
+    skipped_long = len(prompts) - len(keep)
+    keep.sort(key=lambda k: len(prompts[k]))
+
+    def greedy(idx, max_new):
+        """Greedy-decode prompts idx (left-padded batches); return {k: tokens before the stop}
+        and the set of k that hit a stop token."""
+        gens, stopped = {}, set()
+        for b in range(0, len(idx), args.batch_size):
+            batch = idx[b:b + args.batch_size]
+            width = max(len(prompts[k]) for k in batch)
+            ids = torch.full((len(batch), width), pad, dtype=torch.long)
+            mask = torch.zeros((len(batch), width), dtype=torch.long)
+            for r, k in enumerate(batch):
+                ids[r, width - len(prompts[k]):] = torch.tensor(prompts[k])
+                mask[r, width - len(prompts[k]):] = 1
+            out = model.generate(ids.cuda(), attention_mask=mask.cuda(), do_sample=False,
+                                 max_new_tokens=max_new, eos_token_id=stops, pad_token_id=pad)
+            for r, k in enumerate(batch):
+                gen = out[r, width:].tolist()
+                cut = next((j for j, t in enumerate(gen) if t in stops), None)
+                if cut is not None:
+                    stopped.add(k)
+                gens[k] = gen[:cut] if cut is not None else gen[:max_new]
+        return gens, stopped
+
+    results = []
+    if args.mode == "generate":
+        with torch.no_grad():
+            # Most answers are short. Decode everything briefly, then redo only rows that have
+            # not reached a stop token with the full budget (equivalent under greedy decoding).
+            gens, stopped = greedy(keep, min(args.first_pass_tokens, args.max_new_tokens))
+            long_rows = [k for k in keep if k not in stopped]
+            if long_rows and args.first_pass_tokens < args.max_new_tokens:
+                more, _ = greedy(long_rows, args.max_new_tokens)
+                gens.update(more)
+        for k in keep:
+            text = tok.decode(gens[k], skip_special_tokens=True)
+            pred = parse_generated(text)
+            results.append({"row_index": rows[k], "answer": str(answers[k]),
+                            "prediction": None if pred is None else str(pred),
+                            "correct": pred == answers[k], "parsed": pred is not None,
+                            "generated_text": text})
+    else:
+        # Teacher-forced check, one forward pass per row: greedy decoding emits exactly the
+        # answer iff the argmax at every answer position is the answer's next token and the
+        # argmax after the last digit is a stop token. (Only differs from `generate` mode if a
+        # model emits leading zeros, which int() parsing would forgive.)
+        ans_ids = {k: tok.encode(str(answers[k]), add_special_tokens=False) for k in keep}
+        stop_t = torch.tensor(stops, device="cuda")
+        with torch.no_grad():
+            for b in range(0, len(keep), args.batch_size):
+                batch = keep[b:b + args.batch_size]
+                seqs = [prompts[k] + ans_ids[k] for k in batch]
+                width = max(map(len, seqs))
+                ids = torch.full((len(batch), width), pad, dtype=torch.long)
+                mask = torch.zeros((len(batch), width), dtype=torch.long)
+                for r, s in enumerate(seqs):  # right padding is fine for teacher forcing
+                    ids[r, :len(s)] = torch.tensor(s)
+                    mask[r, :len(s)] = 1
+                top = model(ids.cuda(), attention_mask=mask.cuda()).logits.argmax(-1)
+                for r, k in enumerate(batch):
+                    n_p, ans = len(prompts[k]), ans_ids[k]
+                    pred = top[r, n_p - 1:n_p + len(ans)]
+                    ok = (len(ans) < args.max_new_tokens
+                          and bool((pred[:-1] == torch.tensor(ans, device="cuda")).all())
+                          and bool(torch.isin(pred[-1], stop_t)))
+                    results.append({"row_index": rows[k], "answer": str(answers[k]),
+                                    "correct": ok, "parsed": True})
+
+    correct = sum(r["correct"] for r in results)
+    summary = {
+        "checkpoint": args.hf_dir, "data": args.data, "documents": len(prompts),
+        "evaluated": len(results), "skipped_long": skipped_long,
+        "parsed": sum(r["parsed"] for r in results), "correct": correct,
+        "accuracy": correct / len(results) if results else 0.0,
+        "max_new_tokens": args.max_new_tokens, "dtype": args.dtype, "mode": args.mode,
+        "seconds": round(time.perf_counter() - started, 1),
+    }
+    with open(args.out_prefix + ".rows.jsonl", "w") as f:
+        for r in sorted(results, key=lambda r: r["row_index"]):
+            f.write(json.dumps(r) + "\n")
+    with open(args.out_prefix + ".summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+    print(json.dumps(summary))
+
+
+if __name__ == "__main__":
+    main()

eval_results.txt

@@ -0,0 +1,38 @@
+NextTerm-2 evaluation results (released bf16 weights; greedy decoding).
+NextTerm-440M / NextTerm-47M / Naive2 rows are from the NextTerm-440M model card.
+
+OEIS-Eval-Neo:
+NextTerm-2-440M - 36.25%
+NextTerm-2-50M - 31.56%
+NextTerm-2-5M - 23.38%
+NextTerm-440M - 34.43%
+NextTerm-47M - 29.49%
+
+Ryskina & Knight (2021):
+NextTerm-2-440M - 66.67%
+NextTerm-2-50M - 66.67%
+NextTerm-2-5M - 42.11%
+NextTerm-440M - 52.63%
+NextTerm-47M - 70.18%
+
+M1 Competition 111 MAPE (macro; canonical greedy; lower is better):
+Naive2 - 17.7987
+NextTerm-2-440M - 17.1105
+NextTerm-2-50M - 18.3615
+NextTerm-2-5M - 20.8527
+NextTerm-440M - 17.6239
+NextTerm-47M - 18.7621
+
+M1 Competition 111 by frequency (macro MAPE):
+Naive2 - monthly 17.3871 / quarterly 19.5958 / yearly 17.1314
+NextTerm-2-440M - monthly 18.5442 / quarterly 16.6647 / yearly 12.7485
+NextTerm-2-50M - monthly 20.7720 / quarterly 16.6578 / yearly 12.1251
+NextTerm-2-5M - monthly 22.3369 / quarterly 23.4687 / yearly 12.7977
+NextTerm-440M - monthly 18.3407 / quarterly 19.0475 / yearly 13.5498
+NextTerm-47M - monthly 21.2719 / quarterly 16.0270 / yearly 13.3741
+
+Polynomial continuation evals (accuracy; 200 samples per k):
+Arithmetic (k=2-25): NextTerm-2-440M - 96.15%; NextTerm-2-50M - 96.31%; NextTerm-2-5M - 78.21%; NextTerm-440M - 94.38%; NextTerm-47M - 94.15%
+Quadratic (k=3-25): NextTerm-2-440M - 94.39%; NextTerm-2-50M - 84.11%; NextTerm-2-5M - 23.87%; NextTerm-440M - 86.39%; NextTerm-47M - 81.07%
+Cubic (k=4-25): NextTerm-2-440M - 88.70%; NextTerm-2-50M - 63.95%; NextTerm-2-5M - 1.89%; NextTerm-440M - 75.20%; NextTerm-47M - 37.43%
+Quartic (k=5-25): NextTerm-2-440M - 83.74%; NextTerm-2-50M - 47.86%; NextTerm-2-5M - 0.50%; NextTerm-440M - 67.83%; NextTerm-47M - 15.17%

eval_ryskina.py

@@ -0,0 +1,102 @@
+"""Ryskina & Knight (2021) next-term eval for byte-level spark-gpt OEIS checkpoints.
+
+Each item is a short sequence. The prompt is every term but the last, written
+"t1,t2,...,tk," with a trailing comma. The answer is the last term.
+
+Two scores per item:
+  exact   greedy decode until the next comma or EOS; correct iff the text equals
+          str(answer). This matches the NextTerm model-card protocol.
+  logprob log P(answer is the next term) in nats: the sum of the log-probs of the
+          answer's bytes, plus log(P(",") + P(EOS)) at the position after it. A
+          continuous score, much less noisy than 57 binary outcomes.
+
+Encoding is BOS + UTF-8 bytes, exactly as spark-gpt trains. The script checks
+that the exported HF tokenizer agrees.
+
+Usage: python eval_ryskina.py <hf_dir> <ryskina.jsonl> <out_prefix>
+Writes <out_prefix>.items.jsonl and <out_prefix>.summary.json.
+"""
+import json
+import math
+import statistics
+import sys
+
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MAX_NEW_TOKENS = 40  # longest answer is far shorter; a runaway decode just scores wrong
+
+
+def main():
+    hf_dir, data_path, out_prefix = sys.argv[1:4]
+    tok = AutoTokenizer.from_pretrained(hf_dir)
+    model = AutoModelForCausalLM.from_pretrained(hf_dir, dtype=torch.float32).cuda().eval()
+    bos, eos = tok.bos_token_id, tok.eos_token_id
+    comma = ord(",")
+
+    def encode(text):
+        ids = [bos] + list(text.encode("utf-8"))
+        assert tok(text).input_ids == ids, f"tokenizer disagrees on {text!r}"
+        return ids
+
+    items = [json.loads(line)["seq"] for line in open(data_path)]
+    results = []
+    with torch.no_grad():
+        for idx, seq in enumerate(items):
+            prompt_text = ",".join(str(x) for x in seq[:-1]) + ","
+            answer = str(seq[-1])
+            prompt = encode(prompt_text)
+
+            # exact match: greedy until comma / EOS
+            ids = torch.tensor([prompt], device="cuda")
+            generated = []
+            for _ in range(MAX_NEW_TOKENS):
+                nxt = int(model(ids).logits[0, -1].argmax())
+                if nxt in (comma, eos):
+                    break
+                generated.append(nxt)
+                ids = torch.cat([ids, torch.tensor([[nxt]], device="cuda")], dim=1)
+            prediction = bytes(b for b in generated if b < 256).decode("utf-8", "replace")
+
+            # log P(answer is the next term)
+            ans_ids = list(answer.encode("utf-8"))
+            full = torch.tensor([prompt + ans_ids], device="cuda")
+            logp = torch.log_softmax(model(full).logits[0].float(), dim=-1)
+            start = len(prompt) - 1  # logits at position i predict token i + 1
+            lp_answer = sum(float(logp[start + j, t]) for j, t in enumerate(ans_ids))
+            end = logp[start + len(ans_ids)]
+            lp_stop = float(torch.logaddexp(end[comma], end[eos]))
+            lp = lp_answer + lp_stop
+
+            results.append({
+                "idx": idx,
+                "prompt": prompt_text,
+                "answer": answer,
+                "prediction": prediction,
+                "correct": prediction == answer,
+                "logprob": lp,
+                "logprob_answer_bytes": lp_answer,
+                "logprob_stop": lp_stop,
+            })
+
+    lps = [r["logprob"] for r in results]
+    n_correct = sum(r["correct"] for r in results)
+    summary = {
+        "checkpoint": hf_dir,
+        "items": len(results),
+        "exact_correct": n_correct,
+        "exact_acc": n_correct / len(results),
+        "mean_logprob": statistics.fmean(lps),
+        "median_logprob": statistics.median(lps),
+        "mean_answer_prob": statistics.fmean(math.exp(x) for x in lps),
+    }
+    with open(out_prefix + ".items.jsonl", "w") as f:
+        for r in results:
+            f.write(json.dumps(r) + "\n")
+    with open(out_prefix + ".summary.json", "w") as f:
+        json.dump(summary, f, indent=2)
+    print(json.dumps(summary, indent=2))
+
+
+if __name__ == "__main__":
+    main()

gen_poly_evals.py

@@ -0,0 +1,52 @@
+"""Write the NextTerm-440M card's polynomial-continuation eval sets as jsonl.
+
+Reproduces arithmetic_eval.py's sampling exactly (same tasks, coefficient ranges, per-task
+RNG random.Random(seed + degree), same draw order): for each prior-term count k from the
+task's minimum up to --max-terms, --samples-per-k sequences f(0), ..., f(k) of a random
+non-negative-integer polynomial; the prompt is the first k terms and the answer is f(k).
+Each row is {"id", "seq", "k"} with seq = prompt + [answer], so scripts/eval_neo.py
+(last-term exact match; teacher-forced greedy) scores it directly.
+
+Degrees 5 and 6 (quintic, sextic; --extended) are an optional extension, not part of the
+published polynomial evals: the x^5 and
+x^6 coefficients are drawn from [0, 9] like x^4's, before the card's draws (highest degree
+first, as the card orders them), so degrees 1-4 stay byte-identical.
+
+Usage: python gen_poly_evals.py <out_dir> [--extended] [--seed 0 --max-terms 25 --samples-per-k 200]
+"""
+import argparse
+import json
+import random
+from pathlib import Path
+
+COEFF_MAX = {"g": 9, "f": 9, "a": 9, "b": 99, "c": 999, "d": 9999, "e": 99999}  # g: x^6, f: x^5
+CARD_TASKS = (("arithmetic", 1, 2), ("quadratic", 2, 3), ("cubic", 3, 4), ("quartic", 4, 5))
+EXTENDED_TASKS = (("quintic", 5, 6), ("sextic", 6, 7))
+
+p = argparse.ArgumentParser()
+p.add_argument("out_dir")
+p.add_argument("--seed", type=int, default=0)
+p.add_argument("--max-terms", type=int, default=25)
+p.add_argument("--samples-per-k", type=int, default=200)
+p.add_argument("--extended", action="store_true", help="write quintic and sextic instead")
+args = p.parse_args()
+out = Path(args.out_dir)
+out.mkdir(parents=True, exist_ok=True)
+
+for name, degree, min_terms in (EXTENDED_TASKS if args.extended else CARD_TASKS):
+    rng = random.Random(args.seed + degree)
+    with open(out / f"poly_{name}.jsonl", "w") as f:
+        for k in range(min_terms, args.max_terms + 1):
+            for i in range(args.samples_per_k):
+                g = rng.randint(0, COEFF_MAX["g"]) if degree >= 6 else 0
+                f5 = rng.randint(0, COEFF_MAX["f"]) if degree >= 5 else 0
+                # identical draw order to arithmetic_eval.sample_coefficients
+                a = rng.randint(0, COEFF_MAX["a"]) if degree >= 4 else 0
+                b = rng.randint(0, COEFF_MAX["b"]) if degree >= 3 else 0
+                c = rng.randint(0, COEFF_MAX["c"]) if degree >= 2 else 0
+                d = rng.randint(0, COEFF_MAX["d"]) if degree >= 1 else 0
+                e = rng.randint(0, COEFF_MAX["e"])
+                seq = [g * x**6 + f5 * x**5 + a * x**4 + b * x**3 + c * x**2 + d * x + e
+                       for x in range(k + 1)]
+                f.write(json.dumps({"id": f"{name}_k{k}_{i}", "seq": seq, "k": k}) + "\n")
+    print(name, (args.max_terms - min_terms + 1) * args.samples_per_k, "rows")

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "bos_token_id": 256,
+  "eos_token_id": 257,
+  "pad_token_id": 258,
+  "transformers_version": "5.11.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9f360e1ef6d63e659abd54e959de5935b7d07b58dfc3aaaf9570c44a38d12fda
+size 882030896

ryskina_knight_57.jsonl

@@ -0,0 +1,57 @@
+{"seq": [1, 2, 3, 4, 5], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [0, 2, 4, 6, 8], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [12, 19, 26, 33, 40], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [0, 1, 1, 2, 3, 5, 8], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [13, 15, 20, 22, 27, 29], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [32, 35, 39, 44, 50], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [1, 1, 8, 5, 15, 9, 22], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [89, 76, 65, 52, 41], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [17, -24, 41, -65, 106, -171], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [95, 89, 77, 59, 35], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [1, 3, 6, 10, 15, 21], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [1, 4, 10, 20, 35, 56], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [11, 23, 38, 11, 23, 38], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [6, 28, 12, 14, 24, 7, 48], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [7, 19, 43, 91, 187], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [17, 32, 61, 118, 231], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [23, 25, 28, 32, 34, 37, 41], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [1, 4, 9, 16, 25, 36], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [32, 16, 20, 10, 14, 7, 11], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [1, 1, 2, 6, 24, 120], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [65536, 256, 16, 4], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [32, 112, 392, 1372, 4802, 16807], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [12, 19, 38, 45, 90, 97], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [3, 8, 7, 15, 15, 29, 31, 57], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [1, 3, 19, 49, 93, 151], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [2, 6, 12, 20, 30, 42, 56], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [7, 14, 19, 29, 40, 44, 52], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [819, 648, 497, 366, 255], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [87, 165, 726, 1353, 4884], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [15, 47, 31, 54, 73, 15], "source": "https://www.nibcode.com/en/psychometric-training/test-of-numerical-sequence"}
+{"seq": [4, 8, 16, 32, 64, 128], "source": "https://www.syvum.com/cgi/online/serve.cgi/iq/ar_series3.html?question_hide"}
+{"seq": [2, 3, 5, 7, 11, 13, 17, 19, 23], "source": "https://www.syvum.com/cgi/online/serve.cgi/iq/ar_series3.html?question_hide"}
+{"seq": [500, 475, 425, 350, 250, 125], "source": "https://www.syvum.com/cgi/online/serve.cgi/iq/ar_series3.html?question_hide"}
+{"seq": [200, 196, 180, 116, -140], "source": "https://www.syvum.com/cgi/online/serve.cgi/iq/ar_series3.html?question_hide"}
+{"seq": [2, 3, 4, 12, 6, 23, 8, 36, 10, 51], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [16, 18, 29, 49, 78], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [4, 11, 25, 53, 109], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [29, 41, 53, 65, 77, 89], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [2, 4, 8, 10, 20, 22], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [2, 3, 1, 4, 0, 5, -1], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [6, 18, 0, 24, -6, 30, -12], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [6, 3, 12, 6, 24, 12, 48], "source": "https://www.12minprep.com/knowledge-hub/number-series-practice/#sample-questions-horizontal"}
+{"seq": [11, 21, 31, 41, 51], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [48, 42, 36, 30, 24], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [3, 9, 27, 81, 243], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [2, 8, 4, 16, 8, 32], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [249, 254, 127, 132, 66], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [8, 15, 12, 15, 16, 15, 20], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [22, 2, 22, 4, 22, 8, 22, 16], "source": "https://iqtestprep.com/number-sequence-test/"}
+{"seq": [324, 361, 400, 441, 484], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [512, 729, 1000, 1331], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [4, 36, 324, 2916, 26244], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [4, 9, 14, 19, 24, 29, 34], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [1, 3, 6, 10, 15, 21], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [10, 22, 46, 94, 190, 382], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [1, 4, 8, 11, 22, 25, 50], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}
+{"seq": [1, 2, 6, 36, 44, 440, 452], "source": "https://www.hitbullseye.com/Number-Series-Tricks.php"}

tokenizer.json

@@ -0,0 +1 @@
+{"version": "1.0", "truncation": null, "padding": null, "added_tokens": [{"id": 256, "content": "<bos>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true}, {"id": 257, "content": "<eos>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true}, {"id": 258, "content": "<pad>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": false, "special": true}], "normalizer": null, "pre_tokenizer": {"type": "ByteLevel", "add_prefix_space": false, "trim_offsets": true, "use_regex": true}, "post_processor": {"type": "TemplateProcessing", "single": [{"SpecialToken": {"id": "<bos>", "type_id": 0}}, {"Sequence": {"id": "A", "type_id": 0}}], "pair": [{"SpecialToken": {"id": "<bos>", "type_id": 0}}, {"Sequence": {"id": "A", "type_id": 0}}, {"SpecialToken": {"id": "<bos>", "type_id": 1}}, {"Sequence": {"id": "B", "type_id": 1}}], "special_tokens": {"<bos>": {"id": "<bos>", "ids": [256], "tokens": ["<bos>"]}}}, "decoder": {"type": "ByteLevel", "add_prefix_space": false, "trim_offsets": true, "use_regex": true}, "model": {"type": "BPE", "dropout": null, "unk_token": null, "continuing_subword_prefix": null, "end_of_word_suffix": null, "fuse_unk": false, "byte_fallback": false, "ignore_merges": false, "vocab": {"Ā": 0, "ā": 1, "Ă": 2, "ă": 3, "Ą": 4, "ą": 5, "Ć": 6, "ć": 7, "Ĉ": 8, "ĉ": 9, "Ċ": 10, "ċ": 11, "Č": 12, "č": 13, "Ď": 14, "ď": 15, "Đ": 16, "đ": 17, "Ē": 18, "ē": 19, "Ĕ": 20, "ĕ": 21, "Ė": 22, "ė": 23, "Ę": 24, "ę": 25, "Ě": 26, "ě": 27, "Ĝ": 28, "ĝ": 29, "Ğ": 30, "ğ": 31, "Ġ": 32, "!": 33, "\"": 34, "#": 35, "$": 36, "%": 37, "&": 38, "'": 39, "(": 40, ")": 41, "*": 42, "+": 43, ",": 44, "-": 45, ".": 46, "/": 47, "0": 48, "1": 49, "2": 50, "3": 51, "4": 52, "5": 53, "6": 54, "7": 55, "8": 56, "9": 57, ":": 58, ";": 59, "<": 60, "=": 61, ">": 62, "?": 63, "@": 64, "A": 65, "B": 66, "C": 67, "D": 68, "E": 69, "F": 70, "G": 71, "H": 72, "I": 73, "J": 74, "K": 75, "L": 76, "M": 77, "N": 78, "O": 79, "P": 80, "Q": 81, "R": 82, "S": 83, "T": 84, "U": 85, "V": 86, "W": 87, "X": 88, "Y": 89, "Z": 90, "[": 91, "\\": 92, "]": 93, "^": 94, "_": 95, "`": 96, "a": 97, "b": 98, "c": 99, "d": 100, "e": 101, "f": 102, "g": 103, "h": 104, "i": 105, "j": 106, "k": 107, "l": 108, "m": 109, "n": 110, "o": 111, "p": 112, "q": 113, "r": 114, "s": 115, "t": 116, "u": 117, "v": 118, "w": 119, "x": 120, "y": 121, "z": 122, "{": 123, "|": 124, "}": 125, "~": 126, "ġ": 127, "Ģ": 128, "ģ": 129, "Ĥ": 130, "ĥ": 131, "Ħ": 132, "ħ": 133, "Ĩ": 134, "ĩ": 135, "Ī": 136, "ī": 137, "Ĭ": 138, "ĭ": 139, "Į": 140, "į": 141, "İ": 142, "ı": 143, "IJ": 144, "ij": 145, "Ĵ": 146, "ĵ": 147, "Ķ": 148, "ķ": 149, "ĸ": 150, "Ĺ": 151, "ĺ": 152, "Ļ": 153, "ļ": 154, "Ľ": 155, "ľ": 156, "Ŀ": 157, "ŀ": 158, "Ł": 159, "ł": 160, "¡": 161, "¢": 162, "£": 163, "¤": 164, "¥": 165, "¦": 166, "§": 167, "¨": 168, "©": 169, "ª": 170, "«": 171, "¬": 172, "Ń": 173, "®": 174, "¯": 175, "°": 176, "±": 177, "²": 178, "³": 179, "´": 180, "µ": 181, "¶": 182, "·": 183, "¸": 184, "¹": 185, "º": 186, "»": 187, "¼": 188, "½": 189, "¾": 190, "¿": 191, "À": 192, "Á": 193, "Â": 194, "Ã": 195, "Ä": 196, "Å": 197, "Æ": 198, "Ç": 199, "È": 200, "É": 201, "Ê": 202, "Ë": 203, "Ì": 204, "Í": 205, "Î": 206, "Ï": 207, "Ð": 208, "Ñ": 209, "Ò": 210, "Ó": 211, "Ô": 212, "Õ": 213, "Ö": 214, "×": 215, "Ø": 216, "Ù": 217, "Ú": 218, "Û": 219, "Ü": 220, "Ý": 221, "Þ": 222, "ß": 223, "à": 224, "á": 225, "â": 226, "ã": 227, "ä": 228, "å": 229, "æ": 230, "ç": 231, "è": 232, "é": 233, "ê": 234, "ë": 235, "ì": 236, "í": 237, "î": 238, "ï": 239, "ð": 240, "ñ": 241, "ò": 242, "ó": 243, "ô": 244, "õ": 245, "ö": 246, "÷": 247, "ø": 248, "ù": 249, "ú": 250, "û": 251, "ü": 252, "ý": 253, "þ": 254, "ÿ": 255, "<bos>": 256, "<eos>": 257, "<pad>": 258}, "merges": []}}

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "pad_token": "<pad>",
+  "model_max_length": 4096,
+  "clean_up_tokenization_spaces": false
+}