| --- |
| language: |
| - en |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - inferno |
| - small-language-model |
| - gqa |
| - custom_code |
| - arithmark |
| datasets: |
| - AlgoDriveAI/SLM_v6 |
| model-index: |
| - name: Inferno-Exp-1.7 |
| results: |
| - task: |
| type: text-generation |
| name: Arithmetic word-problem continuation |
| dataset: |
| type: AxiomicLabs/Arithmark-3.0 |
| name: ArithMark 3.0 |
| split: test |
| revision: 6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746 |
| metrics: |
| - type: acc_norm |
| name: Length-normalized accuracy |
| value: 85.7 |
| - type: acc |
| name: Raw continuation accuracy |
| value: 85.6 |
| - task: |
| type: text-generation |
| name: Zero-shot continuation scoring |
| dataset: |
| type: Rowan/hellaswag |
| name: HellaSwag |
| split: validation |
| metrics: |
| - type: acc_norm |
| name: Normalized accuracy (0-shot) |
| value: 33.68 |
| - task: |
| type: text-generation |
| name: Zero-shot continuation scoring |
| dataset: |
| type: allenai/ai2_arc |
| name: ARC-Easy |
| split: test |
| metrics: |
| - type: acc_norm |
| name: Normalized accuracy (0-shot) |
| value: 51.6 |
| - task: |
| type: text-generation |
| name: Zero-shot continuation scoring |
| dataset: |
| type: allenai/ai2_arc |
| name: ARC-Challenge |
| split: test |
| metrics: |
| - type: acc_norm |
| name: Normalized accuracy (0-shot) |
| value: 34.04 |
| - task: |
| type: text-generation |
| name: Zero-shot continuation scoring |
| dataset: |
| type: ybisk/piqa |
| name: PIQA |
| split: validation |
| metrics: |
| - type: acc_norm |
| name: Normalized accuracy (0-shot) |
| value: 64.47 |
| --- |
| |
| # Inferno-Exp-1.7 |
|
|
| **Inferno-Exp-1.7** is AlgoDriveAI's 393.6M-parameter, math-first decoder-only base language model. |
| This release publishes the selected `32_indx.pt` snapshot at **continuation step 275,000**, |
| with **4,096-token context** and **85.70% ArithMark 3.0 normalized accuracy**. |
| It continues the Deep400M training line represented by [Inferno-Exp-1.6](https://huggingface.co/AlgoDriveAI/Inferno-Exp-1.6). |
| It is a base text-completion model; it has no chat template or instruction tuning. |
|
|
| | Property | Value | |
| |---|---| |
| | Unique parameters | 393,610,240 | |
| | Layers × width | 32 × 1024 | |
| | Attention | GQA: 16 query / 4 KV heads, head dimension 64 | |
| | Position encoding | Full-head, half-split RoPE; base 10000 | |
| | Feed-forward | Canonical same-token SwiGLU; inner width 2816 | |
| | Context | 4,096 tokens | |
| | Vocabulary | 32,064; Mistral tokenizer with training additions and padding | |
| | Embeddings | Input/output weights tied | |
| | Released weights | Float32 Safetensors | |
| | Source checkpoint | `32_indx.pt`, step 275,000, saved epoch index 1 | |
|
|
| Parameter accounting: 32,833,536 tied token-embedding parameters + 360,775,680 parameters |
| across 32 transformer blocks + 1,024 final normalization parameters = **393,610,240**. |
| The LM head is not counted twice. |
|
|
| ## Compared with Inferno-Exp-1.6 |
|
|
| | Benchmark | Inferno-Exp-1.6 | Inferno-Exp-1.7 | Change (points) | |
| |---|---:|---:|---:| |
| | ArithMark 3.0 (acc_norm) | 83.10% | **85.70%** | +2.60 | |
| | HellaSwag (acc_norm) | 29.80% | **33.68%** | +3.88 | |
| | ARC-Easy (acc_norm) | 47.01% | **51.60%** | +4.59 | |
| | ARC-Challenge (acc_norm) | 30.12% | **34.04%** | +3.92 | |
| | PIQA (acc_norm) | 60.45% | **64.47%** | +4.02 | |
| |
| The ArithMark result was verified during this export. The four language scores are the full-set |
| training evaluations recorded at the same step, 275,000. They are not a new post-export harness run. |
| The previous release reports language evaluation batch size 16; the training evaluations here use 8. |
| |
| Using these rounded scores and the training script's Open SLM Intelligence Index formula gives |
| **32.02**. This is a locally calculated summary, not an official leaderboard submission or rank. |
| |
| ## Usage |
| |
| ```python |
| # pip install "transformers==4.57.6" torch sentencepiece protobuf safetensors |
| import torch |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| repo = "AlgoDriveAI/Inferno-Exp-1.7" |
| tok = AutoTokenizer.from_pretrained(repo) |
| model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, |
| torch_dtype=torch.float32) |
| device = "cuda" if torch.cuda.is_available() else "cpu" |
| model = model.to(device).eval() |
| inputs = tok("Sam had 47 marbles. His sister gave him 28 more. Now Sam has", |
| return_tensors="pt", add_special_tokens=False).to(device) |
| with torch.inference_mode(): |
| output = model.generate(**inputs, max_new_tokens=80, do_sample=False, |
| use_cache=False, pad_token_id=tok.pad_token_id) |
| print(tok.decode(output[0], skip_special_tokens=True)) |
| ``` |
| |
| The custom Transformers implementation is reused from the pinned 1.6 release, with this |
| checkpoint's 4,096-token configuration and new weights. The export checks strict tensor loading, |
| compares its logits with the supplied trainer, reloads the saved Safetensors, and scores the |
| exported model on ArithMark. Generation re-reads the context each step; no KV cache is implemented. |
| |
| ## Try it: interactive demo (Gradio) |
| |
| Run a small web app with a prompt box, sampling sliders, live streaming, a **Stop** button, |
| and token/speed readouts: |
| |
| 1. Open a new notebook at [Google Colab](https://colab.research.google.com/) and choose a GPU |
| (**Runtime → Change runtime type → T4 GPU**). CPU also works, but generation is slower. |
| 2. Expand the script below, paste it into one code cell, and run it. |
| 3. Open the `*.gradio.live` link printed by the cell. |
| |
| The script loads the published Inferno-Exp-1.7 checkpoint and reads its **4,096-token context** |
| from the model configuration. The public model needs no Hugging Face token. |
| If the cell asks for a restart after changing Transformers, use **Runtime → Restart session** |
| and run that cell again. |
| |
| <details> |
| <summary>Show the Gradio demo script (one Colab cell)</summary> |
| |
| ```python |
| #!/usr/bin/env python3 |
| # -*- coding: utf-8 -*- |
| """ |
| Inferno-Exp-1.7 — simple Gradio inference (loads like any Hub user, no pushing) |
| ============================================================================== |
| Loads Inferno-Exp-1.7 with plain Transformers (AutoModelForCausalLM + |
| trust_remote_code=True) and serves a small Gradio app: open prompt box, |
| sampling controls, streaming output with a Stop button, token/speed readout. |
| |
| Works with either source: |
| • the Hub repo: MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.7" |
| • the local export folder the push script builds (test before uploading): |
| MODEL_SOURCE = "/content/Inferno-Exp-1.7-release" |
| |
| The public 1.7 model needs no token. For a private alternative, use an existing |
| Hub login, set the HF_TOKEN environment variable, or set HF_TOKEN below. |
|
|
| It is a base model: it continues text, it does not chat. Write the start of the text |
| you want (e.g. a word problem followed by "Answer:"). |
| Hardcoded constants, no argparse. Paste into one Colab cell and run; open the share link. |
| """ |
| # ============================================================================= |
| # SETTINGS — edit these |
| # ============================================================================= |
| MODEL_SOURCE = "AlgoDriveAI/Inferno-Exp-1.7" # Hub repo id (or a local folder with the same files) |
| MODEL_REVISION = "a24c67451dc7df3b3069e9ce54724ca815d96fda" # verified 1.7 weights |
| HF_TOKEN = "" # optional; otherwise use env var / existing Hub login |
| MODEL_DISPLAY_NAME = "Inferno-Exp-1.7" |
| |
| # Default slider values (all adjustable in the UI) |
| DEF_MAX_NEW_TOKENS = 256 |
| DEF_TEMPERATURE = 0.8 |
| DEF_TOP_K = 40 |
| DEF_TOP_P = 0.95 |
| DEF_REPETITION_PENALTY = 1.1 |
| DEF_SEED = -1 # -1 = different every run |
| DTYPE = "auto" # "auto" (bf16 on newer GPUs, fp32 on CPU) / "float32" / "bfloat16" / "float16" |
|
|
| GRADIO_SHARE = True # public *.gradio.live link (needed in Colab) |
| GRADIO_PORT = 7860 |
| INSTALL_DEPENDENCIES = True |
| |
| # ============================================================================= |
| # RUN (no edits needed below) |
| # ============================================================================= |
| import os, sys, time, threading, subprocess, queue |
| os.environ.setdefault("USE_TF", "0") |
| os.environ.setdefault("TOKENIZERS_PARALLELISM", "false") |
| |
| try: |
| import google.colab # noqa: F401 |
| IN_COLAB = True |
| except ImportError: |
| IN_COLAB = False |
| if IN_COLAB and INSTALL_DEPENDENCIES: |
| import importlib.metadata as _md |
| |
| def _version(name): |
| try: |
| return _md.version(name) |
| except _md.PackageNotFoundError: |
| return None |
| |
| _before = _version("transformers") |
| subprocess.check_call([sys.executable, "-m", "pip", "install", "--quiet", |
| "transformers==4.57.6", "safetensors", "huggingface_hub>=0.25,<1", |
| "gradio>=4.44,<7", "sentencepiece", "protobuf"]) |
| if _before != _version("transformers") and "transformers" in sys.modules: |
| raise SystemExit("Transformers was just changed to a compatible version, but the old one is still loaded. " |
| "Restart the session (Runtime → Restart session) and run this cell again.") |
| |
| import torch |
| import gradio as gr |
| from transformers import (AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList, |
| TextIteratorStreamer) |
| |
| DEVICE = "cuda" if torch.cuda.is_available() else "cpu" |
| STOP_EVENT = threading.Event() |
|
|
|
|
| def get_token(): |
| # token=None lets the Hub use an existing login; public downloads work without one. |
| return HF_TOKEN.strip() or os.environ.get("HF_TOKEN") or None |
| |
| |
| def pick_dtype(): |
| if DTYPE != "auto": |
| return {"float32": torch.float32, "bfloat16": torch.bfloat16, "float16": torch.float16}[DTYPE] |
| if DEVICE == "cuda": |
| return torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16 |
| return torch.float32 |
| |
|
|
| class StopOnEvent(StoppingCriteria): |
| """Ends generate() as soon as the Stop button sets STOP_EVENT.""" |
| def __call__(self, input_ids, scores, **kwargs): |
| return STOP_EVENT.is_set() |
| |
|
|
| def load(): |
| token = None if os.path.isdir(MODEL_SOURCE) else get_token() |
| dtype = pick_dtype() |
| revision = None if os.path.isdir(MODEL_SOURCE) else MODEL_REVISION |
| tok = AutoTokenizer.from_pretrained(MODEL_SOURCE, revision=revision, token=token, use_fast=True) |
| if tok.pad_token_id is None: |
| tok.pad_token = tok.eos_token |
| model = AutoModelForCausalLM.from_pretrained(MODEL_SOURCE, revision=revision, token=token, trust_remote_code=True, |
| torch_dtype=dtype).to(DEVICE).eval() |
| return model, tok, dtype |
| |
|
|
| MODEL, TOK, MODEL_DTYPE = load() |
| CFG = MODEL.config |
| N_PARAMS = sum(p.numel() for p in MODEL.parameters()) |
| CONTEXT = CFG.max_position_embeddings |
| print(f"Loaded {MODEL_SOURCE}: {N_PARAMS / 1e6:.1f}M params · {DEVICE} · {MODEL_DTYPE}") |
| |
| |
| def generate(prompt, max_new, temperature, top_k, top_p, rep_penalty, seed): |
| STOP_EVENT.clear() |
| if not prompt: |
| yield "", "Type a prompt first." |
| return |
| ids = TOK(prompt, return_tensors="pt", add_special_tokens=False).to(DEVICE) |
| room = CONTEXT - ids["input_ids"].shape[1] |
| if room <= 0: |
| yield prompt, f"Prompt is longer than the model's {CONTEXT}-token context; shorten it." |
| return |
| seed = int(seed) |
| if seed >= 0: |
| torch.manual_seed(seed) |
| greedy = temperature <= 0 |
| sampling = {} if greedy else dict(temperature=float(temperature), top_k=int(top_k), top_p=float(top_p)) |
| streamer = TextIteratorStreamer(TOK, skip_prompt=False, skip_special_tokens=True, timeout=1.0) |
| result = {} |
| |
| def worker(): |
| try: |
| with torch.inference_mode(): |
| result["ids"] = MODEL.generate( |
| **ids, max_new_tokens=min(int(max_new), room), do_sample=not greedy, |
| repetition_penalty=float(rep_penalty), eos_token_id=TOK.eos_token_id, pad_token_id=TOK.pad_token_id, |
| use_cache=False, streamer=streamer, stopping_criteria=StoppingCriteriaList([StopOnEvent()]), **sampling) |
| except Exception as exc: |
| result["error"] = exc |
| streamer.end() # unblock the UI if the generation worker fails |
| |
|
|
| started = time.time() |
| thread = threading.Thread(target=worker, daemon=True) |
| thread.start() |
| text = "" |
| try: |
| while True: |
| try: |
| piece = next(streamer) |
| except queue.Empty: |
| if not thread.is_alive(): |
| break |
| continue |
| except StopIteration: |
| break |
| text += piece |
| yield text, f"generating… {time.time() - started:.1f}s" |
| thread.join() |
| finally: |
| if thread.is_alive(): |
| STOP_EVENT.set() |
| if "error" in result: |
| exc = result["error"] |
| yield text or prompt, f"Generation failed: {type(exc).__name__}: {exc}" |
| return |
| new = result["ids"].shape[1] - ids["input_ids"].shape[1] if "ids" in result else 0 |
| took = time.time() - started |
| status = "stopped" if STOP_EVENT.is_set() else "done" |
| yield text or prompt, (f"**{status}** · {new} tokens · {took:.1f}s · {new / max(took, 1e-6):.1f} tok/s · " |
| f"{'greedy' if greedy else f'T={temperature} · top-k {int(top_k)} · top-p {top_p}'} · seed {seed}") |
| |
|
|
| def output_box(): |
| kwargs = dict(label="Output (prompt + continuation)", lines=16, max_lines=30, interactive=False) |
| for extra in (dict(show_copy_button=True), dict(buttons=["copy"]), {}): |
| try: |
| return gr.Textbox(**kwargs, **extra) |
| except TypeError: |
| continue |
| |
|
|
| def build_ui(): |
| gr_major = int(gr.__version__.split(".")[0]) |
| theme = gr.themes.Soft(primary_hue="indigo", secondary_hue="blue") |
| blocks_kwargs = {} if gr_major >= 6 else dict(theme=theme) # Gradio 6 moved theme to launch() |
| launch_kwargs = dict(theme=theme) if gr_major >= 6 else {} |
| with gr.Blocks(title=MODEL_DISPLAY_NAME, **blocks_kwargs) as demo: |
| gr.Markdown(f"# 🔥 {MODEL_DISPLAY_NAME}\n" |
| f"{N_PARAMS / 1e6:.1f}M-parameter math-first base model · `{CFG.num_hidden_layers}L × {CFG.hidden_size}d` · " |
| f"{CFG.num_attention_heads}Q/{CFG.num_key_value_heads}KV heads · context {CONTEXT} · " |
| f"continues text (not a chat model)") |
| with gr.Row(): |
| with gr.Column(scale=3): |
| prompt = gr.Textbox(label="Prompt", lines=6, placeholder="Write the beginning of the text to continue…") |
| with gr.Row(): |
| run = gr.Button("Generate", variant="primary") |
| stop = gr.Button("Stop", variant="stop") |
| output = output_box() |
| status = gr.Markdown("") |
| with gr.Column(scale=1): |
| max_new = gr.Slider(16, CONTEXT, value=min(DEF_MAX_NEW_TOKENS, CONTEXT), step=16, label="Max new tokens") |
| temperature = gr.Slider(0.0, 1.5, value=DEF_TEMPERATURE, step=0.05, label="Temperature (0 = greedy)") |
| top_k = gr.Slider(0, 200, value=DEF_TOP_K, step=1, label="Top-k (0 = off)") |
| top_p = gr.Slider(0.05, 1.0, value=DEF_TOP_P, step=0.01, label="Top-p") |
| rep = gr.Slider(1.0, 1.5, value=DEF_REPETITION_PENALTY, step=0.01, label="Repetition penalty") |
| seed = gr.Number(value=DEF_SEED, precision=0, label="Seed (-1 = random)") |
| with gr.Accordion("Model info", open=False): |
| gr.Markdown(f"- **Source:** `{MODEL_SOURCE}`\n- **Parameters:** `{N_PARAMS:,}`\n" |
| f"- **Device / dtype:** `{DEVICE}` / `{str(MODEL_DTYPE).replace('torch.', '')}`\n" |
| f"- **Vocab:** `{CFG.vocab_size:,}` · EOS `{TOK.eos_token}`") |
| inputs = [prompt, max_new, temperature, top_k, top_p, rep, seed] |
| events = dict(inputs=inputs, outputs=[output, status], concurrency_limit=1, concurrency_id="inferno-generation") |
| run.click(generate, **events) |
| prompt.submit(generate, **events) |
| stop.click(lambda: STOP_EVENT.set(), queue=False) |
| return demo, launch_kwargs |
| |
|
|
| if __name__ == "__main__": |
| demo, launch_kwargs = build_ui() |
| demo.queue().launch(share=GRADIO_SHARE, server_port=GRADIO_PORT, server_name="0.0.0.0", **launch_kwargs) |
| ``` |
| |
| </details> |
|
|
| [Download the same script](inference/gradio_demo.py). |
|
|
| For this base model, write the beginning of the text you want continued, for example: |
| `Sam had 47 marbles. His sister gave him 28 more. Now Sam has`. |
| Set temperature to **0** for greedy decoding, or increase it for more varied stories. |
|
|
| ## Evaluation |
|
|
| All reported accuracies are percentages. Chance is 25% for ArithMark, HellaSwag and ARC; |
| 50% for PIQA. |
|
|
| ### ArithMark 3.0 |
|
|
| The official, unmodified [AxiomicLabs scorer](https://huggingface.co/datasets/AxiomicLabs/Arithmark-3.0/blob/6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746/bencharithmark-3.py) |
| at commit `6f6e59dd9b7e2c63455f7af7f838f9ecc3d0a746` scores 1,000 four-choice continuation word problems. |
| Settings: float32 weights, TF32 enabled as in the official entry point, fast training tokenizer, |
| no added BOS token, batch size 8 examples (32 choices), max context 1,024, |
| raw summed and per-token length-normalized log-likelihood. No few-shot examples. |
|
|
| | Metric | Score | |
| |---|---:| |
| | Normalized accuracy (primary) | **85.70%** (857/1000) | |
| | Raw accuracy | **85.60%** (856/1000) | |
| | Binomial standard error (normalized) | ±1.11 points | |
| | Random-choice baseline | 25.00% | |
|
|
| Dataset SHA-256: `bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435`. |
| Full scores, predictions, category data and runtime settings are in |
| [`evaluation/arithmark-3_results.json`](evaluation/arithmark-3_results.json). |
|
|
| #### Per-category results |
|
|
| | Category | N | Raw | Normalized | |
| |---|---:|---:|---:| |
| | elementary_school_math_continuation::addition::grades_1_2::easy | 128 | 90.62% | 90.62% | |
| | elementary_school_math_continuation::comparison::grades_2_3::medium | 44 | 100.00% | 100.00% | |
| | elementary_school_math_continuation::comparison_difference::grades_2_3::medium | 48 | 100.00% | 100.00% | |
| | elementary_school_math_continuation::data::grades_2_3::easy | 43 | 90.70% | 90.70% | |
| | elementary_school_math_continuation::division::grades_3_4::medium | 54 | 85.19% | 85.19% | |
| | elementary_school_math_continuation::fractions_counting::grades_3_4::medium | 50 | 58.00% | 58.00% | |
| | elementary_school_math_continuation::geometry_area::grades_4_5::medium | 52 | 100.00% | 100.00% | |
| | elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium | 45 | 80.00% | 80.00% | |
| | elementary_school_math_continuation::measurement::grades_2_3::easy | 76 | 96.05% | 96.05% | |
| | elementary_school_math_continuation::money::grades_3_4::medium | 64 | 100.00% | 100.00% | |
| | elementary_school_math_continuation::multiplication::grades_3_4::medium | 74 | 100.00% | 100.00% | |
| | elementary_school_math_continuation::patterns::grades_3_4::medium | 53 | 84.91% | 84.91% | |
| | elementary_school_math_continuation::subtraction::grades_1_2::easy | 117 | 77.78% | 77.78% | |
| | elementary_school_math_continuation::time::grades_2_3::easy | 55 | 100.00% | 100.00% | |
| | elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium | 46 | 34.78% | 34.78% | |
| | elementary_school_math_continuation::two_step_addition::grades_2_3::medium | 19 | 94.74% | 94.74% | |
| | elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium | 32 | 31.25% | 34.38% | |
|
|
| ### Language understanding |
|
|
| Source: [`evaluation/language_step_275000.json`](evaluation/language_step_275000.json), |
| selected from the trainer's `lmeval_history.json` at the released checkpoint's step. |
| `lm-evaluation-harness`: zero-shot, full sets (`limit=None`), `acc_norm`, float32, |
| batch size 8, max length 4096. Item counts below are the full task split sizes. |
| Per-example predictions and harness standard errors were not retained by that training logger. |
|
|
| | Task | Items | acc_norm | |
| |---|---:|---:| |
| | HellaSwag (validation) | 10,042 | 33.68% | |
| | ARC-Easy (test) | 2,376 | 51.60% | |
| | ARC-Challenge (test) | 1,172 | 34.04% | |
| | PIQA (validation) | 1,838 | 64.47% | |
| |
| To run a fresh evaluation of the published model: |
| |
| ```bash |
| lm_eval --model hf \ |
| --model_args "pretrained=AlgoDriveAI/Inferno-Exp-1.7,trust_remote_code=True,dtype=float32" \ |
| --tasks hellaswag,arc_easy,arc_challenge,piqa --num_fewshot 0 --batch_size 8 |
| ``` |
| |
| ### Tokenizer reproducibility |
| |
| The primary ArithMark result uses `LlamaTokenizerFast` from the exact saved training tokenizer. |
| The earlier notebook runner described its requested tokenizer as “slow”; that label did not |
| establish the actual tokenizer backend in its newer Transformers environment. During release |
| verification under pinned Transformers 4.57.6, the fast tokenizer reproduced the reported |
| 85.70% normalized / 85.60% raw result, including all 17 category scores. A genuine |
| `LlamaTokenizer` slow-tokenizer check with the same weights gave 85.80% normalized / 86.00% raw. |
| These are different tokenization paths, not different weights. The slow-tokenizer reference is |
| retained in `evaluation/arithmark-3_slow-tokenizer-reference.json` and is not mixed into the |
| primary result. In this release, the actual tokenizer class is recorded explicitly. |
|
|
| - [`evaluation/evaluate_arithmark3.py`](evaluation/evaluate_arithmark3.py): standalone evaluation runner |
| configured for this exact snapshot. |
| - [`release_manifest.json`](release_manifest.json): SHA-256 and size for every release artifact. |
|
|
| Exact supplied training source SHA-256: `ce792e3792a02edc3f31584294f4dfc68282d2f07f945168ec7ec905abf305f9`. |
| The source is supplied for reproducibility; its historical default settings are not rewritten to |
| pretend they were the saved checkpoint configuration. The released inference weights omit |
| optimizer/RNG state. |
|
|
| The root tokenizer uses the same vocabulary/EOS as training, with automatic BOS insertion |
| disabled and BOS set to the training document EOS for tools that need a start-token ID, |
| following the 1.6 export. |
|
|
| ## Limitations |
|
|
| This math-first base model can produce incorrect arithmetic and unreliable text. Its general |
| language results are modest, and it is not trained to behave as a chat assistant. |
| ArithMark was monitored during training and checkpoint selection, so its score is not an |
| untouched holdout estimate. The four language tasks were also monitored during training. |
|
|
