N-Q-26M-Chat

A 26M-class Russian/English chat model trained from scratch on one V100, with exact tool calls

This is a research release, not a product. At 26M parameters the model is fluent in form but often factually wrong, repetitive under greedy decoding, and it does not solve multi-step word problems (GSM8K ≈ 2%). What it does reliably is call a tool: arithmetic, dates, units, sorting and the like are computed exactly by a fixed circuit, not by the network. See Honest results before drawing conclusions.

What this is

N-Q is the Q Project's take on MiniMind: a ladder of small Russian/English language models trained from scratch on a single 16GB V100, with the whole pipeline — tokenizer → pretrain → SFT → reasoning → LoRA → DPO → GRPO/PPO → agentic RL → distillation — in plain, readable PyTorch. This is the chat model of the smallest rung; the base model is N-Q-26M.

The architecture was chosen by an ablation, not assumed. At 26M and 0.5B tokens each (identical data, schedule and batch) we trained three candidates:

candidate	held-out loss (EN / RU-edu / RU-wiki, mean)	V100 throughput (eager)
A — Q-164M design (ternary weights, frozen 512-bit fingerprint vocabulary, GQA)	3.061	~111k tok/s
B — mini Qwen3.8-Flash-Next (Gated DeltaNet 3:1 gated attention, Gated Residual, n-gram embedding, Muon)	2.616	~19k tok/s (→ 85k with torch.compile)
C — B + ternary weights + fingerprint vocabulary	2.990 @ 4000 steps, behind B at every eval	~17k tok/s

B won by 0.45 nats over A and ~0.2 over C at every checkpoint, and is the architecture of the whole ladder. Q's exact circuits are kept in every variant: the model writes <CALC>expr<EQ>, a deterministic executor writes the exact result and <ECALC>, and the model continues from there.

Honest results

capability	result	note
fixed 16-task tool-calling probe (8 RU + 8 EN, greedy)	15/16	the miss: "30 days after January 15, 2027" — the circuit computes 2027-02-14 correctly, the model then writes "June 14" in its sentence
GSM8K test, exact final number (500 EN / 500 RU, thinking on)	2.0% / 2.2%	essentially not solved: the model calls the calculator correctly but cannot plan multi-step solutions; some hits are lucky final numbers
GSM8K after GRPO / agentic RL on top of this model	2.2% / 2.0% · 1.6% / 1.4%	within noise (±0.6 pp) — at this size RL has almost no signal to learn from, so the released chat model is the DPO checkpoint
base-model held-out loss (EN / RU-edu / RU-wiki)	2.381 / 2.651 / 2.323	perplexity 10.8 / 14.2 / 10.2 on this 8,256-token tokenizer (not comparable across tokenizers)

An earlier version of our evaluation scored GSM8K at 16–20% because a numeric reference counted as correct when it appeared anywhere in the answer as a substring ("84" inside "184"). That was a scoring bug, not a capability; the numbers above use the final-number check only.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("q-project/N-Q-26M-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("q-project/N-Q-26M-Chat", trust_remote_code=True)

msgs = [{"role": "user", "content": "Сколько будет 4821 умножить на 37?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(ids, max_new_tokens=128, temperature=0.7, top_p=0.9, tokenizer=tok)   # tokenizer=tok switches the exact circuits on
print(tok.decode(out[0, ids["input_ids"].shape[1]:], skip_special_tokens=False))
# <CALC>4821*37<EQ>178377<ECALC><ETHINK>4821 умножить на 37 равно 178377.<EOT>

• Pass tokenizer=tok. Without it there is no executor and the model has to "compute" the number after <EQ> itself — it will usually be wrong (4821*37 → 104621).
• Thinking: apply_chat_template(..., enable_thinking=True) opens a <THINK> block; leaving it unset (or False) lets the model decide. In this checkpoint False does not force a no-thinking answer — the SFT data never contained an empty <THINK><ETHINK> block, so forcing one is out of distribution.
• The model has its own generate() (recurrent Gated DeltaNet state + KV cache for the attention layers); it accepts temperature, top_p, top_k, repetition_penalty, max_new_tokens, stop_ids (default <EOT>, <eos>).

Packed single-file format (model.qpack)

The same QPACK1 container as Q-164M and Q-Omni: config, tokenizer, generation config, weights and a self-test in one mmap-friendly file (89 MiB vs 177 MiB safetensors).

• A repo or folder that has model.qpack and no *.safetensors loads with AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True); explicit: NQ4ForCausalLM.from_qpack("model.qpack").
• python qpack.py --selftest model.qpack prints SELFTEST 5/5 (top-1 and top-5 logits against the original fp32 model).
• Matrices are stored in fp16; 1-D tensors (norms, decay parameters) in fp32; the n-gram hash tables (prime sizes, offsets, multipliers) as exact int64 — a float round-trip would silently change the hashes. Greedy generation from model.qpack alone was identical to the safetensors weights on all prompts we checked.
• There is no torch-free runtime for this architecture yet (Q-Omni's qpack_run.py implements Q-Omni only).

Architecture

• 8 layers in the pattern LLLA ×2: three Gated DeltaNet linear-attention layers (3 key / 6 value heads, head dim 64, short causal conv k=4, sigmoid-gated RMSNorm output) then one gated attention layer (8 query / 2 KV heads, head dim 64, zero-centred QK-norm, partial RoPE 25%, θ=1e6, elementwise sigmoid output gate).
• hidden 512, SwiGLU MLP 512→704→512, Gated Residual with 4 parallel residual streams, zero-centred RMSNorm.
• Hashed n-gram embedding (PLE) injected after layer 2: 3-gram hashes into prime-sized tables, 16.8M parameters counted separately from the 29.7M backbone (the same accounting Qwen uses for its n-gram tables).
• Trained with Muon on the matrices + AdamW on the rest, WSD schedule, fp16 autocast on V100; torch.compile took training from 19k to 85k tokens/s.
• Context: 4096 positions in the config, trained at 1024.

Token layout

ids	content
0–8191	byte-level BPE, Russian + English, digits split one per token; 0 <pad>, 1 <bos>, 2 <eos>
8192–8255	control: <USER> <MODEL> <EOT> <THINK> <ETHINK> <CALC> <EQ> <ECALC> <NEED> <QUOTE> <SYSTEM> and reserved <R11>…<R63>

Chat format: <bos>[<SYSTEM>…<EOT>]<USER>…<EOT><MODEL>[<THINK>…<ETHINK>]answer<EOT>.

Training

stage	data	V100 time
pretrain (base model)	2.0B tokens: 50% FineWeb-Edu (EN), 40% Cultura-Ru-Edu, 10% Russian Wikipedia; 5% of rows are dialogues / <CALC> rows from step 1	6.6 h
SFT	150k conversations (SmolTalk, GrandMaster-PRO-MAX, saiga_scored, own <CALC> corpus RU/EN, identity)	49 min
reasoning	60k: 60% <THINK> traces (russian-reasoning), 40% SFT mix	18 min
DPO (this checkpoint)	19.6k preference pairs (saiga_preferences, UltraFeedback), β=0.1	50 min

GRPO, PPO and agentic-RL checkpoints were trained too (GSM8K / GSM8K-ru, verifiable rewards, circuits live during rollouts) and evaluated above; they are not released because they do not improve on this one at 26M.

Limitations

• Facts are frequently wrong ("the capital of France is…" goes off the rails); thin general knowledge; repetition under greedy decoding; long answers drift; weak creative writing.
• Does not solve multi-step math word problems; the circuits make single computations exact, not reasoning.
• The model sometimes ignores a correct circuit result when writing its final sentence (see the date example).
• enable_thinking=False is not a hard switch in this checkpoint (see Usage).
• No safety tuning. Research and education only.

License and data terms

Code and weights: Apache-2.0 (see LICENSE). Training data keeps its own terms — check each dataset card before commercial use: FineWeb-Edu (ODC-By 1.0), Cultura-Ru-Edu, Russian Wikipedia (CC BY-SA / GFDL), SmolTalk, GrandMaster-PRO-MAX, saiga_scored, saiga_preferences, russian-reasoning, UltraFeedback, GSM8K / GSM8K-ru.