Têtu v1
Têtu v1 is a small French language model with recurrent depth: a shared core is applied several times per token before the next word is produced. It was trained from scratch on a single consumer GPU (RTX 5070, 12 GB). A doubt head reads how the recurrent state converges and estimates, for every generated token, the probability that it is wrong.
This repository contains version 1.1, which extends the context to 32,768 tokens.
Model details
| Developer | tetu-AI |
| Architecture | Decoder-only transformer with a looped mixture-of-experts core |
| Parameters | 181M total, 96M active per iteration |
| Context length | 32,768 tokens |
| Languages | French first, English |
| Vocabulary | 32,768 tokens (French and English, digits split individually) |
| Weights | bfloat16 (model.safetensors, 393 MB) |
| License | CC BY-NC 4.0 |
The core is applied 8 times per token by default (configurable at inference). It attends to a sliding window of recent tokens, while the first and last blocks see the full context. Inference includes an exact calculator for the arithmetic the model writes, and an optional reasoning mode that samples 16 answers in parallel and takes a vote, with the doubt head as tie-breaker.
Training
| Stage | Data |
|---|---|
| Pre-training | About 2B tokens: FineWeb-Edu (English), FineWeb-2 (French), FineMath |
| Instruction tuning | Filtered French conversations; program-generated and verified data (school math, word problems, multi-turn follow-ups, math concepts; every equation is recomputed before training); GSM8K train |
| Long-context tuning (v1.1) | Synthetic long conversations with questions about their beginning, long documents, position skipping |
GSM8K test and MGSM were not used for training.
Evaluation
| Benchmark | v1.0 | v1.1 |
|---|---|---|
| Multi-turn follow-ups (1,475 turns)* | 93.2 % | 94.0 % |
| Math concepts, held-out phrasings (350)* | 75.4 % | 83.1 % |
| Template math (215)* | 79.1 % | 80.9 % |
| MGSM-fr, greedy (250) | 4.4 % | 4.8 % |
| MGSM-fr, 16-sample vote (250) | 10.4 % | 11.2 % |
* Test sets from the same generators as the training data, with phrasings reserved for testing. They measure skills the model was taught, not general ability.
Comparisons between versions are paired (sign test) with the same decoder. For reference, Qwen2.5-0.5B scores 17.2 % on MGSM-fr with the same protocol. v1.0 is the previous release and is not included here.
Recall of a fact stated at the beginning of a conversation: 98 % at about 830 tokens, 88 % at about 3,600 and 64 % at about 29,600 (v1.0: 54 %, 40 % and 18 %).
Limitations
- It confidently invents facts on rare topics (see the Les Misérables example above). The doubt head often flags these tokens, but not always. Do not rely on it for factual, medical, legal or financial use.
- Multi-step reasoning on new word problems is weak (at most 11 % on MGSM-fr).
- Its conversation data is mostly French; English chat is limited and it was not trained on code.
- With about 2B training tokens, its general knowledge is far below models trained on trillions of tokens.
Usage
The architecture is not supported by transformers; use the bundled inference code.
git clone https://huggingface.co/tetu-AI/Tetu-v1 && cd Tetu-v1
pip install -r requirements.txt
python chat.py # --cpu without a GPU; --loops N to change the number of iterations
In the chat, /think toggles the reasoning mode (16 parallel samples and a vote) and /reset clears the
conversation. On GPU, word problems switch to the reasoning mode automatically.
import json, torch
from safetensors.torch import load_file
from tetu.model import Tetu, TetuConfig
meta = json.load(open("config.json"))
model = Tetu(TetuConfig(**meta["config"])).eval()
model.load_state_dict({k: v.float() for k, v in load_file("model.safetensors").items()})
License
CC BY-NC 4.0. The instruction-tuning mix includes non-commercial data (Alpaca) and synthetic conversations.
Citation
@misc{tetu2026,
title = {Têtu: a small French looped language model with a doubt head},
author = {tetu-AI},
year = {2026},
url = {https://huggingface.co/tetu-AI/Tetu-v1}
}
- Downloads last month
- 40
Evaluation results
- accuracy (greedy) on MGSM (French)test set self-reported4.800
- accuracy (16-sample majority vote) on MGSM (French)test set self-reported11.200



