Têtu v1

Têtu v1 is a small French language model with recurrent depth: a shared core is applied several times per token before the next word is produced. It was trained from scratch on a single consumer GPU (RTX 5070, 12 GB). A doubt head reads how the recurrent state converges and estimates, for every generated token, the probability that it is wrong.

This repository contains version 1.1, which extends the context to 32,768 tokens.

Doubt head on unedited outputs

Model details

Developer tetu-AI
Architecture Decoder-only transformer with a looped mixture-of-experts core
Parameters 181M total, 96M active per iteration
Context length 32,768 tokens
Languages French first, English
Vocabulary 32,768 tokens (French and English, digits split individually)
Weights bfloat16 (model.safetensors, 393 MB)
License CC BY-NC 4.0

Architecture

The core is applied 8 times per token by default (configurable at inference). It attends to a sliding window of recent tokens, while the first and last blocks see the full context. Inference includes an exact calculator for the arithmetic the model writes, and an optional reasoning mode that samples 16 answers in parallel and takes a vote, with the doubt head as tie-breaker.

Training

Stage Data
Pre-training About 2B tokens: FineWeb-Edu (English), FineWeb-2 (French), FineMath
Instruction tuning Filtered French conversations; program-generated and verified data (school math, word problems, multi-turn follow-ups, math concepts; every equation is recomputed before training); GSM8K train
Long-context tuning (v1.1) Synthetic long conversations with questions about their beginning, long documents, position skipping

GSM8K test and MGSM were not used for training.

Evaluation

Evaluation results

Benchmark v1.0 v1.1
Multi-turn follow-ups (1,475 turns)* 93.2 % 94.0 %
Math concepts, held-out phrasings (350)* 75.4 % 83.1 %
Template math (215)* 79.1 % 80.9 %
MGSM-fr, greedy (250) 4.4 % 4.8 %
MGSM-fr, 16-sample vote (250) 10.4 % 11.2 %

* Test sets from the same generators as the training data, with phrasings reserved for testing. They measure skills the model was taught, not general ability.

Comparisons between versions are paired (sign test) with the same decoder. For reference, Qwen2.5-0.5B scores 17.2 % on MGSM-fr with the same protocol. v1.0 is the previous release and is not included here.

Long-context recall

Recall of a fact stated at the beginning of a conversation: 98 % at about 830 tokens, 88 % at about 3,600 and 64 % at about 29,600 (v1.0: 54 %, 40 % and 18 %).

Limitations

  • It confidently invents facts on rare topics (see the Les Misérables example above). The doubt head often flags these tokens, but not always. Do not rely on it for factual, medical, legal or financial use.
  • Multi-step reasoning on new word problems is weak (at most 11 % on MGSM-fr).
  • Its conversation data is mostly French; English chat is limited and it was not trained on code.
  • With about 2B training tokens, its general knowledge is far below models trained on trillions of tokens.

Usage

The architecture is not supported by transformers; use the bundled inference code.

git clone https://huggingface.co/tetu-AI/Tetu-v1 && cd Tetu-v1
pip install -r requirements.txt
python chat.py            # --cpu without a GPU; --loops N to change the number of iterations

In the chat, /think toggles the reasoning mode (16 parallel samples and a vote) and /reset clears the conversation. On GPU, word problems switch to the reasoning mode automatically.

import json, torch
from safetensors.torch import load_file
from tetu.model import Tetu, TetuConfig

meta = json.load(open("config.json"))
model = Tetu(TetuConfig(**meta["config"])).eval()
model.load_state_dict({k: v.float() for k, v in load_file("model.safetensors").items()})

License

CC BY-NC 4.0. The instruction-tuning mix includes non-commercial data (Alpaca) and synthetic conversations.

Citation

@misc{tetu2026,
  title  = {Têtu: a small French looped language model with a doubt head},
  author = {tetu-AI},
  year   = {2026},
  url    = {https://huggingface.co/tetu-AI/Tetu-v1}
}
Downloads last month
40
Safetensors
Model size
0.2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Evaluation results