Tetu-Code-2 / README.md
tetu-AI's picture
cleaning
941a017 verified
|
Raw History Blame Contribute Delete
6.63 kB
---
license: apache-2.0
language:
- en
- fr
- code
pipeline_tag: text-generation
tags:
- code
- coding-agent
- small-language-model
- trained-from-scratch
- custom-architecture
- tool-use
- opencode
model-index:
- name: Tetu-Code-2
results:
- task:
type: text-generation
dataset:
name: HumanEval+
type: evalplus/humanevalplus
metrics:
- type: pass@1
name: pass@1 (greedy, chat prompt)
value: 15.9
- task:
type: text-generation
dataset:
name: MBPP+
type: evalplus/mbppplus
metrics:
- type: pass@1
name: pass@1 (greedy, chat prompt)
value: 30.7
---
# Têtu Code 2
Têtu Code 2 is a small coding agent trained from scratch. It is designed to run in the terminal through
[OpenCode](https://opencode.ai), where it reads and edits files, runs commands and reports what it did.
The model was trained from random initialization; no pretrained weights were used. Qwen3.6-35B-A3B (Apache 2.0)
served as a teacher for the agent demonstrations, and every demonstration was checked by a program before training.
Training ran on a single RTX 5070 (12 GB) plus about 30 hours of cloud GPU time.
## Model details
| | |
|---|---|
| Developer | tetu-AI |
| Architecture | Decoder-only transformer (custom) |
| Parameters | 1.15B total, 475M active per token (the rest is a lookup memory with negligible compute cost) |
| Context length | 65,536 tokens |
| Languages | Python first, plus HTML, CSS, JavaScript and others; instructions in English or French |
| Vocabulary | 32,768 tokens |
| Weights | bfloat16 (`model.safetensors`) |
| License | Apache 2.0 |
Capabilities:
- tool use through OpenCode: read, write, edit, bash, glob, grep;
- a short written reasoning between `<|think|>` and `<|/think|>` before each action;
- a per-token confidence head that estimates the probability of an error;
- an OpenAI-compatible server (`tetu_code.serve`) for the OpenCode 1.x and 2.x tool formats, which keeps the KV
cache across turns and stops repeated identical tool calls.
## Training
| Stage | Data |
|---|---|
| Pre-training | 7.08B tokens, mostly code (Python first, then web and other languages), with code-related web pages, French and English web text, and math |
| Annealing | The final quarter adds teacher projects, exercises whose solutions pass their tests, agent trajectories and commits |
| Agent fine-tuning | 2,093 verified teacher runs on multi-file projects (CLI tools, games, websites, APIs, bug fixes, tests), plus DAgger states in which the model acts and the teacher supplies the correct action |
## Evaluation
### Code generation (EvalPlus)
![HumanEval+ pass@1: Têtu Code 2 15.9 %, above SantaCoder-1.1B and StarCoderBase-1B](assets/humaneval-plus.png)
![MBPP+ pass@1: Têtu Code 2 30.7 %, above Têtu Code 1](assets/mbpp-plus.png)
| Model | Size | Training tokens | HumanEval+ | MBPP+ |
|---|---|---|---|---|
| DeepSeek-Coder-1.3B-instruct | 1.3B | 2T | 60.4 | 54.8 |
| DeepSeek-Coder-1.3B-base | 1.3B | 2T | 25.6 | 47.9 |
| Qwen2.5-Coder-0.5B (base) | 0.5B | 5.5T | 23.8 | 47.1 |
| Têtu Code 1 (previous model) | 193M active | 7B | 18.3 | 27.5 |
| **Têtu Code 2** | **475M active** | **7.08B** | **15.9** | **30.7** |
| SantaCoder-1.1B | 1.1B | 236B | 14.0 | – |
| StarCoderBase-1B | 1B | 1T | 12.2 | – |
| InCoder-1.3B | 1.3B | ~60B | 11.0 | – |
| CodeGen2-1B | 1B | ~400B | 9.1 | – |
Têtu Code results use EvalPlus 0.3.1 with greedy decoding, one sample per problem and a chat prompt. A repeated run
gave HumanEval+ 14.6 and MBPP+ 30.7, so differences of about 1.5 points are within noise. Other results are taken from
the official EvalPlus leaderboard (23 Oct. 2026); the Qwen2.5-Coder figures are from its authors.
Têtu Code 2 scores above the 2023 1B code models (SantaCoder, StarCoderBase-1B) after 30 to 140 times fewer training
tokens, and below DeepSeek-Coder-1.3B and Qwen2.5-Coder-0.5B, which were trained on 280 to 780 times more tokens.
### Agent tasks
![40 held-out projects in OpenCode: teacher 29/40, Têtu Code 2 8/40, Têtu Code 1 0/40](assets/agent.png)
| | Têtu Code 1 | Têtu Code 2 | Teacher (Qwen3.6-35B-A3B) |
|---|---|---|---|
| 40 held-out multi-file projects in OpenCode 2.1.4 (hidden tests) | 0/40 | **8/40** | 29/40 |
| Single-function agent benchmark (OpenCode 1.19.4) | 46.5 % | 28.5 % | 85.5 % |
| Python validation loss (pre-training) | 0.821 | **0.777** | – |
Throughput on an RTX 5070: about 143 tokens/s at 2k context and 71 tokens/s at 16k.
## Limitations
- As a small model, it often starts from incorrect logic or uses undefined variables.
- When its own tests fail, it tends to re-run commands rather than fix the code.
- In multi-turn conversations it can miss a correction or a follow-up request.
- It is tuned as an agent: in the plain chat script (`chat.py`) it often still tries to call tools. Use it through
OpenCode.
- It sometimes writes files outside the project directory; the bundled server maps such paths back into the project.
- Run it in a sandbox or a disposable directory, not unattended on important code.
## Usage
The architecture is not supported by `transformers`; use the bundled inference code.
```bash
git clone https://huggingface.co/tetu-AI/Tetu-Code-2 && cd Tetu-Code-2
pip install -r requirements.txt
python chat.py # terminal chat without tools (--cpu without a GPU)
python -m tetu_code.serve --model . --port 8070 # OpenAI-compatible server for OpenCode
```
`opencode.json`:
```json
"provider": {"tetu": {"npm": "@ai-sdk/openai-compatible", "name": "Têtu Code",
"options": {"baseURL": "http://127.0.0.1:8070/v1", "apiKey": "x"},
"models": {"tetu-code": {"name": "Têtu Code", "tool_call": true,
"limit": {"context": 65536, "output": 4096}}}}}
```
From Python:
```python
import torch
from tetu_code.generate import Decoder, load
model, tok = load(".", device="cuda") # or device="cpu", dtype=torch.float32
dec = Decoder(model, tok, max_len=16384)
```
## License
Apache 2.0.
Teachers (both Apache 2.0): Qwen3.6-35B-A3B for the agent runs, Qwen2.5-7B-Instruct for general conversation.
Training data: github-code-clean (permissively licensed files only), FineWeb-Edu, FineWeb2-HQ and FineMath (ODC-By),
OpenCodeInstruct, Nemotron-SWE trajectories and CommitPackFT. Check the terms of each dataset before commercial use.
## Citation
```bibtex
@misc{tetucode2026,
title = {Têtu Code 2: a small coding agent trained from scratch},
author = {tetu-AI},
year = {2026},
url = {https://huggingface.co/tetu-AI/Tetu-Code-2}
}
```