Tetu-Code-2 / README.md
tetu-AI's picture
cleaning
941a017 verified
|
Raw History Blame Contribute Delete
6.63 kB
metadata
license: apache-2.0
language:
  - en
  - fr
  - code
pipeline_tag: text-generation
tags:
  - code
  - coding-agent
  - small-language-model
  - trained-from-scratch
  - custom-architecture
  - tool-use
  - opencode
model-index:
  - name: Tetu-Code-2
    results:
      - task:
          type: text-generation
        dataset:
          name: HumanEval+
          type: evalplus/humanevalplus
        metrics:
          - type: pass@1
            name: pass@1 (greedy, chat prompt)
            value: 15.9
      - task:
          type: text-generation
        dataset:
          name: MBPP+
          type: evalplus/mbppplus
        metrics:
          - type: pass@1
            name: pass@1 (greedy, chat prompt)
            value: 30.7

Têtu Code 2

Têtu Code 2 is a small coding agent trained from scratch. It is designed to run in the terminal through OpenCode, where it reads and edits files, runs commands and reports what it did.

The model was trained from random initialization; no pretrained weights were used. Qwen3.6-35B-A3B (Apache 2.0) served as a teacher for the agent demonstrations, and every demonstration was checked by a program before training. Training ran on a single RTX 5070 (12 GB) plus about 30 hours of cloud GPU time.

Model details

Developer tetu-AI
Architecture Decoder-only transformer (custom)
Parameters 1.15B total, 475M active per token (the rest is a lookup memory with negligible compute cost)
Context length 65,536 tokens
Languages Python first, plus HTML, CSS, JavaScript and others; instructions in English or French
Vocabulary 32,768 tokens
Weights bfloat16 (model.safetensors)
License Apache 2.0

Capabilities:

  • tool use through OpenCode: read, write, edit, bash, glob, grep;
  • a short written reasoning between <|think|> and <|/think|> before each action;
  • a per-token confidence head that estimates the probability of an error;
  • an OpenAI-compatible server (tetu_code.serve) for the OpenCode 1.x and 2.x tool formats, which keeps the KV cache across turns and stops repeated identical tool calls.

Training

Stage Data
Pre-training 7.08B tokens, mostly code (Python first, then web and other languages), with code-related web pages, French and English web text, and math
Annealing The final quarter adds teacher projects, exercises whose solutions pass their tests, agent trajectories and commits
Agent fine-tuning 2,093 verified teacher runs on multi-file projects (CLI tools, games, websites, APIs, bug fixes, tests), plus DAgger states in which the model acts and the teacher supplies the correct action

Evaluation

Code generation (EvalPlus)

HumanEval+ pass@1: Têtu Code 2 15.9 %, above SantaCoder-1.1B and StarCoderBase-1B

MBPP+ pass@1: Têtu Code 2 30.7 %, above Têtu Code 1

Model Size Training tokens HumanEval+ MBPP+
DeepSeek-Coder-1.3B-instruct 1.3B 2T 60.4 54.8
DeepSeek-Coder-1.3B-base 1.3B 2T 25.6 47.9
Qwen2.5-Coder-0.5B (base) 0.5B 5.5T 23.8 47.1
Têtu Code 1 (previous model) 193M active 7B 18.3 27.5
Têtu Code 2 475M active 7.08B 15.9 30.7
SantaCoder-1.1B 1.1B 236B 14.0 –
StarCoderBase-1B 1B 1T 12.2 –
InCoder-1.3B 1.3B ~60B 11.0 –
CodeGen2-1B 1B ~400B 9.1 –

Têtu Code results use EvalPlus 0.3.1 with greedy decoding, one sample per problem and a chat prompt. A repeated run gave HumanEval+ 14.6 and MBPP+ 30.7, so differences of about 1.5 points are within noise. Other results are taken from the official EvalPlus leaderboard (23 Oct. 2026); the Qwen2.5-Coder figures are from its authors.

Têtu Code 2 scores above the 2023 1B code models (SantaCoder, StarCoderBase-1B) after 30 to 140 times fewer training tokens, and below DeepSeek-Coder-1.3B and Qwen2.5-Coder-0.5B, which were trained on 280 to 780 times more tokens.

Agent tasks

40 held-out projects in OpenCode: teacher 29/40, Têtu Code 2 8/40, Têtu Code 1 0/40

Têtu Code 1 Têtu Code 2 Teacher (Qwen3.6-35B-A3B)
40 held-out multi-file projects in OpenCode 2.1.4 (hidden tests) 0/40 8/40 29/40
Single-function agent benchmark (OpenCode 1.19.4) 46.5 % 28.5 % 85.5 %
Python validation loss (pre-training) 0.821 0.777 –

Throughput on an RTX 5070: about 143 tokens/s at 2k context and 71 tokens/s at 16k.

Limitations

  • As a small model, it often starts from incorrect logic or uses undefined variables.
  • When its own tests fail, it tends to re-run commands rather than fix the code.
  • In multi-turn conversations it can miss a correction or a follow-up request.
  • It is tuned as an agent: in the plain chat script (chat.py) it often still tries to call tools. Use it through OpenCode.
  • It sometimes writes files outside the project directory; the bundled server maps such paths back into the project.
  • Run it in a sandbox or a disposable directory, not unattended on important code.

Usage

The architecture is not supported by transformers; use the bundled inference code.

git clone https://huggingface.co/tetu-AI/Tetu-Code-2 && cd Tetu-Code-2
pip install -r requirements.txt
python chat.py                                   # terminal chat without tools (--cpu without a GPU)
python -m tetu_code.serve --model . --port 8070  # OpenAI-compatible server for OpenCode

opencode.json:

"provider": {"tetu": {"npm": "@ai-sdk/openai-compatible", "name": "Têtu Code",
  "options": {"baseURL": "http://127.0.0.1:8070/v1", "apiKey": "x"},
  "models": {"tetu-code": {"name": "Têtu Code", "tool_call": true,
                           "limit": {"context": 65536, "output": 4096}}}}}

From Python:

import torch
from tetu_code.generate import Decoder, load
model, tok = load(".", device="cuda")          # or device="cpu", dtype=torch.float32
dec = Decoder(model, tok, max_len=16384)

License

Apache 2.0.

Teachers (both Apache 2.0): Qwen3.6-35B-A3B for the agent runs, Qwen2.5-7B-Instruct for general conversation.

Training data: github-code-clean (permissively licensed files only), FineWeb-Edu, FineWeb2-HQ and FineMath (ODC-By), OpenCodeInstruct, Nemotron-SWE trajectories and CommitPackFT. Check the terms of each dataset before commercial use.

Citation

@misc{tetucode2026,
  title  = {Têtu Code 2: a small coding agent trained from scratch},
  author = {tetu-AI},
  year   = {2026},
  url    = {https://huggingface.co/tetu-AI/Tetu-Code-2}
}