thealper2/t5-small-commitbench

google-t5/t5-small fine-tuned on Maxscha/commitbench for commit message generation: given a git diff, generate the commit message describing it.

Task format

Text-to-text. The input is a task prefix followed by the raw git diff, the target is the commit message.

generate commit message: <git diff>

Training data

Maxscha/commitbench official splits, used unchanged:

Split	Examples in split	Examples used
train	1,165,213	500,000
validation	249,689	2,000
test	249,688	not used for training

Languages covered by the dataset: Python, JavaScript, PHP, Ruby, Java, Go.

Training configuration

Setting	Value
Base model	google-t5/t5-small
Parameters	60.5M
Max source length	512 tokens
Max target length	64 tokens
Per-device batch size	32
Gradient accumulation	1
Effective batch size	32
Learning rate	3e-05
LR schedule	linear
Warmup ratio	0.05
Weight decay	0.01
Epochs	2.0
Label smoothing	0.0
Gradient clipping	1.0
Mixed precision	bf16
Seed	42
Optimizer	AdamW
Training time	1.219 h
Hardware	NVIDIA GeForce RTX 5060 Ti (15.9 GB)

Truncation at these limits (measured on a 50k sample with the T5 tokenizer):

• 0.7% of the diffs exceed 512 source tokens.
• 4.47% of the commit messages exceed 64 target tokens.

Results

• Final training loss: 3.5762
• Best validation loss: 3.2414

Test split (20,000 examples), beam search with num_beams=4:

Metric	Value
rouge1	19.31
rouge2	4.668
rougeL	17.42
rougeLsum	17.42
bleu	2.148
exact_match	0.04
gen_len_words_mean	5.005
ref_len_words_mean	11.27

Per programming language:

Language	n	ROUGE-1	ROUGE-2	ROUGE-L	BLEU	Exact match
Python	5,722	21.20	6.05	19.29	2.73	0.04
JavaScript	4,468	18.86	4.05	17.07	2.01	0.02
PHP	3,489	17.04	3.46	15.29	1.64	0.09
Ruby	2,808	22.08	5.79	19.65	2.44	0.04
Java	1,799	15.19	2.61	13.58	1.02	0.06
Go	1,714	18.65	4.45	16.75	2.11	0.00

ROUGE and BLEU are lexical-overlap metrics. They do not fully capture whether a commit message describes a change correctly, and generic messages can score well.

Usage

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model_id = "thealper2/t5-small-commitbench"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

diff = open("change.patch").read()
inputs = tokenizer(
    "generate commit message: " + diff,
    max_length=512,
    truncation=True,
    return_tensors="pt",
)
output = model.generate(
    **inputs,
    num_beams=4,
    max_new_tokens=64,
    length_penalty=1.0,
    no_repeat_ngram_size=3,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Default generation settings: num_beams=4, max_new_tokens=64, min_new_tokens=0, length_penalty=1.0, no_repeat_ngram_size=3, do_sample=False (deterministic).

Limitations

• CommitBench replaces identifying literals with placeholder tokens: every diff contains <HASH> instead of commit hashes, and 26.5% of the reference messages contain <I> (numbers), <URL> or <EMAIL>. The model therefore also generates these tokens, e.g. Bumped version to <I>.
• The T5 sentencepiece vocabulary does not cover every character used in source code (curly braces, backslashes, angle brackets), so about 2.35% of the input tokens become <unk>. This limits how precisely the model can read a diff.
• Diffs longer than 512 tokens are truncated; the tail of the change is not visible to the model.
• CommitBench splits are random over commits, not over repositories: 98.6% of the test examples come from repositories that also appear in the training split. No (diff, message) pair is shared across splits, but the reported scores partly reflect familiarity with a project's commit style rather than generalization to unseen code.
• The dataset is English-only and covers six languages; behaviour on other languages or on very large multi-file changes is untested.
• CommitBench is released under CC BY-NC 4.0, which restricts commercial use of the data.

Reproducibility

• python: 3.12.3
• torch: 2.11.0+cu128
• transformers: 5.17.0
• datasets: 4.3.0
• tokenizers: 0.23.2
• seed: 42