smart-commit-0.5b-v0.3

smart-commit-0.5b-v0.1 with one more training step, DPO on the model's own samples, for smart-commit, a CLI that writes git commit messages in the repository's style with a small model running locally. Version 0.3: still experimental, while the training recipe changes. (The 0.2 line is a different one: trained without the rules in the prompt. This keeps them, as 0.1 does.)

What the step does: the model samples six replies for each of 5,500 commits it was not trained on, a set of checks marks those that copy a recent subject, name a detail the diff does not contain, say nothing, use a scope the repository never used or a type other than the real commit's, and the model is trained to prefer the reply that passes them all. The replies that fail those checks almost disappear; the type of the commit does not improve significantly.

It is trained on smart-commit's own prompt, byte for byte, with the full rules (--rules full).

Files

File	What	SHA-256
model.safetensors (+ config and tokenizer)	merged weights, bf16	bfa263bb653c8b0266ea31a8a0d31402c11ec6a6c1413b1fb8f79ea406ec3ff1
smart-commit-0.5b-v0.3-q4_k_m.gguf	Q4_K_M for llama.cpp, 398 MB	764940aaec5d3aae4587ce032c1f0437454ff3fa3666bd6c5ff10b2df7acfc3b
training/	run.json and the log of the DPO run, and the two LoRA adapters (SFT, then DPO)

Use with smart-commit

hf download LucasBRT/smart-commit-0.5b-v0.3 smart-commit-0.5b-v0.3-q4_k_m.gguf --local-dir ~/models
SMART_COMMIT_GGUF=~/models/smart-commit-0.5b-v0.3-q4_k_m.gguf smart-commit

It runs on smart-commit's llama.cpp devices (CPU and Vulkan GPUs). The OpenVINO export is not validated for this model.

Evaluation

eval-v1: 500 commits from the test split of CommitChronicle (repositories never seen in training), 250 following Conventional Commits and 250 free-form; eval-short: 197 commits from young repositories (1–5 earlier subjects; 47 Conventional, 150 free). Greedy decoding, Vulkan, smart-commit 0.9.0 plus the commits of main up to 236d51f, with its retries and file-list fallback; the same binary and settings for the three models. The columns without "(free)" or "(young)" are the 250 Conventional commits of eval-v1.

With smart-commit's default file list (annotated), the one a user gets:

Model	type_match	type_macro_f1	scope_known	first_try	fallback	ungrounded	word_f1	first_try (free)	word_f1 (free)	type_match (young)	fallback (young)	fallback (young, free)
Qwen2.5-Coder-0.5B-Instruct, re-quantized like this model (control)	0.352	0.200	0.876	0.336	0.632	0.296	0.159	0.692	0.102	0.468	0.894	0.340
smart-commit-0.5b-v0.1	0.648	0.546	0.852	0.960	0.028	0.116	0.342	0.928	0.234	0.638	0.085	0.080
smart-commit-0.5b-v0.3	0.668	0.552	0.840	0.996	0.004	0.008	0.354	0.996	0.250	0.596	0.021	0.000

With the plain file list, the one it was trained on:

Model	type_match	type_macro_f1	scope_known	first_try	fallback	ungrounded	word_f1	first_try (free)	word_f1 (free)	type_match (young)	fallback (young)	fallback (young, free)
Qwen2.5-Coder-0.5B-Instruct, re-quantized like this model (control)	0.344	0.201	0.884	0.312	0.648	0.308	0.161	0.696	0.101	0.426	0.872	0.333
smart-commit-0.5b-v0.1	0.628	0.531	0.844	0.960	0.028	0.104	0.333	0.916	0.230	0.617	0.085	0.100
smart-commit-0.5b-v0.3	0.664	0.557	0.840	0.996	0.004	0.008	0.347	0.996	0.247	0.617	0.021	0.000

Against v0.1, paired, on the 250 Conventional commits of eval-v1 with the annotated list: ungrounded +1/−28 (p = 0.0), first_copied +0/−9 (p = 0.004), first_try +9/−0 (p = 0.004), fallback +0/−6 (p = 0.031); type_match +11/−6 (p = 0.332) and scope_known +2/−5 (p = 0.453), neither significant.

• type_match: same Conventional type as the real commit. scope_known: no scope, or one the repository already used. first_try: the first reply was usable; fallback: no reply was, and smart-commit wrote the message from the file list. ungrounded: no content word of the description is in the diff, the paths or the history. word_f1: word overlap with the real subject, a rough proxy for content.
• The control row is the base model quantized as this one: it falls back to the file list for most commits in this setup, and these numbers are not comparable to older ones measured with an earlier smart-commit.
• It also uses far fewer words that are in nothing the model was given (history, paths and diff): 0.082 of the words of its Conventional replies on eval-v1, against 0.151 for v0.1 and 0.327 for the real commits. It says what the diff says rather than what the change means.

Training

• Starting point: smart-commit-0.5b-v0.1 (the SFT adapter in training/ is the one it was trained with).
• Data: CommitChronicle (revision 5fd076e). 5,500 commits of its train split, none of those v0.1 trained on and none from the evaluation repositories: 4,500 from 680 repositories that follow Conventional Commits (at most 8 each) and 1,000 from young repositories, six samples of v0.1 for each (temperature 0.7, fixed seeds). Added to them, the free-style commits (repositories that do not follow Conventional Commits) mined in an earlier run from its test split, outside the evaluation repositories, four samples each. Of the 3172 pairs for training (and 514 of other repositories for validation) 905 are of those free-style commits, cut so that the chosen and the rejected replies have the same distribution of type and of scope (pairs file sha256 8d33514b190028ba…).
• Method: LoRA r=16, alpha=32, dropout 0.05 on all linear layers, on the merged v0.1; sigmoid DPO loss, beta 0.1, lr 2e-05 cosine, 198 steps of 16 pairs, plus the per-token loss of the chosen reply with weight 0.2; fp16 on a Kaggle Tesla T4, 31 minutes. torch 2.11.0+cu128, transformers 5.20.1, peft 0.21.2. One of six configurations, chosen on a validation set of 500 commits of other repositories.
• Export: merged into the bf16 weights and quantized to Q4_K_M with llama.cpp b11381 (training/export_gguf.py --adapter SFT --adapter DPO --merged).

Training on a GPU is not bit-for-bit reproducible; generation is (greedy). The recipe, the data selection, the first sweeps that failed and the scripts are in training/ (H12 of HYPOTHESES.md).

Limitations

• English messages only.
• Young repositories: on the 47 Conventional commits of eval-short it does not improve the type and the macro F1 falls; with fewer than 10 earlier commits the data is too small to say more.
• The type of the commit is as unreliable as v0.1's: about two in three right on eval-v1.
• It can still invent details the diff does not show, and it does not know when the staged changes are several commits.
• One seed per configuration: differences of a few cases between runs are noise.

Licence and attribution

The weights are released under Apache-2.0, like the base model. CommitChronicle (JetBrains Research) has no single licence: it gathers commits from repositories under MIT, Apache-2.0 and BSD-3-Clause, and asks that uses follow their terms, attribution included. The dataset records the repository and licence of every commit (repo and license fields), which is the provenance of the training data.