smart-commit-1.5b-v0.3

smart-commit-1.5b-v0.1 with one more training step, DPO on the model's own samples, for smart-commit, a CLI that writes git commit messages in the repository's style with a small model running locally. Version 0.3: still experimental, while the training recipe changes. (The 0.2 line is a different one: trained without the rules in the prompt. This keeps them, as 0.1 does.)

What the step does: the model samples six replies for each of 5,500 commits it was not trained on, a set of checks marks those that copy a recent subject, name a detail the diff does not contain, say nothing, use a scope the repository never used or a type other than the real commit's, and the model is trained to prefer the reply that passes them all. The replies that fail those checks almost disappear; the type of the commit does not improve significantly.

It is trained on smart-commit's own prompt, byte for byte, with the full rules (--rules full).

Files

File	What	SHA-256
model.safetensors (+ config and tokenizer)	merged weights, bf16	7876df94271bb81c28e59d9f2d40a2349daa761b2979be980113aa4463df5eb8
smart-commit-1.5b-v0.3-q4_k_m.gguf	Q4_K_M for llama.cpp, 986 MB	92b0743ad80b2954b3af79af9f0550aca97d3ae589a03b260b6b94008e530d48
training/	run.json and the log of the DPO run, and the two LoRA adapters (SFT, then DPO)

Use with smart-commit

hf download LucasBRT/smart-commit-1.5b-v0.3 smart-commit-1.5b-v0.3-q4_k_m.gguf --local-dir ~/models
SMART_COMMIT_GGUF=~/models/smart-commit-1.5b-v0.3-q4_k_m.gguf smart-commit

It runs on smart-commit's llama.cpp devices (CPU and Vulkan GPUs). The OpenVINO export is not validated for this model.

Evaluation

eval-v1: 500 commits from the test split of CommitChronicle (repositories never seen in training), 250 following Conventional Commits and 250 free-form; eval-short: 197 commits from young repositories (1–5 earlier subjects; 47 Conventional, 150 free). Greedy decoding, Vulkan, smart-commit 0.9.0 plus the commits of main up to 236d51f, with its retries and file-list fallback; the same binary and settings for the three models. The columns without "(free)" or "(young)" are the 250 Conventional commits of eval-v1.

With smart-commit's default file list (annotated), the one a user gets:

Model	type_match	type_macro_f1	scope_known	first_try	fallback	ungrounded	word_f1	first_try (free)	word_f1 (free)	type_match (young)	fallback (young)	fallback (young, free)
Qwen2.5-Coder-1.5B-Instruct, re-quantized like this model (control)	0.436	0.190	0.604	0.844	0.056	0.144	0.249	0.848	0.141	0.149	0.043	0.060
smart-commit-1.5b-v0.1	0.676	0.581	0.852	0.928	0.028	0.076	0.362	0.888	0.256	0.681	0.085	0.047
smart-commit-1.5b-v0.3	0.696	0.604	0.868	0.996	0.004	0.020	0.366	0.996	0.266	0.702	0.021	0.000

With the plain file list, the one it was trained on:

Model	type_match	type_macro_f1	scope_known	first_try	fallback	ungrounded	word_f1	first_try (free)	word_f1 (free)	type_match (young)	fallback (young)	fallback (young, free)
Qwen2.5-Coder-1.5B-Instruct, re-quantized like this model (control)	0.428	0.184	0.576	0.836	0.060	0.160	0.238	0.868	0.138	0.170	0.064	0.053
smart-commit-1.5b-v0.1	0.660	0.568	0.848	0.924	0.024	0.068	0.360	0.896	0.255	0.723	0.064	0.040
smart-commit-1.5b-v0.3	0.692	0.589	0.864	0.996	0.004	0.020	0.366	0.996	0.265	0.681	0.021	0.000

Against v0.1, paired, on the 250 Conventional commits of eval-v1 with the annotated list: ungrounded +0/−14 (p = 0.0), first_copied +0/−17 (p = 0.0), first_try +17/−0 (p = 0.0), fallback +0/−6 (p = 0.031); type_match +9/−4 (p = 0.267) and scope_known +6/−2 (p = 0.289), neither significant.

• type_match: same Conventional type as the real commit. scope_known: no scope, or one the repository already used. first_try: the first reply was usable; fallback: no reply was, and smart-commit wrote the message from the file list. ungrounded: no content word of the description is in the diff, the paths or the history. word_f1: word overlap with the real subject, a rough proxy for content.
• The control row is the base model quantized as this one, run through the same prompt: these numbers are not comparable to older ones measured with an earlier smart-commit.
• It also uses far fewer words that are in nothing the model was given (history, paths and diff): 0.130 of the words of its Conventional replies on eval-v1, against 0.145 for v0.1 and 0.327 for the real commits. It says what the diff says rather than what the change means.

Training

• Starting point: smart-commit-1.5b-v0.1 (the SFT adapter in training/ is the one it was trained with).
• Data: CommitChronicle (revision 5fd076e). 5,500 commits of its train split, none of those v0.1 trained on and none from the evaluation repositories: 4,500 from 680 repositories that follow Conventional Commits (at most 8 each) and 1,000 from young repositories, six samples of v0.1 for each (temperature 0.7, fixed seeds). Added to them, the free-style commits (repositories that do not follow Conventional Commits) mined in an earlier run from its test split, outside the evaluation repositories, four samples each. Of the 3091 pairs for training (and 500 of other repositories for validation) 885 are of those free-style commits, cut so that the chosen and the rejected replies have the same distribution of type and of scope (pairs file sha256 f46cc34102074804…).
• Method: LoRA r=16, alpha=32, dropout 0.05 on all linear layers, on the merged v0.1; sigmoid DPO loss, beta 0.1, lr 2e-05 cosine, 193 steps of 16 pairs, plus the per-token loss of the chosen reply with weight 0.5; fp16 on a Kaggle Tesla T4, 81 minutes. torch 2.11.0+cu128, transformers 5.20.1, peft 0.21.2. One of two configurations (--sft-weight 0.2 and 0.5, the 0.5B sweep's best beta and learning rate, one on each GPU), chosen on data/val1, 500 commits of other repositories, before the evaluation sets were read.
• Export: merged into the bf16 weights and quantized to Q4_K_M with llama.cpp b11381 (training/export_gguf.py --adapter SFT --adapter DPO --merged).

Training on a GPU is not bit-for-bit reproducible; generation is (greedy). The recipe, the data selection, the first sweeps that failed and the scripts are in training/ (H12 of HYPOTHESES.md).

Limitations

• English messages only.
• Young repositories: on the 47 Conventional commits of eval-short the type is neither better nor worse (a few cases either way, none significant); with fewer than 10 earlier commits the data is too small to say more.
• The type of the commit is as unreliable as v0.1's: about two in three right on eval-v1.
• It can still invent details the diff does not show, and it does not know when the staged changes are several commits.
• One seed per configuration: differences of a few cases between runs are noise.

Licence and attribution

The weights are released under Apache-2.0, like the base model. CommitChronicle (JetBrains Research) has no single licence: it gathers commits from repositories under MIT, Apache-2.0 and BSD-3-Clause, and asks that uses follow their terms, attribution included. The dataset records the repository and licence of every commit (repo and license fields), which is the provenance of the training data.