DevTools Assistant 360M (fused)

A fine-tune of SmolLM2-360M-Instruct that picks the right developer tool on devtools.tools for a problem described in plain English, and says so in a parseable form.

This is the fused model (LoRA merged into the base weights), intended as the input to mlc_llm convert_weight for in-browser serving with WebLLM.

What it is for

The site has 210 tools. Asked "my json is one huge line and i can't read it", the model should name the JSON Formatter — not explain how to format JSON by hand.

It does not memorise the catalog. Each turn, BM25 retrieves 8 candidate tools and they are injected into the system prompt; the model's job is to choose among them, or to decline when none fit. That division is what makes a 360M model viable for this at all.

Results

Measured on 226 held-out conversations that none of the three systems had seen. "Tool accuracy" means the cited tool slug exactly matches the gold one.

	tool accuracy	abstains when it should
BM25 retrieval alone	47.8%	0%
stock SmolLM2-360M-Instruct	4.0%	—
this model (fused)	66.4%	93.8%
the LoRA adapter before fusing	67.7%	93.8%

The 1.3-point gap between the adapter and the fused model is numerical noise from the merge, not lost training: 10 records were fixed and 13 broken, a bidirectional split. Lost training would collapse the score toward the 4.0% baseline.

Retrieval recall on the same catalog is 70.6% @1 and 94.5% @8, so most of the remaining error is the model failing to pick a candidate that was in front of it, not the candidate being absent.

Where it is weak

Per-slice, on the same test set:

slice	accuracy
routing	61.6%
FAQ-style questions	69.2%
common mistakes	100%
abstain (nothing on the site fits)	100%
asking a clarifying question	0%
chaining two tools for one task	0%

The last two did not transfer. Growing those slices 11x and 6x was not enough: at 114 and 75 records against 2492 routing records, the dominant pattern — "name one tool, confidently" — wins. Fixing it needs a much larger share of the dataset, not more epochs.

Language

English only, in practice. SmolLM2 was trained overwhelmingly on English and the training set is English throughout. For other languages, start from a multilingual base.

Training

LoRA via mlx_lm.lora on Apple Silicon (unsloth is CUDA-only), then mlx_lm.fuse.


rank / scale / dropout	8 / 20.0 / 0.0
layers	32
iterations	1500
learning rate	1e-4
batch size	4
max sequence length	2048
prompt masking	on

Prompt masking matters here: the injected catalog is roughly 60% of each record's tokens, and training on it teaches the model to reproduce catalogs.

The checkpoint was selected by validation tool accuracy, not validation loss. Loss is not usable on this dataset — 37% of validation records share an exact answer string with another (the refusal used for abstain repeats 9 times), so a model can lower loss by memorising two strings. Checkpoints 1250 and 1500 scored identically on the test set, so the selection yielded no measurable gain.

Prompt format

ChatML, the base model's own template. The system prompt carries the 8 retrieved candidates; the model answers in prose and marks a recommendation with a [TOOL]{"slug":…,"href":…}[/TOOL] block. Consumers should verify the slug against their catalog rather than trusting it — the block is a claim, not a guarantee.