teex-pt/EuroLLM-1.7B-AMALIA-draft-pilot

EuroLLM-1.7B — AMALIA-9B speculative-decoding draft model (research pilot)

A LoRA adapter for utter-project/EuroLLM-1.7B-Instruct, trained to align the small model's outputs to AMALIA-9B-0626-DPO's own greedy-decoded phrasing, for use as a draft model in speculative decoding (mlx_lm.generate --draft-model). Part of the same pilot series as teex-pt/AMALIA-9B-0626-DPO-LoRA-honesty-pilot; full methodology and narrative: github.com/teex-pt/pt-amalia.

Update: the first benchmark (default --num-draft-tokens 2) showed a net slowdown. A follow-up sweep across --num-draft-tokens values found the relationship is monotonic — lower is strictly better — and --num-draft-tokens 1 is a genuine net speedup (1.06x average). Use 1, not the default 2. See below for the full sweep.

What it is

Trained via knowledge distillation: AMALIA-9B's own greedy completions for all 819 prompts in our verified mix-v4 dataset (arithmetic, format, honesty/refusals, boundary pairs) became the LoRA fine-tuning targets for EuroLLM-1.7B-Instruct. The idea: a draft model that predicts AMALIA-9B's exact token sequence, not just plausible pt-PT text, should get more of its speculative tokens accepted by the target model during verification.

Measured speculative-decoding speedup (M5 Pro, target = AMALIA-9B 8-bit)

--num-draft-tokens 1 (recommended)

Prompt type (generation length)	Speedup
General knowledge (long, explanatory)	1.25x
Honesty / fake-entity (medium)	1.19x
Arithmetic, bare answer (~3-5 tokens)	0.75x
Average	1.06x — net speedup

Full sweep — speedup is monotonically decreasing in N

N	General Physics	Honesty	Arithmetic	Average
1	1.25x	1.19x	0.75x	1.06x
2 (mlx-lm's default)	1.17x	1.17x	0.59x	0.98x
3	1.03x	0.97x	0.43x	0.81x
4	0.92x	0.88x	0.37x	0.72x
6	0.73x	0.69x	0.27x	0.56x
8	0.57x	0.53x	0.19x	0.43x
12	0.56x	0.55x	0.20x	0.44x

Use --num-draft-tokens 1. mlx-lm's own default is 2, which is already close to the floor but not optimal here — every value above 1 makes things worse, monotonically, for every prompt type we tested.

Arithmetic still doesn't cross 1.0x even at the floor. This is the more interesting finding: we also benchmarked a non-distilled variant (LoRA on mix-v4's original answers, not AMALIA's own outputs) and got an identical 0.43x at N=2 — ruling out phrasing/style mismatch as the cause. Speculative decoding has fixed per-step overhead (draft proposes tokens, target verifies in a batch); a 3-5 token answer doesn't have enough length to amortize that overhead, no matter how well-aligned the draft is or how low you set N. This is a structural limit of the technique for short outputs, not a training problem. If your use case is dominated by short bare-answer completions (arithmetic, single-value extraction), consider skipping speculative decoding for those prompts entirely rather than tuning around it.

Side finding: this adapter is worse at our own pt-PT harness than the non-distilled version

Metric	Non-distilled (eurollm-1.7b-lora-v4)	This adapter (distilled)
honesty	82.0%	50.0%
overall	47.3%	34.2%

Expected: this adapter imitates AMALIA-9B's own imperfect greedy outputs (AMALIA's honesty rate is 82–96%, not 100%) rather than our hand-verified templates. "Aligned to a target model's token stream" and "verifiably correct" are different objectives — this experiment separates them cleanly. If you want a EuroLLM-1.7B pt-PT adapter for its own sake (not as a draft model), the non-distilled version scores better on every axis we measured.

Usage

pip install mlx-lm
mlx_lm.generate --model amalia-mlx-8bit --draft-model <this-repo> \
    --num-draft-tokens 1 \
    --prompt "Explica-me o que é a gravidade." --max-tokens 200

Attribution

Base model by the AMALIA team and Utter Project (both Apache 2.0). Adapter, distillation pipeline, and benchmarks: teex-pt.