LFM 2.5 Audio 1.5B — ASR LoRA (Voice → Home Assistant Commands)
LoRA adapter fine-tuned on LFM2.5-Audio-1.5B to convert spoken Home Assistant commands into structured function calls (HassLightTurnOn|$area=hall).
Trained entirely on Apple Silicon using MLX. No cloud GPU required.
What It Does
Takes audio of a spoken command and outputs a structured function call:
Input audio: "switch on the light in the hall"
Output: HassLightTurnOn|$area=hall
Output format: FunctionName|$arg1=val1|$arg2=val2 — pipe-delimited, parseable.
Training Details
| Value | |
|---|---|
| Base model | mlx-community/LFM2.5-Audio-1.5B-8bit |
| Dataset | Paulescu/OHF-Voice-audio-20260504 (train split, 950 samples) |
| LoRA rank | 16 |
| LoRA alpha | 32.0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Trainable params | 884,736 / 168,786,688 total (0.52%) |
| Optimizer | AdamW, lr=5e-5, cosine decay |
| Warmup steps | 250 |
| Effective batch size | 8 (grad_accumulation=8) |
| Total steps | 10,000 |
| Hardware | Apple Silicon (M-series) |
| Framework | MLX |
Evaluation Results
Evaluated on Paulescu/OHF-Voice-audio-20260504 test split, 205 stratified samples (5 per function), system prompt "Perform ASR.", temperature 0.0.
| Metric | This adapter | Cookbook baseline (full FT, A100) |
|---|---|---|
| Format compliance | 99.0% | 99.7% |
| Function-name accuracy | 82.0% | 98.8% |
| Argument accuracy | 61.0% | ~97% |
The gap vs the cookbook is expected — they use full fine-tuning (all ~1.5B params) on an A100 with batch 32. This adapter trains 0.52% of parameters on a local Mac.
See also v2 — trained on the full 49,909-sample dataset, achieving 92.2% function-name and 70.0% argument accuracy with only 4,000 steps. Data volume proved more impactful than training duration.
Key Finding: Modality Positioning
LFM 2.5 Audio's __call__ interface concatenates audio embeddings after all text tokens, meaning the model cannot attend to audio when predicting the assistant response. For ASR fine-tuning to work, you must call model._prefill with explicit modalities so audio is positioned inside the user turn (before the assistant tokens) — matching the inference-time ChatState layout.
This is implemented in train/losses/lfm_audio_loss.py.
Usage
import mlx.core as mx
from mlx_audio.sts.models.lfm_audio import LFM2AudioModel, LFM2AudioProcessor
from mlx_audio.sts.models.lfm_audio.processor import ChatState
from mlx_audio.sts.models.lfm_audio.model import LFMModality
# Load base model + adapter
processor = LFM2AudioProcessor.from_pretrained("mlx-community/LFM2.5-Audio-1.5B-8bit")
model = LFM2AudioModel.from_pretrained("mlx-community/LFM2.5-Audio-1.5B-8bit")
# Apply LoRA and load weights
from train.lora import apply_lora, load_adapters, LoRAConfig
apply_lora(model, LoRAConfig(model_type="lfm_audio", rank=16, alpha=32.0))
load_adapters(model, "path/to/adapters.safetensors")
model.eval()
# Transcribe
audio_mx = mx.array(audio_numpy) # float32, any sample rate
chat = ChatState(processor)
chat.new_turn("system"); chat.add_text("Perform ASR."); chat.end_turn()
chat.new_turn("user"); chat.add_audio(audio_mx, sample_rate=sr); chat.end_turn()
chat.new_turn("assistant")
output = ""
for token, modality in model.generate_from_chat_state(
chat, mode="sequential", max_new_tokens=64, temperature=0.0, top_k=1
):
if modality == LFMModality.TEXT:
tok_id = int(token.item())
if tok_id == 7: # <|im_end|>
break
output += processor.tokenizer.decode([tok_id])
print(output) # e.g. HassLightTurnOn|$area=hall
Or run the included demo:
git clone https://github.com/akashicMarga/mlx-audio-train
cd mlx-audio-train
python scripts/lfm_asr_demo.py --adapter path/to/checkpoint-final
# open http://localhost:7860
Reproduce / Experiment
Full training code and configs are in akashicMarga/mlx-audio-train.
Things worth experimenting with:
- Higher LoRA rank (32, 64) — more capacity for memorising exact argument patterns
- Full fine-tuning — remove LoRA and train all params (needs more VRAM/RAM)
- More data — the full OHF-Voice dataset has 55K samples; we used 950
- Different system prompt — the model is sensitive to the exact system prompt used at training vs inference
# Quick 200-step smoke test (~5 min on M-series)
python scripts/train.py --config configs/lfm_audio_asr_test.yaml
# Full 10k-step run (~12 hours on M-series)
caffeinate -i python scripts/train.py --config configs/lfm_audio_asr_10k.yaml
Citation / Acknowledgements
- Base model: LiquidAI/LFM2.5-Audio-1.5B
- Dataset: Paulescu/OHF-Voice-audio-20260504
- Cookbook reference: Liquid4All/cookbook
- Training framework: mlx-audio
Quantized