--- library_name: transformers license: other license_name: lfm1.0 license_link: LICENSE pipeline_tag: any-to-any tags: - liquid - lfm3 - audio - speech - edge - conversational - multilingual - custom_code language: - en - ar - da - de - el - fi - fr - he - it - ja - ko - nl - pl - pt - es - sv - tr - zh ---

API Try Docs Blog

# LFM3-Audio-2B LFM3-Audio-2B is a **compact, fully featured audio model that runs locally on laptops and edge devices**. It takes text or audio as input and responds with text, speech, or both. - **Voice chat:** ask questions aloud and receive streaming replies in text and speech. For text-only responses, the model can reason before answering. - **Multilingual speech:** transcribe, synthesize, and translate speech across 18 languages. - **Audio understanding and reasoning:** interpret recordings and answer questions about the speech, sounds, and events they contain. - **Tool calling:** turn spoken requests into tool calls to control apps and devices, with text or speech alongside the calls. LFM3-Audio-2B introduces an in-house audio encoder and LiquidCodec audio tokenizer, improving speech quality over LFM2.5-Audio. Find more information about the LFM3 family in [our release blog post](https://www.liquid.ai/blog/lfm3). ## Model details

The [LFM3 family](https://huggingface.co/collections/LiquidAI/lfm3) includes base, text, vision, audio, omni, and decision models.

Property	Specification
Parameters	2B total
Architecture	LFM3 backbone, causal audio encoder, and LiquidCodec audio tokenizer
Input	Text or audio; audio encoder operates at 16 kHz
Output	Text, 24 kHz speech, or interleaved text and speech
Context length	32K tokens (input + output)
Streaming	Causal audio encoding, incremental prefill, and streaming speech generation
Speech languages / locales	Arabic, Danish, German, Greek, British English, American English, Finnish, French, Hebrew, Italian, Japanese, Korean, Dutch, Polish, Portuguese, Spanish, Swedish, Turkish, Chinese
License	LFM Open License v1.0

LFM3-Audio-2B is designed for local agents, tool calling, structured outputs, and domain-specific fine-tuning. Use retrieval or search for knowledge-heavy tasks. It is not recommended for agentic coding or complex multi-step reasoning due to its small size. ## Inference

Choose a checkpoint variant for your inference framework and hardware.

Checkpoint	Framework	Description
BF16	vLLM, SGLang, Transformers	BF16 weights for GPU inference.
GGUF	llama.cpp	Local inference on CPUs and GPUs.
MLX	MLX-VLM	4-bit quantization for Apple Silicon.
ONNX	ONNX Runtime	Cross-platform inference with ONNX Runtime.

### Transformers quick start Install the dependencies, then load the model and processor with the checkpoint's custom code. The codec implementation is included in this repository; the processor loads its weights from `audio_tokenizer/`. ```bash pip install "transformers>=5.20.2" torch accelerate librosa soundfile ``` Load the model and processor directly from the Hugging Face Hub: ```python import torch import soundfile as sf from transformers import AutoModelForMultimodalLM, AutoProcessor model_id = "LiquidAI/LFM3-Audio-2B" device = "cuda" if torch.cuda.is_available() else "cpu" dtype = torch.bfloat16 if device == "cuda" else torch.float32 model = AutoModelForMultimodalLM.from_pretrained( model_id, dtype=dtype, device_map=device, trust_remote_code=True ).eval() processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) codec = processor.audio_tokenizer.to(device=device).eval() ``` **Audio sampling parameters:** use `audio_temperature=1.0`, `audio_top_k=128`, `audio_repetition_penalty=1.1`, and `audio_repetition_window=4096` for speech synthesis and interleaved text and speech. For text-only output, set `suppress_mixed=True` and `suppress_audio_out=True`. ### Usage examples The examples below reuse the model, codec, and processor loaded above. Replace `audio.wav`, `question.wav`, or `recording.wav` with your own audio files.

Speech recognition (ASR)

To use the model for transcription, use the system prompt `Perform ASR in .`, where `` is the English name of the language. To let the model infer the language, use `Perform ASR.`. Below, we transcribe a `.wav` file containing English speech. ```python conversation = [ {"role": "system", "content": "Perform ASR in English."}, {"role": "user", "content": [{"type": "audio", "audio": "audio.wav"}]}, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=1024, suppress_mixed=True, suppress_audio_out=True, return_dict_in_generate=True, ) text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True, ) print(text) ```

Live speech recognition (LiveASR)

LiveASR transcribes speech while it is still being spoken, instead of waiting for the full recording. Use the system prompt `Perform live ASR.`. LiveASR currently supports English only. The model processes audio in chunks of about 0.96 s and writes the text for each chunk as soon as the chunk arrives. `model.transcribe_live()` runs one chunk at a time and returns a state that you pass to the next call. Below, we simulate a microphone by feeding a `.wav` file in 100 ms pieces. ```python import librosa tokenizer = processor.tokenizer chunk_frames = 8 * model.config.interleaved_n_audio prompt = tokenizer( "<|startoftext|><|im_start|>system\nPerform live ASR.<|im_end|>\n<|im_start|>assistant\n", add_special_tokens=False, return_tensors="pt", ).input_ids.to(device) suppress = tokenizer.convert_tokens_to_ids(["<|mixed_end|>", "<|endoftext|>"]) stop = tokenizer.convert_tokens_to_ids(["<|endoftext|>", "<|im_start|>"]) state, feature_cache = None, None mel = torch.empty((1, processor.feature_extractor.feature_size, 0), device=device) def transcribe(frames, is_final=False): global state output = model.transcribe_live( frames, state, input_ids=prompt if state is None else None, is_final=is_final, max_new_tokens=96 if is_final else 48, suppress_token_ids=suppress, stop_token_ids=stop, ) state = output.state print(tokenizer.decode(output.sequences[0], skip_special_tokens=True), end="", flush=True) waveform, _ = librosa.load("audio.wav", sr=16_000) for start in range(0, len(waveform), 1_600): # 100 ms microphone chunks features, feature_cache = processor.feature_extractor.stream( waveform[start:start + 1_600], feature_cache ) mel = torch.cat((mel, features.input_features.to(device)), dim=-1) while mel.shape[-1] >= chunk_frames: transcribe(mel[..., :chunk_frames]) mel = mel[..., chunk_frames:] # flush the remaining audio and end the transcript features, feature_cache = processor.feature_extractor.stream([], feature_cache, is_final=True) mel = torch.cat((mel, features.input_features.to(device)), dim=-1) while mel.shape[-1] > chunk_frames: transcribe(mel[..., :chunk_frames]) mel = mel[..., chunk_frames:] transcribe(mel, is_final=True) print() ``` For streams longer than 30 seconds, pass `input_ids=prompt` again to start a new text context. The encoder keeps following the audio stream.

Speech synthesis (TTS)

For speech synthesis, use the system prompt `Perform TTS in using a voice.`. Use a language or locale from the model details above and a `female` or `male` voice. Prefill the assistant turn with `<|audio_start|>`. ```python conversation = [ {"role": "system", "content": "Perform TTS in American English using a female voice."}, {"role": "user", "content": "The quick brown fox jumps over the lazy dog"}, {"role": "assistant", "content": "<|audio_start|>"}, ] inputs = processor.apply_chat_template( conversation, continue_final_message=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=256, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096, return_dict_in_generate=True, ) waveform = codec.decode( outputs.audio_codes[:1, :, :outputs.audio_lengths[0]] ).audio_values[0] # 24 kHz sf.write("tts.wav", waveform.float().cpu().numpy(), 24_000) ```

Voice chat: speech and text responses

In this mode, the model will respond to the user in both speech and text. The recommended system prompt is `Respond with interleaved text and audio.`. User input can be either text or audio, and we prefill the assistant turn with a special token `<|mixed_start|>`. The assistant responds in a conversational style with interleaved text and speech. This example saves the complete response. ```python conversation = [ {"role": "system", "content": "Respond with interleaved text and audio."}, {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, {"role": "assistant", "content": "<|mixed_start|>"}, ] inputs = processor.apply_chat_template( conversation, continue_final_message=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=1024, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096, return_dict_in_generate=True, ) text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True, ) waveform = codec.decode( outputs.audio_codes[:1, :, :outputs.audio_lengths[0]] ).audio_values[0] # 24 kHz print(text) sf.write("reply.wav", waveform.float().cpu().numpy(), 24_000) ``` For streaming applications, `model.generate()` accepts `streamer` for text tokens and `audio_streamer` for speech frames. LiquidCodec supports incremental decoding of those frames.

Speech-to-text chat, with or without reasoning

For a text-only answer, no specific system prompt is needed. Prefill `` to let the model reason before answering, or `` for a direct response. ```python use_thinking = True conversation = [ {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, {"role": "assistant", "content": "" if use_thinking else ""}, ] inputs = processor.apply_chat_template( conversation, continue_final_message=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=1024, suppress_mixed=True, suppress_audio_out=True, return_dict_in_generate=True, ) text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True, ) if use_thinking: _, separator, text = text.partition("") if not separator: raise RuntimeError("Reasoning has not finished; increase max_new_tokens.") print(text.strip()) ```

Audio understanding

Include the recording and a question in the user turn. Questions can address speech, music, background sounds, or events. No specific system prompt is required; to ground answers in the recording, use `Answer questions about the provided audio clip. Base every answer only on what can be heard.`. ```python conversation = [ { "role": "system", "content": "Answer questions about the provided audio clip. Base every answer only on what can be heard.", }, { "role": "user", "content": [ {"type": "audio", "audio": "recording.wav"}, {"type": "text", "text": "What is happening in this recording?"}, ], }, ] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=1024, suppress_mixed=True, suppress_audio_out=True, return_dict_in_generate=True, ) print(processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True, )) ``` For multiple-choice questions, include the choices in the text part and ask the model to select an answer.

Translation

Use a system prompt that specifies the source language, target language, and output format: - **Speech to text:** `Translate the following speech from {source} to {target} in text form.` - **Text to text:** `Translate the following {source} text into {target}.` Replace `{source}` and `{target}` with the English names of the source and target languages. Follow the ASR example with the translation prompt; for text input, set the user message's `content` to the text to translate. Translation supports the speech languages listed above; translation to or from English generally works best.

Tool calling from spoken requests

Pass function schemas through `tools=...` to `apply_chat_template()`. The chat template appends them to the system prompt, one JSON definition per line inside `…`. Requests may be spoken or written. With the system prompt `Respond with function calls, and interleaved text and audio.\n\nList of tools:`, the model either calls tools or answers with interleaved text and speech. Do not prefill the assistant turn: the model chooses whether to start with `` or `<|mixed_start|>`. A tool call looks like this, with multiple calls separated by a blank line: ``` 30 ``` Values other than strings are written as JSON. Your application parses and runs the calls, then appends the assistant's `tool_calls` (with `arguments` as a dict) and a `tool` message with the results to the conversation. The template wraps the results in `…`. Generate again with the same `tools`: the model either makes more calls or answers. ```python import json import re tools = [{ "type": "function", "function": { "name": "set_volume", "description": "Set the device speaker volume.", "parameters": { "type": "object", "properties": { "volume": {"type": "integer", "description": "Volume from 0 to 100."} }, "required": ["volume"], }, }, }] functions = {"set_volume": lambda volume: {"volume": volume}} # your implementations CALL_RE = re.compile(r"\s*\n]+)>(.*?)\s*", re.DOTALL) PARAM_RE = re.compile(r"\n]+)>\n?(.*?)\n?", re.DOTALL) def parse_tool_calls(text): calls = [] for name, body in CALL_RE.findall(text): arguments = {} for key, value in PARAM_RE.findall(body): try: arguments[key] = json.loads(value) except json.JSONDecodeError: arguments[key] = value # plain string calls.append({"type": "function", "function": {"name": name, "arguments": arguments}}) return calls def respond(conversation): inputs = processor.apply_chat_template( conversation, tools=tools, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(device) with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=1024, audio_temperature=1.0, audio_top_k=128, audio_repetition_penalty=1.1, audio_repetition_window=4096, return_dict_in_generate=True, ) text = processor.decode( outputs.sequences[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True, ) return text, outputs conversation = [ { "role": "system", "content": "Respond with function calls, and interleaved text and audio.\n\nList of tools:", }, {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]}, ] text, outputs = respond(conversation) while calls := parse_tool_calls(text): results = [functions[c["function"]["name"]](**c["function"]["arguments"]) for c in calls] conversation += [ {"role": "assistant", "tool_calls": calls}, {"role": "tool", "content": json.dumps(results)}, ] text, outputs = respond(conversation) print(text) with torch.no_grad(): waveform = codec.decode( outputs.audio_codes[:1, :, :outputs.audio_lengths[0]] ).audio_values[0] # 24 kHz sf.write("reply.wav", waveform.float().cpu().numpy(), 24_000) ``` **Text-only answers:** use the system prompt `List of tools:` (keep `tools=`) and pass `suppress_mixed=True, suppress_audio_out=True` to `generate()`. Calls and results are handled the same way.

## Fine-tuning

LFM3-Audio-2B is designed for **easy customization** for a given domain, task, or application with Hugging Face Transformers.

Training	Use case	Notebook
SFT	Audio-to-function calling for smart home control
SFT	Text-to-speech (TTS)

## Performance

### Benchmarks LFM3-Audio-2B improves over LFM2.5-Audio in speech quality, spoken instruction following, and audio understanding, while supporting multilingual speech and tool calling.

Benchmark	LFM3-Audio 2B	LFM2.5-Audio 1.5B	Gemma 4 E2B	Gemma 4 E4B	Qwen2.5-Omni 3B	NVIDIA Audex 2B	Voxtral Mini 3B
Total parameters	2B	1.5B	5B	8B	6B	3B	5B
English ASR WER ↓	6.23	6.13	8.26	7.82	6.30	5.99	5.51
Multilingual ASR WER / CER ↓	6.03	—	8.08	6.54	7.41	6.40	7.76
English TTS UTMOSv2 ↑	3.83	3.22	—	—	3.42	—	—
VoiceBench ↑	73.13	55.17	63.28	72.90	66.37	22.55	65.78
Tool calling internal eval ↑	58.50	—	55.60	72.40	44.30	—	44.50
Audio understanding MMAR ↑	50.60	25.85	44.49	49.10	54.41	53.21	50.10

Evaluation details

English ASR: average word error rate (WER) in percent on the ESB / Open ASR benchmark suite, lower is better.
Multilingual ASR: mean error rate in percent on FLEURS across English, Spanish, Portuguese, German, French, Italian, Chinese, Japanese, and Korean, lower is better. WER is used for all languages except Chinese, Japanese, and Korean, which use character error rate (CER).
English TTS: UTMOSv2 speech quality score on a 1–5 scale, higher is better.
VoiceBench: audio input, text output, percentage score, higher is better.
Tool calling: internal evaluation with audio input, percentage score, higher is better.
Audio understanding: MMAR, percentage score, higher is better.

### Streaming performance The causal encoder processes live microphone input and allows the backbone to prefill while you speak, reducing the wait after you finish. Time to first token can be **as low as 50 ms**. LiquidCodec converts generated tokens into speech at about **87× real time on a single CPU thread on Apple M4 Max**. It supports streaming decoding so speech can be played as it is generated. ## Architecture

LFM3-Audio-2B architecture: 16 kHz audio input passes through a causal, streaming audio encoder into the LFM3 backbone, and text input enters the backbone directly. The backbone outputs text and tool calls, or speech tokens that LiquidCodec, at 12 tokens per 80 ms, decodes into 24 kHz speech.

**Causal audio encoder:** our LFM-based encoder combines sliding-window attention and short convolution layers to keep memory utilization bounded. Its fully causal design supports live audio input and incremental prefill. **LiquidCodec:** our in-house audio tokenizer powers multilingual speech generation in both LFM3-Audio and LFM3-Omni. Built on the LFM3 architecture, it represents each **80 ms of speech with 12 discrete tokens** and supports streaming decoding. **Parallel speech prediction:** multiple audio tokens are predicted iteratively in parallel, replacing the sequential prediction used by previous LFM Audio models. As the backbone generates tokens, the detokenizer progressively turns them into speech faster than real time, keeping conversations responsive. ## Citation

```bibtex @article{liquidAI2026LFM3, author = {Liquid AI}, title = {Introducing LFM3: Every Modality for Specialized Intelligence}, journal = {Liquid AI Blog}, year = {2026}, note = {www.liquid.ai/blog/lfm3}, } ``` For questions and feedback, join [Discord](https://discord.com/invite/liquid-ai). For custom deployment and fine-tuning support, [contact Liquid AI](https://www.liquid.ai/connect?utm_source=huggingface&utm_medium=referral&utm_campaign=huggingface).