---
language:
- en
pipeline_tag: audio-text-to-text
tags:
- audio
- speech
- voice-assistant
- voicebench
- gemma
---
# LFG-2
LFG-2 is a speech-in / text-out voice assistant: a trained audio projector bridging a
Gemma E4B audio encoder to a Gemma 4 31B language model. The audio encoder and
the language model are **frozen**; only the projector is trained, so the model
inherits the full text-side reasoning of Gemma 4 31B while learning to listen.
- **Input:** spoken English audio (16 kHz mono).
- **Output:** text. The model reasons in a hidden thought channel (emitting a
`…` transcription of what it heard as a comprehension check),
then produces the final answer.
## Usage
```bash
pip install -U transformers accelerate librosa
# On a bare machine, install the torch trio together first (matching CUDA build):
# pip install torch torchvision torchaudio
```
LFG-2 loads exactly like base Gemma 4 E4B audio — `AutoProcessor` +
`AutoModelForMultimodalLM` — the only difference is **`trust_remote_code=True`**,
which lets the repo's bundled model class install the trained projector for you.
```python
import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM
REPO = "glenn2/LFG-2"
processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForMultimodalLM.from_pretrained(
REPO, dtype="auto", device_map="auto", trust_remote_code=True,
)
# Ask a question with your voice (audio can be a local path or a URL; 16 kHz mono).
messages = [
{"role": "system", "content": "You are a helpful voice assistant. Answer the user's spoken question clearly and concisely."},
{"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
messages, tokenize=True, return_dict=True, return_tensors="pt",
add_generation_prompt=True, enable_thinking=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=2048)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
# parse_response strips the <|channel>thought… block; keep the answer.
print(processor.parse_response(response)["content"])
```
Without trust_remote_code (manual projector install)
If you'd rather not run repo code, load the base model normally and install the
projector yourself (needs `huggingface_hub`):
```python
import sys, torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor, AutoModelForMultimodalLM
local = snapshot_download("glenn2/LFG-2")
processor = AutoProcessor.from_pretrained(local)
model = AutoModelForMultimodalLM.from_pretrained(local, dtype="auto", device_map="auto")
sys.path.insert(0, local)
from deep_projector import install_deep_projector
ckpt = torch.load(f"{local}/projector_final.pt", map_location=model.device)
deep = install_deep_projector(model, hidden=ckpt["config"]["hidden"],
n_hidden_layers=ckpt["config"]["mlp_layers"])
deep.load_state_dict(ckpt["state_dict"], strict=True)
# ... then apply_chat_template / generate / parse_response as above.
```
### Notes
- **Audio format:** 16 kHz mono. Resample first (e.g. `librosa.load(path, sr=16000)`).
- **Thinking:** with `enable_thinking=True` the model emits
`<|channel>thought … ` (including a `…` transcript of
the audio) before the answer. `processor.parse_response(...)["content"]`
returns just the final answer; strip any residual `…` if present.
- **Decoding:** greedy (`do_sample=False`) is reproducible and used for all
benchmark numbers below. For more varied generation use Gemma-4 sampling:
`do_sample=True, temperature=1.0, top_p=0.95, top_k=64`.
- **Stop token:** generation ends on `` (the Gemma 4 turn terminator).
- **Long answers / runaway thinking:** cap total length with `max_new_tokens`
and optionally force-close the thought channel after a fixed budget by writing
a small `LogitsProcessor` that forces the `` token once N tokens have
been generated (see `deep_projector.py` / the training repo for the reference
`ThinkingBudgetProcessor`).
## VoiceBench
Evaluated with the official [VoiceBench](https://github.com/MatthewCYM/VoiceBench)
protocol (greedy decoding, thinking enabled).
| Subset | Score |
| --- | ---: |
| AlpacaEval | 4.67 |
| CommonEval | 4.26 |
| WildVoice | 4.23 |
| SD-QA (USA) | 73.42 |
| MMSU | 85.20 |
| OpenBookQA | 93.63 |
| BBH | 87.10 |
| IFEval | 84.54 |
| AdvBench | 95.77 |
| **Overall** | **86.98** |
## Intended use & limitations
- Designed for **English** spoken questions/instructions → text answers.
- Not a transcription service (though it transcribes internally); not intended
for non-speech audio, speaker ID, or languages other than English.