README.md

---
language:
- en
pipeline_tag: audio-text-to-text
tags:
- audio
- speech
- voice-assistant
- voicebench
- gemma
---
# LFG-2


![LFG-2-silver](https://cdn-uploads.huggingface.co/production/uploads/61b37e66986f43ddf4956d21/3W5Xz_7oM94PRMUstX6yx.png)


LFG-2 is a speech-in / text-out voice assistant: a trained audio projector bridging a
Gemma E4B audio encoder to a Gemma 4 31B language model. The audio encoder and
the language model are **frozen**; only the projector is trained, so the model
inherits the full text-side reasoning of Gemma 4 31B while learning to listen.

- **Input:** spoken English audio (16 kHz mono).
- **Output:** text. The model reasons in a hidden thought channel (emitting a
  `<heard>…</heard>` transcription of what it heard as a comprehension check),
  then produces the final answer.


## Usage

```bash
pip install -U transformers accelerate librosa

# On a bare machine, install the torch trio together first (matching CUDA build):
#   pip install torch torchvision torchaudio
```

LFG-2 loads exactly like base Gemma 4 E4B audio — `AutoProcessor` +
`AutoModelForMultimodalLM` — the only difference is **`trust_remote_code=True`**,
which lets the repo's bundled model class install the trained projector for you.

```python
import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM

REPO = "glenn2/LFG-2"
processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForMultimodalLM.from_pretrained(
    REPO, dtype="auto", device_map="auto", trust_remote_code=True,
)

# Ask a question with your voice (audio can be a local path or a URL; 16 kHz mono).
messages = [
    {"role": "system", "content": "You are a helpful voice assistant. Answer the user's spoken question clearly and concisely."},
    {"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
    messages, tokenize=True, return_dict=True, return_tensors="pt",
    add_generation_prompt=True, enable_thinking=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=2048)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)

# parse_response strips the <|channel>thought…<channel|> block; keep the answer.
print(processor.parse_response(response)["content"])
```

<details>
<summary>Without <code>trust_remote_code</code> (manual projector install)</summary>

If you'd rather not run repo code, load the base model normally and install the
projector yourself (needs `huggingface_hub`):

```python
import sys, torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor, AutoModelForMultimodalLM

local = snapshot_download("glenn2/LFG-2")
processor = AutoProcessor.from_pretrained(local)
model = AutoModelForMultimodalLM.from_pretrained(local, dtype="auto", device_map="auto")

sys.path.insert(0, local)
from deep_projector import install_deep_projector
ckpt = torch.load(f"{local}/projector_final.pt", map_location=model.device)
deep = install_deep_projector(model, hidden=ckpt["config"]["hidden"],
                              n_hidden_layers=ckpt["config"]["mlp_layers"])
deep.load_state_dict(ckpt["state_dict"], strict=True)
# ... then apply_chat_template / generate / parse_response as above.
```

</details>

### Notes

- **Audio format:** 16 kHz mono. Resample first (e.g. `librosa.load(path, sr=16000)`).
- **Thinking:** with `enable_thinking=True` the model emits
  `<|channel>thought … <channel|>` (including a `<heard>…</heard>` transcript of
  the audio) before the answer. `processor.parse_response(...)["content"]`
  returns just the final answer; strip any residual `<heard>…</heard>` if present.
- **Decoding:** greedy (`do_sample=False`) is reproducible and used for all
  benchmark numbers below. For more varied generation use Gemma-4 sampling:
  `do_sample=True, temperature=1.0, top_p=0.95, top_k=64`.
- **Stop token:** generation ends on `<turn|>` (the Gemma 4 turn terminator).
- **Long answers / runaway thinking:** cap total length with `max_new_tokens`
  and optionally force-close the thought channel after a fixed budget by writing
  a small `LogitsProcessor` that forces the `<channel|>` token once N tokens have
  been generated (see `deep_projector.py` / the training repo for the reference
  `ThinkingBudgetProcessor`).

## VoiceBench

Evaluated with the official [VoiceBench](https://github.com/MatthewCYM/VoiceBench)
protocol (greedy decoding, thinking enabled).

| Subset | Score |
| --- | ---: |
| AlpacaEval | 4.67 |
| CommonEval | 4.26 |
| WildVoice | 4.23 |
| SD-QA (USA) | 73.42 |
| MMSU | 85.20 |
| OpenBookQA | 93.63 |
| BBH | 87.10 |
| IFEval | 84.54 |
| AdvBench | 95.77 |
| **Overall** | **86.98** |

## Intended use & limitations

- Designed for **English** spoken questions/instructions → text answers.
- Not a transcription service (though it transcribes internally); not intended
  for non-speech audio, speaker ID, or languages other than English.