File size: 5,015 Bytes
767ac28 8ea5f17 441a9bd c17ae3c 441a9bd 767ac28 441a9bd 767ac28 d416979 767ac28 441a9bd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 | ---
language:
- en
pipeline_tag: audio-text-to-text
tags:
- audio
- speech
- voice-assistant
- voicebench
- gemma
---
# LFG-2
LFG-2 is a speech-in / text-out voice assistant: a trained audio projector bridging a
Gemma E4B audio encoder to a Gemma 4 31B language model. The audio encoder and
the language model are **frozen**; only the projector is trained, so the model
inherits the full text-side reasoning of Gemma 4 31B while learning to listen.
- **Input:** spoken English audio (16 kHz mono).
- **Output:** text. The model reasons in a hidden thought channel (emitting a
`<heard>…</heard>` transcription of what it heard as a comprehension check),
then produces the final answer.
## Usage
```bash
pip install -U transformers accelerate librosa
# On a bare machine, install the torch trio together first (matching CUDA build):
# pip install torch torchvision torchaudio
```
LFG-2 loads exactly like base Gemma 4 E4B audio — `AutoProcessor` +
`AutoModelForMultimodalLM` — the only difference is **`trust_remote_code=True`**,
which lets the repo's bundled model class install the trained projector for you.
```python
import torch
from transformers import AutoProcessor, AutoModelForMultimodalLM
REPO = "glenn2/LFG-2"
processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForMultimodalLM.from_pretrained(
REPO, dtype="auto", device_map="auto", trust_remote_code=True,
)
# Ask a question with your voice (audio can be a local path or a URL; 16 kHz mono).
messages = [
{"role": "system", "content": "You are a helpful voice assistant. Answer the user's spoken question clearly and concisely."},
{"role": "user", "content": [{"type": "audio", "audio": "question.wav"}]},
]
inputs = processor.apply_chat_template(
messages, tokenize=True, return_dict=True, return_tensors="pt",
add_generation_prompt=True, enable_thinking=True,
).to(model.device)
input_len = inputs["input_ids"].shape[-1]
outputs = model.generate(**inputs, max_new_tokens=2048)
response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
# parse_response strips the <|channel>thought…<channel|> block; keep the answer.
print(processor.parse_response(response)["content"])
```
<details>
<summary>Without <code>trust_remote_code</code> (manual projector install)</summary>
If you'd rather not run repo code, load the base model normally and install the
projector yourself (needs `huggingface_hub`):
```python
import sys, torch
from huggingface_hub import snapshot_download
from transformers import AutoProcessor, AutoModelForMultimodalLM
local = snapshot_download("glenn2/LFG-2")
processor = AutoProcessor.from_pretrained(local)
model = AutoModelForMultimodalLM.from_pretrained(local, dtype="auto", device_map="auto")
sys.path.insert(0, local)
from deep_projector import install_deep_projector
ckpt = torch.load(f"{local}/projector_final.pt", map_location=model.device)
deep = install_deep_projector(model, hidden=ckpt["config"]["hidden"],
n_hidden_layers=ckpt["config"]["mlp_layers"])
deep.load_state_dict(ckpt["state_dict"], strict=True)
# ... then apply_chat_template / generate / parse_response as above.
```
</details>
### Notes
- **Audio format:** 16 kHz mono. Resample first (e.g. `librosa.load(path, sr=16000)`).
- **Thinking:** with `enable_thinking=True` the model emits
`<|channel>thought … <channel|>` (including a `<heard>…</heard>` transcript of
the audio) before the answer. `processor.parse_response(...)["content"]`
returns just the final answer; strip any residual `<heard>…</heard>` if present.
- **Decoding:** greedy (`do_sample=False`) is reproducible and used for all
benchmark numbers below. For more varied generation use Gemma-4 sampling:
`do_sample=True, temperature=1.0, top_p=0.95, top_k=64`.
- **Stop token:** generation ends on `<turn|>` (the Gemma 4 turn terminator).
- **Long answers / runaway thinking:** cap total length with `max_new_tokens`
and optionally force-close the thought channel after a fixed budget by writing
a small `LogitsProcessor` that forces the `<channel|>` token once N tokens have
been generated (see `deep_projector.py` / the training repo for the reference
`ThinkingBudgetProcessor`).
## VoiceBench
Evaluated with the official [VoiceBench](https://github.com/MatthewCYM/VoiceBench)
protocol (greedy decoding, thinking enabled).
| Subset | Score |
| --- | ---: |
| AlpacaEval | 4.67 |
| CommonEval | 4.26 |
| WildVoice | 4.23 |
| SD-QA (USA) | 73.42 |
| MMSU | 85.20 |
| OpenBookQA | 93.63 |
| BBH | 87.10 |
| IFEval | 84.54 |
| AdvBench | 95.77 |
| **Overall** | **86.98** |
## Intended use & limitations
- Designed for **English** spoken questions/instructions → text answers.
- Not a transcription service (though it transcribes internally); not intended
for non-speech audio, speaker ID, or languages other than English.
|