| --- |
| license: apache-2.0 |
| language: |
| - en |
| base_model: |
| - google/gemma-4-26B-A4B-it |
| tags: |
| - Audio-Text-to-Text |
| - multimodal |
| --- |
|  |
|
|
| ## Model Details |
|
|
| * **Model Name:** LFG-1 (Listening Fusion Gemma) |
| * **Model Type:** Multimodal Conversational Audio-Language Model |
| * **Background:** LFG-1 was built as a personal learning project, trained entirely on Apple Silicon at my house on nights and weekends. It isn't an official model. |
| * **Architecture:** LFG-1 bridges the **Gemma 4 E2B audio encoder** with the **Gemma 4 26B-A4B text model**. The connection is made via a custom-trained projection layer, allowing the language model to natively ingest and understand raw acoustic features without relying on an intermediate text transcript. |
| * **Framework:** MLX (`mlx-vlm`) |
| * **License:** Apache 2.0 |
|
|
| ## Intended Use |
|
|
| * **Primary Use Case:** Real-time conversational audio applications running locally on Mac hardware. |
| * **Capabilities:** Native speech-to-response generation, streaming text output, and multimodal support (simultaneous audio and image input). Because the model processes raw acoustic data, it captures conversational pacing, pauses, and tone that traditional Speech-to-Text (STT) pipelines strip away. |
| * **Untouched Text Reasoning:** LFG-1 keeps Gemma 4’s text backbone frozen during audio-projection training, so its core language and reasoning capabilities are expected to remain essentially unchanged while gaining a trained audio input pathway. |
|
|
| ## Language Support |
|
|
| * **Current Status:** The audio projection layer is currently trained exclusively on **English**. |
| * **Future Roadmap:** This is an active learning project (hence the "1" in LFG-1). Future iterations will focus on expanding the audio training data. Requests, suggestions, and dataset recommendations for additional languages are highly welcome in the repository Issues! |
|
|
| ## Hardware Requirements |
|
|
| Due to the size of the combined weights (~48 GB), LFG-1 requires substantial unified memory. |
| * **Minimum Requirement:** Apple Silicon with at least **64 GB of Unified Memory**. |
| * **Storage:** ~50 GB of disk space for local weights. |
|
|
| ## Install |
|
|
| Install `mlx-vlm`, then clone and install the LFG wrapper code: |
|
|
| ```bash |
| pip install mlx-vlm |
| git clone https://github.com/codemadeio/LFG-1.git |
| cd LFG-1 |
| pip install -e |
| ``` |
|
|
| ## Usage |
|
|
| LFG-1 is designed as a drop-in model type for `mlx-vlm`. Once the `lfg` module is registered, it can be loaded like any other Hugging Face model. |
|
|
| ```python |
| |
| from lfg import register |
| register() |
| |
| from mlx_vlm import load, generate |
| |
| model, processor = load("glenn2/LFG-1") |
| |
| response = generate( |
| model, processor, |
| prompt="<|audio|>", |
| audio=["recording.wav"], |
| max_tokens=512, |
| ) |
| print(response.text) |
| |
| ``` |
|
|
| ## Example |
| ```python |
| git clone https://github.com/codemadeio/LFG-1.git |
| cd LFG-1/examples |
| pip install flask |
| brew install ffmpeg |
| python app.py |
| # open http://127.0.0.1:5001 |
| ``` |
|
|
|
|
|
|
|
|
|
|
