README.md

---
pipeline_tag: automatic-speech-recognition
license: apache-2.0
base_model: ibm-granite/granite-speech-4.1-2b-plus
library_name: zeromodels
tags:
- keras
- zeromodels
- granite-speech-plus
- speech-llm
- automatic-speech-recognition
- audio
- arxiv:2505.08699
- pytorch
- jax
- tf
---

## ***See [our collection](https://huggingface.co/collections/zeromodels/granite-speech-plus-6a8eaf2de7ec075355e7176c) for all versions of Granite Speech Plus.***

# Run Granite Speech Plus with Keras 3: JAX, PyTorch, or TensorFlow

[![GitHub](https://img.shields.io/badge/GitHub-ZeroModels-black?logo=github)](https://github.com/IMvision12/ZeroModels) [![Docs](https://img.shields.io/badge/Docs-Granite%20Speech%20Plus-blue)](https://imvision12.github.io/ZeroModels/granite_speech_plus/) [![Collection](https://img.shields.io/badge/HF-Granite%20Speech%20Plus%20collection-yellow)](https://huggingface.co/collections/zeromodels/granite-speech-plus-6a8eaf2de7ec075355e7176c)

# zeromodels/granite_speech_4_1_2b_plus

Paper: [Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities (arXiv:2505.08699)](https://arxiv.org/abs/2505.08699) · [HF Papers](https://huggingface.co/papers/2505.08699)

Granite Speech Plus is the **Granite 4.0-based** speech-aware LLM successor to Granite Speech: a conformer CTC encoder and Q-Former projector feed audio embeddings into `<|audio|>` slots of a Granite decoder. You ask for what you want in words (transcribe, summarize, answer). LoRA is fully merged; no adapter toggle.

For more details on the model, please go to the upstream [model card](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus).

Pure-**Keras 3** conversion of [`ibm-granite/granite-speech-4.1-2b-plus`](https://huggingface.co/ibm-granite/granite-speech-4.1-2b-plus) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.

This is a **speech LLM** checkpoint (`GraniteSpeechPlusConditionalGenerate`) on **Granite 4.0 2B**. Prefer `load_dtype="bfloat16"`.

## ✨ Quick start

```python
import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

import keras
import numpy as np
import soundfile as sf
from zeromodels.models.granite_speech_plus import (
    GraniteSpeechPlusConditionalGenerate,
    GraniteSpeechPlusProcessor,
)

model = GraniteSpeechPlusConditionalGenerate.from_weights(
    "zeromodels/granite_speech_4_1_2b_plus", load_dtype="bfloat16"
)
processor = GraniteSpeechPlusProcessor.from_weights("zeromodels/granite_speech_4_1_2b_plus")

audio, sr = sf.read("your_audio.wav", dtype="float32")  # 16 kHz mono

# Instruction in words: this is a speech LLM, not fixed-task ASR.
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "audio"},
            {
                "type": "text",
                "text": "can you transcribe the speech into a written format?",
            },
        ],
    }
]

inputs = processor(conversation=conversation, audio=audio, sampling_rate=sr)
out = model.generate(**inputs, max_new_tokens=64)
ids = np.asarray(keras.ops.convert_to_numpy(out))[0].tolist()
print(repr(processor.tokenizer.decode(ids)))
```

Load any Granite Speech Plus variant the same way with `from_weights("zeromodels/<variant>")`:

| Variant | Hub | Base LLM |
|---|---|---|
| `granite_speech_4_1_2b_plus` | [`zeromodels/granite_speech_4_1_2b_plus`](https://huggingface.co/zeromodels/granite_speech_4_1_2b_plus) | Granite 4.0 2B |

## Tips

- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Pass audio via `audio=` + `sampling_rate=`; put only an `{"type": "audio"}` marker in the conversation (do not embed the waveform).
- Change the text instruction to get a different answer over the same clip.
- See [Granite Speech Plus docs](https://imvision12.github.io/ZeroModels/granite_speech_plus/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `GraniteSpeechPlusConditionalGenerate.from_weights("hf:ibm-granite/granite-speech-4.1-2b-plus")`.

## Special Thanks

A huge thank you to the IBM Granite authors for creating and releasing these models.

License: Apache 2.0.