Granite-Speech-3.3-2b — ONNX (speech-llm)

ONNX export of ibm-granite/granite-speech-3.3-2b for onnx-asr with the speech-llm model type (requires the speech-llm model-family patches, see TigreGotico/onnx-asr PR #3).

The source model applies a LoRA adapter to the language model only when the turn contains audio. Audio is always present for ASR, so the adapter is merged into the base weights before the export and the decoder is a plain causal LM. Merged output matches the adapter path character for character.

Graphs

Graph	Inputs	Outputs
`encoder.onnx`	`input_features` `(1, N)` f32 raw 16 kHz waveform	`audio_embeds` `(1, L, 2048)` f32
`embed_tokens.onnx`	`input_ids` `(1, S)` i64	`inputs_embeds` `(1, S, 2048)` f32
`decoder.onnx`	`inputs_embeds` `(1, S, 2048)`, `attn_bias` `(1, 1, S, P+S)`, `position_ids` `(1, S)`, `past_key_values.{0..39}.{key,value}` `(1, 8, P, 64)`	`logits` `(1, S, 49160)`, `present.{0..39}.{key,value}` `(1, 8, P+S, 64)`

decoder.onnx is a merged prefill and decode graph: a zero-length past is valid, so there is no use_cache_branch. Prompt token ids (the Granite chat template with the transcription instruction) are baked into config.json, so no tokenizer encoder is needed at runtime.

The Granite feature extractor (torchaudio mel spectrogram, n_fft 512, window 400, hop 160, 80 HTK mel bins, log10 with an 8 dB floor, then pairs of frames stacked into 160 features) matches no onnx-asr preprocessor. It is computed inside encoder.onnx instead, and the model declares "preprocessor": "identity", so the runtime hands the raw waveform to the graph. The audio length stays dynamic; there is no fixed 30 s window.

Usage

import onnx_asr
model = onnx_asr.load_model("speech-llm", "path/to/this/repo")  # or quantization="int8"
print(model.recognize("audio_16khz.wav"))

Accuracy

Google FLEURS validation clips, two en_us and two pt_br, greedy decoding. fp32 matches native transformers (with the LoRA adapter active, fp32 CPU) character for character on all four clips.

int8 is dynamic weight quantization. The text graphs survive it well, but the conformer encoder does not: quantizing it whole destroys the audio embeddings (cosine 0.43 against fp32). Keeping the in-graph mel front end in fp32 raises that to 0.96, which is what this repo ships. Even then int8 changes wording on longer clips. Use fp32 when the transcript matters.

Clip	fp32	int8
en_1	identical to native	identical to native
en_2	identical to native	"the predominant flavors being peanuts, chili, sugar"
pt_1	identical to native	"que oriente e pite as ciencias"
pt_2	identical to native	"espiar e uma atividade", "skibums"

Files

File	Size
`encoder.onnx` + `encoder.onnx_data`	0.7 MB + 1.90 GB
`encoder_int8.onnx`	480.6 MB
`embed_tokens.onnx`	402.7 MB
`embed_tokens_int8.onnx`	100.7 MB
`decoder.onnx` + `decoder.onnx_data`	2.1 MB + 10.13 GB
`decoder_int8.onnx` + `decoder_int8.onnx_data`	3.1 MB + 2.53 GB
`vocab.json`	875 kB
`config.json`	1 kB

fp32 is about 12.4 GB in total, int8 about 3.1 GB.

Limitations

Batch 1. recognize_batch loops over the waveforms; because the preprocessor is the identity, a padded batch would send its zero padding to the encoder, so batch entries must have equal length to be exact.
Greedy decoding only. No beam search and no timestamps.
English, French, German, Spanish and Portuguese, as in the source model. Speech translation is not wired up: the baked prompt is the transcription instruction.
Free-text prompts would need a tokenizer encoder at runtime.

License: Apache-2.0, inherited from the source model.