--- license: apache-2.0 library_name: transformers base_model: google/gemma-4-E4B-it base_model_relation: finetune pipeline_tag: any-to-any language: - en tags: - decisions - typed-decisions - audio - vision - calibration - on-device - smaug - abacusai ---

State-of-the-art audio decisions in 4B parameters, and it sees and reads too.
60.1 Earshot audio skill, ahead of Gemini 4 Argon (59.4) and Gemini 3.8 Flash (50.5), and better calibrated than both.

## 1. Model Introduction Smaug-Decisions is a 4B decision model that hears, sees and reads. A fine-tune of Google's [Gemma 4 E4B-it](https://huggingface.co/google/gemma-4-E4B-it) by Abacus.AI, it answers a typed question about **audio, an image or text** with a calibrated probability per option, in one forward pass and with no generated text.

Four example decisions over audio, an image and text, each answered with a probability per option in one forward pass

60.1 Earshot audio skill, three input types, one forward pass per decision, 0.069 calibration error

It is the only 4B decision model with results on all three input types. On audio it is ahead of the frontier APIs: **60.1** skill on **Earshot**, our 6,671-item audio decision benchmark, against 59.4 for Gemini 4 Argon and 50.5 for Gemini 3.8 Flash, and better calibrated than both (ECE 0.069 vs 0.123 and 0.204). On the public Jev Decision Index it scores **57.9** on images and **56.6** on text, the best Gemma 4 E4B on both. ```python import sys from huggingface_hub import snapshot_download sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"])) from smaug_decide import SmaugDecide # ships with this repo model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions") model("call.wav", "Has the speaker finished their turn?") # -> {'Yes': 0.93, 'No': 0.07} ```

Every 4B decision model by input type: Smaug-Decisions is the only one with audio, image and text results

## 2. Highlights - **Ahead of the frontier APIs on audio,** and better calibrated, so the probabilities work as thresholds. - **Hears, sees and reads:** audio up to 30 s, images and text through one prompt and one readout. - **One pass, no decoding:** about 200 ms per audio decision on an H100. - **Runs on phones:** [Smaug-Decisions-LiteRT](https://huggingface.co/abacusai/Smaug-Decisions-LiteRT) is 3.45 GB, keeps 59.4 audio skill and answers a text question in about 0.6 s on an iPhone 14 Pro Max, fully offline. ## 3. Model Summary

Architecture	Gemma 4 E4B-it (language model + audio and vision encoders); letter-logit readout
Parameters	4B effective, 8.6B stored (3.5B of the stored weights are embedding tables that are looked up, not computed; the Hugging Face counter shows the stored total)
Input	Audio up to 30 s (16 kHz mono), an image (560 image tokens), or text, plus a question with 2 to 20 options
Output	A calibrated probability per option (softmax for one answer, sigmoid for select-all)
Hardware	NVIDIA GPU (bfloat16), Apple Silicon (MPS) and CPU through Transformers; phones through Smaug-Decisions-LiteRT
Precision	bfloat16
Formats	Transformers (repo root); LiteRT-LM for phones (Smaug-Decisions-LiteRT)
Base Model	`google/gemma-4-E4B-it`
Adaptation	Full fine-tune

## 4. Evaluation ### Audio: Earshot **Earshot** is our audio decision benchmark: 6,671 items and 17.2 hours of audio in four tracks: **conversation dynamics** (turn completion, next speaker, speaker count, speech activity; from VoxConverse, AMI and Smart Turn), **external turn completion** (Smart Turn v3.2 test), **sound and music** (genre, instrument, tempo, scene, sound events) and a small **pilot** track. **Audio skill** is the mean task skill per track, averaged over the tracks: 0 is guessing, 100 is perfect. No benchmark recording or item is in the training data.

Earshot audio skill per track, and calibration error

Earshot table

	Smaug-Decisions	Gemini 4 Argon (API)	Gemini 3.8 Flash (API)	AudioJev (Qwen2.5-Omni-3B)	Gemma 4 E4B-it (zero-shot)
Earshot (audio decisions)
Audio skill (headline) ↑	60.1	59.4	50.5	43.6	5.0
ECE ↓	0.069	0.123	0.204	0.191	0.464
Per track (skill) ↑
Conversation dynamics (3,000)	48.2	49.6	48.5	26.7	3.6
External turn completion (1,000)	80.8	76.5	81.8	62.0	11.2
Sound & music (1,115)	66.1	64.7	42.4	57.6	5.2
Pilot (80)	45.2	47.0	29.1	28.2	0.1

Bold = best in the row. Shading is relative within each row (darker = better). Every model scored on the same 6,671 items with the same harness; the APIs received the audio and returned JSON probabilities. Skill = (accuracy − chance) / (1 − chance), times 100.

Need it faster? [Smaug-Audio-Decisions-0.7B](https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B) scores 53.0 on Earshot at 9.7 ms; in front of this model as a [cascade](https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B#the-cascade) it matches this model's accuracy (60.1) at 130 ms instead of 210 ms. ### Images and text: the public Jev Decision Index The matrix at the top has the headline numbers: 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both (Winnow-E4B, the other Gemma 4 E4B fine-tune on the boards: 48.1 and 41.7). Images: our run of the official vision kit (0.4, public sets) at 560 image tokens.

Full Decision Index breakdown

Text decision skill per area

	Smaug-Decisions (audio + image + text)	Perplexity Decider Flash (4B)	vLLM-SR d3-mini (4B)	RSI-Jev v6.1 VL (4B)	Winnow (Gemma 4 E4B)
Jev Decision Index 0.4.1, text (same 35 public benchmarks)
Public skill ↑	56.6	61.8	56.7	52.9	41.7
Knowledge & reasoning	35.1	46.0	44.3	35.2	25.3
Language	66.4	67.2	60.4	59.2	47.5
Retrieval	64.7	65.2	63.6	57.1	43.8
Tools	72.9	81.0	72.0	68.4	62.5
Arts	41.6	46.6	37.7	45.7	26.9
Earshot (audio decisions)
Audio skill ↑	60.1	—	—	—	—

Bold = best in the row. Other models: the public Decision Index board as of 2026-10-28 (text 0.4.1). Smaug-Decisions: our runs of the official Decision Index kit (text on a 3,000-row sample of the public suite, compared on the same 35 benchmarks); self-reported, not board submissions. — = no published result.

## 5. Training Approach - **Readout, not generation.** The answer is read from the LM-head logits of the option letters at the answer position (tanh softcap 30, then softmax; select-all uses a sigmoid with one learned threshold). A decision costs one prefill. - **Decision-tuned and calibrated.** Full fine-tune of Gemma 4 E4B-it on a broad mix of typed decision tasks over audio, images and text, with calibration as an explicit objective and options shuffled so the model learns the options, not their positions. ## 6. Limitations - **Decisions, not text:** each call answers one typed question with a probability per option; the model does not write text. - **Per request:** one audio clip of up to 30 s (longer audio keeps its last 30 s), one image at 560 image tokens, up to 20 options. - **Thresholds:** set decision thresholds on your own data for high-stakes uses. ## 7. Usage ```bash pip install "transformers>=5.20" torch soundfile pillow huggingface_hub ``` ```python import sys from huggingface_hub import snapshot_download sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"])) from smaug_decide import SmaugDecide model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions") # CUDA, Apple Silicon (MPS) or CPU, picked automatically model("call.wav", "Has the speaker finished their turn?") # predicate -> {'Yes': p, 'No': 1-p} model("song.wav", "Which genre is playing?", ["rock", "jazz", "classical"]) # choice -> probabilities sum to 1 model("room.wav", "Which sounds are present?", ["speech", "music", "alarm"], multi=True) # select-all -> one probability each model("receipt.jpg", "Does the total exceed $50?") # images model("The order never arrived.", "How upset is the customer?", ["calm", "annoyed", "angry"], descriptions=["neutral tone", "complains", "threatens to leave"], rules="Judge the tone, not the facts.") ``` `from_pretrained` downloads the checkpoint (about 16 GB). Any runtime can reproduce the readout: run the prompt in `smaug_decide.render`, take the LM-head logits of the tokens `A`..`T` at the last position, apply the softcap from `readout.json`, then softmax (one answer) or `sigmoid(z - b)` with `b` from `readout.json` (select-all). With plain Transformers, `AutoModelForMultimodalLM.from_pretrained("abacusai/Smaug-Decisions", dtype="bfloat16")` loads the weights. ### On phones [Smaug-Decisions-LiteRT](https://huggingface.co/abacusai/Smaug-Decisions-LiteRT) is this model in one **3.45 GB** file for Google's on-device runtime LiteRT-LM, running on the phone GPU. At about a fifth of the size it keeps almost all of this model's quality. It runs in the Cascade app on iPhone and Android next to [Smaug-Audio-Decisions-0.7B](https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B), which answers the easy audio questions first. App source: [github.com/abacusai/Smaug-Decisions-Mobile](https://github.com/abacusai/Smaug-Decisions-Mobile). | | Smaug-Decisions (16 GB) | Smaug-Decisions-LiteRT (3.45 GB) | change | |---|---|---|---| | Earshot audio skill | 60.1 | **59.4** | −0.7 | | Earshot calibration error (ECE, lower is better) | 0.069 | **0.074** | +0.005 | | Decision Index text | 56.6 | **53.9** | −2.7 | | Decision Index images, 560 image tokens^* | 57.2 | **55.1** | −2.1 | | Decision Index images, 280 image tokens^* | 54.8 | **51.8** | −3.0 | Same items, same harness; the phone-build scores come from a PyTorch replica of the file's quantized weights. ^*10 of the 11 public vision sets (Winoground left out; this model's 11-set index is 57.9). | iPhone 14 Pro Max (6 GB RAM, phone GPU) | measured | |---|---| | first load (one time) / later loads | 13-21 s / 0.48 s | | text decision | ~0.6 s | | audio decision | ~1.6 s | | image decision (280 image tokens) | ~7.4 s | | peak app memory, with Smaug-Audio-Decisions-0.7B loaded | 1.35 GB | | storage | the 3.45 GB file + a ~2.4 GB cache written on first load | ## 8. License Apache 2.0, the same license as the base model, Gemma 4 E4B-it. ## 9. Citation ```bibtex @misc{abacusai2026smaugdecisions, title = {Smaug-Decisions: calibrated typed decisions over audio, images and text}, author = {Abacus.AI}, year = {2026}, url = {https://huggingface.co/abacusai/Smaug-Decisions} } ``` ## 10. Contact Us If you have any questions, please reach out at [Abacus.AI](https://abacus.ai).