Smaug-Decisions
Collection
Jev-type calibrated decision models for audio, images and text. Ahead of Gemini 4 Argon on audio decisions; runs offline on phones. • 4 items • Updated
State-of-the-art audio decisions in 4B parameters, and it sees and reads too.
60.1 Earshot audio skill, ahead of Gemini 4 Argon (59.4) and Gemini 3.8 Flash (50.5), and better calibrated than both.
1. Model Introduction
Smaug-Decisions is a 4B decision model that hears, sees and reads. A fine-tune of Google's Gemma 4 E4B-it by Abacus.AI, it answers a typed question about audio, an image or text with a calibrated probability per option, in one forward pass and with no generated text.
It is the only 4B decision model with results on all three input types. On audio it is ahead of the frontier APIs: 60.1 skill on Earshot, our 6,671-item audio decision benchmark, against 59.4 for Gemini 4 Argon and 50.5 for Gemini 3.8 Flash, and better calibrated than both (ECE 0.069 vs 0.123 and 0.204). On the public Jev Decision Index it scores 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both.
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))
from smaug_decide import SmaugDecide # ships with this repo
model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions")
model("call.wav", "Has the speaker finished their turn?") # -> {'Yes': 0.93, 'No': 0.07}
2. Highlights
- Ahead of the frontier APIs on audio, and better calibrated, so the probabilities work as thresholds.
- Hears, sees and reads: audio up to 30 s, images and text through one prompt and one readout.
- One pass, no decoding: about 200 ms per audio decision on an H100.
- Runs on phones: Smaug-Decisions-LiteRT is 3.45 GB, keeps 59.4 audio skill and answers a text question in about 0.6 s on an iPhone 14 Pro Max, fully offline.
3. Model Summary
| Architecture | Gemma 4 E4B-it (language model + audio and vision encoders); letter-logit readout |
| Parameters | 4B effective, 8.6B stored (3.5B of the stored weights are embedding tables that are looked up, not computed; the Hugging Face counter shows the stored total) |
| Input | Audio up to 30 s (16 kHz mono), an image (560 image tokens), or text, plus a question with 2 to 20 options |
| Output | A calibrated probability per option (softmax for one answer, sigmoid for select-all) |
| Hardware | NVIDIA GPU (bfloat16), Apple Silicon (MPS) and CPU through Transformers; phones through Smaug-Decisions-LiteRT |
| Precision | bfloat16 |
| Formats | Transformers (repo root); LiteRT-LM for phones (Smaug-Decisions-LiteRT) |
| Base Model | google/gemma-4-E4B-it |
| Adaptation | Full fine-tune |
4. Evaluation
Audio: Earshot
Earshot is our audio decision benchmark: 6,671 items and 17.2 hours of audio in four tracks: conversation dynamics (turn completion, next speaker, speaker count, speech activity; from VoxConverse, AMI and Smart Turn), external turn completion (Smart Turn v3.2 test), sound and music (genre, instrument, tempo, scene, sound events) and a small pilot track. Audio skill is the mean task skill per track, averaged over the tracks: 0 is guessing, 100 is perfect. No benchmark recording or item is in the training data.
Earshot table
| Smaug-Decisions | Gemini 4 Argon (API) |
Gemini 3.8 Flash (API) |
AudioJev (Qwen2.5-Omni-3B) |
Gemma 4 E4B-it (zero-shot) |
|
|---|---|---|---|---|---|
| Earshot (audio decisions) | |||||
| Audio skill (headline) ↑ | 60.1 | 59.4 | 50.5 | 43.6 | 5.0 |
| ECE ↓ | 0.069 | 0.123 | 0.204 | 0.191 | 0.464 |
| Per track (skill) ↑ | |||||
| Conversation dynamics (3,000) | 48.2 | 49.6 | 48.5 | 26.7 | 3.6 |
| External turn completion (1,000) | 80.8 | 76.5 | 81.8 | 62.0 | 11.2 |
| Sound & music (1,115) | 66.1 | 64.7 | 42.4 | 57.6 | 5.2 |
| Pilot (80) | 45.2 | 47.0 | 29.1 | 28.2 | 0.1 |
Bold = best in the row. Shading is relative within each row (darker = better). Every model scored on the same 6,671 items with the same harness; the APIs received the audio and returned JSON probabilities. Skill = (accuracy − chance) / (1 − chance), times 100.
Need it faster? Smaug-Audio-Decisions-0.7B scores 53.0 on Earshot at 9.7 ms; in front of this model as a cascade it matches this model's accuracy (60.1) at 130 ms instead of 210 ms.
Images and text: the public Jev Decision Index
The matrix at the top has the headline numbers: 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both (Winnow-E4B, the other Gemma 4 E4B fine-tune on the boards: 48.1 and 41.7). Images: our run of the official vision kit (0.4, public sets) at 560 image tokens.
Full Decision Index breakdown
| Smaug-Decisions (audio + image + text) |
Perplexity Decider Flash (4B) |
vLLM-SR d3-mini (4B) |
RSI-Jev v6.1 VL (4B) |
Winnow (Gemma 4 E4B) |
|
|---|---|---|---|---|---|
| Jev Decision Index 0.4.1, text (same 35 public benchmarks) | |||||
| Public skill ↑ | 56.6 | 61.8 | 56.7 | 52.9 | 41.7 |
| Knowledge & reasoning | 35.1 | 46.0 | 44.3 | 35.2 | 25.3 |
| Language | 66.4 | 67.2 | 60.4 | 59.2 | 47.5 |
| Retrieval | 64.7 | 65.2 | 63.6 | 57.1 | 43.8 |
| Tools | 72.9 | 81.0 | 72.0 | 68.4 | 62.5 |
| Arts | 41.6 | 46.6 | 37.7 | 45.7 | 26.9 |
| Earshot (audio decisions) | |||||
| Audio skill ↑ | 60.1 | — | — | — | — |
Bold = best in the row. Other models: the public Decision Index board as of 2026-10-28 (text 0.4.1). Smaug-Decisions: our runs of the official Decision Index kit (text on a 3,000-row sample of the public suite, compared on the same 35 benchmarks); self-reported, not board submissions. — = no published result.
5. Training Approach
- Readout, not generation. The answer is read from the LM-head logits of the option letters at the answer position (tanh softcap 30, then softmax; select-all uses a sigmoid with one learned threshold). A decision costs one prefill.
- Decision-tuned and calibrated. Full fine-tune of Gemma 4 E4B-it on a broad mix of typed decision tasks over audio, images and text, with calibration as an explicit objective and options shuffled so the model learns the options, not their positions.
6. Limitations
- Decisions, not text: each call answers one typed question with a probability per option; the model does not write text.
- Per request: one audio clip of up to 30 s (longer audio keeps its last 30 s), one image at 560 image tokens, up to 20 options.
- Thresholds: set decision thresholds on your own data for high-stakes uses.
7. Usage
pip install "transformers>=5.20" torch soundfile pillow huggingface_hub
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))
from smaug_decide import SmaugDecide
model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions") # CUDA, Apple Silicon (MPS) or CPU, picked automatically
model("call.wav", "Has the speaker finished their turn?") # predicate -> {'Yes': p, 'No': 1-p}
model("song.wav", "Which genre is playing?", ["rock", "jazz", "classical"]) # choice -> probabilities sum to 1
model("room.wav", "Which sounds are present?", ["speech", "music", "alarm"], multi=True) # select-all -> one probability each
model("receipt.jpg", "Does the total exceed $50?") # images
model("The order never arrived.", "How upset is the customer?", ["calm", "annoyed", "angry"],
descriptions=["neutral tone", "complains", "threatens to leave"], rules="Judge the tone, not the facts.")
from_pretrained downloads the checkpoint (about 16 GB). Any runtime can reproduce the readout: run the prompt in
smaug_decide.render, take the LM-head logits of the tokens A..T at the last position, apply the softcap from readout.json,
then softmax (one answer) or sigmoid(z - b) with b from readout.json (select-all). With plain Transformers,
AutoModelForMultimodalLM.from_pretrained("abacusai/Smaug-Decisions", dtype="bfloat16") loads the weights.
On phones
Smaug-Decisions-LiteRT is this model in one 3.45 GB file for Google's on-device runtime LiteRT-LM, running on the phone GPU. At about a fifth of the size it keeps almost all of this model's quality. It runs in the Cascade app on iPhone and Android next to Smaug-Audio-Decisions-0.7B, which answers the easy audio questions first. App source: github.com/abacusai/Smaug-Decisions-Mobile.
| Smaug-Decisions (16 GB) | Smaug-Decisions-LiteRT (3.45 GB) | change | |
|---|---|---|---|
| Earshot audio skill | 60.1 | 59.4 | −0.7 |
| Earshot calibration error (ECE, lower is better) | 0.069 | 0.074 | +0.005 |
| Decision Index text | 56.6 | 53.9 | −2.7 |
| Decision Index images, 560 image tokens* | 57.2 | 55.1 | −2.1 |
| Decision Index images, 280 image tokens* | 54.8 | 51.8 | −3.0 |
Same items, same harness; the phone-build scores come from a PyTorch replica of the file's quantized weights. *10 of the 11 public vision sets (Winoground left out; this model's 11-set index is 57.9).
| iPhone 14 Pro Max (6 GB RAM, phone GPU) | measured |
|---|---|
| first load (one time) / later loads | 13-21 s / 0.48 s |
| text decision | ~0.6 s |
| audio decision | ~1.6 s |
| image decision (280 image tokens) | ~7.4 s |
| peak app memory, with Smaug-Audio-Decisions-0.7B loaded | 1.35 GB |
| storage | the 3.45 GB file + a ~2.4 GB cache written on first load |
8. License
Apache 2.0, the same license as the base model, Gemma 4 E4B-it.
9. Citation
@misc{abacusai2026smaugdecisions,
title = {Smaug-Decisions: calibrated typed decisions over audio, images and text},
author = {Abacus.AI},
year = {2026},
url = {https://huggingface.co/abacusai/Smaug-Decisions}
}
10. Contact Us
If you have any questions, please reach out at Abacus.AI.
- Downloads last month
- 13
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("abacusai/Smaug-Decisions") model = AutoModelForMultimodalLM.from_pretrained("abacusai/Smaug-Decisions", device_map="auto")