Release
659438c 3 days ago
28.2 kB
---
license: apache-2.0
library_name: transformers
base_model: google/gemma-4-E4B-it
base_model_relation: finetune
pipeline_tag: any-to-any
language:
- en
tags:
- decisions
- typed-decisions
- audio
- vision
- calibration
- on-device
- smaug
- abacusai
---
<div align="center">
  <picture>
      <img src="https://cdn-avatars.huggingface.co/v1/production/uploads/63128dd099791aa61d180c72/dJ6uR23m09M-YEafBpmea.png" width="20%" alt="Smaug-Decisions">
  </picture>
</div>
<hr>
<div align="center" style="line-height:1">
  <a href="https://abacus.ai" target="_blank"><img alt="Homepage" src="https://img.shields.io/badge/Homepage-Abacus.AI-white?color=1783ff&logoColor=white"/></a>
  <a href="https://huggingface.co/abacusai" target="_blank"><img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Abacus.AI-ffc107?color=ffc107&logoColor=white"/></a>
</div>
<div align="center" style="line-height: 1;">
  <a href="LICENSE"><img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-f5de53?&color=f5de53"/></a>
  <a href="https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B" target="_blank"><img alt="Fast tier" src="https://img.shields.io/badge/Fast_tier-Smaug--Audio--Decisions--0.7B-1783ff?&color=1783ff"/></a>
  <a href="https://huggingface.co/abacusai/Smaug-Decisions-LiteRT" target="_blank"><img alt="On phones" src="https://img.shields.io/badge/On_phones-Smaug--Decisions--LiteRT-1783ff?&color=1783ff"/></a>
</div>
<p align="center"><b>State-of-the-art audio decisions in 4B parameters, and it sees and reads too.</b><br>
60.1 Earshot audio skill, ahead of Gemini 4 Argon (59.4) and Gemini 3.8 Flash (50.5), and better calibrated than both.</p>

## 1. Model Introduction

Smaug-Decisions is a 4B decision model that hears, sees and reads. A fine-tune of Google's
[Gemma 4 E4B-it](https://huggingface.co/google/gemma-4-E4B-it) by Abacus.AI, it answers a typed question about **audio, an image or
text** with a calibrated probability per option, in one forward pass and with no generated text.

<p align="center"><img src="figures/demo.svg" width="100%" alt="Four example decisions over audio, an image and text, each answered with a probability per option in one forward pass"></p>

<p align="center"><img src="figures/stats.svg" width="100%" alt="60.1 Earshot audio skill, three input types, one forward pass per decision, 0.069 calibration error"></p>

It is the only 4B decision model with results on all three input types. On audio it is ahead of the frontier APIs: **60.1** skill on
**Earshot**, our 6,671-item audio decision benchmark, against 59.4 for Gemini 4 Argon and 50.5 for Gemini 3.8 Flash, and better
calibrated than both (ECE 0.069 vs 0.123 and 0.204). On the public Jev Decision Index it scores **57.9** on images and **56.6** on text,
the best Gemma 4 E4B on both.

```python
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))
from smaug_decide import SmaugDecide                                   # ships with this repo
model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions")
model("call.wav", "Has the speaker finished their turn?")               # -> {'Yes': 0.93, 'No': 0.07}
```

<p align="center"><img src="figures/matrix.svg" width="100%" alt="Every 4B decision model by input type: Smaug-Decisions is the only one with audio, image and text results"></p>

## 2. Highlights

- **Ahead of the frontier APIs on audio,** and better calibrated, so the probabilities work as thresholds.
- **Hears, sees and reads:** audio up to 30 s, images and text through one prompt and one readout.
- **One pass, no decoding:** about 200 ms per audio decision on an H100.
- **Runs on phones:** [Smaug-Decisions-LiteRT](https://huggingface.co/abacusai/Smaug-Decisions-LiteRT) is 3.45 GB, keeps 59.4 audio skill and answers
  a text question in about 0.6 s on an iPhone 14 Pro Max, fully offline.

## 3. Model Summary

<div align="center">
<table>
<tbody>
<tr><td align="center"><strong>Architecture</strong></td><td align="center">Gemma 4 E4B-it (language model + audio and vision encoders); letter-logit readout</td></tr>
<tr><td align="center"><strong>Parameters</strong></td><td align="center">4B effective, 8.6B stored (3.5B of the stored weights are embedding tables that are looked up, not computed; the Hugging Face counter shows the stored total)</td></tr>
<tr><td align="center"><strong>Input</strong></td><td align="center">Audio up to 30 s (16 kHz mono), an image (560 image tokens), or text, plus a question with 2 to 20 options</td></tr>
<tr><td align="center"><strong>Output</strong></td><td align="center">A calibrated probability per option (softmax for one answer, sigmoid for select-all)</td></tr>
<tr><td align="center"><strong>Hardware</strong></td><td align="center">NVIDIA GPU (bfloat16), Apple Silicon (MPS) and CPU through Transformers; phones through <a href="https://huggingface.co/abacusai/Smaug-Decisions-LiteRT">Smaug-Decisions-LiteRT</a></td></tr>
<tr><td align="center"><strong>Precision</strong></td><td align="center">bfloat16</td></tr>
<tr><td align="center"><strong>Formats</strong></td><td align="center">Transformers (repo root); LiteRT-LM for phones (<a href="https://huggingface.co/abacusai/Smaug-Decisions-LiteRT">Smaug-Decisions-LiteRT</a>)</td></tr>
<tr><td align="center"><strong>Base Model</strong></td><td align="center"><code>google/gemma-4-E4B-it</code></td></tr>
<tr><td align="center"><strong>Adaptation</strong></td><td align="center">Full fine-tune</td></tr>
</tbody>
</table>
</div>

## 4. Evaluation

### Audio: Earshot

**Earshot** is our audio decision benchmark: 6,671 items and 17.2 hours of audio in four tracks: **conversation dynamics** (turn
completion, next speaker, speaker count, speech activity; from VoxConverse, AMI and Smart Turn), **external turn completion** (Smart
Turn v3.2 test), **sound and music** (genre, instrument, tempo, scene, sound events) and a small **pilot** track. **Audio skill** is
the mean task skill per track, averaged over the tracks: 0 is guessing, 100 is perfect. No benchmark recording or item is in the
training data.

<p align="center"><img src="figures/earshot.svg" width="100%" alt="Earshot audio skill per track, and calibration error"></p>

<details>
<summary><b>Earshot table</b></summary>

<div style="max-width:1100px;margin:0 auto">
<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
<thead><tr><th style="padding:10px 6px;text-align:left;border-bottom:2px solid #2a78d6;width:26%"></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;font-weight:600;background:rgba(42,120,214,0.10);">Smaug-Decisions</th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Gemini 4 Argon<div style="font-size:10px;font-weight:400;color:#6B6B6B">(API)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Gemini 3.8 Flash<div style="font-size:10px;font-weight:400;color:#6B6B6B">(API)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">AudioJev<div style="font-size:10px;font-weight:400;color:#6B6B6B">(Qwen2.5-Omni-3B)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Gemma 4 E4B-it<div style="font-size:10px;font-weight:400;color:#6B6B6B">(zero-shot)</div></th>
</tr></thead>
<tbody>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Earshot (audio decisions)</td></tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Audio skill (headline) &uarr;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.47)"><strong>60.1</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.46)">59.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.40)">50.5</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.34)">43.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">5.0</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">ECE &darr;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.47)"><strong>0.069</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.41)">0.123</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.33)">0.204</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.34)">0.191</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">0.464</td>
</tr>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Per track (skill) &uarr;</td></tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Conversation dynamics (3,000)</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.46)">48.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>49.6</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.46)">48.5</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.26)">26.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">3.6</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">External turn completion (1,000)</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.46)">80.8</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.44)">76.5</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>81.8</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.35)">62.0</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">11.2</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Sound &amp; music (1,115)</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.47)"><strong>66.1</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.46)">64.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.31)">42.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.41)">57.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">5.2</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Pilot (80)</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.45)">45.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>47.0</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.31)">29.1</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.30)">28.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">0.1</td>
</tr>
</tbody></table>
<div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best in the row. Shading is relative within each row (darker = better). Every model scored on the same 6,671 items with the same harness; the APIs received the audio and returned JSON probabilities. Skill = (accuracy &minus; chance) / (1 &minus; chance), times 100.</div>
</div>

</details>

Need it faster? [Smaug-Audio-Decisions-0.7B](https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B) scores 53.0 on Earshot at 9.7 ms; in front of this
model as a [cascade](https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B#the-cascade) it matches this model's accuracy (60.1) at 130 ms
instead of 210 ms.

### Images and text: the public Jev Decision Index

The matrix at the top has the headline numbers: 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both (Winnow-E4B, the
other Gemma 4 E4B fine-tune on the boards: 48.1 and 41.7). Images: our run of the official vision kit (0.4, public sets) at 560 image
tokens.

<details>
<summary><b>Full Decision Index breakdown</b></summary>

<p align="center"><img src="figures/text.svg" width="100%" alt="Text decision skill per area"></p>

<div style="max-width:1100px;margin:0 auto">
<table style="width:100%;table-layout:fixed;border-collapse:collapse;font-size:13px">
<thead><tr><th style="padding:10px 6px;text-align:left;border-bottom:2px solid #2a78d6;width:26%"></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;font-weight:600;background:rgba(42,120,214,0.10);">Smaug-Decisions<div style="font-size:10px;font-weight:400;color:#6B6B6B">(audio + image + text)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Perplexity Decider Flash<div style="font-size:10px;font-weight:400;color:#6B6B6B">(4B)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">vLLM-SR d3-mini<div style="font-size:10px;font-weight:400;color:#6B6B6B">(4B)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">RSI-Jev v6.1 VL<div style="font-size:10px;font-weight:400;color:#6B6B6B">(4B)</div></th>
<th style="padding:10px 4px;text-align:center;font-weight:500;border-bottom:2px solid #2a78d6;color:#2a78d6;font-size:13px;">Winnow<div style="font-size:10px;font-weight:400;color:#6B6B6B">(Gemma 4 E4B)</div></th>
</tr></thead>
<tbody>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Jev Decision Index 0.4.1, text (same 35 public benchmarks)</td></tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Public skill &uarr;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.36)">56.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>61.8</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.36)">56.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.28)">52.9</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">41.7</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Knowledge &amp; reasoning</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.25)">35.1</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>46.0</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.44)">44.3</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.25)">35.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">25.3</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Language</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.45)">66.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>67.2</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.33)">60.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.30)">59.2</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">47.5</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Retrieval</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.46)">64.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>65.2</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.44)">63.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.31)">57.1</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">43.8</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Tools</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.29)">72.9</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>81.0</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.27)">72.0</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.18)">68.4</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">62.5</td>
</tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Arts</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.36)">41.6</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.47)"><strong>46.6</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.28)">37.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.45)">45.7</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;background:rgba(42,120,214,0.05)">26.9</td>
</tr>
<tr><td colspan="6" style="padding:7px 10px;font-weight:600;color:#1a5fb4;background:#e8f0fc;border-bottom:1px solid rgba(42,120,214,0.2)">Earshot (audio decisions)</td></tr>
<tr><td style="padding:8px 6px;padding-left:16px;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600">Audio skill &uarr;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;font-weight:600;background:rgba(42,120,214,0.47)"><strong>60.1</strong></td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;color:#9aa0a6">&mdash;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;color:#9aa0a6">&mdash;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;color:#9aa0a6">&mdash;</td>
<td style="padding:8px 6px;text-align:center;border-bottom:1px solid rgba(128,128,128,0.15);font-size:14px;color:#9aa0a6">&mdash;</td>
</tr>
</tbody></table>
<div style="font-size:11.5px;color:#6B6B6B;margin-top:6px">Bold = best in the row. Other models: the public Decision Index board as of 2026-10-28 (text 0.4.1). Smaug-Decisions: our runs of the official Decision Index kit (text on a 3,000-row sample of the public suite, compared on the same 35 benchmarks); self-reported, not board submissions. &mdash; = no published result.</div>
</div>

</details>

## 5. Training Approach

- **Readout, not generation.** The answer is read from the LM-head logits of the option letters at the answer position (tanh
  softcap 30, then softmax; select-all uses a sigmoid with one learned threshold). A decision costs one prefill.
- **Decision-tuned and calibrated.** Full fine-tune of Gemma 4 E4B-it on a broad mix of typed decision tasks over audio, images and
  text, with calibration as an explicit objective and options shuffled so the model learns the options, not their positions.

## 6. Limitations

- **Decisions, not text:** each call answers one typed question with a probability per option; the model does not write text.
- **Per request:** one audio clip of up to 30 s (longer audio keeps its last 30 s), one image at 560 image tokens, up to 20 options.
- **Thresholds:** set decision thresholds on your own data for high-stakes uses.

## 7. Usage

```bash
pip install "transformers>=5.20" torch soundfile pillow huggingface_hub
```

```python
import sys
from huggingface_hub import snapshot_download
sys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))
from smaug_decide import SmaugDecide
model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions")      # CUDA, Apple Silicon (MPS) or CPU, picked automatically
model("call.wav", "Has the speaker finished their turn?")                                  # predicate -> {'Yes': p, 'No': 1-p}
model("song.wav", "Which genre is playing?", ["rock", "jazz", "classical"])                 # choice -> probabilities sum to 1
model("room.wav", "Which sounds are present?", ["speech", "music", "alarm"], multi=True)    # select-all -> one probability each
model("receipt.jpg", "Does the total exceed $50?")                                          # images
model("The order never arrived.", "How upset is the customer?", ["calm", "annoyed", "angry"],
      descriptions=["neutral tone", "complains", "threatens to leave"], rules="Judge the tone, not the facts.")
```

`from_pretrained` downloads the checkpoint (about 16 GB). Any runtime can reproduce the readout: run the prompt in
`smaug_decide.render`, take the LM-head logits of the tokens `A`..`T` at the last position, apply the softcap from `readout.json`,
then softmax (one answer) or `sigmoid(z - b)` with `b` from `readout.json` (select-all). With plain Transformers,
`AutoModelForMultimodalLM.from_pretrained("abacusai/Smaug-Decisions", dtype="bfloat16")` loads the weights.

### On phones

[Smaug-Decisions-LiteRT](https://huggingface.co/abacusai/Smaug-Decisions-LiteRT) is this model in one **3.45 GB** file for Google's on-device
runtime LiteRT-LM, running on the phone GPU. At about a fifth of the size it keeps almost all of this model's quality. It runs in
the Cascade app on iPhone and Android next to [Smaug-Audio-Decisions-0.7B](https://huggingface.co/abacusai/Smaug-Audio-Decisions-0.7B), which answers the easy
audio questions first. App source: [github.com/abacusai/Smaug-Decisions-Mobile](https://github.com/abacusai/Smaug-Decisions-Mobile).

| | Smaug-Decisions (16 GB) | Smaug-Decisions-LiteRT (3.45 GB) | change |
|---|---|---|---|
| Earshot audio skill | 60.1 | **59.4** | &minus;0.7 |
| Earshot calibration error (ECE, lower is better) | 0.069 | **0.074** | +0.005 |
| Decision Index text | 56.6 | **53.9** | &minus;2.7 |
| Decision Index images, 560 image tokens<sup>*</sup> | 57.2 | **55.1** | &minus;2.1 |
| Decision Index images, 280 image tokens<sup>*</sup> | 54.8 | **51.8** | &minus;3.0 |

Same items, same harness; the phone-build scores come from a PyTorch replica of the file's quantized weights. <sup>*</sup>10 of the 11 public
vision sets (Winoground left out; this model's 11-set index is 57.9).
| iPhone 14 Pro Max (6 GB RAM, phone GPU) | measured |
|---|---|
| first load (one time) / later loads | 13-21 s / 0.48 s |
| text decision | ~0.6 s |
| audio decision | ~1.6 s |
| image decision (280 image tokens) | ~7.4 s |
| peak app memory, with Smaug-Audio-Decisions-0.7B loaded | 1.35 GB |
| storage | the 3.45 GB file + a ~2.4 GB cache written on first load |
## 8. License
Apache 2.0, the same license as the base model, Gemma 4 E4B-it.
## 9. Citation
```bibtex
@misc{abacusai2026smaugdecisions,
  title  = {Smaug-Decisions: calibrated typed decisions over audio, images and text},
  author = {Abacus.AI},
  year   = {2026},
  url    = {https://huggingface.co/abacusai/Smaug-Decisions}
}
```
## 10. Contact Us
If you have any questions, please reach out at [Abacus.AI](https://abacus.ai).