---
license: apache-2.0
base_model: google/embeddinggemma-2
library_name: transformers
pipeline_tag: text-classification
language:
- en
tags:
- decision-model
- calibration
- embeddinggemma
datasets:
- LocalLLaMA/typed-decisions
- google/boolq
- nyu-mll/glue
- mteb/banking77
- clinc/clinc_oos
---

# embeddinggemma2-decision

A decision model built on the text encoder of `google/embeddinggemma-2` (~270M parameters). It reads an input, a set of questions and their options in a single forward pass and returns a calibrated probability for every option. It never generates text. The head follows the Clef design: option and question token spans are mean-pooled and scored by a small MLP.

Question types: `choice` (pick one of named options), `noul` (true / false), `score` (ordered levels).

## Usage

```python
from huggingface_hub import hf_hub_download
import importlib.util
spec = importlib.util.spec_from_file_location("decision_model", hf_hub_download("qnaug/embeddinggemma2-decision", "decision_model.py"))
dm_mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(dm_mod)
dm = dm_mod.DecisionModel("qnaug/embeddinggemma2-decision")
dm.decide("Hi, I was charged twice for invoice #4411. Please refund the duplicate today.", {
    "team": {"type": "choice", "instructions": "Which team should handle this?",
             "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages", "sales": "pricing"}},
    "refund": {"type": "noul", "instructions": "The customer asks for a refund."},
})
```

`domain="extra"` (default) uses the temperatures calibrated on natural text; `domain="typed"` uses those calibrated on JSON state like typed-decisions. Run in float32 or bfloat16, never float16 (the base model returns NaN).

## Results

typed-decisions test accuracy **74.3%** (random 31.8%, majority label 47.9%).

### typed-decisions (test)

Accuracy 74.3%, ECE 0.029 after calibration.

| Confidence threshold | Decisions handled | Accuracy on handled |
|---|---|---|
| 0.5 | 94.7% | 76.0% |
| 0.6 | 78.8% | 80.2% |
| 0.7 | 62.4% | 83.9% |
| 0.8 | 45.0% | 87.4% |
| 0.9 | 26.6% | 91.3% |

### boolq, mnli, banking77, clinc150 (test)

Accuracy 80.4%, ECE 0.039 after calibration.

| Confidence threshold | Decisions handled | Accuracy on handled |
|---|---|---|
| 0.5 | 90.6% | 83.7% |
| 0.6 | 74.4% | 87.2% |
| 0.7 | 59.5% | 92.6% |
| 0.8 | 48.2% | 96.9% |
| 0.9 | 40.1% | 98.4% |

## Limitations

- Trained on English data only. Other languages (e.g. Vietnamese) are untested and the calibration does not carry over.
- Calibration is only valid for data like the training sources. Recalibrate on a few hundred examples of your own data before relying on the probabilities.
- Yes/no reading comprehension on free text is the weakest part (boolq / mnli ≈ 70%).
- Labels in typed-decisions have low inter-annotator agreement, so accuracy has a ceiling well below 100%.

## Training

Text encoder of `google/embeddinggemma-2` with the vocabulary embedding frozen, trained with soft-label cross-entropy (typed-decisions) and one-hot labels (other datasets). Intent datasets show the gold label plus random distractors, so the model learns to choose among arbitrary option lists. Per-source temperature scaling on a held-out split.

Dataset licenses differ (typed-decisions Apache-2.0, BoolQ CC BY-SA 3.0, banking77 CC BY 4.0, CLINC150 CC BY 3.0, MNLI mixed); check them for your use.