---
license: apache-2.0
library_name: alben
pipeline_tag: text-classification
tags:
  - system-one
  - decision-model
  - calibration
  - text-classification
  - zero-shot-classification
  - multilingual
language:
  - en
  - zh
  - de
  - fr
  - es
  - ja
  - ko
  - ar
  - hi
  - th
---

# Alben

**An open, self-hosted decision model.** Give it a piece of text and a set of typed
questions, and it answers them — a label, a rating on an ordered scale, or a
probability. No API key, no per-token billing, no data leaving your machine.

Alben answers questions **you define at request time**. You are not restricted to a
fixed taxonomy: pass any set of options, with a description of what each one means,
and the model scores the text against them. Up to 255 options, or an ordered scale of
2–10 levels, or a yes/no.

It speaks the same HTTP protocol as TypeSafe's System One API, so existing clients —
including the official Python and JavaScript SDKs — work against it unchanged. That is
a compatibility property, deliberately maintained; it is not the point of the project.
The point is a small, inspectable, self-hostable model that stays useful when the
question is one nobody anticipated.

```bash
pip install alben
```

```python
from alben import Alben

a = Alben()                    # fetches this checkpoint on first use

answer = a.predict(
    "We were billed twice for March. Refund today or we cancel.",
    {
        "team": {
            "type": "choice",
            "instructions": "Which team should handle this?",
            "criteria": {"billing": "invoices and refunds",
                         "tech": "bugs and outages"},
        },
        "urgency": {
            "type": "score",
            "instructions": "How urgent is this?",
            "criteria": ["can wait", "this week", "today", "immediately"],
        },
        "churn_risk": {
            "type": "noul",
            "instructions": "Did they threaten to cancel?",
        },
    },
)

print(answer["answers"]["team"]["choice"])        # billing
print(answer["answers"]["urgency"]["score"])       # 1.35 — between two levels
print(answer["answers"]["churn_risk"]["noul"])     # 0.38
```

The response is the same shape the System One API returns (`model` / `answers` /
`usage`, probabilities rounded to two decimals) — deliberately, so the official SDKs
parse it without changes.

---

## What is in this checkpoint

`bge-small-zh-v1.5`-based sentence encoder with a trained projection head, fine-tuned
end-to-end on questions the model had to answer from the option descriptions alone.

| | |
|---|---|
| Base encoder | `bge-small-zh-v1.5` — BAAI, MIT. Fetched at runtime, **not redistributed here** |
| Training examples | **56160** |
| Distinct question sets | **1882** — each with its own label names and wording |
| Answer types | choice (≤255 options) · ordered score (2–10 levels) · yes/no |
| Calibration | temperature scaling, fitted on a text-disjoint held-out set |
| Footprint | ~92 MB checkpoint + ~92 MB encoder, runs on CPU or CUDA |
| License | Apache-2.0 (this checkpoint) |

### Training data

Assembled from the **train splits** of public research datasets —
[MASSIVE](https://github.com/alexa/massive) (10 languages),
[BANKING77](https://github.com/PolyAI-LDN/task-specific-datasets),
[AG News](https://huggingface.co/datasets/fancyzhx/ag_news),
[DAIR Emotion](https://huggingface.co/datasets/dair-ai/emotion),
[SST-5](https://huggingface.co/datasets/SetFit/sst5) — plus synthetic question sets
generated for this project. Evaluation splits were held out and decontaminated;
3 upstream duplicates between SST-5 train and test were removed from the training side.

---

## Measured accuracy

Public benchmarks, **test splits only**:

| Benchmark | Alben | Random | Release gate |
|---|---|---|---|
| MASSIVE intent, English (60 intents) | **0.6975** | 0.0303 | ≥ 0.60 |
| MASSIVE intent, 9 other languages | **0.3883** | 0.0303 | ≥ 0.30 |
| BANKING77 (77 classes) | **0.8817** | 0.0500 | ≥ 0.70 |
| SST-5 (5-level ordered) | **0.3133** | 0.2000 | ≥ 0.25 |

Per language, MASSIVE intent — the spread is worth seeing before you deploy:

| Language | MASSIVE intent |
|---|---|
| `en-US` | 0.6975 |
| `zh-CN` | 0.8025 |
| `de-DE` | 0.4600 |
| `fr-FR` | 0.4550 |
| `es-ES` | 0.5050 |
| `ja-JP` | 0.6350 |
| `ko-KR` | 0.1475 |
| `ar-SA` | 0.2125 |
| `hi-IN` | 0.1125 |
| `th-TH` | 0.1650 |

On held-out question sets from the same task family (0% text overlap with training,
label-balanced, real question wording): **0.6465** average, **0.6178** worst.

Every gate above is checked by a script before release — `python -u release.py verify`,
which exits non-zero on failure. The gates were fixed **before** the model was trained.

---

## Calibrated confidence

Confidence is temperature-scaled on a held-out set that shares no text with training.
Accuracy is unchanged by calibration; only the confidence moves.

| | Accuracy | Mean confidence | ECE ↓ |
|---|---|---|---|
| Raw | 0.7019 | 0.8167 | 0.1273 |
| Calibrated (T = 1.65) | 0.7019 | 0.7089 | **0.0508** |

Expected calibration error falls by **60%**. A confidence threshold is
therefore meaningful for routing: act above it, escalate below it.

---

## Honest limitations

We would rather you know these before you build on it.

| Scenario | Works? | Measured |
|---|---|---|
| Questions whose labels resemble what it trained on | **Yes** | 0.62–0.88 on unseen text |
| Same task, different language | **Yes, unevenly** | 0.70 (en) down to 0.11 (hi) across 10 languages |
| **A genuinely new question type** | **No — near chance** | 0.3150 vs 0.3211 random |
| **New question type + a few dozen labelled examples** | **Yes** | 0.4917 → 0.9917 |

In one line: **it generalises across text and language, not across question types.**

We checked the second row rather than assuming it. Splitting BANKING77's 77 labels in
half — train on 38, evaluate on the other 39 with zero training examples — gives 0.5208,
against 0.4725 for the frozen encoder and 0.9158 for a model that saw all 77. So most of
the headline accuracy comes from having seen that label set, not from generalising to a
new one. That is why the next section exists.

### Teaching a new question type

This is a first-class operation, not a workaround:

```bash
python -m alben.teach --examples my_labelled_rows.jsonl --out my-dim.pt
```

```python
a.teach(examples=my_labelled_rows, out="my-dim.pt", epochs=12)
# -> {"before": 0.513, "after": 0.663, "forgetting_delta": -0.020}
```

A few dozen examples are enough. Two properties matter, and both are measured:

- **Reported honestly** — `teach` returns *before*, *after*, and a forgetting probe
  computed on old data that was **not** replayed into training. Reporting only *after*
  would make any model look good.
- **Local gains** — untrained control questions move by **+0.0000**. Teaching one
  question type does not teach another. Teach the type you need.

---

## Reproduce every number

```bash
git clone https://github.com/XiaoBinGan/Alben && cd Alben
pip install -e ".[dev]"

python -u release.py verify     # the six acceptance gates, exits non-zero on failure
python -u release.py status     # what is deployed right now
python -u corpus/am_i_general.py  # the three senses of "general", with evidence
```

---

## Attribution

- **This checkpoint**: Apache-2.0.
- **Base encoder**: [`BAAI/bge-small-zh-v1.5`](https://huggingface.co/BAAI/bge-small-zh-v1.5)
  (Beijing Academy of Artificial Intelligence), MIT. Fetched at runtime; **not**
  redistributed in this repository.
- **Training text**: MASSIVE (CC BY 4.0), BANKING77 (CC BY 4.0), AG News, DAIR Emotion,
  SST-5 — train splits only, under their respective licenses.
- **Protocol compatibility**: Alben implements the publicly documented request and response
  shapes of the TypeSafe System One API so existing clients interoperate. It is an
  independent implementation: no TypeSafe weights, no TypeSafe training data, and no
  affiliation or endorsement. Outputs are not expected to match the hosted service.

---

## 中文说明

Alben 是一个**可自托管的决策模型**。给它一段文本和一组你自己定义的问题，它返回
**经过校准的**答案 —— 分类、有序打分、或是/否概率。不用 API key，不按 token 计费，
数据不出你的机器。

**重点是「问题由你在运行时定义」**：传任意一组选项、每个选项的描述，模型据此给
文本打分。最多 255 个选项，或 2–10 级有序打分，或是/否。

它说同一套 HTTP 协议，所以现成客户端（含官方 SDK）不用改就能用。**兼容是刻意维护的
特性，但不是这个项目的重点。**

| 项 | 值 |
|---|---|
| 训练样本 | **56160** |
| 不同问题集 | **1882** |
| 底模 | `bge-small-zh-v1.5`（BAAI，MIT，**不随本仓库分发**） |
| 许可 | Apache-2.0 |

**能力边界（必须一起读）**：它跨**文本**和**语言**泛化，**不跨问题类型**泛化。
遇到全新类型的问题时接近随机（0.3150 vs 随机 0.3211）；但给几十条标注做热启动微调
就能学会（0.4917 → 0.9917），而且**收益只限教过的那个类型**（未教过的对照问题 +0.0000）。

完整数据、复现命令、以及我们**明确排除掉的方向**，见
[GitHub 仓库](https://github.com/XiaoBinGan/Alben)。