docs • nullsilver.com

alpha-sys-1-1.6B

alpha-sys-1 is a multimodal, Jev-compatible System One model. It takes a state, which may contain text, an image, or both, together with a question that has a fixed set of answers, and returns a probability distribution over those answers in one forward pass. It generates no text.

The model is trained for calibrated probabilities: across a large group of similar examples where it assigns an answer a probability near 80%, that answer should be correct in roughly 80% of cases. Calibration degrades when the input differs substantially from the training data, so check the probabilities on data from the intended application.

Base	LiquidAI/LFM2.5-VL-1.6B
Tuning	LoRA rank 32, lr 1e-4, merged into the base weights
Checkpoint	alpha-sys-1-260919, revision 260919, seed 1 of 3
Input	text, one image, or both; English
Output	probabilities over the answer space
Sizes	450M · 1.6B · 3B

Question types

Questions follow TypeSafe's System One format: a request contains one state and any number of questions, so a question written for Jev runs here as is. images is an extra field for multimodal inputs.

type	answer space	returns
choice	named options, up to 26	probabilities over the options, choice (argmax)
noul	a statement	noul, P(true)
score	ordered levels, lowest first	probabilities over the levels, score (expected level index)

The answer is read from the next-token logits for the answer labels (A, B, … or No/Yes), renormalised over the valid labels. Each question is answered independently: one question's answer is never context for another.

confidence is 1 - H(p)/log(n), computed from the distribution. It is not a separate prediction.

A noul probability near 0.5 means the model is uncertain.

Usage

alpha_sys_1.py in this repository renders questions the way the model was trained on them, batches the questions on one state, and returns answers in the System One shape.

from huggingface_hub import hf_hub_download
import importlib.util, sys
spec = importlib.util.spec_from_file_location("alpha_sys_1", hf_hub_download("nullsilver/alpha-sys-1-1.6B", "alpha_sys_1.py", revision="260919"))
alpha_sys_1 = importlib.util.module_from_spec(spec); spec.loader.exec_module(alpha_sys_1)

m = alpha_sys_1.SystemOne("nullsilver/alpha-sys-1-1.6B", revision="260919")
m.system_one({
    "state": {"subject": "Duplicate charge on invoice #4411",
              "body": "We were billed twice for March. Refund the duplicate today or we cancel our plan."},
    "questions": {
        "department": {"type": "choice", "instructions": "Which department should handle this email?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
                                    "sales": "pricing, new contracts", "other": "everything else"}},
        "urgency": {"type": "score", "instructions": "How urgent is this request?",
                    "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
        "churn_risk": {"type": "noul", "instructions": "The user threatens to cancel or leave."}}})
# {"department": {"choice": "billing", "probabilities": {"billing": 0.98, "technical": 0.00, "sales": 0.01, "other": 0.01}, "confidence": 0.92},
#  "urgency": {"score": 1.60, "probabilities": [0.10, 0.20, 0.70], ...},
#  "churn_risk": {"noul": 0.59}}

Email triage is not one of the training environments; the output above is what this checkpoint returns on it, not a tuned result.

Without the client, use this prompt format. A different format gives less reliable probabilities.

import string, torch
from transformers import AutoModelForImageTextToText, AutoProcessor

repo, rev = "nullsilver/alpha-sys-1-1.6B", "260919"
processor = AutoProcessor.from_pretrained(repo, revision=rev)
processor.tokenizer.padding_side = "left"
model = AutoModelForImageTextToText.from_pretrained(
    repo, revision=rev, dtype=torch.bfloat16, device_map="auto").eval()

def render(state, q):
    parts = [state] if state else []
    if q["type"] == "noul":
        c = q.get("criteria") or {}
        clar = "".join(f"\n{lab} means: {c[k]}" for lab, k in (("Yes", "true"), ("No", "false")) if c.get(k))
        parts.append(f"Statement: {q['instructions']}{clar}\nIs the statement true? Answer with Yes or No only.")
        return "\n\n".join(parts), ["No", "Yes"]
    crit = q["criteria"]
    items = list(crit.items()) if isinstance(crit, dict) else [(o, None) for o in crit]
    labels = list(string.ascii_uppercase[:len(items)])
    lines = [f"{lab}. {o}" + (f": {d}" if d else "") for lab, (o, d) in zip(labels, items)]
    parts.append(q["instructions"] + "\n" + "\n".join(lines) + "\nAnswer with the letter only.")
    return "\n\n".join(parts), labels

@torch.inference_mode()
def ask(q, state="", image=None):
    text, labels = render(state, q)
    content = ([{"type": "image", "image": image}] if image is not None else []) + [{"type": "text", "text": text}]
    inputs = processor.apply_chat_template(
        [[{"role": "user", "content": content}]], add_generation_prompt=True, tokenize=True,
        return_dict=True, processor_kwargs={"return_tensors": "pt"}).to(model.device)
    logits = model(**inputs, logits_to_keep=1).logits[0, -1].float()
    ids = [processor.tokenizer.encode(lab, add_special_tokens=False)[0] for lab in labels]
    return torch.softmax(logits[ids], -1).tolist()

p = ask({"type": "noul", "instructions": "The message conveys urgency"},
        state="Our API integration started returning 500 errors an hour before launch.")
urgent = p[1]  # P(Yes)

• A dict state is rendered one field per line, as key: value.
• Training images smaller than 256 px were upscaled to 256 px.
• For several questions on one state, batch them with padding_side="left".
• In bfloat16, probabilities move by up to a few hundredths with batch composition and padding.

Training

Training uses cross-entropy between the model's distribution and a target y_soft. The target is one-hot when a dataset provides one answer, and the annotator distribution when several annotations are available. Options are shuffled on every draw, the vision tower is frozen, and environments are sampled in proportion to the square root of their size.

environment	modality	type	label
mcq (ARC-Easy, SciQ, OpenBookQA, CommonsenseQA)	text	choice	one-hot
ChaosNLI (100-annotator items)	text	choice	annotator distribution
CivilComments-WILDS	text	noul	annotator share
STS-B	text	score	annotator mean
Folktables (ACS income, California 2014)	tabular as text	noul	outcome
CIFAR-10	image	choice	one-hot
Camelyon17-WILDS	image	noul	outcome

Three random seeds were trained. The released checkpoint is the seed with the lowest mean development loss across environments.

Evaluation

Each test split was read once per checkpoint. Reported intervals are 95% clustered bootstrap intervals, clustered on the relevant dataset group: question, comment, hospital, or state-year.

Compare models on NLL and Brier score. ECE is reported alongside them and is misleading on its own: a model that always predicts the base rate can have a low ECE.

The tables carry two reference points. The base rate is the constant predictor: it answers every question with the label frequencies of the training split (for example "toxic" 14% of the time on CivilComments, whatever the comment says), or uniformly when the options are shuffled. Any model should beat it. Base + T is the untuned LFM2.5-VL-1.6B, read the same way as the tuned model, with its label logits divided by one scalar temperature chosen to minimise NLL on the environment's development split.

Trained environments.

environment	NLL	NLL, base + T	Brier	ECE	AUROC	acc
mcq	0.427	0.527	0.218	0.012	0.877	0.847
ChaosNLI	0.784	0.835	0.171	0.022	0.670	0.680
CivilComments	0.323	0.629	0.044	0.064	0.922	0.933
STS-B	1.033	1.749	0.302	0.045	0.621	0.572
Folktables	0.448	0.612	0.296	0.013	0.761	0.778
CIFAR-10 (+C)	0.253	0.526	0.111	0.006	0.947	0.922
Camelyon17	0.156	0.649	0.090	0.013	0.907	0.938

ChaosNLI, CivilComments and STS-B have soft labels from multiple annotations, so top-label ECE does not fully measure calibration. On these, use NLL and KL divergence to the annotator distribution.

Unseen tasks. Not in training.

task	type	NLL	NLL, base + T	base rate
BoolQ	noul	0.479	0.457	0.665
Yelp review stars	score	1.110	1.226	1.609

Distribution shift. CIFAR-10-C.

	clean	sev. 1	2	3	4	5
accuracy	0.982	0.952	0.935	0.917	0.892	0.848
mean confidence	0.978	0.956	0.939	0.923	0.898	0.860

Other System One models. NLL on the text environments, same test splits, same readout. The other alpha-sys-1 sizes on the table are their released seeds. Qwen3.8-27B is the open 27B generalist, read at its first answer token with reasoning off, plus a dev-fitted temperature.

environment	alpha-sys-1-450M	alpha-sys-1-1.6B (this)	Qwen3.8-27B + T	base rate
mcq	0.727	0.427	0.159	1.439
ChaosNLI	0.902	0.784	0.706	0.938
CivilComments	0.324	0.323	0.473	0.425
STS-B	1.107	1.033	1.347	1.727
Folktables	0.436	0.448	0.472	0.683
BoolQ (unseen)	0.661	0.479	0.316	0.665
Yelp review stars (unseen)	1.467	1.110	0.858	1.609

Per-hospital, per-state-year and per-identity-group tables, the three-seed gate tables and the full comparison against other System One models (hosted and open) are in the repository under runs/.

Limitations

On a task that differs substantially from the training environments, do not assume this model stays calibrated; measure it against the base model's calibration. In leave-one-domain-out tests at 1.6B, a model tuned on the other environments beat the untuned base with a transferred temperature on one held-out environment out of three, and on the two unseen tasks above the trained-on-all checkpoints match the base model and do not beat it. With a few hundred labelled examples from your own task, fit a temperature on them: divide the label logits by one scalar chosen to minimise NLL on those examples (alpha_sys_1.fit_temperature), then pass it as SystemOne(..., temperature=T).

• Knowledge depends on model size: on MCQ, the untuned 3B base outperforms this tuned model.
• When the model does not know an answer, its distribution is close to uniform.
• Under strong distribution shift, such as CIFAR-10-C at severity 5, confidence remains higher than accuracy.
• Reversing the option order changes the top answer on 11% of MCQ items, mostly among low-confidence examples.
• The answer space is capped at 26 options.
• Fine-tuning used English data only and at most one image per question.

Related work

The interface follows TypeSafe's Jev (a hosted System One model, the state / questions request shape). Reading an answer distribution from the label-token logits of one forward pass is the readout of Kadavath et al. (2022, Language Models (Mostly) Know What They Know) and of the LLM-as-a-Verifier line of work, which scores rubric levels from the logits of letter tokens. That calibration improves with size, and that a temperature fitted on one domain transfers badly to another, is Jiang et al. (2021, How Can We Know When Language Models Know?). Training on a proper scoring rule against annotator distributions is why a fixed answer space and calibration are non-conflicting (Kalai and Vempala, 2024, Calibrated Language Models Must Hallucinate). Base models: Liquid AI's LFM2.5-VL.

License

This model is derived from LiquidAI/LFM2.5-VL-1.6B and is released under the LFM Open License v1.0.

Citation

@misc{alphasys1,
  title  = {alpha-sys-1: a small calibrated System One model},
  author = {Nullsilver},
  year   = {2026},
  url    = {https://huggingface.co/collections/nullsilver/alpha-sys-1}
}