Update README.md

README.md

@@ -2,4 +2,293 @@
 license: other
 license_name: lfm1.0
 license_link: LICENSE
+base_model:
+- LiquidAI/LFM2.5-VL-450M
+base_model_relation: finetune
+pipeline_tag: image-text-to-text
+library_name: transformers
+language:
+- en
+tags:
+- calibration
+- classification
+- system-one
+- jev-compatible
+datasets:
+- allenai/ai2_arc
+- allenai/sciq
+- allenai/openbookqa
+- tau/commonsense_qa
+- uoft-cs/cifar10
+- mteb/stsbenchmark-sts
 ---
+
+<div align="center">
+  <img
+    src="https://huggingface.co/nullsilver/alpha-sys-1-scratch/resolve/main/alpha-sys-1-banner-light.png"
+    alt="alpha-sys-1 banner"
+    style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;"
+  />
+  <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;">
+    <a href="https://docs.liquid.ai/lfm/getting-started/welcome"><strong>docs</strong></a> •
+    <a href="https://nullsilver.com"><strong>nullsilver.com</strong></a>
+  </div>
+</div>
+
+# alpha-sys-1-450M
+
+alpha-sys-1 is a multimodal, Jev-compatible **System One model**. It takes a state, which
+may contain text, an image, or both, together with a question that has a fixed set of
+answers, and returns a probability distribution over those answers in one forward pass. It
+generates no text.
+
+The model is trained for calibrated probabilities: across a large group of similar
+examples where it assigns an answer a probability near 80%, that answer should be correct
+in roughly 80% of cases. Calibration degrades when the input differs substantially from
+the training data, so check the probabilities on data from the intended application.
+
+| | |
+|---|---|
+| Base | [LiquidAI/LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) |
+| Tuning | LoRA rank 32, lr 1e-4, merged into the base weights |
+| Checkpoint | `alpha-sys-1-260919`, revision `260919`, seed 1 of 3 |
+| Input | text, one image, or both; English |
+| Output | probabilities over the answer space |
+| Sizes | [450M](https://huggingface.co/nullsilver/alpha-sys-1-450M) · [1.6B](https://huggingface.co/nullsilver/alpha-sys-1-1.6B) · [3B](https://huggingface.co/nullsilver/alpha-sys-1-3B) |
+
+## Question types
+
+Questions follow TypeSafe's System One format: a request contains one `state` and any
+number of `questions`, so a question written for Jev runs here as is. `images` is an extra
+field for multimodal inputs.
+
+| type | answer space | returns |
+|---|---|---|
+| `choice` | named options, up to 26 | `probabilities` over the options, `choice` (argmax) |
+| `noul` | a statement | `noul`, P(true) |
+| `score` | ordered levels, lowest first | `probabilities` over the levels, `score` (expected level index) |
+
+The answer is read from the next-token logits for the answer labels (`A`, `B`, … or
+`No`/`Yes`), renormalised over the valid labels. Each question is answered independently:
+one question's answer is never context for another.
+
+> [!NOTE]
+> `confidence` is `1 - H(p)/log(n)`, computed from the distribution. It is not a separate
+> prediction.
+
+> [!TIP]
+> A `noul` probability near 0.5 means the model is uncertain.
+
+## Usage
+
+`alpha_sys_1.py` in this repository renders questions the way the model was trained on
+them, batches the questions on one state, and returns answers in the System One shape.
+
+```python
+from huggingface_hub import hf_hub_download
+import importlib.util, sys
+spec = importlib.util.spec_from_file_location("alpha_sys_1", hf_hub_download("nullsilver/alpha-sys-1-450M", "alpha_sys_1.py", revision="260919"))
+alpha_sys_1 = importlib.util.module_from_spec(spec); spec.loader.exec_module(alpha_sys_1)
+
+m = alpha_sys_1.SystemOne("nullsilver/alpha-sys-1-450M", revision="260919")
+m.system_one({
+    "state": {"subject": "Duplicate charge on invoice #4411",
+              "body": "We were billed twice for March. Refund the duplicate today or we cancel our plan."},
+    "questions": {
+        "department": {"type": "choice", "instructions": "Which department should handle this email?",
+                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages",
+                                    "sales": "pricing, new contracts", "other": "everything else"}},
+        "urgency": {"type": "score", "instructions": "How urgent is this request?",
+                    "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]},
+        "churn_risk": {"type": "noul", "instructions": "The user threatens to cancel or leave."}}})
+# {"model": ..., "answers": {"department": {"choice": "billing", "probabilities": {...}, "confidence": ...},
+#                           "urgency": {"score": 1.7, ...}, "churn_risk": {"noul": 0.83}}}
+```
+
+Without the client, use this prompt format. A different format gives less reliable
+probabilities.
+
+```python
+import string, torch
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+repo, rev = "nullsilver/alpha-sys-1-450M", "260919"
+processor = AutoProcessor.from_pretrained(repo, revision=rev)
+processor.tokenizer.padding_side = "left"
+model = AutoModelForImageTextToText.from_pretrained(
+    repo, revision=rev, dtype=torch.bfloat16, device_map="auto").eval()
+
+def render(state, q):
+    parts = [state] if state else []
+    if q["type"] == "noul":
+        c = q.get("criteria") or {}
+        clar = "".join(f"\n{lab} means: {c[k]}" for lab, k in (("Yes", "true"), ("No", "false")) if c.get(k))
+        parts.append(f"Statement: {q['instructions']}{clar}\nIs the statement true? Answer with Yes or No only.")
+        return "\n\n".join(parts), ["No", "Yes"]
+    crit = q["criteria"]
+    items = list(crit.items()) if isinstance(crit, dict) else [(o, None) for o in crit]
+    labels = list(string.ascii_uppercase[:len(items)])
+    lines = [f"{lab}. {o}" + (f": {d}" if d else "") for lab, (o, d) in zip(labels, items)]
+    parts.append(q["instructions"] + "\n" + "\n".join(lines) + "\nAnswer with the letter only.")
+    return "\n\n".join(parts), labels
+
+@torch.inference_mode()
+def ask(q, state="", image=None):
+    text, labels = render(state, q)
+    content = ([{"type": "image", "image": image}] if image is not None else []) + [{"type": "text", "text": text}]
+    inputs = processor.apply_chat_template(
+        [[{"role": "user", "content": content}]], add_generation_prompt=True, tokenize=True,
+        return_dict=True, processor_kwargs={"return_tensors": "pt"}).to(model.device)
+    logits = model(**inputs, logits_to_keep=1).logits[0, -1].float()
+    ids = [processor.tokenizer.encode(lab, add_special_tokens=False)[0] for lab in labels]
+    return torch.softmax(logits[ids], -1).tolist()
+
+p = ask({"type": "noul", "instructions": "The message conveys urgency"},
+        state="Our API integration started returning 500 errors an hour before launch.")
+urgent = p[1]  # P(Yes)
+```
+
+> [!NOTE]
+> - A dict `state` is rendered one field per line, as `key: value`.
+> - Training images smaller than 256 px were upscaled to 256 px.
+> - For several questions on one state, batch them with `padding_side="left"`.
+> - In bfloat16, probabilities move by up to a few hundredths with batch composition and
+>   padding.
+
+## Training
+
+Training uses cross-entropy between the model's distribution and a target `y_soft`. The
+target is one-hot when a dataset provides one answer, and the annotator distribution when
+several annotations are available. Options are shuffled on every draw, the vision tower is
+frozen, and environments are sampled in proportion to the square root of their size.
+
+| environment | modality | type | label |
+|---|---|---|---|
+| mcq (ARC-Easy, SciQ, OpenBookQA, CommonsenseQA) | text | choice | one-hot |
+| ChaosNLI (100-annotator items) | text | choice | annotator distribution |
+| CivilComments-WILDS | text | noul | annotator share |
+| STS-B | text | score | annotator mean |
+| Folktables (ACS income, California 2014) | tabular as text | noul | outcome |
+| CIFAR-10 | image | choice | one-hot |
+| Camelyon17-WILDS | image | noul | outcome |
+
+Three random seeds were trained. The released checkpoint is the seed with the lowest mean
+development loss across environments.
+
+## Evaluation
+
+Each test split was read once per checkpoint. Reported intervals are 95% clustered
+bootstrap intervals, clustered on the relevant dataset group: question, comment, hospital,
+or state-year.
+
+> [!IMPORTANT]
+> Compare models on NLL and Brier score. ECE is reported alongside them and is misleading
+> on its own: a model that always predicts the base rate can have a low ECE.
+
+The tables carry two reference points. The **base rate** is the constant predictor: it
+answers every question with the label frequencies of the training split (for example
+"toxic" 14% of the time on CivilComments, whatever the comment says), or uniformly when
+the options are shuffled. Any model should beat it. **Base + T** is the untuned
+LFM2.5-VL-450M, read the same way as the tuned model, with its label logits divided by
+one scalar temperature chosen to minimise NLL on the environment's development split.
+
+**Trained environments.**
+
+| environment | NLL | NLL, base + T | Brier | ECE | AUROC | acc |
+|---|---|---|---|---|---|---|
+| mcq | **0.727** | 0.929 | 0.380 | 0.067 | 0.816 | 0.722 |
+| ChaosNLI | **0.902** | 0.941 | 0.245 | 0.079 | 0.591 | 0.575 |
+| CivilComments | **0.324** | 0.581 | 0.044 | 0.073 | 0.906 | 0.940 |
+| STS-B | **1.107** | 1.782 | 0.325 | 0.046 | 0.622 | 0.562 |
+| Folktables | **0.436** | 0.693 | 0.286 | 0.013 | 0.763 | 0.789 |
+| CIFAR-10 (+C) | **0.509** | 0.804 | 0.212 | 0.040 | 0.918 | 0.851 |
+| Camelyon17 | **0.202** | 0.693 | 0.110 | 0.033 | 0.887 | 0.930 |
+
+> [!NOTE]
+> ChaosNLI, CivilComments and STS-B have soft labels from multiple annotations, so
+> top-label ECE does not fully measure calibration. On these, use NLL and KL divergence to
+> the annotator distribution.
+
+**Unseen tasks.** Not in training.
+
+| task | type | NLL | NLL, base + T | base rate |
+|---|---|---|---|---|
+| BoolQ | noul | **0.661** | 0.646 | 0.665 |
+| Yelp review stars | score | **1.467** | 1.455 | 1.609 |
+
+**Distribution shift.** CIFAR-10-C.
+
+| | clean | sev. 1 | 2 | 3 | 4 | 5 |
+|---|---|---|---|---|---|---|
+| accuracy | 0.960 | 0.907 | 0.872 | 0.841 | 0.795 | 0.724 |
+| mean confidence | 0.962 | 0.928 | 0.903 | 0.882 | 0.856 | 0.811 |
+
+**Other System One models.** NLL on the text environments, same test splits, same
+readout. The other alpha-sys-1 sizes on the table are their released seeds. `Qwen3.8-27B` is the open 27B
+generalist, read at its first answer token with reasoning off, plus a dev-fitted
+temperature.
+
+| environment | alpha-sys-1-450M (this) | alpha-sys-1-1.6B | Jev 1.13.0 | Qwen3.8-27B + T | base rate |
+|---|---|---|---|---|---|
+| mcq | **0.727** | 0.427 | 0.129 | 0.159 | 1.439 |
+| ChaosNLI | **0.902** | 0.784 | 0.929 | 0.706 | 0.938 |
+| CivilComments | **0.324** | 0.323 | 0.671 | 0.473 | 0.425 |
+| STS-B | **1.107** | 1.033 | 1.222 | 1.347 | 1.727 |
+| Folktables | **0.436** | 0.448 | 0.534 | 0.472 | 0.683 |
+| BoolQ (unseen) | **0.661** | 0.479 | 0.225 | 0.316 | 0.665 |
+| Yelp review stars (unseen) | **1.467** | 1.110 | 0.900 | 0.858 | 1.609 |
+
+Per-hospital, per-state-year and per-identity-group tables, the three-seed gate tables and
+the full comparison against other System One models (hosted and open) are in the
+[repository](https://github.com/nullsilver/alpha-sys-1) under `runs/`.
+
+## Limitations
+
+> [!WARNING]
+> On a task that differs substantially from the training environments, do not assume this
+> model stays calibrated; measure it against the base model's calibration. In
+> leave-one-domain-out tests at 1.6B, a model tuned on the other environments beat the
+> untuned base with a transferred temperature on one held-out environment out of three,
+> and on the two unseen tasks above the trained-on-all checkpoints match the base model and do not beat it.
+> With a few hundred labelled examples from your own task, fit a temperature on them:
+> divide the label logits by one scalar chosen to minimise NLL on those examples
+> (`alpha_sys_1.fit_temperature`), then pass it as `SystemOne(..., temperature=T)`.
+
+- Knowledge depends on model size: on MCQ, the untuned 1.6B base outperforms this tuned
+  model.
+- When the model does not know an answer, its distribution is close to uniform.
+- Under strong distribution shift, such as CIFAR-10-C at severity 5, confidence remains
+  higher than accuracy.
+- Reversing the option order changes the top answer on 15% of MCQ items, mostly
+  among low-confidence examples.
+- The answer space is capped at 26 options.
+- Fine-tuning used English data only and at most one image per question.
+
+## Related work
+
+The interface follows TypeSafe's Jev (a hosted System One model, the `state` / `questions`
+request shape). Reading an answer distribution from the label-token logits of one forward
+pass is the readout of Kadavath et al. (2022, *Language Models (Mostly) Know What They
+Know*) and of the LLM-as-a-Verifier line of work, which scores rubric levels from the
+logits of letter tokens. That calibration improves with size, and that a temperature
+fitted on one domain transfers badly to another, is Jiang et al. (2021, *How Can We Know
+When Language Models Know?*). Training on a proper scoring rule against annotator
+distributions is why a fixed answer space and calibration are non-conflicting (Kalai and
+Vempala, 2024, *Calibrated Language Models Must Hallucinate*). Base models: Liquid AI's
+LFM2.5-VL.
+
+## License
+
+This model is derived from LiquidAI/LFM2.5-VL-450M and is released under the [LFM Open License
+v1.0](LICENSE).
+
+## Citation
+
+```bibtex
+@misc{alphasys1,
+  title  = {alpha-sys-1: a small calibrated System One model},
+  author = {Nullsilver},
+  year   = {2026},
+  url    = {https://huggingface.co/collections/nullsilver/alpha-sys-1}
+}
+```