Card: point at LiquidAI/d1-3B, date the 0.2.1 claim

README.md

@@ -31,7 +31,6 @@ tags:
 - calibration
 - system-one
 - multimodal
-- decision-model
 ---
 
 <div align="center">
@@ -50,22 +49,21 @@ tags:
 # d1-3B
 
 d1-3B is a 3B parameter **decision model** built on [LFM2.5-VL-3B](https://huggingface.co/LiquidAI/LFM2.5-VL-3B).
-You give it a state (text, JSON, images, or a mix) and a set of questions. It returns calibrated,
-typed answers in **one forward pass with zero output tokens**.
+You give it a state (text, JSON, images, or a mix) and a set of named questions. It returns calibrated,
+typed answers in **one forward pass with zero output tokens**: every answer is read directly from the
+model's distribution over the options, with no generation and no parsing.
 
-- **Best decision model under 10B on the Decision Index 0.2.1**: 48.57, ahead of every 4B and 9B model
-  and of Decider 35B-A3B (47.11).
-- **Multimodal**: images and text in the same state. It scores 74.1 on 11 public image benchmarks
+- **Best decision model under 10B on the Decision Index 0.2.1** (leaderboard of 2026-09-28): 47.12, ahead
+  of every 4B and 9B model and level with Decider 35B-A3B (47.11).
+- **Multimodal**: images and text in the same state. It scores 73.7 on 11 public image benchmarks
   (LFM2.5-VL-3B: 73.9).
 - **Fast**: 8 ms a decision on an NVIDIA RTX 4090, 9 ms on an AMD MI325X, 30 ms on an Apple M5 Pro.
 
-Find more information about open d1 in our [blog post](https://www.liquid.ai/blog/open-d1).
-
-![image](https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/r2H7UlZ_m48SYAhZWIHiu.png)
+![Decision Index against model size](assets/di_pareto.png)
 
 > [!NOTE]
 > 💻 **Demos**: Try d1-3B in a Hugging Face space without any setup:
-> [**Open d1 Arcade**](https://huggingface.co/spaces/LiquidAI/system-one-arcade): Collection of 10 demos using d1-3B
+> **TODO: add demo**
 
 
 ## 🗒️ Model Details
@@ -78,9 +76,11 @@ Find more information about open d1 in our [blog post](https://www.liquid.ai/blo
 d1-3B is a multimodal decision model with the following features:
 
 - **Total parameters**: 3.12B
+- **LM backbone**: LFM2.5-2.6B, 30 layers (22 double-gated short convolution blocks + 8 GQA)
 - **Vision encoder**: SigLIP2 NaFlex shape-optimized 400M
 - **Context length**: 32,768 tokens
 - **Vocabulary size**: 128,000
+- **Output**: no generated tokens. Each answer is a probability distribution over the question's options.
 
 We recommend d1-3B wherever a pipeline needs a yes/no, a pick from named options, or a rating:
 routing and triage, moderation, intent and topic classification, extraction checks, reranking, LLM-as-a-judge
@@ -97,47 +97,36 @@ pip install "transformers>=5.14" torch torchvision pillow
 The model ships its own code, so load it with `trust_remote_code=True`:
 
 ```python
+import io
+import urllib.request
+
 import torch
+from PIL import Image
 from transformers import AutoModel
-from transformers.image_utils import load_image
 
 device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
-dtype = torch.float32 if device == "cpu" else torch.bfloat16
-model = AutoModel.from_pretrained("LiquidAI/d1-3B", trust_remote_code=True, dtype=dtype).to(device)
+model = AutoModel.from_pretrained("LiquidAI/d1-3B", trust_remote_code=True,
+                                  dtype=torch.float32 if device == "cpu" else torch.bfloat16).to(device)
 
-# Text: several named questions over one state, answered in one pass
+# Several named questions over one text state, answered in one pass
 questions = {
-    "refund": {
-        "type": "noul",
-        "instructions": "Is the customer asking for a refund?",
-    },
-    "team": {
-        "type": "choice",
-        "instructions": "Which team should handle this?",
-        "criteria": {
-            "billing": "Charges, refunds, invoices",
-            "technical": "App or site faults",
-            "fraud": "Suspected unauthorised use",
-        },
-    },
-    "urgency": {
-        "type": "score",
-        "instructions": "How urgent is this?",
-        "criteria": ["Can wait", "Today", "Blocking the customer now"],
-    },
+    "refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},
+    "team": {"type": "choice", "instructions": "Which team should handle this?",
+             "criteria": {"billing": "Charges, refunds, invoices", "technical": "App or site faults",
+                          "fraud": "Suspected unauthorised use"}},
+    "urgency": {"type": "score", "instructions": "How urgent is this?",
+                "criteria": ["Can wait", "Today", "Blocking the customer now"]},
 }
 print(model.system_one("I was charged twice this month, please refund one of them.", questions))
 
-# Image: the photo is the whole state
-image = load_image("http://images.cocodataset.org/val2017/000000039769.jpg")  # two cats on a sofa
-cats = {
-    "type": "choice",
-    "instructions": "How many cats are there?",
-    "criteria": {"one": "One", "two": "Two", "more": "Three or more"},
-}
-print(model.system_one(None, {"cats": cats}, images=[image]))
+# An image as the whole state
+url = "http://images.cocodataset.org/val2017/000000039769.jpg"  # two cats on a sofa
+photo = Image.open(io.BytesIO(urllib.request.urlopen(url).read()))
+print(model.system_one(None, {"cats": {"type": "choice", "instructions": "How many cats are there?",
+                                       "criteria": {"one": "One", "two": "Two", "more": "Three or more"}}},
+                       images=[photo]))
 
-# Batch: many requests, packed together with no padding
+# Many requests, packed together with no padding
 tickets = ["Where is my parcel? It was due Monday.", "The app crashes when I open settings."]
 print(model.system_one_batch([(t, {"team": questions["team"]}) for t in tickets]))
 ```
@@ -163,32 +152,16 @@ Each call returns `{"answers": {name: answer}, "usage": {"input_tokens": n, "out
 
 ## ⚡ Speed
 
-Warm calls, one request at a time: a single question, three questions over one state, a 3.4k-token
-state and a 384 px image. The last column is throughput with 64 states packed into one pass.
-
-### Edge Inference
-
-We measure latency on an Apple M5 Pro and, in collaboration with NVIDIA, on an NVIDIA Jetson AGX Thor,
-a Jetson AGX Orin 64 GB and a Jetson Orin Nano.
+Warm calls in bf16, median of 20 runs:
 
 | | one question | 3 questions, one pass | 3.4k-token state | 384 px image | 64 states, packed |
 |---|---:|---:|---:|---:|---:|
-| Apple M5 Pro (`mps`) | 30 ms | 41 ms | 640 ms | 62 ms | 78 / s |
-| NVIDIA Jetson AGX Thor | 16 ms | 20 ms | 220 ms | 35 ms | 262 / s |
-| NVIDIA Jetson AGX Orin 64 GB | 26 ms | 35 ms | 560 ms | 83 ms | 110 / s |
-| NVIDIA Jetson Orin Nano | 50 ms | 73 ms | 1640 ms | 202 ms | 38 / s |
+| NVIDIA RTX 4090 | **8.0 ms** | 21 ms | 102 ms | 17 ms | 475 / s |
+| AMD MI325X | 9.2 ms | 14 ms | 44 ms | 18 ms | 1,106 / s |
+| Apple M5 Pro (`mps`) | 30 ms | 41 ms | 0.64 s | 62 ms | 78 / s |
 
-### GPU Inference
-
-We measure latency on an NVIDIA RTX 4090 and an AMD MI325X, in bf16, median of 20 runs.
-
-| | one question | 3 questions, one pass | 3.4k-token state | 384 px image | 64 states, packed |
-|---|---:|---:|---:|---:|---:|
-| NVIDIA RTX 4090 | 8 ms | 21 ms | 102 ms | 17 ms | 475 / s |
-| AMD MI325X | 9 ms | 14 ms | 44 ms | 18 ms | 1,106 / s |
-
-On NVIDIA GPUs, `model.compile(mode="reduce-overhead")` runs single questions as CUDA graphs (the RTX 4090
-row uses it). Without it, a single question takes 16 ms. The first call with a new shape pays for kernel
+On NVIDIA GPUs, `model.compile(mode="reduce-overhead")` runs single questions as CUDA graphs. The RTX 4090
+row uses it; without it a single question takes 16 ms. The first call with a new shape pays for kernel
 selection or compilation, so warm up the shapes you serve.
 
 ## 📊 Performance
@@ -197,12 +170,15 @@ All results are on public benchmarks.
 
 ### Decision Index 0.2.1
 
-We scored d1-3B with the official scorer (not a leaderboard submission). All other rows come from the public leaderboard v0.2.1.
+The [Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) casts 38 public
+benchmarks as 120,226 typed decisions, scored as chance-corrected skill. We scored d1-3B with the official
+scorer; it is not a leaderboard submission. All other rows come from the public leaderboard
+(snapshot 2026-09-28). Sizes are served parameters.
 
 | Model | Size | Decision Index | Knowledge | Language | Retrieval | Tools | Arts |
 |---|---:|---:|---:|---:|---:|---:|---:|
 | Winnow-12B | 12B | 50.02 | 33.8 | 56.0 | 54.0 | 71.0 | 30.0 |
-| **d1-3B** | **3B** | **48.57** | 23.8 | 56.4 | 52.8 | **74.5** | **36.3** |
+| **d1-3B** | **3B** | **47.12** | 27.8 | 53.5 | 52.2 | 66.7 | **34.6** |
 | Decider 35B-A3B | 36B | 47.11 | 31.8 | 55.5 | 54.7 | 56.5 | 32.6 |
 | JPT-9B | 9.7B | 46.89 | 31.7 | 56.7 | 44.6 | 67.0 | 28.6 |
 | Decision 1.0 Lux | 9.7B | 43.49 | 30.9 | 48.0 | 50.0 | 57.2 | 26.4 |
@@ -214,22 +190,23 @@ We scored d1-3B with the official scorer (not a leaderboard submission). All oth
 
 ### Benchmarks as decisions
 
-Besides the Decision Index, we added a few other internal evaluations based on public benchmarks.
+Evaluation splits are cast as typed decisions, with the same rows for every model. These scores follow the
+decision protocol, not each benchmark's own.
 
 | Benchmark | d1-3B | Decider 4B | Decider 2B |
 |---|---:|---:|---:|
-| SQuAD 2.0 | **85.3** | 76.0 | 67.7 |
-| Civil Comments | 93.0 | 92.8 | **93.6** |
-| MASSIVE intent | 87.3 | **88.3** | 81.1 |
-| HelpSteer2 | 36.7 | **42.0** | 32.0 |
-| PubMedQA | **66.0** | 63.3 | 65.7 |
-| BoolQ | 86.7 | **89.0** | 87.3 |
-| XNLI | 85.0 | **88.6** | 85.0 |
-| PAWS-X | **76.9** | 69.8 | 59.5 |
-| **Mean** | **77.1** | 76.2 | 71.5 |
-
-d1-3B also scores 71.8 on [DecisionBench](https://huggingface.co/datasets/Hanno-Labs/decision-bench) (eng v1,
-all 23,900 rows) and 69.3 on [Fast Decisions](https://huggingface.co/datasets/fastino/fast-decisions)
+| SQuAD 2.0 | **83.3** | 76.0 | 67.7 |
+| Civil Comments | 93.3 | 92.8 | **93.6** |
+| MASSIVE intent | 86.9 | **88.3** | 81.1 |
+| HelpSteer2 | 38.0 | **42.0** | 32.0 |
+| PubMedQA | **68.3** | 63.3 | 65.7 |
+| BoolQ | 86.3 | **89.0** | 87.3 |
+| XNLI | 85.6 | **88.6** | 85.0 |
+| PAWS-X | **76.4** | 69.8 | 59.5 |
+| **Mean** | **77.3** | 76.2 | 71.5 |
+
+d1-3B also scores 69.5 on [DecisionBench](https://huggingface.co/datasets/Hanno-Labs/decision-bench) (eng v1,
+all 23,900 rows) and 68.8 on [Fast Decisions](https://huggingface.co/datasets/fastino/fast-decisions)
 (dev split).
 
 ### Vision
@@ -239,22 +216,29 @@ compared with the base model:
 
 | Benchmark | d1-3B | LFM2.5-VL-3B |
 |---|---:|---:|
-| AI2D | 79.9 | 80.9 |
-| BLINK | 59.2 | 58.7 |
-| CV-Bench | 82.1 | 87.6 |
-| HallusionBench | 65.3 | 65.0 |
-| MMBench | 84.9 | 84.3 |
-| MME | 82.1 | 82.4 |
-| MMStar | 59.9 | 61.2 |
-| MMVP | 77.0 | 73.7 |
+| AI2D | 80.1 | 80.9 |
+| BLINK | 58.2 | 58.7 |
+| CV-Bench | 82.7 | 87.6 |
+| HallusionBench | 63.8 | 65.0 |
+| MMBench | 84.3 | 84.3 |
+| MME | 81.7 | 82.4 |
+| MMStar | 59.4 | 61.2 |
+| MMVP | 76.3 | 73.7 |
 | POPE | 88.5 | 90.1 |
-| VisualWebBench | 71.4 | 78.3 |
-| VL-RewardBench | 65.0 | 50.9 |
-| **Mean** | **74.1** | 73.9 |
-| [ImajevBench](https://huggingface.co/datasets/mohit67890/imajev-bench) (dev and calibration, 253 rows) | 64.0 | 66.8 |
+| VisualWebBench | 71.6 | 78.3 |
+| VL-RewardBench | 63.6 | 50.9 |
+| **Mean** | **73.7** | 73.9 |
+| [ImajevBench](https://huggingface.co/datasets/mohit67890/imajev-bench) (dev and calibration, 253 rows) | 66.8 | 66.8 |
 
 With the images removed, the same questions score 45.1, so the answers come from the images.
 
+## ⚠️ Limitations
+
+- **Knowledge-heavy questions**: larger models stay ahead on hard knowledge (for example MMLU-Pro and GPQA).
+- **Thresholds**: calibrate any threshold on data from your own deployment before gating on it.
+- **No text output**: d1-3B answers questions over fixed options. Use [LFM2.5-VL-3B](https://huggingface.co/LiquidAI/LFM2.5-VL-3B)
+  for open-ended generation.
+
 ## 📬 Contact
 
 - Got questions or want to connect? [Join our Discord community](https://discord.com/invite/liquid-ai)
@@ -262,16 +246,6 @@ With the images removed, the same questions score 45.1, so the answers come from
 
 ## Citation
 
-```bibtex
-@article{liquidAI2026opend1,
-  author  = {Liquid AI},
-  title   = {Open d1: Edge decision models for text, vision, and audio},
-  journal = {Liquid AI Blog},
-  year    = {2026},
-  note    = {https://www.liquid.ai/blog/d1-open},
-}
-```
-
 ```bibtex
 @article{liquidai2025lfm2,
   title   = {LFM2 Technical Report},

assets/d1-3b-smash.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08f69b1a67bf32cd18fad32b5872519b580be9e94bafbe26a13d5cf0aaa49c1f
+size 53062187

assets/ood_tasks.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b491dfa851157e12ba2bee33a394c9565408247cf0004e493730622c62878e42
+size 8931625

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:50e03317847caf6df9a9aee27ed40f20554a86a21e60d1d47ba41a422b546c0c
+oid sha256:042be2b86f2140cf7e696d9d26c0dd8d2f0f8f943b783eb9ffa34903b41c98bb
 size 6247065504