d1-2B

Co-authored-by: mlabonne <mlabonne@users.noreply.huggingface.co>
Co-authored-by: Aurelien-Lac <Aurelien-Lac@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,71 @@
+LFM Open License v1.0 
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by this document.
+
+"Licensor" shall mean Liquid AI, Inc.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work.
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+"Commercial Use" shall mean any use of the Work for direct or indirect commercial advantage or monetary compensation.
+
+"Qualified Non-Profit Organization" shall mean a Legal Entity that is organized and operated exclusively for religious, charitable, scientific, testing for public safety, literary, or educational purposes, and which is exempt from federal income tax under Section 501(c)(3) of the United States Internal Revenue Code of 1986, as amended, or any equivalent non-profit or charitable organization in a foreign jurisdiction.
+
+"Non-Commercial or Research Purposes" shall mean purposes that do not involve any use of the Work or a Derivative Work for Commercial Use.
+
+"Threshold" shall mean annual revenue of 10 million United States dollars ($10,000,000) or more.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, including the Commercial Use limitation set forth in Section 5, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License, including the Commercial Use limitation set forth in Section 5, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and
+
+(b) You must cause any modified files to carry prominent notices stating that You changed the files; and
+
+(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+
+(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+5. Commercial Use Limitation.
+
+(a) The rights granted under this License for Commercial Use are conditioned upon You or Your Legal Entity not exceeding the Threshold.
+
+(b) Any Commercial Use of the Work or a Derivative Work by a Legal Entity that exceeds the Threshold is not licensed under this Agreement.
+
+(c) The Threshold shall not apply to a Qualified Non-Profit Organization's use of the Work or a Derivative Work for Non-Commercial or Research Purposes.
+
+6. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+7. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except for the reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+8. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+9. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+10. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+11. Termination. This License will terminate automatically and immediately if You fail to comply with any of its terms and conditions. Upon termination, You must cease all use of the Work and any Derivative Works and delete all copies in Your possession.
+
+END OF TERMS AND CONDITIONS

README.md

@@ -0,0 +1,593 @@
+---
+library_name: transformers
+license: other
+license_name: lfm1.0
+license_link: LICENSE
+pipeline_tag: image-text-to-text
+tags:
+- liquid
+- lfm3
+- edge
+- decision
+- classification
+- extraction
+- calibration
+- system-one
+- multimodal
+- decision-model
+- custom_code
+language:
+- en
+- ar
+- zh
+- fr
+- de
+- it
+- ja
+- ko
+- pt
+- es
+- vi
+- th
+- id
+- hi
+- ru
+- pl
+---
+
+<div align="center">
+  <img
+    src="https://huggingface.co/LiquidAI/d1-2B/resolve/main/assets/liquid-ai-banner.webp"
+    alt="Liquid AI"
+    style="width: 100%; max-width: 100%; height: auto; display: block; margin: 0;"
+  />
+  <div align="center" style="display: flex; justify-content: center; align-items: center; gap: 1.25em; margin: 8px 0 1.7rem; padding: 0; line-height: 1.3;">
+    <a href="https://console.liquid.ai/dashboard" style="display: inline-flex; align-items: center; gap: 6px;"><img src="https://huggingface.co/LiquidAI/d1-2B/resolve/main/assets/icons/api.svg" width="16" height="16" alt="" aria-hidden="true" style="display: block; flex: 0 0 16px; width: 16px; height: 16px; margin: 0;" /><strong>API</strong></a>
+    <a href="https://playground.liquid.ai/" style="display: inline-flex; align-items: center; gap: 6px;"><img src="https://huggingface.co/LiquidAI/d1-2B/resolve/main/assets/icons/try.svg" width="16" height="16" alt="" aria-hidden="true" style="display: block; flex: 0 0 16px; width: 16px; height: 16px; margin: 0;" /><strong>Try</strong></a>
+    <a href="https://docs.liquid.ai/lfm/getting-started/welcome" style="display: inline-flex; align-items: center; gap: 6px;"><img src="https://huggingface.co/LiquidAI/d1-2B/resolve/main/assets/icons/docs.svg" width="16" height="16" alt="" aria-hidden="true" style="display: block; flex: 0 0 16px; width: 16px; height: 16px; margin: 0;" /><strong>Docs</strong></a>
+    <a href="https://www.liquid.ai/blog/open-d1" style="display: inline-flex; align-items: center; gap: 6px;"><img src="https://huggingface.co/LiquidAI/d1-2B/resolve/main/assets/icons/blog.svg" width="16" height="16" alt="" aria-hidden="true" style="display: block; flex: 0 0 16px; width: 16px; height: 16px; margin: 0;" /><strong>Blog</strong></a>
+  </div>
+</div>
+
+# d1-2B
+
+d1-2B is a **decision model for text, pictures, and video**, built on [LFM3-VL-2B](https://huggingface.co/LiquidAI/LFM3-VL-2B). It returns calibrated answers in one forward pass, without generating tokens.
+
+- **Best-in-class:** it leads the open d1 models on average and on our board of over a hundred decision benchmarks, ahead of d1-3B at under two thirds of its size.
+- **Beyond classification:** it also extracts text, numbers, and dates with their span, locates objects in pictures, and estimates quantities with an interval.
+- **Millisecond decisions:** 5.7 ms per decision on an NVIDIA H100 and 3.8 ms on an AMD MI325X, with any number of questions about one state in one call.
+
+Find more information about open d1 in [our release blog post](https://www.liquid.ai/blog/open-d1).
+
+## Model details
+<hr style="margin: -0.7em 0 1.2em 0; border: 0; border-top: 1px solid rgba(127, 127, 127, 0.3);" />
+
+The [LFM3 family](https://huggingface.co/collections/LiquidAI/lfm3) includes base, text, vision, audio, omni, and decision models. d1-2B decides over text, pictures, and video, and [d1-Omni-2B](https://huggingface.co/LiquidAI/d1-Omni-2B) adds audio.
+
+<table style="width: 100%; display: table; border: 0; border-radius: 0; border-collapse: collapse;">
+<thead style="border: 0;">
+<tr style="border: 0;">
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Property</th>
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Specification</th>
+</tr>
+</thead>
+<tbody style="border: 0;">
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Checkpoint</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">Post-trained decision model, built on <a href="https://huggingface.co/LiquidAI/LFM3-VL-2B">LFM3-VL-2B</a></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Parameters</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">1.92B</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Architecture</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">LFM3 backbone, 24 layers: 13 gated short convolutions, 5 sliding-window attention, 6 global attention</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Context length</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">128K tokens</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Vocabulary</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">128,000 tokens</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Input</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">A state (text or JSON), pictures or a video, and any number of named questions</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Output</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">Typed answers with their probabilities, with no generated tokens</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>Languages</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;">English, Arabic, Chinese, French, German, Italian, Japanese, Korean, Portuguese, Spanish, Vietnamese, Thai, Indonesian, Hindi, Russian, Polish</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><strong>License</strong></td>
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="LICENSE">LFM Open License v1.0</a></td>
+</tr>
+</tbody>
+</table>
+
+d1-2B is designed for any pipeline step that needs a decision: routing and triage, moderation and guardrails, intent and topic classification, extraction from documents and forms, judging and reranking, and visual inspection. It is not a chat model.
+
+Knowledge-heavy questions favor larger models, and answers get less reliable past about 64K tokens of state. Estimates are reliable on counts and scores, less so on free quantities.
+
+## Inference
+<hr style="margin: -0.7em 0 1.2em 0; border: 0; border-top: 1px solid rgba(127, 127, 127, 0.3);" />
+
+d1-2B runs in [Transformers](https://docs.liquid.ai/deployment/gpu-inference/transformers) with its own model code. It reads its answers from a single forward pass, so there are no sampling parameters to set.
+
+**Transformers quick start:**
+
+```bash
+pip install "transformers>=5.19" torch torchvision pillow
+```
+
+```python
+import torch
+from transformers import AutoModel
+
+device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
+dtype = torch.float32 if device == "cpu" else torch.bfloat16
+model = AutoModel.from_pretrained("LiquidAI/d1-2B", trust_remote_code=True, dtype=dtype).to(device)
+
+# Several named questions over one state, answered in one pass
+questions = {
+    "refund": {"type": "noul", "instructions": "Is the customer asking for a refund?"},  # yes/no: P(yes)
+    "team": {"type": "choice", "instructions": "Which team should handle this?",
+             "criteria": {"billing": "Charges, refunds, invoices", "shipping": "Delivery and parcels",
+                          "technical": "App or site faults"}},
+    "topics": {"type": "choice", "multiple": True, "instructions": "Which topics does the message raise?",
+               "criteria": {"billing": "Payments and charges", "delivery": "Shipping and parcels",
+                            "login": "Signing in to the account"}},
+    "urgency": {"type": "score", "instructions": "How urgent is this ticket?", "criteria": ["low", "medium", "high"]},
+}
+out = model.system_one("I was charged twice and my parcel never arrived. Please refund one charge.", questions)
+a = out["answers"]
+urgency = a["urgency"]["legend"][str(int(a["urgency"]["score"]))]
+print(round(a["refund"]["noul"], 2), a["team"]["choice"], a["topics"]["choices"], urgency)
+# 1.0 billing ['billing', 'delivery'] medium
+```
+
+<details>
+<summary><strong>Values from a document</strong></summary>
+
+```python
+invoice = "ACME SA\nInvoice 2026-117\nTotal due: 1 234,50 EUR\nPay before 03/04/2026 (day/month/year)"
+out = model.system_one(invoice, {
+    "payee": {"type": "text", "instructions": "Who issued the invoice?"},
+    "total": {"type": "number", "instructions": "How much is due?"},
+    "due": {"type": "date", "instructions": "By when must it be paid?"},
+    "vat": {"type": "number", "instructions": "How much VAT is charged?"},
+})
+a = out["answers"]
+print(a["payee"]["text"], a["total"]["number"], a["due"]["date"], a["vat"]["number"])
+# ACME SA 1234.5 2026-04-03 None
+```
+
+</details>
+
+<details>
+<summary><strong>Pictures and boxes</strong></summary>
+
+Count, locate one object, or list every instance (`max` reads at most that many):
+
+```python
+from transformers.image_utils import load_image
+
+image = load_image("http://images.cocodataset.org/val2017/000000039769.jpg")  # two cats on a sofa
+out = model.system_one(None, {
+    "cats": {"type": "number", "instructions": "How many cats are there?", "range": [0, 10], "step": 1},
+    "remote": {"type": "box", "instructions": "Where is the remote control on the left?"},
+    "every_cat": {"type": "boxes", "instructions": "Every cat.", "max": 4},
+}, media=[image])
+a = out["answers"]
+print(a["cats"]["number"], a["every_cat"]["count"], {k: round(v, 2) for k, v in a["remote"]["box"].items()})
+# 2.0 2 {'top': 0.15, 'bottom': 0.24, 'left': 0.06, 'right': 0.28}
+```
+
+</details>
+
+<details>
+<summary><strong>Many requests in one call</strong></summary>
+
+```python
+tickets = ["Where is my parcel? It was due Monday.", "The app crashes when I open settings."]
+for out in model.system_one_batch([(t, {"team": questions["team"]}) for t in tickets]):
+    print(out["answers"]["team"]["choice"])
+# shipping
+# technical
+```
+
+</details>
+
+<details>
+<summary><strong>Calls</strong></summary>
+
+<table style="width: 100%; display: table; border: 0; border-radius: 0; border-collapse: collapse;">
+<thead style="border: 0;">
+<tr style="border: 0;">
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Call</th>
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">What it does</th>
+</tr>
+</thead>
+<tbody style="border: 0;">
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>system_one(state, questions, media=None)</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">Named questions over one state, in one pass. <code>media</code> lists pictures (PIL images or <code>{&quot;type&quot;: &quot;image&quot;, &quot;path&quot;: ...}</code>) and videos as their frames (<code>{&quot;type&quot;: &quot;video&quot;, &quot;video&quot;: [PIL images], &quot;fps&quot;: 24}</code>).</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>system_one_batch([(state, questions[, media]), ...])</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">Many requests in one call, batched.</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>set_size=</code> (either call)</td>
+<td style="border: 0; padding: 0.55em 0.75em;">Questions read together after the state. The default is the model's trained size (16). Larger is faster and costs some accuracy on text.</td>
+</tr>
+</tbody>
+</table>
+
+A state is a string, any JSON value, or `None` when the media are the whole state. Each call returns `{"answers": {name: answer}, "usage": {"input_tokens": n, "output_tokens": 0}}`.
+
+</details>
+
+<details>
+<summary><strong>Question types</strong></summary>
+
+<table style="width: 100%; display: table; border: 0; border-radius: 0; border-collapse: collapse;">
+<thead style="border: 0;">
+<tr style="border: 0;">
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);"><code>type</code></th>
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Question</th>
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Answer</th>
+</tr>
+</thead>
+<tbody style="border: 0;">
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>noul</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">Yes or no</td>
+<td style="border: 0; padding: 0.55em 0.75em;"><code>noul</code>: P(yes)</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>choice</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">One of the options <code>criteria</code> names (<code>{name: description}</code>)</td>
+<td style="border: 0; padding: 0.55em 0.75em;"><code>choice</code>, <code>confidence</code>, <code>probabilities</code></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>choice</code> with <code>multiple: true</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">Any of the options <code>criteria</code> names</td>
+<td style="border: 0; padding: 0.55em 0.75em;"><code>choices</code>, <code>probabilities</code> (each option's own P(yes))</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>score</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">One of 2 to 10 ordered levels, with <code>criteria</code> listing them lowest first</td>
+<td style="border: 0; padding: 0.55em 0.75em;"><code>score</code> (the median level), <code>confidence</code>, <code>probabilities</code>, <code>legend</code></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>text</code>, <code>number</code>, <code>date</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">A value to read from the state</td>
+<td style="border: 0; padding: 0.55em 0.75em;">The field of its type (a number with its <code>unit</code>, a date in ISO 8601), <code>span</code>, <code>found</code></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>number</code> with <code>range</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">An estimate, <code>step</code> apart</td>
+<td style="border: 0; padding: 0.55em 0.75em;"><code>number</code> (the median), <code>interval</code> (p10, p90)</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><code>box</code>, <code>boxes</code></td>
+<td style="border: 0; padding: 0.55em 0.75em;">One object, or every instance (16 at most, or <code>max</code>), in the one picture</td>
+<td style="border: 0; padding: 0.55em 0.75em;"><code>box</code> (<code>top</code>, <code>bottom</code>, <code>left</code>, <code>right</code> as fractions of the picture), <code>found</code>, <code>confidence</code>, or <code>boxes</code> and <code>count</code></td>
+</tr>
+</tbody>
+</table>
+
+A value or a box is missing when `found` is under 0.5, and it is then `None`. Calibrate any decision threshold on data from your own use.
+
+</details>
+
+## Performance
+<hr style="margin: -0.7em 0 1.2em 0; border: 0; border-top: 1px solid rgba(127, 127, 127, 0.3);" />
+
+### Benchmarks
+
+d1-2B leads the open d1 models on our decision board, on average, and on most single benchmarks, ahead of d1-3B with under two thirds of its parameters.
+
+<div role="region" tabindex="0" style="width: 100%; max-width: 100%; overflow-x: auto; -webkit-overflow-scrolling: touch;">
+<table style="width: 100%; display: table; border: 0; border-radius: 0; border-collapse: collapse;">
+<thead style="border: 0;">
+<tr style="border: 0;">
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Benchmark</th>
+<th scope="col" style="text-align: center; background-color: rgba(108, 79, 224, 0.08); padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);"><strong>d1<br>2B</strong></th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">d1-omni<br>600M</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">d1-Omni<br>2B</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">d1<br>3B</th>
+</tr>
+</thead>
+<tbody style="border: 0;">
+<tr style="border: 0;"><th scope="rowgroup" style="text-align: left; background-color: transparent; padding: 0.55em 0.75em; border: 0; font-weight: 600; padding-top: 1em; padding-bottom: 0.5em;">Decisions</th><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em; background-color: rgba(108, 79, 224, 0.08);"></td><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em;"></td><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em;"></td><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em;"></td></tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://huggingface.co/spaces/multimodalart/jev-decision-index">Decision Index</a> public suite (v0.3)</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>51.1</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">18.3</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>49.0</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>49.0</u></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://huggingface.co/datasets/Hanno-Labs/decision-bench">DecisionBench (Hanno Labs)</a> v1</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>77.1</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">58.7</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>76.4</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">73.1</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://decisionbench.ai">Decision Bench (decisionbench.ai)</a> v4</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>85.2</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">57.9</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>84.7</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">83.6</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://huggingface.co/datasets/fastino/fast-decisions">Fast Decisions</a></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><u>58.8</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">–</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">56.8</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><strong>59.3</strong></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://github.com/fstandhartinger/jevbench">JevBench</a> v1.4.1</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>77.9</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">58.0</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">75.3</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>76.2</u></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://huggingface.co/datasets/mohit67890/imajev-bench">ImajevBench</a> v2.0-lite</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>75.5</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">21.7</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>72.3</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">64.0</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://evals.typesafe.ai">WorkflowEvals</a></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>81.6</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">48.8</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>77.1</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">74.4</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;"><a href="https://huggingface.co/datasets/LocalLLaMA/typed-decisions">Typed Decisions</a></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;">61.3</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">57.5</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>63.1</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><strong>65.0</strong></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">Our board (over a hundred public and private benchmarks)</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>63.2</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">–</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>61.8</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">58.8</td>
+</tr>
+</tbody>
+<tbody style="border: 0;">
+<tr style="border: 0;"><th scope="rowgroup" style="text-align: left; background-color: transparent; padding: 0.55em 0.75em; border: 0; font-weight: 600; padding-top: 1em; padding-bottom: 0.5em; border-top: 1px solid rgba(127, 127, 127, 0.25);">Vision</th><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em; background-color: rgba(108, 79, 224, 0.08); border-top: 1px solid rgba(127, 127, 127, 0.25);"></td><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);"></td><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);"></td><td aria-hidden="true" style="border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);"></td></tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">AI2D</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><u>76.7</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">37.1</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">71.5</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><strong>79.9</strong></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">BLINK</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>66.2</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">37.7</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>63.5</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">59.2</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">CountQA</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>51.4</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">26.8</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>45.4</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">42.6</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">CV-Bench</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>86.1</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">40.8</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>84.0</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">82.1</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">ERQA</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>45.8</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">27.3</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>42.5</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">39.8</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">HallusionBench</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>72.1</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">52.6</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>67.5</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">65.3</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">MMBench</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>87.9</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">54.2</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>85.2</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">84.9</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">MME</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>85.4</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">61.0</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>83.9</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">82.1</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">MMMU-Pro</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>26.4</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">11.9</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>24.1</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">18.5</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">MMRB2</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>54.2</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">48.3</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>52.6</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">49.5</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">MMStar</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>67.5</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">31.9</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>61.3</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">59.9</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">MMVP</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>82.7</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">54.0</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>81.3</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">77.0</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">NaturalBench</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>81.5</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">58.8</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>80.0</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">79.8</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">POPE</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><u>88.7</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">82.7</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><strong>89.9</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">88.6</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">V*</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>76.4</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">35.6</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><strong>76.4</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>73.8</u></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">VisualPuzzles</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>29.9</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">26.2</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">26.9</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>29.5</u></td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">VisualWebBench</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>96.2</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">14.9</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>88.6</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">71.4</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">VL-RewardBench</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em;"><strong>69.3</strong></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">48.1</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;"><u>68.8</u></td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">65.0</td>
+</tr>
+</tbody>
+<tbody style="border: 0;">
+<tr style="border: 0;"><td style="border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);"><strong>Average</strong></td><td style="text-align: center; font-variant-numeric: tabular-nums; background-color: rgba(108, 79, 224, 0.08); border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);"><strong>70.2</strong></td><td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);">42.8</td><td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);"><u>67.7</u></td><td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em; border-top: 1px solid rgba(127, 127, 127, 0.25);">65.4</td></tr>
+</tbody>
+</table>
+</div>
+
+<details>
+<summary><small>Evaluation details</small></summary>
+
+<p><small>All four models are scored on the same items with the same scoring, so a few of d1-3B's and d1-omni-600M's numbers differ slightly from their own cards. d1-omni-600M has no Fast Decisions or board score. Average is the unweighted mean of the public benchmarks that all four models have, so it leaves out Fast Decisions and our board. Image benchmarks are read as decisions over each benchmark's options, on at most 1,000 rows each. Three of them are saturated: POPE, MME, and VisualWebBench.</small></p>
+
+</details>
+
+### Speed
+
+Warm calls after `model.compile()` in bf16, median of 20 calls, and best of three runs on the H100 and the M5 Pro. Compiling takes about a minute on the first calls.
+
+<div role="region" tabindex="0" style="width: 100%; max-width: 100%; overflow-x: auto; -webkit-overflow-scrolling: touch;">
+<table style="width: 100%; display: table; border: 0; border-radius: 0; border-collapse: collapse; min-width: 40rem; font-size: calc(0.875em - 1pt);">
+<thead style="border: 0;">
+<tr style="border: 0;">
+<th scope="col" style="text-align: left; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Hardware</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">One question</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">3 questions</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">5K-token state</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">384 px picture</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Box</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">Boxes</th>
+<th scope="col" style="text-align: center; padding: 0.55em 0.75em; border: 0; border-bottom: 1px solid rgba(127, 127, 127, 0.55);">64 states in one call</th>
+</tr>
+</thead>
+<tbody style="border: 0;">
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">NVIDIA H100 PCIe</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">5.7 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">7.6 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">52 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">12.3 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">16.1 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">24.4 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">1,132 / s</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">AMD MI325X</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">3.8 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">5.6 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">44 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">8.7 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">11.2 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">16.5 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">1,699 / s</td>
+</tr>
+<tr style="border: 0;">
+<td style="border: 0; padding: 0.55em 0.75em;">Apple M5 Pro (<code>mps</code>)</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">18.6 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">26.5 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">753 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">45.2 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">53.5 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">100 ms</td>
+<td style="text-align: center; font-variant-numeric: tabular-nums; border: 0; padding: 0.55em 0.75em;">109 / s</td>
+</tr>
+</tbody>
+</table>
+</div>
+
+## Citation
+<hr style="margin: -0.7em 0 1.2em 0; border: 0; border-top: 1px solid rgba(127, 127, 127, 0.3);" />
+
+```bibtex
+@article{liquidAI2026opend1,
+  author  = {Liquid AI},
+  title   = {Open d1: Edge decision models for text, vision, and audio},
+  journal = {Liquid AI Blog},
+  year    = {2026},
+  note    = {https://www.liquid.ai/blog/open-d1},
+}
+```
+
+```bibtex
+@article{liquidAI2026LFM3,
+  author  = {Liquid AI},
+  title   = {Introducing LFM3: Every Modality for Specialized Intelligence},
+  journal = {Liquid AI Blog},
+  year    = {2026},
+  note    = {https://www.liquid.ai/blog/lfm3},
+}
+```
+
+For questions and feedback, join [Discord](https://discord.com/invite/liquid-ai). For custom deployment and fine-tuning support, [contact Liquid AI](https://www.liquid.ai/contact).

api.py

@@ -0,0 +1,214 @@
+"""The System One API: named questions over a state in, typed answers out.
+
+    model.system_one(state, {"refund": {"type": "noul", "instructions": "Is this a refund request?"}})
+    # {"answers": {"refund": {"type": "noul", "noul": p}}, "usage": {"input_tokens": n, "output_tokens": 0}}
+
+A state is a string, any JSON value, or None with media alone; `media` holds its pictures, audio clips and videos
+(`media`). A question is a dict in the Decision Index's schema (`type`, `instructions`, `criteria`; spoken
+instructions a media dict) or one of `prompt`'s classes.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from typing import Any
+
+from .media import Media, media_list
+from .prompt import (
+    DEFAULT_STATE_STYLE,
+    Box,
+    Boxes,
+    Choice,
+    Estimate,
+    Noul,
+    Question,
+    Read,
+    as_question,
+    check_question,
+    state_text,
+    typed,
+)
+
+# How a score question's distribution becomes its answer. `median`, the default: the first level whose cumulative
+# probability reaches one half, optimal for absolute error (ordinal MAE). `mean`: the expected level, which a caller
+# rounds (the Decision Index's rule), optimal only for squared error. `argmax`: the likeliest level. `median` and
+# `argmax` answer a level, which rounding leaves as it is.
+SCORE_READOUTS = ("mean", "median", "argmax")
+SCORE_READOUT = "median"
+# An explicit "I don't know" option: a choice option whose text (its description, else its name) says the answerer
+# does not know. Not an option saying the evidence is insufficient or that a request cannot be served: those are
+# answers. `abstain_below` (off by default) picks it when the best other option's probability, renormalised over the
+# other options, is below the threshold.
+IDK_TEXTS = frozenset({"i don't know", "i do not know", "i don't know the answer", "i do not know the answer"})
+
+
+def abstain_option(criteria: Mapping[str, str | None] | None) -> str | None:
+    """The name of a choice's explicit "I don't know" option (the English `IDK_TEXTS`), or None."""
+    if not isinstance(criteria, Mapping):
+        return None
+    for name, text in criteria.items():
+        if str(text or name).strip().lower().replace("\u2019", "'").rstrip(".!") in IDK_TEXTS:
+            return name
+    return None
+
+
+def choice_index(probs: Sequence[float], abstain: int | None = None, abstain_below: float | None = None) -> int:
+    """A choice's answer: the likeliest option; with `abstain` (the "I don't know" option's index) and
+    `abstain_below`, that option when the best other option holds less than `abstain_below` of the other options'
+    mass."""
+    best = max(range(len(probs)), key=probs.__getitem__)
+    if abstain is None or abstain_below is None:
+        return best
+    others = [p for i, p in enumerate(probs) if i != abstain]
+    total = sum(others)
+    return abstain if total <= 0 or max(others) < abstain_below * total else best
+
+
+def score_level(probs: Sequence[float], readout: str = SCORE_READOUT) -> int:
+    """A score question's answer as a level index under `readout` (`mean`: rounded, clamped to the levels)."""
+    total = sum(probs) or 1.0
+    if readout == "mean":
+        return max(0, min(len(probs) - 1, int(round(sum(i * p / total for i, p in enumerate(probs))))))
+    if readout == "median":
+        mass = 0.0
+        for i, p in enumerate(probs):
+            mass += p / total
+            if mass >= 0.5:
+                return i
+        return len(probs) - 1
+    if readout == "argmax":
+        return max(range(len(probs)), key=probs.__getitem__)
+    raise ValueError(f"score_readout {readout!r}: one of {SCORE_READOUTS}")
+
+
+def answer(q: Question, probs: Sequence[float] | Sequence[Sequence[float]] | Mapping, score_readout: str = SCORE_READOUT,
+           abstain_below: float | None = None, context: str = "", calibration=None, edges: str = "median") -> dict:
+    """A noul's P(yes); a choice's pick (`choice_index`) and its probabilities; a score's level under
+    `score_readout` (`mean`: the expected level, unrounded); a typed answer's probabilities through `calibration`'s
+    tables (a read is found, a box drawn, an option chosen, at 0.5 after them; an estimate's quantiles read at the
+    levels its table gives). `edges`: how a box's edges are read (`box.EDGE_READS`, the checkpoint's `d1_box_edges`)."""
+    through = calibration.through if calibration is not None else lambda name, value: value
+    if isinstance(q, Box):
+        from .box import answer as box_answer
+
+        return box_answer(probs, through, edges)
+    if isinstance(q, Boxes):
+        from .box import answer_many
+
+        return answer_many(probs, through, edges)
+    if isinstance(q, Estimate):
+        value = q.grid.read(probs, None if calibration is None else lambda share: calibration.level("estimate", share))
+        return {"type": "number", "number": value["median"], "interval": [value["p10"], value["p90"]], "unit": q.unit}
+    if isinstance(q, Read):
+        from .parsing import parse_date, parse_number, written_unit
+
+        found = through(f"found:{q.type}", probs["found"])
+        present = found >= 0.5
+        text = probs["text"] if present else None
+        span = probs["span"] if present else None
+        value, unit = text, q.unit
+        if text is not None and q.type == "number":
+            unit = unit or written_unit(text)
+            value = parse_number(text, context, unit=q.unit)
+        elif text is not None and q.type == "date":
+            value, _ = parse_date(text, context)
+        result = {"type": q.type, q.type: value, "span": span, "found": found}
+        if q.type != "text":
+            result["text"] = text
+        if present and "path" in probs:
+            result["path"] = probs["path"]
+        if q.type == "number":
+            result["unit"] = unit
+        return result
+    if isinstance(q, Noul):
+        return {"type": "noul", "noul": probs[0]}
+    if isinstance(q, Choice) and q.multiple:
+        probs = [through("multiple", p) for p in probs]
+        return {"type": "choice", "choices": [name for name, p in zip(q.criteria, probs) if p >= 0.5],
+                "probabilities": dict(zip(q.criteria, probs))}
+    best = max(range(len(probs)), key=probs.__getitem__)
+    if isinstance(q, Choice):
+        names = list(q.criteria)
+        idk = abstain_option(q.criteria) if abstain_below is not None else None
+        pick = choice_index(probs, None if idk is None else names.index(idk), abstain_below)
+        return {"type": "choice", "choice": names[pick], "confidence": probs[pick],
+                "probabilities": dict(zip(names, probs))}
+    level = (sum(i * p for i, p in enumerate(probs)) if score_readout == "mean"
+             else float(score_level(probs, score_readout)))
+    return {"type": "score", "score": level, "confidence": probs[best],
+            "probabilities": {str(i): p for i, p in enumerate(probs)},
+            "legend": {str(i): text for i, text in enumerate(q.criteria)}}
+
+
+def request_media(request: tuple) -> list[Media]:
+    """A `(state, questions)` or `(state, questions, media)` request's media."""
+    return media_list(request[2] if len(request) > 2 else None)
+
+
+class SystemOneApi:
+    """`system_one` and `system_one_batch`, and the plain probabilities under them, over a model's
+    `run(requests)`: each `(state, questions, media)` request's probabilities (in its questions' order)
+    and the tokens it read."""
+
+    serves_typed = True  # an engine without the slots and state positions of slots.py says False
+
+    def ask(self, q: Mapping | Question) -> Question:
+        """A caller's question, refused with what to change, or when this engine cannot serve its type."""
+        check_question(q, getattr(self, "tokenizer", None))
+        q = as_question(q)
+        if typed(q) and not self.serves_typed:
+            raise ValueError(f"{type(self).__name__} serves noul, choice and score questions only")
+        return q
+
+    def named(self, questions: Mapping[str, Any]) -> dict[str, Question]:
+        """A request's named questions (`ask`), an error naming the question it is about."""
+        if not isinstance(questions, Mapping):
+            raise ValueError(f'questions are a dict of name to question, {{"refund": {{"type": "noul", "instructions": '
+                             f'"Is it a refund request?"}}}}, not {type(questions).__name__}')
+        out = {}
+        for name, q in questions.items():
+            try:
+                out[name] = self.ask(q)
+            except ValueError as e:
+                raise ValueError(f"question {name!r}: {e}") from None
+        return out
+
+    def answer(self, state: Any, q: Question, probs, score_readout: str = SCORE_READOUT,
+               abstain_below: float | None = None) -> dict:
+        """One question's answer as `system_one` gives it: a read's value parsed in its state, a typed answer's
+        probabilities through the served calibration's tables."""
+        context = state_text(state, getattr(self, "state_style", DEFAULT_STATE_STYLE)) if isinstance(q, Read) else ""
+        return answer(q, probs, score_readout, abstain_below, context, getattr(self, "calibration", None),
+                      getattr(getattr(self, "tokenizer", None), "d1_box_edges", "median"))
+
+    def run(self, requests: Sequence[tuple[Any, list[Question], Sequence[Media]]],
+            set_size: int | None = None) -> list[tuple[list[list[float]], int]]:
+        raise NotImplementedError
+
+    def probabilities(self, state: Any, questions: Sequence, media: Sequence | None = None) -> list[list[float]]:
+        """Each question's option probabilities; a multiple choice returns independent P(yes) per option."""
+        return self.run([(state, [self.ask(q) for q in questions], media_list(media))])[0][0]
+
+    def system_one(self, state: Any, questions: Mapping[str, Any], media: Sequence | None = None,
+                   score_readout: str = SCORE_READOUT, abstain_below: float | None = None,
+                   set_size: int | None = None) -> dict:
+        """Named questions over one state, and its media if any (pictures, audio clips, videos), in one forward
+        pass."""
+        return self.system_one_batch([(state, questions, media)], score_readout, abstain_below, set_size)[0]
+
+    def system_one_batch(self, requests: Sequence[tuple], score_readout: str = SCORE_READOUT,
+                         abstain_below: float | None = None, set_size: int | None = None) -> list[dict]:
+        """Many `(state, questions)` or `(state, questions, media)` requests in one call, batched as the engine
+        reads them (`run`); a request with several questions reads its state once a pass.
+        `score_readout` (`SCORE_READOUTS`) sets how a score question answers; `abstain_below` (off by default)
+        when a choice with an "I don't know" option picks it (`choice_index`); `set_size` how many questions are
+        read together after the state, for this call (the model's trained size by default)."""
+        if score_readout not in SCORE_READOUTS:
+            raise ValueError(f"score_readout {score_readout!r}: one of {SCORE_READOUTS}")
+        if abstain_below is not None and not 0 < abstain_below <= 1:
+            raise ValueError(f"abstain_below {abstain_below!r}: a share in (0, 1]")
+        named = [(r[0], self.named(r[1]), request_media(r)) for r in requests]
+        done = self.run([(state, list(qs.values()), media) for state, qs, media in named], set_size=set_size)
+        return [{"answers": {n: self.answer(state, q, p, score_readout, abstain_below) for (n, q), p in zip(qs.items(), probs)},
+                 "usage": {"input_tokens": read, "output_tokens": 0}}
+                for (state, qs, _), (probs, read) in zip(named, done)]

attention.py

@@ -0,0 +1,332 @@
+"""Which tokens see which on a backbone's full-attention layers, in training and serving alike.
+
+| mode | a state token sees | a question token sees |
+|---|---|---|
+| `causal` | the tokens before it | the tokens before it |
+| `state` | the whole state | the whole state, and the question before it |
+| `prompt` | the whole prompt: state, question, options | the whole prompt |
+| `set` | the state and every question | the state and its own question |
+
+`state` reads the state both ways and never shows it the question, so one state's keys still serve every question
+about it. `prompt` reads everything before the answer both ways, the state with its question in view: each question is
+then a prompt of its own, with no prefill shared between the questions of one state. `set` keeps `prompt`'s reading
+and packs a state's questions after it, `[state][q1]..[qN]`, each ending at its own answer slot: one pass, the state
+read once. On these layers a question never attends to another; it hears of them only through the state, which reads
+them all. A lone question reads exactly as under `prompt`.
+
+Only full-attention layers change; recurrent, convolution and sliding-window layers stay causal. Under `set` they
+run over the packed sequence, so a question also hears the ones before it through them; training packs the same way
+(questions of a state in random order), so the served function is the trained one. A
+checkpoint trains and serves with one mode, recorded in its config (`d1_attention`, with `d1_set_size`, the most
+questions `set` packs after one state).
+"""
+
+from __future__ import annotations
+
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+
+import torch
+import torch.nn.functional as F
+from transformers import AttentionInterface, AttentionMaskInterface
+from transformers.integrations.sdpa_attention import sdpa_attention_forward
+from transformers.masking_utils import sdpa_mask
+
+Tensor = torch.Tensor
+SET_SIZE = 4  # the most questions `set` packs after one state when a checkpoint records none (`d1_set_size`)
+TRAINED_SET_SIZE = 128  # the largest cap training takes (`--set-size`), so the most a checkpoint records
+CHUNK = 4096  # queries a `Sees` mask is built for at once
+
+
+def _fused(q: torch.Tensor) -> bool:
+    """Whether the flash kernels take these queries: a GPU in half precision, ROCm or CUDA from Ampere on."""
+    if not q.is_cuda or q.dtype not in (torch.float16, torch.bfloat16):
+        return False
+    return bool(torch.version.hip) or torch.cuda.get_device_capability(q.device)[0] >= 8
+
+
+def _windowed(q: Tensor, k: Tensor, v: Tensor, window: int, scale: float) -> Tensor:
+    """Causal attention within `window`, as local blocks: each block of `window`
+    queries sees its own block and the one before it."""
+    batch, heads, length, dim = q.shape
+    blocks = -(-length // window)
+
+    def split(t: Tensor) -> Tensor:  # (B, H, T, D) -> (B, blocks, H, window, D)
+        t = F.pad(t, (0, 0, 0, blocks * window - length))
+        return t.reshape(batch, t.shape[1], blocks, window, dim).transpose(1, 2)
+
+    def with_previous(t: Tensor) -> Tensor:
+        return torch.cat((F.pad(t, (0, 0, 0, 0, 0, 0, 1, 0))[:, :-1], t), dim=3)
+
+    qb, kb, vb = split(q), with_previous(split(k)), with_previous(split(v))
+    qpos = torch.arange(window, device=q.device)[:, None] + window
+    kpos = torch.arange(2 * window, device=q.device)[None, :]
+    block = torch.arange(blocks, device=q.device)[:, None, None]
+    allowed = (kpos <= qpos) & (kpos > qpos - window) & ((block > 0) | (kpos >= window))  # block 0 has none before it
+    mask = allowed[:, None].repeat(batch, 1, 1, 1)
+    out = F.scaled_dot_product_attention(qb.flatten(0, 1), kb.flatten(0, 1), vb.flatten(0, 1),
+                                         attn_mask=mask, scale=scale, enable_gqa=True)
+    out = out.reshape(batch, blocks, heads, window, dim).transpose(1, 2)
+    return out.reshape(batch, heads, blocks * window, dim)[:, :, :length]
+
+
+def causal_attention(q: Tensor, k: Tensor, v: Tensor, window: int | None, scale: float) -> Tensor:
+    """Right-padded chains: causal, within `window` if set: on the flash kernels their own window at any length (no
+    branch on it, so torch.compile keeps one graph across the window), else blocks."""
+    if window is None or not _fused(q) and q.shape[2] <= window:
+        return F.scaled_dot_product_attention(q, k, v, is_causal=True, scale=scale, enable_gqa=True)
+    if not _fused(q):
+        return _windowed(q, k, v, window, scale)
+    out = torch.ops.aten._flash_attention_forward(*(t.transpose(1, 2) for t in (q, k, v)), None, None, q.shape[2],
+                                                  k.shape[2], 0.0, True, False, scale=scale,
+                                                  window_size_left=window - 1, window_size_right=0)[0]
+    return out.transpose(1, 2)
+
+
+def both_ways(mode: str, tokenizer, ids: list[int], suffix: str) -> int | None:
+    """How many of a prompt's first tokens read each other both ways: none (`causal`), its state, or all of it
+    (`prompt`, and `set` for a lone question)."""
+    if mode == "causal":
+        return None
+    return len(ids) if mode in ("prompt", "set") else state_length(tokenizer, ids, suffix)
+
+
+def state_length(tokenizer, ids: list[int], suffix: str) -> int:
+    """A prompt's tokens before its question: the prompt is the state's text then `suffix`, which opens on a token
+    boundary (the state ends on a blank line), so its own tokens are the prompt's last ones."""
+    return len(ids) - len(tokenizer.encode(suffix, add_special_tokens=False))
+
+
+def pack_set(tokenizer, prefix: str, suffixes: list[str],
+             header: str = "", slot_offsets: list[list[int]] | None = None,
+             first_ids: list[int] | None = None,
+             ) -> tuple[list[int], int, list[int], list[list[int]]]:
+    """`set`'s sequence for one state, in any prompt layout: `prefix` (the state, ending with the question
+    `header`) and the first question tokenized as its lone prompt is, so one question is exactly its `prompt`
+    row; each further question its own group, `header` then its suffix. Returns the ids, the state's length, each
+    token's question (1..N, -1 on the state) and each question's answer positions. `slot_offsets` names slots
+    relative to its suffix's last token (0); without it, each question has that one slot."""
+    if not suffixes:
+        raise ValueError("a question set cannot be empty")
+    offsets = [[0] for _ in suffixes] if slot_offsets is None else slot_offsets
+    if len(offsets) != len(suffixes):
+        raise ValueError("one slot layout is required per question")
+
+    def positions(k: int, length: int, end: int) -> list[int]:
+        own = offsets[k]
+        if (not own or own != sorted(set(own)) or own[-1] != 0
+                or any(isinstance(o, bool) or not isinstance(o, int) or not -length < o <= 0 for o in own)):
+            raise ValueError("slots must be increasing positions inside their question, ending at its last token")
+        return [end + o for o in own]
+
+    ids = tokenizer.encode(prefix + suffixes[0], add_special_tokens=False) if first_ids is None else list(first_ids)
+    state = state_length(tokenizer, ids, suffixes[0])
+    question = [-1] * state + [1] * (len(ids) - state)
+    ends = [positions(0, len(ids) - state, len(ids) - 1)]
+    for k, suffix in enumerate(suffixes[1:], 2):
+        own = tokenizer.encode(header + suffix, add_special_tokens=False)
+        ids, question = ids + own, question + [k] * len(own)
+        ends.append(positions(k - 1, len(own), len(ids) - 1))
+    return ids, state, question, ends
+
+
+def text_backbone(model):
+    """The module that returns hidden states, under any wrapper (DDP, a training wrapper)."""
+    m = getattr(model, "module", model)
+    if hasattr(m, "get_base_model"):
+        m = m.get_base_model()
+    return getattr(m, "model", m)
+
+
+def language_model(model):
+    """The language model under any wrapper and a vision-language model."""
+    m = text_backbone(model)
+    return getattr(m, "language_model", m)
+
+
+def _runs(*parts: tuple[torch.Tensor, torch.Tensor]) -> tuple[torch.Tensor, torch.Tensor, int]:
+    """Segments, segment i the runs `(start[i], length[i])` of every part back to back: their flat indices, their
+    cumulative lengths (int32, as the varlen kernels take them) and the longest."""
+    lengths = sum(n for _, n in parts)
+    cu = F.pad(lengths.cumsum(0), (1, 0))
+    seg = torch.repeat_interleave(torch.arange(len(lengths), device=lengths.device), lengths)
+    off = torch.arange(len(seg), device=seg.device) - cu[seg]
+    idx = torch.zeros_like(off)
+    for start, n in parts:
+        idx = torch.where((off >= 0) & (off < n[seg]), start[seg] + off, idx)
+        off = off - n[seg]
+    return idx, cu.int(), int(lengths.max()) if len(lengths) else 0
+
+
+def _varlen(q: Tensor, k: Tensor, v: Tensor, cu_q: Tensor, cu_k: Tensor, longest: Tensor, causal: bool,
+            scale: float) -> tuple[Tensor, Tensor, Tensor, Tensor]:
+    """Flash attention over packed segments: the output [N, H, D], then what its backward reads (the log-sum-exp
+    [H, N], the RNG state and its companion). Causal aligns each segment's last query to its last key. `longest`: the
+    longest query and key segment, on the CPU. Compiled, the opaque op `d1::varlen`, so the lengths neither break the
+    graph nor recompile it; its backward is flash's own (`_varlen_backward`), as eager's is."""
+    max_q, max_k = longest.tolist()
+    out, lse, rng, unused, _ = torch.ops.aten._flash_attention_forward(q, k, v, cu_q, cu_k, max_q, max_k, 0.0, causal,
+                                                                       False, scale=scale)
+    return out, lse, rng, unused
+
+
+def _varlen_backward(ctx, grad, *_):
+    q, k, v, out, lse, rng, unused, cu_q, cu_k = ctx.saved_tensors
+    max_q, max_k = ctx.longest
+    dq, dk, dv = torch.ops.aten._flash_attention_backward(grad.contiguous(), q, k, v, out, lse, cu_q, cu_k, max_q,
+                                                          max_k, 0.0, ctx.causal, rng, unused, scale=ctx.scale)
+    return dq, dk, dv, None, None, None, None, None
+
+
+def _varlen_context(ctx, inputs, output):
+    q, k, v, cu_q, cu_k, longest, causal, scale = inputs
+    ctx.save_for_backward(q, k, v, *output, cu_q, cu_k)
+    ctx.longest, ctx.causal, ctx.scale = longest.tolist(), causal, scale
+
+
+if not hasattr(torch.ops.d1, "varlen"):  # once a process: two repos' copies of this module share the op
+    _op = torch.library.custom_op("d1::varlen", _varlen, mutates_args=())
+    _op.register_fake(lambda q, k, v, cu_q, cu_k, longest, causal, scale: (
+        torch.empty_like(q), q.new_empty(q.shape[1], q.shape[0], dtype=torch.float32),
+        q.new_empty((), dtype=torch.int64), q.new_empty((), dtype=torch.int64)))
+    _op.register_autograd(_varlen_backward, setup_context=_varlen_context)
+
+
+@dataclass
+class Sees:
+    """The full-attention mask of `state_attention`, never built whole (a [B, 1, T, T] tensor takes 33 GB on a
+    180k-token row): `attend` takes the attention under it, `rows` a block of queries' part of it."""
+
+    state: torch.Tensor  # [B] each row's state length
+    question: torch.Tensor | None  # [B, T] `set`: each token's question, 1..N, and -1 elsewhere
+    real: torch.Tensor  # [B, T] the tokens that are not padding
+    _calls: dict = field(default_factory=dict, init=False, repr=False, compare=False)  # once a device, every layer's
+
+    def __post_init__(self):
+        """On a GPU, the plan on the mask's device, made before any layer: a compiled forward reads it with no graph
+        break. Off a GPU, `attend` masks and never plans."""
+        if self.real.is_cuda:
+            self._calls[self.real.device] = self._plan(self.real.device)
+
+    def rows(self, start: int, width: int, device) -> torch.Tensor:
+        """Which keys queries `start` to `start + width` see, [B, 1, width, T]."""
+        q = torch.arange(start, start + width, device=device)[None, :, None]
+        kv = torch.arange(self.real.shape[1], device=device)[None, None, :]
+        s = self.state.to(device)[:, None, None]
+        if self.question is None:  # causal, and the state both ways
+            m = (kv <= q) | ((q < s) & (kv < s))
+        else:  # the state sees every token and every token the state; a question token its own question
+            g = self.question.to(device)
+            m = (q < s) | (kv < s) | (g[:, start:start + width, None] == g[:, None, :])
+        return (m & self.real.to(device)[:, None, :])[:, None]
+
+    def attend(self, q: Tensor, k: Tensor, v: Tensor, scale: float, window: int | None = None) -> Tensor:
+        """Attention under this mask; q [B, H, T, D], k and v [B, Hkv, T, D]. On the flash kernels, no mask: the
+        states over their keys in one varlen call, the questions over their state and themselves in another. Off
+        them (CPU, fp32), `CHUNK` masked queries at a time. A sliding layer's `window`: causal within it, as every
+        mode keeps it (right-padded rows: a pad never reaches a real token)."""
+        if window is not None:
+            return causal_attention(q, k, v, window, scale)
+        b, h, t, d = q.shape
+        if not _fused(q):
+            starts = range(0, t, CHUNK) if t > CHUNK else (0,)  # a range over t would compile a graph per length
+            return torch.cat([F.scaled_dot_product_attention(q[:, :, s:s + CHUNK], k, v, scale=scale, enable_gqa=True,
+                                                             attn_mask=self.rows(s, min(CHUNK, t - s), q.device))
+                              for s in starts], dim=2)
+        if q.device not in self._calls:  # a model over GPUs: once on each
+            self._calls[q.device] = self._plan(q.device)
+        if self._calls[q.device] is None:  # every row reads all of itself, unpadded: plain attention
+            return F.scaled_dot_product_attention(q, k, v, scale=scale, enable_gqa=True)
+        flat = [x.transpose(1, 2).reshape(b * t, x.shape[1], d) for x in (q, k, v)]
+        out = flat[0].new_zeros(b * t, h, d)
+        varlen = torch.ops.d1.varlen if torch.compiler.is_compiling() else _varlen
+        for qi, cu_q, ki, cu_k, longest, causal in self._calls[q.device]:
+            attended = varlen(flat[0][qi], flat[1][ki], flat[2][ki], cu_q, cu_k, longest, causal, scale)[0]
+            out = out.index_copy(0, qi, attended)
+        return out.view(b, t, h, d).transpose(1, 2)
+
+    def _plan(self, device) -> list | None:
+        """The varlen calls, as (queries, their offsets, keys, theirs, the longest of each, causal): each state over
+        its row (`set`) or itself (`state`, `prompt`); each question over its state then itself, causally under
+        `state`, where the question part is one run a row. Under `set` a row of one question is one segment: all of
+        it sees all of it. None when every row is one such segment with no padding."""
+        real = self.real.to(device)
+        b, t = real.shape
+        n = real.sum(1)
+        s = torch.minimum(self.state.to(device), n)
+        base = torch.arange(b, device=device) * t
+        tail = (real & (torch.arange(t, device=device) >= s[:, None])).flatten().nonzero().squeeze(1)
+        g = (torch.zeros(b, t, dtype=torch.long, device=device) if self.question is None
+             else self.question.to(device)).flatten()[tail]
+        new = F.pad((g[1:] != g[:-1]) | (tail[1:] != tail[:-1] + 1), (1, 0), value=True)[:len(tail)]
+        first = new.nonzero().squeeze(1)
+        start, length = tail[first], torch.diff(F.pad(first, (0, 1), value=len(tail)))
+        if self.question is not None:
+            lone = torch.bincount(start // t, minlength=b) == 1
+            s = torch.where(lone, n, s)
+            start, length = start[~lone[start // t]], length[~lone[start // t]]
+        if not len(start) and bool((s == t).all()):
+            return None
+        calls = [(_runs((base, s)), _runs((base, n if self.question is not None else s)), False)]
+        if len(start):
+            row = start // t
+            calls.append((_runs((start, length)), _runs((base[row], s[row]), (start, length)), self.question is None))
+        return [(qi, cu_q, ki, cu_k, torch.tensor([max_q, max_k]), causal)
+                for (qi, cu_q, max_q), (ki, cu_k, max_k), causal in calls]
+
+
+def d1_attention(module, q, k, v, attention_mask, scaling=None, sliding_window=None, **kwargs):
+    """transformers' attention function for `attn_implementation="d1"` (registered on import): a `Sees` mask attends
+    through `Sees.attend` (a sliding layer within its window), anything else is SDPA's."""
+    if isinstance(attention_mask, Sees):
+        return attention_mask.attend(q, k, v, scaling, sliding_window).transpose(1, 2), None
+    return sdpa_attention_forward(module, q, k, v, attention_mask, scaling=scaling, sliding_window=sliding_window,
+                                  **kwargs)
+
+
+AttentionInterface.register("d1", d1_attention)
+# the masks transformers builds itself under "d1" (no `state_attention`; a tower, an audio encoder) are SDPA's: an
+# implementation without one gets None for every mask, which drops a sliding window and the padding
+AttentionMaskInterface.register("d1", sdpa_mask)
+ATTENTION_KINDS = ("full_attention", "sliding_attention")  # the layer kinds a `Sees` masks; the others take padding
+
+
+def mask_mapping(layer_types, sees: Sees, pad: Tensor) -> dict:
+    """The per-kind masks of a language model under `Sees`, keyed as its `layer_types`: the `Sees` on attention
+    layers, the 2D padding (None without any) on convolution and linear-attention layers."""
+    padding = None if pad.all() else pad
+    return {kind: sees if kind in ATTENTION_KINDS else padding for kind in set(layer_types)}
+
+
+@contextmanager
+def state_attention(model, state_len: torch.Tensor | None, question: torch.Tensor | None = None):
+    """Within the block, the model's language model reads each row's first `state_len` tokens both ways; with
+    `question` (`set`: each token's question, 1..N, and -1 elsewhere), as `Sees` says. The mask is swapped in at the
+    language model, after the multimodal wrapper has placed its media: as the mask of a language model that reads it
+    (`reads_sees`), else as transformers' per-kind masks (`mask_mapping`), which `d1_attention` reads."""
+    if state_len is None:
+        yield
+        return
+
+    def hook(module, args, kwargs):
+        pad, x = kwargs.get("attention_mask"), kwargs.get("input_ids")
+        if isinstance(pad, dict):
+            raise ValueError("state attention replaces a 2D padding mask, not a prepared mapping")
+        if pad is None:
+            x = kwargs["inputs_embeds"] if x is None else x
+            pad = torch.ones(x.shape[:2], dtype=torch.long, device=x.device)
+        sees = Sees(state_len.to(pad.device), question, pad.bool())
+        if getattr(module, "reads_sees", False):
+            kwargs["attention_mask"] = sees
+        elif module.config._attn_implementation == "d1":
+            kinds = getattr(module.config, "layer_types", None) or ("full_attention",)  # a config naming none: all full
+            kwargs["attention_mask"] = mask_mapping(kinds, sees, pad)
+        else:
+            raise ValueError(f"{type(module).__name__} reads `state_attention` with attn_implementation='d1'")
+        return args, kwargs
+
+    handle = language_model(model).register_forward_pre_hook(hook, with_kwargs=True)
+    try:
+        yield
+    finally:
+        handle.remove()

box.py

@@ -0,0 +1,150 @@
+"""Boxes: a presence and four edges, each a distribution over the same grid of fractions of the picture (0 the top
+or left side, 1 the bottom or right): the thousandths 0-999 as the tokenizer writes them on a checkpoint that reads
+numbers (`edge_grid`), else 128 codes.
+
+- An edge is its distribution's median: an unsure edge's mean drifts to the picture's middle. A checkpoint that
+  records `d1_box_edges: interp` reads it inside the median's bin, where the distribution's CDF
+  crosses one half (`EDGE_READS`): the bin's value alone is off by a quarter of a bin on average. One without the
+  record reads the bin's value.
+- A box's confidence: the share of boxes drawn from the edges that overlap it (IoU >= 0.5).
+- `boxes`: every instance, blocks of a presence and four edges; block k is present when there are at least k (its
+  presence is P(count >= k)), the boxes answered in reading order (left edge, then top). A question reads its `max`
+  first blocks, else `INSTANCES`: the count is then at least that when the last is present. Prompt 1 trained 16
+  blocks under a prompt saying "16 at most"; prompt 2 states no count and trains 1 to `MAX_INSTANCES` blocks drawn
+  per row (`prompt.PROMPTS`).
+- Probabilities go through the served calibration's tables (calibration.py)."""
+
+from __future__ import annotations
+
+import math
+from bisect import bisect_left
+from collections.abc import Callable
+from functools import lru_cache
+from itertools import accumulate
+
+from .grid import Grid
+
+EDGES = ("top", "bottom", "left", "right")
+GRID = Grid(0, 1)  # an edge's values as codes, 128 of them
+THOUSANDTHS = Grid(0, 0.999, 1000)  # an edge's values as numbers: 0-999 thousandths ("1000" takes two tokens)
+DRAWS = 512
+INSTANCES = 16  # a `boxes` question's blocks without `max`: 99.97% of COCO val's (picture, class) pairs hold 16 at most
+MAX_INSTANCES = 64  # the most a `max` asks, and a prompt-2 training row draws (dense scenes: 17-60 instances)
+EDGE_READS = ("median", "interp")  # how an edge is read off its distribution; a checkpoint records it (`d1_box_edges`)
+
+
+def edge_grid(tokenizer) -> tuple[Grid, int]:
+    """An edge's grid and the factor its values are written at: the thousandths where the checkpoint reads numbers
+    (`d1_numbers`) and the tokenizer writes each in one token, else the codes."""
+    from .prompt import number_ids
+
+    numbers = getattr(tokenizer, "d1_numbers", False) and number_ids(tokenizer, THOUSANDTHS, 1000) is not None
+    return (THOUSANDTHS, 1000) if numbers else (GRID, 1)
+
+
+def read_as(distribution) -> Grid:
+    """The grid an edge's distribution is over, by its length: the codes' 128 values or the thousandths."""
+    return THOUSANDTHS if len(distribution) == THOUSANDTHS.size else GRID
+
+
+def ordered(a: float, b: float, grid: Grid = GRID) -> tuple[float, float]:
+    """Two edges of one axis, the second below or right of the first: crossed ones become one bin about their middle."""
+    if a < b:
+        return a, b
+    half = grid.step / 2
+    middle = min(1 - half, max(half, (a + b) / 2))
+    return middle - half, middle + half
+
+
+def rectangle(edges: list[float], grid: Grid = GRID) -> dict[str, float]:
+    top, bottom = ordered(*edges[:2], grid)
+    left, right = ordered(*edges[2:], grid)
+    return {"top": top, "bottom": bottom, "left": left, "right": right}
+
+
+def area(box: dict) -> float:
+    return (box["bottom"] - box["top"]) * (box["right"] - box["left"])
+
+
+def iou(a: dict, b: dict) -> float:
+    overlap = (max(0.0, min(a["bottom"], b["bottom"]) - max(a["top"], b["top"]))
+               * max(0.0, min(a["right"], b["right"]) - max(a["left"], b["left"])))
+    return overlap / (area(a) + area(b) - overlap)
+
+
+@lru_cache(maxsize=1)
+def samples() -> tuple[tuple[float, ...], ...]:
+    """`DRAWS` points of a Halton sequence in four dimensions: the same draws every call."""
+    def radical_inverse(i: int, base: int) -> float:
+        value, scale = 0.0, 1.0
+        while i:
+            i, digit = divmod(i, base)
+            scale /= base
+            value += digit * scale
+        return value
+
+    return tuple(tuple(radical_inverse(i, base) for base in (2, 3, 5, 7)) for i in range(1, DRAWS + 1))
+
+
+def confidence(edges: list[list[float]], box: dict) -> float:
+    """The share of boxes drawn from the edge distributions (through each one's CDF) with IoU >= 0.5 against `box`."""
+    cdfs = [list(accumulate(p / total for p in row)) for row, total in zip(edges, map(math.fsum, edges))]
+    grid = read_as(edges[0])
+    values = grid.values
+    hits = sum(iou(rectangle([values[min(grid.size - 1, bisect_left(cdf, u))] for cdf, u in zip(cdfs, point)], grid),
+                   box) >= 0.5 for point in samples())
+    return hits / DRAWS
+
+
+def answer(probabilities: list[list[float]], through: Callable[[str, float], float] = lambda name, value: value,
+           edges: str = "median") -> dict:
+    """The API's box: `probabilities` are presence ([yes, no]) then the four edges' distributions; `through(name, raw)`
+    calibrates `found` ("found:box") and the confidence ("box"); `edges`, how each edge is read (`EDGE_READS`)."""
+    found = through("found:box", probabilities[0][0])
+    if found < 0.5:
+        return {"type": "box", "box": None, "found": found, "confidence": 0.0}
+    box = medians(probabilities[1:], edges)
+    return {"type": "box", "box": box, "found": found, "confidence": through("box", confidence(probabilities[1:], box))}
+
+
+def crossing(probabilities, grid: Grid = GRID) -> float:
+    """Where the distribution's CDF crosses one half, the mass spread evenly over each bin (value +- step / 2)."""
+    total, mass = math.fsum(probabilities), 0.0
+    for value, p in zip(grid.values, probabilities):
+        share = p / total
+        if share > 0 and mass + share >= 0.5:
+            return min(grid.high, max(grid.low, value - grid.step / 2 + (0.5 - mass) / share * grid.step))
+        mass += share
+    return grid.high
+
+
+def medians(edges: list[list[float]], read: str = "median") -> dict[str, float]:
+    """The box at the four edge distributions' medians: each its bin's value, or where its CDF crosses one half
+    inside that bin (`read`)."""
+    if read not in EDGE_READS:
+        raise ValueError(f"box edges read as one of {EDGE_READS}, not {read!r}")
+    grid = read_as(edges[0])
+    at = (lambda row: crossing(row, grid)) if read == "interp" else (lambda row: grid.read(row)["median"])
+    return rectangle([at(row) for row in edges], grid)
+
+
+def answer_many(probabilities: list[list[float]], through: Callable[[str, float], float] = lambda name, value: value,
+                edges: str = "median") -> dict:
+    """The API's `boxes`: `probabilities` are blocks (`INSTANCES`, or a question's `max`) of presence ([yes, no]) and
+    four edges. `presence` is P(count >= k) for each k, calibrated ("found:boxes") and kept decreasing; the boxes are
+    the blocks at >= 0.5, each with that presence as `found` and its confidence ("boxes"), in reading order; `count`
+    is how many, `at_least` when every block read is present (the true count may be higher)."""
+    presence, below = [], 1.0
+    for k in range(0, len(probabilities), 5):
+        below = min(below, through("found:boxes", probabilities[k][0]))
+        presence.append(below)
+    out = []
+    for k, found in enumerate(presence):
+        if found < 0.5:
+            break
+        block = probabilities[5 * k + 1:5 * k + 5]
+        box = medians(block, edges)
+        out.append({"box": box, "found": found, "confidence": through("boxes", confidence(block, box))})
+    out.sort(key=lambda b: (b["box"]["left"], b["box"]["top"]))
+    return {"type": "boxes", "boxes": out, "count": len(out), "at_least": bool(out) and len(out) == len(presence),
+            "presence": presence}

calibration.json

@@ -0,0 +1,380 @@
+{
+ "default": 1.337398453966105,
+ "temperatures": {},
+ "tables": {
+  "found:text": [
+   [
+    0.005682162615729877,
+    0.0
+   ],
+   [
+    0.020574209407310595,
+    0.011363636363636364
+   ],
+   [
+    0.05274184431288053,
+    0.022727272727272728
+   ],
+   [
+    0.12593568082560191,
+    0.11363636363636363
+   ],
+   [
+    0.2609023091129281,
+    0.25
+   ],
+   [
+    0.45711776140061294,
+    0.42045454545454547
+   ],
+   [
+    0.6460263153368776,
+    0.5568181818181818
+   ],
+   [
+    0.7728411636569283,
+    0.7727272727272727
+   ],
+   [
+    0.852111120115627,
+    0.8522727272727273
+   ],
+   [
+    0.9055508368394591,
+    0.9204545454545454
+   ],
+   [
+    0.9416752389886163,
+    0.9318181818181818
+   ],
+   [
+    0.9675597308034246,
+    0.9772727272727273
+   ],
+   [
+    0.9927885918454691,
+    0.9924242424242424
+   ],
+   [
+    0.9997205003886156,
+    1.0
+   ]
+  ],
+  "found:number": [
+   [
+    0.01582507598356401,
+    0.0
+   ],
+   [
+    0.1465578345315797,
+    0.2857142857142857
+   ],
+   [
+    0.29448616611106054,
+    0.5
+   ],
+   [
+    0.6300800165959767,
+    0.7142857142857143
+   ],
+   [
+    0.9313231366021293,
+    0.9285714285714286
+   ],
+   [
+    0.9951998972501911,
+    1.0
+   ]
+  ],
+  "found:date": [
+   [
+    0.008417200646363199,
+    0.0
+   ],
+   [
+    0.5494083234419426,
+    0.5
+   ],
+   [
+    0.9654906044403712,
+    0.9166666666666666
+   ],
+   [
+    0.9966836659113566,
+    1.0
+   ]
+  ],
+  "found:box": [
+   [
+    0.0010319100776659475,
+    0.0027472527472527475
+   ],
+   [
+    0.0162678323005357,
+    0.11235955056179775
+   ],
+   [
+    0.1641463061831407,
+    0.4157303370786517
+   ],
+   [
+    0.6041711138630492,
+    0.8541666666666666
+   ],
+   [
+    0.8735690557675277,
+    0.978021978021978
+   ],
+   [
+    0.9579824398960495,
+    0.9887640449438202
+   ],
+   [
+    0.9952215192477699,
+    1.0
+   ]
+  ],
+  "box": [
+   [
+    0.08177083333333333,
+    0.18333333333333332
+   ],
+   [
+    0.35257161458333336,
+    0.4
+   ],
+   [
+    0.5962035123966942,
+    0.6694214876033058
+   ],
+   [
+    0.7902631915983607,
+    0.7786885245901639
+   ],
+   [
+    0.8755763319672131,
+    0.8688524590163934
+   ],
+   [
+    0.9086263020833333,
+    0.9
+   ],
+   [
+    0.9435433650362319,
+    0.9492753623188406
+   ],
+   [
+    0.9652183219178082,
+    0.9726027397260274
+   ],
+   [
+    0.9831365659916201,
+    0.9888268156424581
+   ],
+   [
+    0.9948308270676691,
+    1.0
+   ]
+  ],
+  "estimate": [
+   [
+    0.030000000000000002,
+    0.22420634920634921
+   ],
+   [
+    0.09,
+    0.28869047619047616
+   ],
+   [
+    0.15000000000000002,
+    0.3551587301587302
+   ],
+   [
+    0.21000000000000002,
+    0.4007936507936508
+   ],
+   [
+    0.27,
+    0.4603174603174603
+   ],
+   [
+    0.32999999999999996,
+    0.5238095238095238
+   ],
+   [
+    0.39,
+    0.5922619047619048
+   ],
+   [
+    0.44999999999999996,
+    0.6448412698412699
+   ],
+   [
+    0.51,
+    0.7033730158730159
+   ],
+   [
+    0.57,
+    0.7579365079365079
+   ],
+   [
+    0.63,
+    0.7986111111111112
+   ],
+   [
+    0.69,
+    0.8452380952380952
+   ],
+   [
+    0.75,
+    0.8869047619047619
+   ],
+   [
+    0.81,
+    0.9246031746031746
+   ],
+   [
+    0.87,
+    0.9523809523809523
+   ],
+   [
+    0.93,
+    0.9751984126984127
+   ],
+   [
+    0.9799999999999999,
+    0.9940476190476191
+   ]
+  ],
+  "multiple": [
+   [
+    0.0016627926872626571,
+    0.0
+   ],
+   [
+    0.008286236297705318,
+    0.0024813895781637717
+   ],
+   [
+    0.013284336191710702,
+    0.0049504950495049506
+   ],
+   [
+    0.017877624036637375,
+    0.015
+   ],
+   [
+    0.023793328700436385,
+    0.018518518518518517
+   ],
+   [
+    0.03199311782995977,
+    0.03398058252427184
+   ],
+   [
+    0.04889508202323689,
+    0.05459057071960298
+   ],
+   [
+    0.07631002315865278,
+    0.07407407407407407
+   ],
+   [
+    0.10610537429711313,
+    0.1262135922330097
+   ],
+   [
+    0.1516745984883731,
+    0.15865384615384615
+   ],
+   [
+    0.2304679394076778,
+    0.21393034825870647
+   ],
+   [
+    0.374861408323991,
+    0.3417085427135678
+   ],
+   [
+    0.6689872755703948,
+    0.7
+   ],
+   [
+    0.976994335126831,
+    0.9690721649484536
+   ]
+  ],
+  "found:boxes": [
+   [
+    0.0022886859880368398,
+    0.0
+   ],
+   [
+    0.11311487904813239,
+    0.0891089108910891
+   ],
+   [
+    0.39857997008840157,
+    0.3786407766990291
+   ],
+   [
+    0.8182158553298949,
+    0.8514851485148515
+   ],
+   [
+    0.980104905128495,
+    0.9702970297029703
+   ],
+   [
+    0.9981022046260327,
+    1.0
+   ]
+  ],
+  "boxes": [
+   [
+    0.00048828125,
+    0.0
+   ],
+   [
+    0.008246527777777778,
+    0.05555555555555555
+   ],
+   [
+    0.0777587890625,
+    0.3125
+   ],
+   [
+    0.20135498046875,
+    0.46875
+   ],
+   [
+    0.3692626953125,
+    0.5625
+   ],
+   [
+    0.50830078125,
+    0.625
+   ],
+   [
+    0.5980224609375,
+    0.6875
+   ],
+   [
+    0.6732177734375,
+    0.8125
+   ],
+   [
+    0.8321126302083334,
+    0.9375
+   ],
+   [
+    0.9644757699275363,
+    0.9710144927536232
+   ],
+   [
+    0.9945746527777778,
+    1.0
+   ]
+  ]
+ }
+}

calibration.py

@@ -0,0 +1,77 @@
+"""Temperature scaling of option scores, and tables for the typed answers' probabilities.
+
+- `calibration.json` holds a global temperature and optional per-type ones; scaling keeps an argmax but can move a
+  score's median or expected level.
+- A typed probability goes through its table, a monotone map fitted on held-out rows: `found:text`,
+  `found:number`, `found:date`, `found:box`, `box` (a box's confidence), `found:boxes` (P(count >= k)), `boxes`
+  (each listed box's confidence), `multiple` (each option's P(yes)), `estimate` (a quantile level, read backwards by
+  `Calibration.level`).
+"""
+
+from __future__ import annotations
+
+import json
+from bisect import bisect_left
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from .prompt import Noul, Question, Score, cardinality
+
+
+def bucket(kind: str, k: int) -> str:
+    if kind == "noul":
+        return "noul"
+    if kind == "score":
+        return f"score:{k}"
+    if k <= 3:
+        return "choice:small"
+    if k <= 8:
+        return "choice:mid"
+    if k <= 26:
+        return "choice:large"
+    return "choice:inventory"
+
+
+def question_bucket(q: Question) -> str:
+    kind = "noul" if isinstance(q, Noul) else "score" if isinstance(q, Score) else "choice"
+    return bucket(kind, cardinality(q))
+
+
+def through(value: float, knots: list[list[float]] | None) -> float:
+    """`value` through increasing (raw, calibrated) knots, linear between them, flat past the ends."""
+    if not knots:
+        return value
+    index = bisect_left([raw for raw, _ in knots], value)
+    if index == 0:
+        return knots[0][1]
+    if index == len(knots):
+        return knots[-1][1]
+    (a, pa), (b, pb) = knots[index - 1], knots[index]
+    return pa + (value - a) * (pb - pa) / (b - a)
+
+
+@dataclass
+class Calibration:
+    temperatures: dict[str, float] = field(default_factory=dict)
+    default: float = 1.0
+    counts: dict[str, int] = field(default_factory=dict)
+    tables: dict[str, list[list[float]]] = field(default_factory=dict)
+
+    def temperature(self, q: Question) -> float:
+        return self.temperatures.get(question_bucket(q), self.default)
+
+    def apply(self, q: Question, scores: list[float]) -> list[float]:
+        t = self.temperature(q)
+        return scores if t == 1.0 else [s / t for s in scores]
+
+    def through(self, name: str, value: float) -> float:
+        return through(value, self.tables.get(name))
+
+    def level(self, name: str, share: float) -> float:
+        """The level whose observed share is `share`: the table read backwards (both columns increase)."""
+        return through(share, [[y, x] for x, y in self.tables.get(name) or ()])
+
+    @classmethod
+    def load(cls, path: Path | str) -> "Calibration":
+        d = json.loads(Path(path).read_text())
+        return cls(d["temperatures"], d.get("default", 1.0), d.get("counts", {}), d.get("tables", {}))

chat_template.jinja

@@ -0,0 +1,286 @@
+{{- bos_token -}}
+{%- set preserve_thinking = preserve_thinking | default(false) -%}
+{%- set enable_thinking = enable_thinking | default(true) -%}
+
+{%- macro render_value(v) -%}
+    {%- if v is string -%}
+        {{- v -}}
+    {%- elif v is boolean -%}
+        {{- "true" if v else "false" -}}
+    {%- elif v is none -%}
+        {{- "null" -}}
+    {%- elif v is mapping or (v is sequence and v is not string) -%}
+        {{- v | tojson -}}
+    {%- else -%}
+        {{- v -}}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- macro parse_content(content) -%}
+    {%- if content is string -%}
+        {{- content -}}
+    {%- elif content is mapping and content.get("type") == "image" -%}
+        {{- "<|vision_start|><|image|><|vision_end|>" -}}
+    {%- elif content is mapping and content.get("type") == "video" -%}
+        {{- "<|vision_start|><|video|><|vision_end|>" -}}
+    {%- elif content is mapping and content.get("type") == "text" -%}
+        {{- (content.get("text") or "") | string -}}
+    {%- elif content is mapping -%}
+        {{- content | tojson -}}
+    {%- elif content is iterable -%}
+        {%- set _ns = namespace(result="") -%}
+        {%- for item in content -%}
+            {%- if item is string -%}
+                {%- set _ns.result = _ns.result + item -%}
+            {%- elif item is mapping and item.get("type") == "image" -%}
+                {%- set _ns.result = _ns.result + "<|vision_start|><|image|><|vision_end|>" -%}
+            {%- elif item is mapping and item.get("type") == "video" -%}
+                {%- set _ns.result = _ns.result + "<|vision_start|><|video|><|vision_end|>" -%}
+            {%- elif item is mapping and item.get("type") == "text" -%}
+                {%- set _ns.result = _ns.result + ((item.get("text") or "") | string) -%}
+            {%- else -%}
+                {%- set _ns.result = _ns.result + (item | tojson) -%}
+            {%- endif -%}
+        {%- endfor -%}
+        {{- _ns.result -}}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- macro render_tool_calls(tool_calls) -%}
+    {%- for tool_call in tool_calls -%}
+        {%- if not loop.first -%}{{- "\n\n" -}}{%- endif -%}
+        {%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
+        {%- set func_args = func.get("arguments") -%}
+        {%- if func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
+            {{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
+        {%- endif -%}
+        {{- "<tool_call>\n<function=" + func["name"] + ">\n" -}}
+        {%- if func_args is mapping -%}
+            {%- for arg_name, arg_value in func_args.items() -%}
+                {{- "<parameter=" + arg_name + ">\n" + render_value(arg_value) + "\n</parameter>\n" -}}
+            {%- endfor -%}
+        {%- endif -%}
+        {{- "</function>\n</tool_call>" -}}
+    {%- endfor -%}
+{%- endmacro -%}
+
+{%- macro tool_to_json(tool) -%}
+    {%- set fn = tool.function if tool.function is defined else tool -%}
+    {%- set ns_t = namespace(first=true) -%}
+    {{- "{" -}}
+    {%- for k, v in fn.items() -%}
+        {%- if k != "defer_loading" -%}
+            {%- if not ns_t.first -%}{{- ", " -}}{%- endif -%}
+            {%- set ns_t.first = false -%}
+            {{- k | tojson -}}{{- ": " -}}{{- v | tojson -}}
+        {%- endif -%}
+    {%- endfor -%}
+    {{- "}" -}}
+{%- endmacro -%}
+
+{#- a tool_reference content part (vLLM, SGLang) names a tool from `tools`,
+    usually a deferred one; it renders as that tool's definition -#}
+{%- macro is_tool_reference(item) -%}
+    {{- "1" if item is mapping and item.get("type") == "tool_reference" else "" -}}
+{%- endmacro -%}
+
+{%- macro referenced_tool_to_json(name) -%}
+    {%- set _ns = namespace(found=none) -%}
+    {%- for tool in tools or [] -%}
+        {%- set fn = tool.function if tool.function is defined else tool -%}
+        {%- if fn.name == name -%}
+            {%- set _ns.found = fn -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- if _ns.found is none -%}
+        {{- raise_exception("tool_reference names a tool that is not in tools: " + name | string) -}}
+    {%- endif -%}
+    {{- tool_to_json(_ns.found) -}}
+{%- endmacro -%}
+
+{%- macro render_tool_response(content) -%}
+    {#- content that already carries the tag (pre-formatted by the caller) is
+        passed through as-is instead of being wrapped a second time -#}
+    {%- if content is string and "<tool_response>" in content -%}
+        {{- content -}}
+    {%- else -%}
+    {{- "<tool_response>\n" -}}
+    {%- if content is string -%}
+        {{- content -}}
+    {%- elif content is mapping -%}
+        {{- parse_content(content) -}}
+    {%- elif content is iterable -%}
+        {%- for item in content -%}
+            {%- if not loop.first -%}{{- "\n" -}}{%- endif -%}
+            {%- if is_tool_reference(item) -%}
+                {#- consecutive references share one <additional_tools> block -#}
+                {%- if loop.first or not is_tool_reference(loop.previtem) -%}
+                    {{- "<additional_tools>\n" -}}
+                {%- endif -%}
+                {{- referenced_tool_to_json(item.name) -}}
+                {%- if loop.last or not is_tool_reference(loop.nextitem) -%}
+                    {{- "\n</additional_tools>" -}}
+                {%- endif -%}
+            {%- elif item is mapping and item.result is defined and item.result is mapping and item.result.tools is defined -%}
+                {{- "<additional_tools>\n" -}}
+                {%- set _tl = item.result.tools -%}
+                {%- if _tl is string -%}
+                    {{- _tl -}}
+                {%- elif _tl is mapping -%}
+                    {{- tool_to_json(_tl) -}}
+                {%- else -%}
+                    {%- for t in _tl -%}
+                        {%- if not loop.first -%}{{- "\n" -}}{%- endif -%}
+                        {{- tool_to_json(t) -}}
+                    {%- endfor -%}
+                {%- endif -%}
+                {{- "\n</additional_tools>" -}}
+            {%- else -%}
+                {{- parse_content(item) -}}
+            {%- endif -%}
+        {%- endfor -%}
+    {%- endif -%}
+    {{- "\n</tool_response>" -}}
+    {%- endif -%}
+{%- endmacro -%}
+
+{%- set ns = namespace(system_prompt="", last_user_index=-1) -%}
+{%- if messages and messages[0]["role"] == "system" -%}
+    {%- if messages[0].get("content") -%}
+        {%- set ns.system_prompt = parse_content(messages[0]["content"]) -%}
+    {%- endif -%}
+    {%- set messages = messages[1:] -%}
+{%- endif -%}
+{%- if tools -%}
+    {%- set tools_ns = namespace(lines=[]) -%}
+    {%- for tool in tools -%}
+        {%- set fn = tool.function if tool.function is defined else tool -%}
+        {%- set _dyn = (fn.defer_loading is defined and fn.defer_loading) or (tool.defer_loading is defined and tool.defer_loading) -%}
+        {%- if not _dyn -%}
+            {%- set tools_ns.lines = tools_ns.lines + [tool_to_json(fn)] -%}
+        {%- endif -%}
+    {%- endfor -%}
+    {%- if tools_ns.lines -%}
+        {%- set ns.system_prompt = ns.system_prompt + ("\n\n" if ns.system_prompt else "") + "<tools>\n" + (tools_ns.lines | join("\n")) + "\n</tools>" -%}
+    {%- endif -%}
+{%- endif -%}
+{%- if ns.system_prompt -%}
+    {{- "<|im_start|>system\n" + ns.system_prompt + "<|im_end|>\n" -}}
+{%- endif -%}
+{%- for message in messages -%}
+    {%- if message["role"] == "user" -%}
+        {%- set ns.last_user_index = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+{#- vLLM's and SGLang's Anthropic /v1/messages adapters split one tool_result that carries
+    tool_reference blocks into consecutive tool messages with the same tool_call_id (text first,
+    references second). Merge such runs into one tool turn so they render exactly like a single
+    OpenAI-format tool message whose content holds the same items. -#}
+{%- set carry = namespace(pending=[]) -%}
+{%- for message in messages -%}
+    {%- set _next = messages[loop.index0 + 1] if not loop.last else none -%}
+    {%- set _merge = message.role == "tool" and not message.get("additional_tools")
+        and _next is not none and _next.role == "tool" and not _next.get("additional_tools")
+        and message.get("tool_call_id") and _next.get("tool_call_id") == message.get("tool_call_id") -%}
+    {%- set _items = [] -%}
+    {%- if message.role == "tool" and not message.get("additional_tools") -%}
+        {%- set _c = message.get("content") -%}
+        {%- if _c is string -%}
+            {%- set _items = [{"type": "text", "text": _c}] if _c else [] -%}
+        {%- elif _c is mapping -%}
+            {%- set _items = [_c] -%}
+        {%- elif _c -%}
+            {%- set _items = _c | list -%}
+        {%- endif -%}
+    {%- endif -%}
+    {%- if _merge -%}
+        {%- set carry.pending = carry.pending + _items -%}
+    {%- else -%}
+    {{- "<|im_start|>" + message.role + "\n" -}}
+    {%- if message.role == "assistant" -%}
+        {%- generation -%}
+        {%- set thinking = message.thinking if message.thinking is defined else message.reasoning if message.reasoning is defined else message.reasoning_content -%}
+        {%- set keep_thinking = preserve_thinking or loop.index0 > ns.last_user_index -%}
+        {%- if thinking is defined and keep_thinking -%}
+            {{- "<think>" + (thinking or "") + "</think>" -}}
+        {%- endif -%}
+        {%- set _cfm_tag = "CONTINUE_FINAL_MESSAGE_TAG " -%}
+        {%- set _has_cfm = false -%}
+        {%- set content = "" -%}
+        {%- if message.get("content") -%}
+            {%- set content = parse_content(message.content) -%}
+        {%- endif -%}
+        {%- if not keep_thinking -%}
+            {%- if "</think>" in content -%}
+                {%- set content = content.split("</think>")[-1] | trim -%}
+            {%- endif -%}
+            {%- set content = "<think></think>" + content -%}
+        {%- endif -%}
+        {%- if content.endswith(_cfm_tag) -%}
+            {%- set _has_cfm = true -%}
+            {%- set _trunc_len = (content | length) - (_cfm_tag | length) -%}
+            {%- set content = content[:_trunc_len] -%}
+        {%- endif -%}
+        {{- content -}}
+        {%- if message.tool_calls -%}
+            {{- render_tool_calls(message.tool_calls) -}}
+        {%- endif -%}
+        {%- if _has_cfm -%}
+            {{- _cfm_tag -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+        {%- endgeneration -%}
+    {%- elif message.role == "tool" -%}
+        {%- if message.get("additional_tools") -%}
+            {%- set _at = message["additional_tools"] -%}
+            {#- definitions may live in additional_tools itself, or in content
+                with additional_tools used as a plain truthy flag -#}
+            {%- if _at is mapping or (_at is sequence and _at is not string) -%}
+                {%- set _src = _at -%}
+            {%- else -%}
+                {%- set _src = message.get("content") -%}
+            {%- endif -%}
+            {#- nothing to reveal: skip the wrapper entirely rather than
+                emitting an empty <additional_tools> block -#}
+            {%- if _src is string and "<tool_response>" in _src -%}
+                {{- _src -}}
+            {%- elif _src is string and "<additional_tools>" in _src -%}
+                {{- "<tool_response>\n" + _src + "\n</tool_response>" -}}
+            {%- elif _src -%}
+                {{- "<tool_response>\n<additional_tools>\n" -}}
+                {%- if _src is string -%}
+                    {{- _src -}}
+                {%- elif _src is mapping -%}
+                    {{- tool_to_json(_src) -}}
+                {%- else -%}
+                    {%- for t in _src -%}
+                        {%- if not loop.first -%}{{- "\n" -}}{%- endif -%}
+                        {{- tool_to_json(t) -}}
+                    {%- endfor -%}
+                {%- endif -%}
+                {{- "\n</additional_tools>\n</tool_response>" -}}
+            {%- endif -%}
+        {%- elif carry.pending -%}
+            {{- render_tool_response(carry.pending + _items) -}}
+            {%- set carry.pending = [] -%}
+        {%- else -%}
+            {#- an empty or missing result still gets the wrapper, so every tool turn has the same shape -#}
+            {{- render_tool_response(message.get("content") or "") -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+    {%- else %}
+        {%- if message.get("content") -%}
+            {{- parse_content(message["content"]) -}}
+        {%- endif -%}
+        {{- "<|im_end|>\n" -}}
+    {%- endif %}
+    {%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+    {{- "<|im_start|>assistant\n" -}}
+    {%- if enable_thinking -%}
+        {{- "<think>" -}}
+    {%- else -%}
+        {{- "<think></think>" -}}
+    {%- endif -%}
+{%- endif -%}

config.json

@@ -0,0 +1,126 @@
+{
+  "architectures": [
+    "D1Model"
+  ],
+  "bos_token_id": 124894,
+  "downsample_factor": 2,
+  "dtype": "bfloat16",
+  "eos_token_id": 124900,
+  "image_token_id": 125030,
+  "model_type": "lfm3_vl",
+  "pad_token_id": 124893,
+  "projector_bias": true,
+  "projector_hidden_act": "gelu",
+  "projector_hidden_size": 4096,
+  "projector_layer_norm_eps": 1e-06,
+  "text_config": {
+    "bos_token_id": 124894,
+    "conv_L_cache": 7,
+    "conv_bias": false,
+    "dtype": "bfloat16",
+    "eos_token_id": 124900,
+    "head_dim": 128,
+    "hidden_size": 2048,
+    "initializer_range": 0.02,
+    "intermediate_size": 7680,
+    "layer_types": [
+      "conv",
+      "conv",
+      "full_attention",
+      "conv",
+      "sliding_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "sliding_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "sliding_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "sliding_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "sliding_attention",
+      "conv",
+      "full_attention",
+      "conv"
+    ],
+    "max_position_embeddings": 128000,
+    "model_type": "lfm3",
+    "norm_eps": 1e-05,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 24,
+    "num_key_value_heads": 4,
+    "pad_token_id": 124893,
+    "rope_parameters": {
+      "full_attention": {
+        "partial_rotary_factor": 0.5,
+        "rope_theta": 1000000.0,
+        "rope_type": "default"
+      },
+      "sliding_attention": {
+        "rope_theta": 10000,
+        "rope_type": "default"
+      }
+    },
+    "sliding_window": 256,
+    "tie_word_embeddings": true,
+    "use_cache": true,
+    "use_sandwich_norm": true,
+    "vocab_size": 128000
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.19.0",
+  "video_token_id": 125029,
+  "vision_config": {
+    "conv_kernel_size": 3,
+    "dtype": "bfloat16",
+    "hidden_size": 1024,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 2560,
+    "layer_types": [
+      "conv",
+      "conv",
+      "full_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "full_attention",
+      "conv",
+      "full_attention",
+      "conv"
+    ],
+    "model_type": "lfm3_vl_vision",
+    "norm_eps": 1e-05,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 14,
+    "num_key_value_heads": 8,
+    "patch_size": 16,
+    "rope_parameters": {
+      "rope_theta": 1000000.0,
+      "rope_type": "axial"
+    },
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 2
+  },
+  "vision_end_token_id": 125010,
+  "vision_start_token_id": 125009,
+  "d1_codes": 2,
+  "d1_attention": "set",
+  "d1_numbers": true,
+  "d1_set_size": 16,
+  "d1_vision_max_pixels": 1638400,
+  "auto_map": {
+    "AutoModel": "modeling_d1_lfm3_vl.D1Model",
+    "AutoConfig": "native_lfm3_vl.Lfm3VlConfig"
+  }
+}

configuration_lfm3.py

@@ -0,0 +1,140 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3/configuration_lfm3.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3/modular_lfm3.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from huggingface_hub.dataclasses import strict
+
+from transformers.configuration_utils import PreTrainedConfig
+from transformers.utils import auto_docstring
+
+
+@auto_docstring(checkpoint="LiquidAI/LFM3-1.7B")
+@strict
+class Lfm3Config(PreTrainedConfig):
+    r"""
+    rope_parameters (`dict[str, RopeParameters]`, *optional*):
+        RoPE parameters keyed by attention layer type, for example `{"full_attention": {"rope_type": "default",
+        "rope_theta": 1000000.0, "partial_rotary_factor": 0.5}, "sliding_attention": {"rope_type": "default",
+        "rope_theta": 10000.0}}`. `conv` layers do not use RoPE and have no entry. Full-attention layers rotate the
+        first `partial_rotary_factor * head_dim` dimensions and leave the rest unrotated.
+    conv_bias (`bool`, *optional*, defaults to `False`):
+        Whether to use bias in the conv layers.
+    conv_L_cache (`int`, *optional*, defaults to 7):
+        L_cache dim in the conv layers.
+    layer_types (`list[str]`, *optional*):
+        Per-layer operator type, each one of `"full_attention"`, `"sliding_attention"` or `"conv"`. Defaults to the
+        LFM3 pattern of a full-attention layer every fourth layer starting from the third, a sliding-attention layer
+        in between, and gated short convolutions elsewhere.
+    sliding_window (`int`, *optional*, defaults to 256):
+        Sliding-window size of the `sliding_attention` layers.
+    use_sandwich_norm (`bool`, *optional*, defaults to `True`):
+        Whether to apply an RMSNorm to the operator and feed-forward outputs before they enter the residual stream.
+
+    ```python
+    >>> from transformers import Lfm3Model, Lfm3Config
+
+    >>> # Initializing a LFM3 model
+    >>> configuration = Lfm3Config()
+
+    >>> # Initializing a model from the LFM3-1.7B style configuration
+    >>> model = Lfm3Model(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```
+    """
+
+    model_type = "lfm3"
+    keys_to_ignore_at_inference = ["past_key_values"]
+    default_theta = 1000000.0
+
+    vocab_size: int = 128_000
+    hidden_size: int = 2048
+    intermediate_size: int = 7680
+    num_hidden_layers: int = 24
+    num_attention_heads: int = 16
+    num_key_value_heads: int = 4
+    max_position_embeddings: int = 128_000
+    initializer_range: float = 0.02
+    norm_eps: float = 0.00001
+    use_cache: bool = True
+    pad_token_id: int | None = 124_893
+    bos_token_id: int | None = 124_894
+    eos_token_id: int | list[int] | None = 124_900
+    tie_word_embeddings: bool = True
+    rope_parameters: dict | None = None
+    conv_bias: bool = False
+    conv_L_cache: int = 7
+    layer_types: list[str] | None = None
+    _attention_layer_types = ("full_attention", "sliding_attention")
+    head_dim: int | None = None
+    sliding_window: int | None = 256
+    use_sandwich_norm: bool = True
+
+    def __post_init__(self, **kwargs):
+        if self.head_dim is None:
+            self.head_dim = self.hidden_size // self.num_attention_heads
+        if self.rope_parameters is None:
+            self.rope_parameters = {
+                "full_attention": {"rope_type": "default", "rope_theta": 1_000_000.0, "partial_rotary_factor": 0.5},
+                "sliding_attention": {"rope_type": "default", "rope_theta": 10_000.0},
+            }
+        if self.layer_types is None:
+            self.layer_types = [
+                "full_attention" if i % 4 == 2 else "sliding_attention" if i % 4 == 0 and i > 0 else "conv"
+                for i in range(self.num_hidden_layers)
+            ]
+
+        self.tie_word_embeddings = kwargs.pop("tie_embedding", self.tie_word_embeddings)
+        super().__post_init__(**kwargs)
+
+    def validate_architecture(self):
+        """Part of ``@strict``-powered validation."""
+        if len(self.layer_types) != self.num_hidden_layers:
+            raise ValueError(
+                f"`layer_types` has {len(self.layer_types)} entries but `num_hidden_layers` is {self.num_hidden_layers}."
+            )
+        allowed_layer_types = (*self._attention_layer_types, "conv")
+        invalid_layer_types = [layer_type for layer_type in self.layer_types if layer_type not in allowed_layer_types]
+        if invalid_layer_types:
+            raise ValueError(f"`layer_types` entries must be one of {allowed_layer_types}; got {invalid_layer_types}.")
+        if "sliding_attention" in self.layer_types and self.sliding_window is None:
+            raise ValueError("`layer_types` has `sliding_attention` layers but `sliding_window` is None.")
+        missing_rope = [
+            layer_type
+            for layer_type in self._attention_layer_types
+            if layer_type in self.layer_types and layer_type not in self.rope_parameters
+        ]
+        if missing_rope:
+            raise ValueError(f"`rope_parameters` has no entry for the {missing_rope} layers in `layer_types`.")
+        for layer_type, rope_params in self.rope_parameters.items():
+            if not isinstance(rope_params, dict):
+                continue
+            if (
+                rope_params.get("rope_type", "default") in ("dynamic", "longrope")
+                and rope_params.get("partial_rotary_factor", 1.0) < 1
+            ):
+                raise ValueError(
+                    f"`{rope_params['rope_type']}` RoPE recomputes its frequencies during the forward pass over the "
+                    f"rotated dimension only and cannot keep LFM3's partial rotation on `{layer_type}` layers."
+                )
+
+
+__all__ = ["Lfm3Config"]

configuration_lfm3_vl.py

@@ -0,0 +1,211 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3_vl/configuration_lfm3_vl.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3_vl/modular_lfm3_vl.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3_vl.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from huggingface_hub.dataclasses import strict
+
+from transformers.configuration_utils import PreTrainedConfig
+from transformers.utils import auto_docstring
+from transformers.models.auto import CONFIG_MAPPING, AutoConfig
+
+
+@auto_docstring(checkpoint="LiquidAI/LFM3-VL-1.9B")
+@strict
+class Lfm3VlVisionConfig(PreTrainedConfig):
+    r"""
+    layer_types (`list[str]`, *optional*):
+        Per-layer operator type, each one of `"full_attention"` or `"conv"`. Defaults to two gated convolution
+        layers followed by alternating attention and convolution layers.
+    patch_size (`int`, *optional*, defaults to 16):
+        The spatial size of each image patch.
+    temporal_patch_size (`int`, *optional*, defaults to 2):
+        The number of video frames packed into one patch.
+    spatial_merge_size (`int`, *optional*, defaults to 2):
+        The side of the square block of patches merged into one language token.
+    conv_kernel_size (`int`, *optional*, defaults to 3):
+        The kernel size of the depthwise 2D convolution layers.
+    norm_eps (`float`, *optional*, defaults to 1e-05):
+        The epsilon used by the RMS normalization layers.
+    """
+
+    model_type = "lfm3_vl_vision"
+    base_config_key = "vision_config"
+    default_theta = 1_000_000.0
+    default_rope_type = "axial"
+    _attention_layer_types = ("full_attention",)
+
+    hidden_size: int = 1024
+    intermediate_size: int = 2560
+    num_hidden_layers: int = 14
+    num_attention_heads: int = 16
+    num_key_value_heads: int = 8
+    layer_types: list[str] | None = None
+    patch_size: int = 16
+    temporal_patch_size: int = 2
+    in_channels: int = 3
+    spatial_merge_size: int = 2
+    conv_kernel_size: int = 3
+    norm_eps: float = 1e-5
+    initializer_range: float = 0.02
+    rope_parameters: dict | None = None
+
+    def __post_init__(self, **kwargs):
+        if self.layer_types is None:
+            self.layer_types = ["conv" if i < 2 or i % 2 else "full_attention" for i in range(self.num_hidden_layers)]
+        super().__post_init__(**kwargs)
+
+    def validate_architecture(self):
+        """Part of ``@strict``-powered validation."""
+        if len(self.layer_types) != self.num_hidden_layers:
+            raise ValueError(
+                f"`layer_types` has {len(self.layer_types)} entries but `num_hidden_layers` is {self.num_hidden_layers}."
+            )
+        allowed_layer_types = (*self._attention_layer_types, "conv")
+        invalid_layer_types = [layer_type for layer_type in self.layer_types if layer_type not in allowed_layer_types]
+        if invalid_layer_types:
+            raise ValueError(f"`layer_types` entries must be one of {allowed_layer_types}; got {invalid_layer_types}.")
+        if self.hidden_size % self.num_attention_heads:
+            raise ValueError("`hidden_size` must be a multiple of `num_attention_heads`.")
+
+
+# The first checkpoints spell every text and `vision_*` field at the top level of the config
+_LEGACY_TEXT_KEYS = (
+    "vocab_size",
+    "hidden_size",
+    "intermediate_size",
+    "num_hidden_layers",
+    "num_attention_heads",
+    "num_key_value_heads",
+    "head_dim",
+    "layer_types",
+    "conv_bias",
+    "conv_L_cache",
+    "sliding_window",
+    "max_position_embeddings",
+    "use_sandwich_norm",
+    "use_cache",
+)
+_LEGACY_VISION_KEYS = {
+    "vision_hidden_size": "hidden_size",
+    "vision_intermediate_size": "intermediate_size",
+    "vision_num_hidden_layers": "num_hidden_layers",
+    "vision_num_attention_heads": "num_attention_heads",
+    "vision_num_key_value_heads": "num_key_value_heads",
+    "vision_layer_types": "layer_types",
+    "vision_patch_size": "patch_size",
+    "vision_temporal_patch_size": "temporal_patch_size",
+    "vision_num_channels": "in_channels",
+    "vision_spatial_merge_size": "spatial_merge_size",
+}
+_LEGACY_DROPPED_KEYS = ("vision_patch_dim", "vision_max_position_embeddings")
+
+
+@auto_docstring(checkpoint="LiquidAI/LFM3-VL-1.9B")
+@strict
+class Lfm3VlConfig(PreTrainedConfig):
+    r"""
+    video_token_id (`int`, *optional*, defaults to 125029):
+        The id of the placeholder token for video patches.
+    vision_start_token_id (`int`, *optional*, defaults to 125009):
+        The id of the token opening an image or a video frame.
+    vision_end_token_id (`int`, *optional*, defaults to 125010):
+        The id of the token closing an image or a video frame.
+    projector_layer_norm_eps (`float`, *optional*, defaults to 1e-06):
+        The epsilon of the layer normalization applied to the vision features before projection.
+    downsample_factor (`int`, *optional*, defaults to 2):
+        The side of the square block of vision tokens merged by the projector, equal to the vision encoder's
+        `spatial_merge_size`.
+    """
+
+    model_type = "lfm3_vl"
+    sub_configs = {"text_config": AutoConfig, "vision_config": Lfm3VlVisionConfig}
+
+    text_config: dict | PreTrainedConfig | None = None
+    vision_config: dict | PreTrainedConfig | None = None
+    image_token_id: int = 125_030
+    video_token_id: int = 125_029
+    vision_start_token_id: int = 125_009
+    vision_end_token_id: int = 125_010
+    projector_hidden_act: str = "gelu"
+    projector_hidden_size: int = 4096
+    projector_bias: bool = True
+    projector_layer_norm_eps: float = 1e-6
+    downsample_factor: int = 2
+    tie_word_embeddings: bool = True
+
+    def __post_init__(self, **kwargs):
+        if "vision_hidden_size" in kwargs:
+            kwargs = self._convert_legacy_flat_config(kwargs)
+
+        if isinstance(self.vision_config, dict):
+            self.vision_config = self.sub_configs["vision_config"](**self.vision_config)
+        elif self.vision_config is None:
+            self.vision_config = self.sub_configs["vision_config"]()
+
+        if isinstance(self.text_config, dict):
+            self.text_config = {"model_type": "lfm3", **self.text_config}
+            self.text_config = CONFIG_MAPPING[self.text_config["model_type"]](**self.text_config)
+        elif self.text_config is None:
+            self.text_config = CONFIG_MAPPING["lfm3"]()
+        self.validate_vision_fields()
+
+        super().__post_init__(**kwargs)
+
+    def validate_vision_fields(self):
+        """Part of `@strict`-powered validation, so `validate()` and `save_pretrained` refuse later edits too."""
+        if self.downsample_factor != self.vision_config.spatial_merge_size:
+            raise ValueError("downsample_factor must match vision_config.spatial_merge_size")
+        # an `inputs_embeds` forward embeds both placeholder ids to find them
+        vocab_size = self.text_config.vocab_size
+        for name in ("image_token_id", "video_token_id"):
+            if not 0 <= getattr(self, name) < vocab_size:
+                raise ValueError(f"{name}={getattr(self, name)} must lie in [0, text_config.vocab_size={vocab_size})")
+
+    def _convert_legacy_flat_config(self, kwargs: dict) -> dict:
+        text_config = {key: kwargs.pop(key) for key in _LEGACY_TEXT_KEYS if key in kwargs}
+        text_config["norm_eps"] = kwargs.pop("rms_norm_eps", 1e-5)
+        text_config["rope_parameters"] = {
+            "full_attention": {
+                "rope_type": "default",
+                "rope_theta": kwargs.pop("full_attention_rope_theta", 1_000_000.0),
+                "partial_rotary_factor": kwargs.pop("partial_rotary_factor", 0.5),
+            },
+            "sliding_attention": {
+                "rope_type": "default",
+                "rope_theta": kwargs.pop("sliding_attention_rope_theta", 10_000.0),
+            },
+        }
+        for key in ("pad_token_id", "bos_token_id", "eos_token_id"):
+            if kwargs.get(key) is not None:
+                text_config[key] = kwargs[key]
+        text_config["tie_word_embeddings"] = self.tie_word_embeddings
+
+        vision_config = {new: kwargs.pop(old) for old, new in _LEGACY_VISION_KEYS.items() if old in kwargs}
+        vision_config["norm_eps"] = text_config["norm_eps"]
+        vision_config["rope_parameters"] = {"rope_type": "axial", "rope_theta": kwargs.pop("vision_rope_theta", 1e6)}
+        for key in _LEGACY_DROPPED_KEYS:
+            kwargs.pop(key, None)
+
+        self.text_config = text_config
+        self.vision_config = vision_config
+        return kwargs
+
+
+__all__ = ["Lfm3VlConfig", "Lfm3VlVisionConfig"]

depthwise.py

@@ -0,0 +1,53 @@
+"""LFM's depthwise causal convolution, the same numbers in every process: on ROCm as shifted sums, never MIOpen's
+kernel, whose solver search runs again for every new length."""
+
+from __future__ import annotations
+
+import torch
+import torch.nn.functional as F
+
+
+def on_rocm(t: torch.Tensor) -> bool:
+    return t.is_cuda and bool(torch.version.hip)
+
+
+def depthwise(padded: torch.Tensor, weight: torch.Tensor) -> torch.Tensor:
+    """Depthwise conv, `(B, C, T + k - 1)` to `(B, C, T)`. On ROCm, `shifted_sum`: MIOpen's depthwise kernel
+    searches a solver for every new length, logs a warning each time, may settle on another one in another process,
+    and takes 1.25 ms at 5k tokens against 0.07 ms fused (MI325X)."""
+    if not on_rocm(padded):
+        return F.conv1d(padded, weight, groups=padded.shape[1])
+    return shifted_sum(padded, weight)
+
+
+def shifted_sum(padded: torch.Tensor, weight: torch.Tensor) -> torch.Tensor:
+    """The depthwise conv as k shifted products summed in fp32: one fp32 copy of `padded`, the shifts views of it."""
+    k, t = weight.shape[-1], padded.shape[-1] - weight.shape[-1] + 1
+    p, w = padded.float(), weight[:, 0, :, None].float()
+    return sum(p[..., i:i + t] * w[:, i] for i in range(k)).to(padded.dtype)
+
+
+def pinned_conv(conv: torch.nn.Conv1d | torch.nn.Conv2d, x: torch.Tensor) -> torch.Tensor:
+    """`conv(x)` without MIOpen on ROCm: a 1x1 conv as the matmul it is, a 2D conv with one input channel a group
+    (`groups` = input channels) as `shifted_sum2d`. MIOpen's solver for the omni downsampler's 1x1 `lin_out` changed
+    from one call to the next within a process (MI325X). Elsewhere, and for any other conv, `conv`'s own forward."""
+    if on_rocm(x) and conv.padding_mode == "zeros" and set(conv.dilation) == {1}:
+        if set(conv.kernel_size) == {1} and set(conv.stride) == {1} and conv.padding in ((0,), (0, 0)) and conv.groups == 1:
+            return F.linear(x.movedim(1, -1), conv.weight.flatten(1), conv.bias).movedim(-1, 1)
+        if x.ndim == 4 and conv.groups == conv.in_channels and not isinstance(conv.padding, str):
+            return shifted_sum2d(x, conv.weight, conv.bias, conv.stride, conv.padding)
+    return type(conv).forward(conv, x)
+
+
+def shifted_sum2d(x: torch.Tensor, weight: torch.Tensor, bias: torch.Tensor | None, stride: tuple[int, int],
+                  padding: tuple[int, int]) -> torch.Tensor:
+    """A 2D conv with one input channel a group, `weight` `(C * m, 1, kh, kw)`, as kh x kw strided products summed in
+    fp32: each of the C input channels feeds its m output channels."""
+    (kh, kw), (sh, sw), (ph, pw) = weight.shape[-2:], stride, padding
+    c = x.shape[1]
+    ho, wo = (x.shape[-2] + 2 * ph - kh) // sh + 1, (x.shape[-1] + 2 * pw - kw) // sw + 1
+    p = F.pad(x.float(), (pw, pw, ph, ph))[:, :, None]
+    w = weight.float().view(c, -1, kh, kw)[..., None, None]
+    out = sum(p[..., i:i + sh * (ho - 1) + 1:sh, j:j + sw * (wo - 1) + 1:sw] * w[:, :, i, j]
+              for i in range(kh) for j in range(kw)).flatten(1, 2)
+    return (out if bias is None else out + bias.float()[:, None, None]).to(x.dtype)

grid.py

@@ -0,0 +1,45 @@
+"""Ordered values read as a distribution over equally spaced bins."""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Callable
+from dataclasses import dataclass
+from functools import cached_property
+from typing import Sequence
+
+
+@dataclass(frozen=True)
+class Grid:
+    low: float
+    high: float
+    size: int = 128
+
+    def __post_init__(self):
+        if self.size < 2 or not self.low < self.high:
+            raise ValueError("a grid needs two values or more, low to high")
+
+    @property
+    def step(self) -> float:
+        return (self.high - self.low) / (self.size - 1)
+
+    @cached_property
+    def values(self) -> tuple[float, ...]:
+        return tuple(self.low + i * self.step for i in range(self.size - 1)) + (self.high,)
+
+    def read(self, probabilities: Sequence[float], level: Callable[[float], float] | None = None) -> dict[str, float]:
+        """The mean, median and central 80 percent interval, on this grid's values; `level`, the level a quantile is
+        read at (a calibration's, increasing)."""
+        total = math.fsum(probabilities)
+        values = self.values
+        out = {"mean": math.fsum(v * (p / total) for v, p in zip(values, probabilities))}
+        at = [(key, level(q) if level else q) for key, q in (("p10", 0.1), ("median", 0.5), ("p90", 0.9))]
+        mass = 0.0
+        for value, p in zip(values, probabilities):
+            mass += p / total
+            for key, quantile in at:
+                if key not in out and mass >= quantile:
+                    out[key] = value
+        for key, _ in at:  # a level near 1 the summed mass falls short of by rounding: the last value
+            out.setdefault(key, values[-1])
+        return out

image_processing_lfm3_vl.py

@@ -0,0 +1,301 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3_vl/image_processing_lfm3_vl.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3_vl/modular_lfm3_vl.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3_vl.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import math
+import warnings
+from collections.abc import Iterable
+
+import torch
+from torchvision.transforms.v2 import functional as tvF
+
+from transformers.image_processing_backends import TorchvisionBackend
+from transformers.image_processing_utils import BatchFeature
+from transformers.image_transforms import group_images_by_shape, reorder_images
+from transformers.image_utils import ImageInput, PILImageResampling, SizeDict
+from transformers.processing_utils import ImagesKwargs, Unpack
+from transformers.utils import TensorType, auto_docstring
+
+
+class Lfm3VlImageProcessorKwargs(ImagesKwargs, total=False):
+    r"""
+    min_pixels (`int`, *optional*, defaults to `56 * 56`):
+        The min pixels of the image to resize the image.
+    max_pixels (`int`, *optional*, defaults to `28 * 28 * 1280`):
+        The max pixels of the image to resize the image.
+    patch_size (`int`, *optional*, defaults to 14):
+        The spatial patch size of the vision encoder.
+    temporal_patch_size (`int`, *optional*, defaults to 2):
+        The temporal patch size of the vision encoder.
+    merge_size (`int`, *optional*, defaults to 2):
+        The merge size of the vision encoder to llm encoder.
+    """
+
+    min_pixels: int
+    max_pixels: int
+    patch_size: int
+    temporal_patch_size: int
+    merge_size: int
+
+
+def smart_resize(
+    height: int, width: int, factor: int = 28, min_pixels: int = 56 * 56, max_pixels: int = 14 * 14 * 4 * 1280
+):
+    """Rescales the image so that the following conditions are met:
+
+    1. Both dimensions (height and width) are divisible by 'factor'.
+
+    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
+
+    3. The aspect ratio of the image is maintained as closely as possible.
+
+    """
+    if max(height, width) / min(height, width) > 200:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
+        )
+    h_bar = round(height / factor) * factor
+    w_bar = round(width / factor) * factor
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+    return h_bar, w_bar
+
+
+@auto_docstring
+class Lfm3VlImageProcessor(TorchvisionBackend):
+    do_resize = True
+    resample = PILImageResampling.BICUBIC
+    size = {"shortest_edge": 65_536, "longest_edge": 16_777_216}
+    default_to_square = False
+    do_rescale = True
+    do_normalize = True
+    image_mean = [0.5, 0.5, 0.5]
+    image_std = [0.5, 0.5, 0.5]
+    do_convert_rgb = True
+    patch_size = 16
+    temporal_patch_size = 2
+    merge_size = 2
+    valid_kwargs = Lfm3VlImageProcessorKwargs
+    model_input_names = ["pixel_values", "image_grid_thw"]
+
+    def __init__(self, **kwargs: Unpack[Lfm3VlImageProcessorKwargs]):
+        # backward compatibility: override size with min_pixels and max_pixels if they are provided
+        size = kwargs.pop("size", None)
+        size = dict(self.size) if size is None else size
+        if (min_pixels := kwargs.pop("min_pixels", None)) is not None:
+            size["shortest_edge"] = min_pixels
+            size.pop("min_pixels", None)
+        if (max_pixels := kwargs.pop("max_pixels", None)) is not None:
+            size["longest_edge"] = max_pixels
+            size.pop("max_pixels", None)
+        super().__init__(size=size, **kwargs)
+
+    def _standardize_kwargs(
+        self,
+        size: int | Iterable[int] | dict[str, int] | SizeDict | None = None,
+        min_pixels: int | None = None,
+        max_pixels: int | None = None,
+        **kwargs,
+    ) -> dict:
+        if min_pixels is not None or max_pixels is not None:
+            warnings.warn(
+                "Passing `min_pixels` and `max_pixels` to a processor call is deprecated and will be removed in v5.23. "
+                "Pass in `size={'longest_edge': xxx, 'shortest_edge': xxx} to override the target size.`",
+                FutureWarning,
+            )
+
+            size_dict = dict(size) if isinstance(size, (dict, SizeDict)) else {}
+            if min_pixels is not None:
+                size_dict["shortest_edge"] = min_pixels
+            if max_pixels is not None:
+                size_dict["longest_edge"] = max_pixels
+            size = SizeDict(**size_dict)
+        return super()._standardize_kwargs(size=size, **kwargs)
+
+    @auto_docstring
+    def preprocess(
+        self,
+        images: ImageInput,
+        **kwargs: Unpack[Lfm3VlImageProcessorKwargs],
+    ) -> BatchFeature:
+        return super().preprocess(images, **kwargs)
+
+    def resize(
+        self,
+        images: "torch.Tensor",
+        size: SizeDict,
+        resample: "PILImageResampling | tvF.InterpolationMode | int | None",
+        factor: int,
+        **kwargs,
+    ) -> "torch.Tensor":
+        """Resize dynamically based on input image aspect ratio."""
+        if not size.shortest_edge or not size.longest_edge:
+            raise ValueError(f"`size` dict must contain 'shortest_edge' and 'longest_edge' keys but got {size}.")
+
+        height, width = images.shape[-2:]
+        resized_height, resized_width = smart_resize(
+            height,
+            width,
+            factor=factor,
+            min_pixels=size.shortest_edge,
+            max_pixels=size.longest_edge,
+        )
+        return super().resize(
+            image=images,
+            size=SizeDict(height=resized_height, width=resized_width),
+            resample=resample,
+        )
+
+    def patchify(
+        self,
+        images: "torch.Tensor",
+        patch_size: int,
+        merge_size: int,
+        temporal_patch_size: int,
+    ) -> tuple["torch.Tensor", int, int]:
+        """Patchifies each image into flat layout of shape (`seq_len`, `patch_dim`) so we can concat dynamically shaped pixels."""
+        batch_size, channel, resized_height, resized_width = images.shape
+        grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+        patches = images.reshape(
+            batch_size,
+            channel,
+            grid_h // merge_size,
+            merge_size,
+            patch_size,
+            grid_w // merge_size,
+            merge_size,
+            patch_size,
+        )
+        # Reorder dimensions to group grid and patch information for subsequent flattening.
+        # [batch, grid_h/merge, grid_w/merge, merge, merge, channel, patch, patch]
+        patches = patches.permute(0, 2, 5, 3, 6, 1, 4, 7)
+
+        flatten_patches = (
+            patches.unsqueeze(6)
+            .expand(-1, -1, -1, -1, -1, -1, temporal_patch_size, -1, -1)
+            .reshape(
+                batch_size,
+                grid_h * grid_w,
+                channel * temporal_patch_size * patch_size * patch_size,
+            )
+        )
+        return flatten_patches, grid_h, grid_w
+
+    def _preprocess(
+        self,
+        images: list["torch.Tensor"],
+        do_resize: bool,
+        size: SizeDict,
+        resample: "PILImageResampling | tvF.InterpolationMode | int | None",
+        do_rescale: bool,
+        rescale_factor: float,
+        do_normalize: bool,
+        image_mean: float | list[float] | None,
+        image_std: float | list[float] | None,
+        patch_size: int,
+        temporal_patch_size: int,
+        merge_size: int,
+        disable_grouping: bool | None,
+        return_tensors: str | TensorType | None,
+        **kwargs,
+    ) -> BatchFeature:
+        grouped_images, grouped_images_index = group_images_by_shape(images, disable_grouping=disable_grouping)
+        resized_images_grouped = {}
+        for shape, stacked_images in grouped_images.items():
+            if do_resize:
+                stacked_images = self.resize(
+                    images=stacked_images,
+                    size=size,
+                    resample=resample,
+                    factor=patch_size * merge_size,
+                )
+            resized_images_grouped[shape] = stacked_images
+        resized_images = reorder_images(resized_images_grouped, grouped_images_index)
+
+        grouped_images, grouped_images_index = group_images_by_shape(resized_images, disable_grouping=disable_grouping)
+        processed_images_grouped = {}
+        processed_grids = {}
+        for shape, stacked_images in grouped_images.items():
+            stacked_images = self.rescale_and_normalize(
+                stacked_images, do_rescale, rescale_factor, do_normalize, image_mean, image_std
+            )
+            patches, grid_h, grid_w = self.patchify(
+                stacked_images,
+                patch_size=patch_size,
+                merge_size=merge_size,
+                temporal_patch_size=temporal_patch_size,
+            )
+
+            processed_images_grouped[shape] = patches
+            processed_grids[shape] = [[1, grid_h, grid_w]] * len(stacked_images)
+
+        processed_images = reorder_images(processed_images_grouped, grouped_images_index)
+        processed_grids_ordered = reorder_images(processed_grids, grouped_images_index)
+        pixel_values = processed_images[0] if len(processed_images) == 1 else torch.cat(processed_images, dim=0)
+        image_grid_thw = torch.tensor(processed_grids_ordered, dtype=torch.long)
+
+        return BatchFeature(
+            data={"pixel_values": pixel_values, "image_grid_thw": image_grid_thw}, tensor_type=return_tensors
+        )
+
+    def get_number_of_image_patches(self, height: int, width: int, images_kwargs: dict | None = None) -> int:
+        """
+        A utility that returns number of image patches for a given image size.
+
+        Note: Do not remove this method! It is used by vLLM to infer the number of patches and placeholders
+        without an image input.
+
+        Args:
+            height (`int`):
+                Height of the input image.
+            width (`int`):
+                Width of the input image.
+            images_kwargs (`dict`, *optional*)
+                Any kwargs to override defaults of the image processor.
+        Returns:
+            `int`: Number of image patches per image.
+        """
+        # `size` is the supported override (`min_pixels` / `max_pixels` are deprecated): count with the budget the
+        # preprocessing applies
+        images_kwargs = dict(images_kwargs or {})
+        size = images_kwargs.pop("size", None) or self.size
+        images_kwargs.setdefault("min_pixels", size["shortest_edge"])
+        images_kwargs.setdefault("max_pixels", size["longest_edge"])
+        images_kwargs = images_kwargs or {}
+        min_pixels = images_kwargs["min_pixels"] if "min_pixels" in images_kwargs else self.size["shortest_edge"]
+        max_pixels = images_kwargs["max_pixels"] if "max_pixels" in images_kwargs else self.size["longest_edge"]
+        patch_size = images_kwargs.get("patch_size", self.patch_size)
+        merge_size = images_kwargs.get("merge_size", self.merge_size)
+
+        factor = patch_size * merge_size
+        resized_height, resized_width = smart_resize(
+            height, width, factor, min_pixels=min_pixels, max_pixels=max_pixels
+        )
+        grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+        return grid_h * grid_w
+
+
+__all__ = ["Lfm3VlImageProcessor"]

integrations_lfm3.py

@@ -0,0 +1,204 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/integrations/lfm3.py:
+# its relative imports made absolute, nothing else changed.
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Inference kernels for LFM3's width-7 depthwise convolution."""
+
+import torch
+
+from transformers.utils import is_triton_available
+
+
+if is_triton_available():
+    import triton
+    import triton.language as tl
+
+    # the length and strides are runtime arguments, or every prompt length would compile its own kernel
+    @triton.jit(do_not_specialize=["T", "X0", "X1", "X2", "S0", "S1", "ST"])
+    def _prefill(
+        X,
+        W,
+        BIAS,
+        SEQ,
+        Y,
+        T,
+        D: tl.constexpr,
+        X0,
+        X1,
+        X2,
+        S0,
+        S1,
+        ST,
+        HAS_BIAS: tl.constexpr,
+        PACKED: tl.constexpr,
+        BLOCK: tl.constexpr,
+    ):
+        # int64 offsets: batch * channels * length can exceed 2**31 elements
+        b, d = tl.program_id(0).to(tl.int64), tl.program_id(1).to(tl.int64)
+        t = tl.program_id(2).to(tl.int64) * BLOCK + tl.arange(0, BLOCK)
+        acc = tl.full((BLOCK,), 0, tl.float32)
+        if PACKED:
+            doc = tl.load(SEQ + b * S0 + (t - T + ST) * S1, (t < T) & (t >= T - ST), other=-1)
+        for k in tl.static_range(7):
+            pos = t + k - 6
+            valid = (t < T) & (pos >= 0)
+            if PACKED:
+                prev = tl.load(SEQ + b * S0 + (pos - T + ST) * S1, valid & (pos >= T - ST), other=-1)
+                valid = valid & (doc == prev)
+            x = tl.load(X + b * X0 + d * X1 + pos * X2, valid, other=0).to(tl.float32)
+            w = tl.load(W + d * 7 + k).to(tl.float32)
+            acc += x * w
+        if HAS_BIAS:
+            acc += tl.load(BIAS + d).to(tl.float32)
+        tl.store(Y + (b * D + d) * T + t, acc, t < T)
+
+    @triton.jit
+    def _decode(
+        X,
+        W,
+        BIAS,
+        STATE,
+        Y,
+        D: tl.constexpr,
+        K: tl.constexpr,
+        X0: tl.constexpr,
+        X1: tl.constexpr,
+        S0: tl.constexpr,
+        S1: tl.constexpr,
+        S2: tl.constexpr,
+        HAS_BIAS: tl.constexpr,
+        BLOCK: tl.constexpr,
+    ):
+        b = tl.program_id(0).to(tl.int64)
+        d = tl.program_id(1).to(tl.int64) * BLOCK + tl.arange(0, BLOCK)
+        k = tl.arange(0, 8)
+        # Load the complete state before writing the shifted window.
+        old = tl.load(STATE + b * S0 + d[:, None] * S1 + k[None, :] * S2, (d[:, None] < D) & (k[None, :] < K), other=0)
+        x = tl.load(X + b * X0 + d * X1, d < D, other=0)
+        shifted = tl.gather(old, tl.broadcast_to(tl.minimum(k + 1, 7)[None, :], (BLOCK, 8)), 1)
+        shifted = tl.where(k[None, :] == K - 1, x[:, None], shifted)
+        # A cache may store either K=6 previous inputs or K=7 including the oldest discarded input.
+        positions = k + K - 6
+        values = tl.gather(old, tl.broadcast_to(tl.minimum(positions, 7)[None, :], (BLOCK, 8)), 1)
+        values = tl.where(k[None, :] == 6, x[:, None], values)
+        weights = tl.load(W + d[:, None] * 7 + k[None, :], (d[:, None] < D) & (k[None, :] < 7), other=0)
+        acc = tl.sum(values.to(tl.float32) * weights.to(tl.float32), 1)
+        if HAS_BIAS:
+            acc += tl.load(BIAS + d, d < D, other=0).to(tl.float32)
+        tl.store(STATE + b * S0 + d[:, None] * S1 + k[None, :] * S2, shifted, (d[:, None] < D) & (k[None, :] < K))
+        tl.store(Y + b * D + d, acc, d < D)
+
+
+def can_use_lfm3_conv(hidden_states, weight):
+    return (
+        is_triton_available()
+        and hasattr(tl, "gather")  # the decode kernel needs Triton >= 3.3
+        and hidden_states.is_cuda
+        and not torch.is_grad_enabled()
+        and hidden_states.dtype == weight.dtype
+        and hidden_states.dtype in (torch.float16, torch.bfloat16, torch.float32)
+        and weight.shape[-1] == 7
+    )
+
+
+def lfm3_causal_conv1d(hidden_states, weight, bias=None, seq_idx=None):
+    batch, channels, length = hidden_states.shape
+    # the kernels index every tensor by the batch and channel of `hidden_states` and cannot bound the others
+    if weight.shape != (channels, 7) or (bias is not None and bias.shape != (channels,)):
+        raise ValueError(
+            f"`weight` must have shape ({channels}, 7) and `bias` ({channels},), got {tuple(weight.shape)}"
+        )
+    if seq_idx is not None and (
+        seq_idx.ndim != 2 or seq_idx.shape[0] not in (1, batch) or seq_idx.dtype not in (torch.int32, torch.int64)
+    ):
+        raise ValueError(
+            f"`seq_idx` must be an int32 or int64 tensor of shape (1, length) or ({batch}, length), got "
+            f"{seq_idx.dtype} {tuple(seq_idx.shape)}"
+        )
+    if any(t is not None and t.device != hidden_states.device for t in (weight, bias, seq_idx)):
+        raise ValueError(
+            f"`weight`, `bias` and `seq_idx` must be on the device of `hidden_states` ({hidden_states.device})"
+        )
+    output = torch.empty_like(hidden_states, memory_format=torch.contiguous_format)
+    # Triton launches on the current device; the model may live on another one (`device_map`, `.to("cuda:1")`).
+    with torch.cuda.device(hidden_states.device):
+        _prefill[(batch, channels, triton.cdiv(length, 128))](
+            hidden_states,
+            # the kernels index `bias` and `weight` densely
+            weight.contiguous(),
+            bias.contiguous() if bias is not None else None,
+            seq_idx,
+            output,
+            length,
+            channels,
+            *hidden_states.stride(),
+            *(
+                (0 if seq_idx.shape[0] == 1 else seq_idx.stride(0), seq_idx.stride(1))
+                if seq_idx is not None
+                else (0, 0)
+            ),
+            seq_idx.shape[-1] if seq_idx is not None else 0,
+            bias is not None,
+            seq_idx is not None,
+            128,
+            enable_fp_fusion=False,
+        )
+    return output
+
+
+def lfm3_causal_conv1d_update(hidden_states, conv_state, weight, bias=None):
+    batch, channels, length = hidden_states.shape
+    # the kernel addresses one position per row and bounds nothing by the other tensors' shapes
+    if length != 1:
+        raise ValueError(f"`hidden_states` must hold one position per row, got {tuple(hidden_states.shape)}")
+    # the 8-wide tile holds a 6- or 7-wide state (see `positions` in `_decode`)
+    if (
+        conv_state.ndim != 3
+        or conv_state.shape[:2] != (batch, channels)
+        or conv_state.shape[-1] not in (6, 7)
+        or weight.shape != (channels, 7)
+    ):
+        raise ValueError(
+            f"`conv_state` must have shape ({batch}, {channels}, 6 or 7) and `weight` ({channels}, 7), got "
+            f"{tuple(conv_state.shape)} and {tuple(weight.shape)}"
+        )
+    if bias is not None and bias.shape != (channels,):
+        raise ValueError(f"`bias` must have shape ({channels},), got {tuple(bias.shape)}")
+    # the kernel writes the state in place: rows or channels sharing memory (an `expand`ed cache) would race
+    extent = 0
+    for stride, size in sorted(zip(conv_state.stride(), conv_state.shape)):
+        if size > 1 and stride <= extent:
+            raise ValueError(f"`conv_state` must not overlap itself, got strides {conv_state.stride()}")
+        extent += stride * (size - 1)
+    if any(t is not None and t.device != hidden_states.device for t in (conv_state, weight, bias)):
+        raise ValueError(
+            f"`conv_state`, `weight` and `bias` must be on the device of `hidden_states` ({hidden_states.device})"
+        )
+    output = torch.empty_like(hidden_states, memory_format=torch.contiguous_format)
+    with torch.cuda.device(hidden_states.device):
+        _decode[(batch, triton.cdiv(channels, 32))](
+            hidden_states,
+            weight.contiguous(),
+            bias.contiguous() if bias is not None else None,
+            conv_state,
+            output,
+            channels,
+            conv_state.shape[-1],
+            *hidden_states.stride()[:2],
+            *conv_state.stride(),
+            bias is not None,
+            32,
+            enable_fp_fusion=False,
+        )
+    return output

json_spans.py

@@ -0,0 +1,59 @@
+"""Spans of a rendered JSON state: where its strings sit and what an escaped slice of one says, without torch."""
+
+from __future__ import annotations
+
+import json
+
+
+def string_paths(text: str) -> list[tuple[int, int, str]]:
+    """String-value spans in rendered JSON, excluding quotes; paths use RFC6901 escaping."""
+    decoder = json.JSONDecoder()
+    ranges = []
+
+    def skip(at):
+        while at < len(text) and text[at].isspace():
+            at += 1
+        return at
+
+    def value(at, path):
+        at = skip(at)
+        if text[at] == "{":
+            at = skip(at + 1)
+            while text[at] != "}":
+                key, end = decoder.raw_decode(text, at)
+                if not isinstance(key, str) or text[skip(end)] != ":":
+                    raise ValueError("invalid JSON object")
+                name = key.replace("~", "~0").replace("/", "~1")
+                at = skip(value(skip(end) + 1, path + "/" + name))
+                if text[at] == ",":
+                    at = skip(at + 1)
+                elif text[at] != "}":
+                    raise ValueError("invalid JSON object separator")
+            return at + 1
+        if text[at] == "[":
+            at, index = skip(at + 1), 0
+            while text[at] != "]":
+                at = skip(value(at, path + "/" + str(index)))
+                index += 1
+                if text[at] == ",":
+                    at = skip(at + 1)
+                elif text[at] != "]":
+                    raise ValueError("invalid JSON array separator")
+            return at + 1
+        item, end = decoder.raw_decode(text, at)
+        if isinstance(item, str):
+            ranges.append((at + 1, end - 1, path))
+        return end
+
+    end = value(0, "")
+    if skip(end) != len(text):
+        raise ValueError("trailing JSON text")
+    return ranges
+
+
+def json_text(escaped: str) -> str:
+    """A slice of a rendered JSON string as the characters it encodes (unchanged when it cuts an escape)."""
+    try:
+        return json.loads(f'"{escaped}"')
+    except ValueError:
+        return escaped

lfm3.py

@@ -0,0 +1,201 @@
+"""LFM3: text (`lfm3`), vision-language (`lfm3_vl`) and omni (`lfm3_omni`) checkpoints on transformers' classes
+(`native_lfm3*`), loaded as their Hub class (`modeling_d1_lfm3*.D1Model`) with d1's attention. d1 adds only:
+
+- the masks: `attention.d1_attention` under `attn_implementation="d1"`, its mask function registered;
+- the short convolution's torch path as `depthwise.depthwise` (ROCm: no MIOpen search), taken with autograd and
+  under torch.compile (`lfm3_conv`); eager inference keeps the classes' fused kernel; the audio encoder's
+  convolutions off MIOpen on ROCm (`lfm3_conv`);
+- the media encoders `SystemOne.compile` compiles beside the language model (`compiled_parts`), and its inductor
+  options (`compile_options`); the vision tower in one graph for every picture size (`lfm3_tower`).
+"""
+
+from __future__ import annotations
+
+import functools
+import importlib
+import importlib.util
+import sys
+
+import torch
+import torch.nn.functional as F
+
+from .depthwise import depthwise, pinned_conv
+from .modeling_d1 import SystemOneModel
+from .prompt import LEADS
+
+MODULES = {"lfm3": "modeling_d1_lfm3", "lfm3_vl": "modeling_d1_lfm3_vl", "lfm3_omni": "modeling_d1_lfm3_omni"}
+# the templates open a reasoning block; a non-thinking reply follows it closed at once
+LEADS.update(dict.fromkeys(MODULES, "<think></think>"))
+
+
+class Lfm3SystemOne(SystemOneModel):
+    """What every LFM3 Hub class adds to transformers': d1's attention and its convolution."""
+
+    attention = "d1"
+    compiled_parts: tuple[str, ...] = ()  # the media encoders `SystemOne.compile` compiles beside the language model
+    # compiled for inference with the weights as constants (inductor's freezing): over the fused projections
+    # (`fuse_projections`) d1-2B on an MI325X reads 3 questions in 5.5 against 6.2 ms, a 5k-token state in 41.8
+    # against 43.3, 64 packed states at 1,427 against 1,390 a second, at the same peak memory (3.65 GiB)
+    compile_options = {"freezing": True}
+    # the patch grids stay on the host: the tower and the image features read their sizes there, without a sync
+    host_inputs = ("image_grid_thw", "video_grid_thw")
+
+    def __init__(self, config):
+        super().__init__(config)
+        lfm3_conv(self)
+        lfm3_tower(self)
+
+    def fuse_projections(self) -> None:
+        """For inference (`SystemOne`): the linears that read one input as one matmul (`fuse_projections`)."""
+        fuse_projections(self)
+
+
+def lfm3_conv(model) -> None:
+    """Each LFM3 short convolution's torch path (`Lfm3ShortConv._conv`) as `depthwise`, and that path under
+    torch.compile: inductor fuses its shifted sums with the gates around them, where the classes' fused kernel is a
+    call of its own (d1-2B on an MI325X, compiled: 49.0 against 43.2 ms on a 5k-token state, 39.2 against 36.0 ms
+    for 64 short states). Eager inference keeps the kernel (10.2 against 11.3 ms a question).
+
+    The omni's audio encoder, off MIOpen on ROCm as well: its LFM2 convolutions (transformers' `Lfm2ShortConv`, which
+    calls its modeling module's `causal_conv1d_fn`, `F.conv1d` without causal-conv1d) as `depthwise`, its downsampler's
+    as `pinned_conv`. MIOpen's solver for the downsampler's 1x1 `lin_out` changed from one call to the next, and a
+    clip's P(yes) by 3.1e-03 (d1-omni-2B on an MI325X)."""
+    for m in model.modules():
+        module = sys.modules[type(m).__module__]
+        if type(m).__name__ == "Lfm3ShortConv":
+            m._conv = functools.partial(_causal_depthwise, m.conv)
+            if not getattr(module.can_use_lfm3_conv, "eager_only", False):
+                module.can_use_lfm3_conv = _eager_only(module.can_use_lfm3_conv)
+        elif type(m).__name__ == "Lfm2ShortConv" and importlib.util.find_spec("causal_conv1d") is None:
+            module.causal_conv1d_fn = _lfm2_conv
+        elif type(m).__name__ == "Lfm3OmniDownsampler":
+            for conv in m.children():
+                if isinstance(conv, torch.nn.modules.conv._ConvNd):
+                    conv.forward = functools.partial(pinned_conv, conv)
+
+
+def lfm3_tower(model) -> None:
+    """The vision tower under torch.compile, one graph for every picture size and no graph break:
+
+    - what its forward derives from the patch grids (positions, `cu_seqlens`, the frames' sizes: a sync and a graph
+      break each), made before it on the host once a grid, the way transformers' `vision_utils` takes them
+      precomputed; the frames' sizes come in as the grid itself, a list (`_frame_shapes`);
+    - its `len(set(frame sizes)) == 1` (one batch for frames of one size) compared by `==`: hashing the sizes makes
+      torch.compile guard on their values, a recompile for every picture size (about 10 s on an MI325X).
+
+    Delete both module patches when transformers' tower compares the sizes and takes the frames' sizes precomputed."""
+    for m in model.modules():
+        if type(m).__name__ == "Lfm3VlVisionModel":
+            m.register_forward_pre_hook(_grid_inputs, with_kwargs=True)
+            module = sys.modules[type(m).__module__]
+            module.set, module.get_vision_frame_shapes = _distinct, _frame_shapes
+
+
+def _grid_inputs(tower, args: tuple, kwargs: dict) -> tuple[tuple, dict]:
+    """The grid's positions, `cu_seqlens`, longest frame and frame sizes, made once a grid."""
+    from transformers.vision_utils import get_max_seqlen, get_vision_cu_seqlens, get_vision_position_ids
+
+    args = list(args)
+    grid = (args.pop(1) if len(args) > 1 else kwargs["grid_thw"]).cpu()  # on the host already when served
+    device = (args[0] if args else kwargs["pixel_values"]).device
+    made = tower.__dict__.setdefault("d1_grid_inputs", {})
+    key = (tuple(grid.flatten().tolist()), device)
+    if key not in made:
+        if len(made) >= 256:
+            made.clear()
+        cu_seqlens = get_vision_cu_seqlens(grid)
+        made[key] = {"cu_seqlens": cu_seqlens.to(device), "max_seqlen": get_max_seqlen(cu_seqlens, tower.config),
+                     "position_ids": get_vision_position_ids(grid, tower.spatial_merge_size).to(device),
+                     "grid_thw": _grid_frames(grid)}
+    kwargs.update(made[key])
+    return tuple(args), kwargs
+
+
+def _frame_shapes(grid) -> list[list[int]]:
+    """`get_vision_frame_shapes`: the grid `_grid_inputs` passes is its frames' sizes already; a tower without the
+    hook (transformers' own model) passes the tensor."""
+    return grid if isinstance(grid, list) else _grid_frames(grid)
+
+
+def _grid_frames(grid: torch.Tensor) -> list[list[int]]:
+    return grid[:, 1:].repeat_interleave(grid[:, 0], dim=0).tolist()
+
+
+# the linears each LFM3 module calls on one input, in the order its forward calls them
+SHARED_INPUT = {"Lfm3Attention": ("q_proj", "k_proj", "v_proj", "gate_proj"), "Lfm3MLP": ("w1", "w3"),
+                "Lfm3VlVisionAttention": ("q_proj", "k_proj", "v_proj"), "Lfm3VlMLP": ("w1", "w3")}
+
+
+def fuse_projections(model) -> None:
+    """Each module's linears on one input (`SHARED_INPUT`) as one matmul, its output split: the first's call computes
+    every part, the others read theirs, the last drops them. Exact, one weight copy, the classes untouched, the
+    state dict the native one (a fused model saves as it loaded). For inference only: training keeps the separate
+    linears, and a group with any layer not a plain linear stays."""
+    for m in model.modules():
+        names = SHARED_INPUT.get(type(m).__name__, ())
+        linears = [getattr(m, n, None) for n in names]
+        if names and all(type(lin) is torch.nn.Linear and lin.bias is None for lin in linears):
+            m.fused = Fused(torch.cat([lin.weight for lin in linears]), names, [lin.out_features for lin in linears])
+            for i, name in enumerate(names):
+                setattr(m, name, Part(m.fused, i))
+            m.register_state_dict_post_hook(_unfused)
+
+
+def _unfused(module, state_dict, prefix, local_metadata) -> None:
+    fused = module.fused
+    weight = state_dict.pop(prefix + "fused.weight")
+    for name, part in zip(fused.names, weight.split(fused.sizes)):
+        state_dict[f"{prefix}{name}.weight"] = part.clone()
+
+
+class Fused(torch.nn.Module):
+    def __init__(self, weight: torch.Tensor, names: tuple[str, ...], sizes: list[int]):
+        super().__init__()
+        self.weight = torch.nn.Parameter(weight, requires_grad=False)
+        self.names, self.sizes, self.parts = names, sizes, None
+
+
+class Part(torch.nn.Module):
+    """One linear of a `Fused` group: the first part computes the group's matmul."""
+
+    def __init__(self, fused: Fused, index: int):
+        super().__init__()
+        self.group, self.index, self.last = [fused], index, index == len(fused.sizes) - 1  # a list: not a submodule
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        fused = self.group[0]
+        if self.index == 0:
+            fused.parts = F.linear(x, fused.weight).split(fused.sizes, dim=-1)
+        out = fused.parts[self.index]
+        if self.last:
+            fused.parts = None
+        return out
+
+
+def _distinct(items) -> list:
+    """`set(items)`'s members by `==`, without hashing them."""
+    seen = []
+    for item in items:
+        if all(item != other for other in seen):
+            seen.append(item)
+    return seen
+
+
+def _eager_only(kernel):
+    def can_use(hidden_states, weight) -> bool:
+        return not torch.compiler.is_compiling() and kernel(hidden_states, weight)
+
+    can_use.eager_only = True
+    return can_use
+
+
+def _causal_depthwise(conv: torch.nn.Conv1d, x: torch.Tensor) -> torch.Tensor:
+    """`conv` over `(B, C, T)`, causal: what `Lfm3ShortConv._conv` computes."""
+    return _lfm2_conv(x, conv.weight[:, 0], conv.bias)
+
+
+def _lfm2_conv(x: torch.Tensor, weight: torch.Tensor, bias=None, seq_idx=None) -> torch.Tensor:
+    """transformers' `causal_conv1d_fn` as `Lfm2ShortConv` calls it: `weight` `(C, k)`, no activation, `seq_idx`
+    unused, as in transformers' own torch path."""
+    out = depthwise(F.pad(x.to(weight.dtype), (weight.shape[-1] - 1, 0)), weight[:, None])
+    return (out if bias is None else out + bias[:, None]).to(x.dtype)

media.py

@@ -0,0 +1,252 @@
+"""Media: pictures, audio clips and videos, each placed in a prompt as its checkpoint's processor expands it.
+
+    media=[picture, {"type": "audio", "path": "call.wav"}, {"type": "video", "path": "clip.mp4"}]
+    {"type": "noul", "instructions": {"type": "audio", "path": "question.wav"}}   # a spoken question
+
+- An item is a PIL image (a picture), or `{"type": "image" | "audio" | "video", "path": ...}`, the decoded value under
+  its type instead of a path (`{"type": "audio", "audio": waveform, "sampling_rate": 16000}`, a PIL image, a video's
+  frames with their `fps`). A spoken state is `state=None` with its clip in `media`.
+- A request's media head its state, in order, as the chat template writes them; a question's spoken instructions sit
+  where its text would. Each item's markup is the processor's own expansion, so a prompt is plain text and is packed,
+  read both ways and pointed into as a text prompt is (`Reader`).
+- Pictures are capped at the checkpoint's pixel budget first (`cap_pixels`); audio is read as 16-bit PCM WAV or a
+  NumPy file and resampled to the feature extractor's rate; a decoded stereo waveform may be (channels, samples) or
+  (samples, channels), as soundfile returns it, and its channels are averaged.
+"""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+KINDS = ("image", "audio", "video")
+PROCESSORS = {"image": "image_processor", "audio": "feature_extractor", "video": "video_processor"}
+INPUTS = {"image": "images", "audio": "audio", "video": "videos"}  # the processor's argument for each kind
+SPOKEN = ""  # a spoken question's place in its suffix until its markup replaces it (`Reader.place`)
+
+
+@dataclass(frozen=True, eq=False)
+class Media:
+    """One picture, audio clip or video: a path, or its decoded value (`sampling_rate`: a waveform's, `fps`: a video's
+    frames')."""
+
+    type: str
+    value: Any
+    sampling_rate: int | None = None
+    fps: float | None = None
+
+
+def as_media(item: Any) -> Media:
+    """A caller's item (a PIL image, or a dict with its `type`), as `Media`; `Media` passes through."""
+    if isinstance(item, Media):
+        return item
+    if isinstance(item, Mapping):
+        kind = item.get("type")
+        if kind not in KINDS or kind not in item and "path" not in item:
+            raise ValueError(f"a media item is a picture or {{'type': one of {KINDS}, 'path' or <type>: its value}}")
+        return Media(kind, item[kind] if kind in item else item["path"], item.get("sampling_rate"), item.get("fps"))
+    if hasattr(item, "convert") and hasattr(item, "size"):  # a PIL image
+        return Media("image", item)
+    raise TypeError(f"a media item is a picture or a dict with its type, not {type(item).__name__}")
+
+
+def media_list(items: Sequence | None) -> list[Media]:
+    return [as_media(x) for x in items or ()]
+
+
+def spoken(q) -> Media | None:
+    """A question's spoken instructions, or None for written ones."""
+    return q.instructions if isinstance(q.instructions, Media) else None
+
+
+def cap_pixels(image, max_pixels: int):
+    """A picture downscaled to at most `max_pixels`, bicubic."""
+    image = image.convert("RGB") if hasattr(image, "convert") else image
+    w, h = image.size
+    if w * h <= max_pixels:
+        return image
+    from PIL import Image
+
+    scale = math.sqrt(max_pixels / (w * h))
+    return image.resize((max(1, int(w * scale)), max(1, int(h * scale))), Image.Resampling.BICUBIC)
+
+
+class Reader:
+    """One checkpoint's media: each item's markup and processor outputs (`encode`), and the model's inputs for a
+    sequence holding several (`inputs`)."""
+
+    def __init__(self, processor, max_pixels: int):
+        self.processor, self.max_pixels = processor, max_pixels
+        self._texts: dict[tuple[int, ...], str] = {}  # an expansion's ids, checked to read back, as text
+
+    def encode(self, items: Sequence[Media]) -> tuple[str, list[dict]]:
+        """The items' markup back to back, each the template's with its placeholder expanded by the processor, and
+        each item's processor outputs."""
+        encoded = [self._encode(m) for m in items]
+        return "".join(text for text, _ in encoded), [out for _, out in encoded]
+
+    def place(self, suffix: str, q) -> tuple[str, list[dict]]:
+        """A question's suffix with its spoken instructions' markup in place of `SPOKEN`, and their outputs."""
+        voice = spoken(q)
+        if voice is None:
+            return suffix, []
+        markup, parts = self.encode([voice])
+        return suffix.replace(SPOKEN, markup), parts
+
+    def inputs(self, parts: Sequence[dict], ids: Sequence[int]) -> dict:
+        """The model's media inputs for a sequence `ids` holding the items `parts` came from, in this order: their
+        processor outputs joined (`join`), and the per-token marks a processor returns (`mm_token_type_ids`) over
+        `ids`."""
+        out = join([{k: v for k, v in p.items() if k not in ("input_ids", "attention_mask", "mm_token_type_ids")}
+                    for p in parts])
+        if any("mm_token_type_ids" in p for p in parts):
+            out["mm_token_type_ids"] = self._marks(ids)
+        if "pixel_attention_mask" in out:  # pictures padded to a fixed count of patches: cut to the real ones
+            n = int(out["pixel_attention_mask"].sum(1).max())
+            out["pixel_values"], out["pixel_attention_mask"] = out["pixel_values"][:, :n], out["pixel_attention_mask"][:, :n]
+        return out
+
+    def batch(self, inputs: Sequence[dict], ids) -> dict:
+        """Several sequences' inputs (`inputs`, in row order) as one forward over their right-padded rows `ids`
+        [B, T] takes them: `join`ed, and the marks over every row when a processor returns them."""
+        out = join([{k: v for k, v in x.items() if k != "mm_token_type_ids"} for x in inputs])
+        if any("mm_token_type_ids" in x for x in inputs):
+            out["mm_token_type_ids"] = self._marks(ids)
+        return out
+
+    def _marks(self, ids):
+        """Which tokens hold a picture (1) or a video (2), as the processors that return such marks write them: over
+        one sequence's ids, [1, T], or a batch's rows, [B, T]."""
+        import torch
+
+        x = torch.as_tensor(ids).cpu()
+        x = x[None] if x.dim() == 1 else x
+        marks = torch.zeros_like(x)
+        for value, name in ((1, "image_token_id"), (2, "video_token_id")):
+            if (token := getattr(self.processor, name, None)) is not None:
+                marks[x == token] = value
+        return marks
+
+    def _encode(self, item: Media) -> tuple[str, dict]:
+        """One item's expanded markup, as text that encodes back to the processor's ids, and its outputs."""
+        if getattr(self.processor, PROCESSORS[item.type], None) is None:
+            raise ValueError(f"this checkpoint takes no {item.type}")
+        template = template_markup(self.processor, item.type)
+        rate = {"sampling_rate": self.processor.feature_extractor.sampling_rate} if item.type == "audio" else {}
+        if item.type == "video" and not isinstance(item.value, str):  # a path is decoded with its own clock
+            if item.fps is None:
+                raise ValueError("a video's frames need their fps")
+            rate = {"video_metadata": [{"fps": item.fps, "total_num_frames": len(item.value)}]}
+        out = self.processor(text=[template], **{INPUTS[item.type]: [[self._decoded(item)]]}, **rate,
+                             return_tensors="pt", add_special_tokens=False)
+        ids = tuple(out["input_ids"][0].tolist())
+        if ids not in self._texts:  # one check an expansion (a picture size, a clip length), not one a call
+            text = self.processor.tokenizer.decode(ids)
+            if tuple(self.processor.tokenizer.encode(text, add_special_tokens=False)) != ids:
+                raise ValueError(f"{type(self.processor).__name__}'s {item.type} markup does not read back as text")
+            if len(self._texts) >= 4096:
+                self._texts.clear()
+            self._texts[ids] = text
+        return self._texts[ids], dict(out)
+
+    def _decoded(self, item: Media):
+        """What the processor takes: a capped RGB picture, a waveform at the feature extractor's rate, a video."""
+        if item.type == "image":
+            from PIL import Image
+
+            image = Image.open(item.value) if isinstance(item.value, str) else item.value
+            return cap_pixels(image, self.max_pixels)
+        if item.type == "audio":
+            rate = self.processor.feature_extractor.sampling_rate
+            wave, given = read_audio(item.value, rate) if isinstance(item.value, str) else (item.value, item.sampling_rate)
+            if given is None:
+                raise ValueError("a waveform needs its sampling_rate")
+            return resampled(wave, given, rate)
+        return item.value
+
+
+def read_audio(path: str, rate: int):
+    """A clip's waveform and its rate: 16-bit PCM WAV, or a NumPy file (int16 or float, at `rate`), with the standard
+    library and NumPy; any other format through transformers' `load_audio` (torchcodec or librosa), at `rate`."""
+    import numpy as np
+
+    if path.endswith(".wav"):
+        import wave
+
+        with wave.open(path, "rb") as f:
+            if f.getsampwidth() != 2:
+                raise ValueError(f"{path}: 16-bit PCM WAV only")
+            pcm = np.frombuffer(f.readframes(f.getnframes()), dtype="<i2").reshape(-1, f.getnchannels())
+            return pcm.T.astype(np.float32) / 32768, f.getframerate()
+    if path.endswith(".npy"):
+        data = np.load(path)
+        return (data.astype(np.float32) / 32768 if data.dtype == np.int16 else data.astype(np.float32)), rate
+    from transformers.audio_utils import load_audio
+
+    return load_audio(path, sampling_rate=rate), rate
+
+
+def resampled(wave, rate: int, target: int, zeros: int = 6, rolloff: float = 0.99):
+    """A mono float32 waveform at `target` (channels averaged), resampled in torch as torchaudio's
+    `functional.resample` does by default: a Hann-windowed sinc over `zeros` zero crossings, cut at `rolloff` of the
+    lower Nyquist, one filter per output phase (no torchaudio: a Hub repo needs no package it does not name)."""
+    import numpy as np
+
+    wave = np.asarray(wave, dtype=np.float32)
+    if wave.ndim == 2:  # (channels, samples) or (samples, channels): a clip never has more channels than samples
+        wave = (wave.T if wave.shape[0] > wave.shape[1] else wave).mean(0)
+    if rate == target:
+        return wave
+    import torch
+
+    gcd = math.gcd(int(rate), int(target))
+    old, new = int(rate) // gcd, int(target) // gcd
+    cut = min(old, new) * rolloff
+    width = math.ceil(zeros * old / cut)
+    t = torch.arange(0, -new, -1, dtype=torch.float32)[:, None, None] / new
+    t = ((t + torch.arange(-width, width + old, dtype=torch.float32)[None, None] / old) * cut).clamp(-zeros, zeros)
+    window = torch.cos(t * math.pi / zeros / 2) ** 2
+    t = t * math.pi
+    kernel = torch.where(t == 0, torch.tensor(1.0), t.sin() / t) * (window * (cut / old))
+    x = torch.nn.functional.pad(torch.from_numpy(wave)[None, None], (width, width + old))
+    out = torch.nn.functional.conv1d(x, kernel, stride=old).transpose(1, 2).reshape(-1)
+    return out[:math.ceil(new * len(wave) / old)].numpy()
+
+
+def template_markup(processor, kind: str) -> str:
+    """One item's markup as the processor's chat template writes it at the head of a user turn."""
+    messages = [{"role": "user", "content": [{"type": kind}, {"type": "text", "text": "\x00"}]}]
+    text = processor.apply_chat_template(messages, add_generation_prompt=False, tokenize=False)
+    head = "<|im_start|>user\n"
+    return text[text.index(head) + len(head):text.index("\x00")]
+
+
+def join(inputs: Sequence[dict]) -> dict:
+    """Media inputs of several items, or of several sequences in row order, as one forward takes them: each modality's
+    tensors joined on their first axis, padded with zeros to the largest along the others (pictures' patches and
+    their mask, clips' mel frames and theirs: `feature_attention_mask`), lists concatenated. A forward fills each
+    modality's placeholders in this order, so items of every kind share it."""
+    import torch
+
+    out = {}
+    for key in dict.fromkeys(k for i in inputs for k in i):
+        values = [i[key] for i in inputs if key in i]
+        out[key] = _join(values) if torch.is_tensor(values[0]) else [v for value in values for v in value]
+    return out
+
+
+def _join(values: list):
+    """Tensors joined on their first axis, each padded with zeros to the largest along the others."""
+    import torch
+
+    if all(v.shape[1:] == values[0].shape[1:] for v in values):
+        return torch.cat(values)
+    size = [max(v.shape[d] for v in values) for d in range(1, values[0].dim())]
+    out = values[0].new_zeros(sum(v.shape[0] for v in values), *size)
+    at = 0
+    for v in values:
+        out[(slice(at, at + v.shape[0]), *(slice(0, n) for n in v.shape[1:]))] = v
+        at += v.shape[0]
+    return out

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5e573687315e82ea80f4f7dd8d76b5cec3b6ee97e3509fd0493cfebb203719f
+size 3836417920

modeling_d1.py

@@ -0,0 +1,73 @@
+"""The System One API on any backbone: each `modeling_d1_<model_type>.py` serves `D1Model(SystemOneModel, <its
+class>)` to `AutoModel.from_pretrained(repo, trust_remote_code=True)`.
+
+    model.system_one(state, {name: question}, media=None)     # one forward pass; media: pictures, audio, video
+    model.system_one_batch([(state, {name: question}), ...])  # many states in one call
+
+A release's `calibration.json` (its temperature) is applied to every answer.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping, Sequence
+from functools import cached_property
+from typing import Any
+
+from .api import SCORE_READOUT
+from .runner import SystemOne
+
+
+class SystemOneModel:
+    """The System One API over any backbone the runner reads. `attention`: the attention function a transformers
+    backbone loads with (`attention.d1_attention`), when it has one."""
+
+    attention: str | None = None
+
+    @classmethod
+    def from_pretrained(cls, *args, revision=None, **kwargs):
+        if cls.attention:
+            kwargs["attn_implementation"] = cls.attention
+        loaded = super().from_pretrained(*args, revision=revision, **kwargs)  # with its loading info when asked
+        # transformers 5.19 resolves the revision to a commit for the load and keeps no `_commit_hash`
+        (loaded[0] if isinstance(loaded, tuple) else loaded).d1_revision = revision
+        return loaded
+
+    def d1_file(self, name: str, required: bool = True) -> str | None:
+        """A file of the loaded revision, not main's: its path, or None for a missing one not `required`."""
+        from transformers.utils import cached_file
+
+        return cached_file(self.name_or_path, name, revision=self._revision(),
+                           _raise_exceptions_for_missing_entries=required)
+
+    def _revision(self) -> str | None:
+        return getattr(self.config, "_commit_hash", None) or getattr(self, "d1_revision", None)
+
+    @cached_property
+    def engine(self) -> SystemOne:
+        """The runner over this model, with the loaded revision's tokenizer and temperature."""
+        from transformers import AutoTokenizer
+
+        from .calibration import Calibration
+
+        found = self.d1_file("calibration.json", required=False)
+        return SystemOne(model=self.eval(), tokenizer=AutoTokenizer.from_pretrained(self.name_or_path,
+                                                                                  revision=self._revision()),
+                         calibration=Calibration.load(found) if found else None, revision=self._revision())
+
+    def compile(self, mode: str | None = None) -> None:
+        """torch.compile what `system_one` runs (`SystemOne.compile`): fused kernels, with no CUDA graphs under `set` attention."""
+        self.engine.compile(mode)
+
+    def system_one(self, state: Any, questions: Mapping[str, Any], media: Sequence | None = None,
+                   score_readout: str = SCORE_READOUT, set_size: int | None = None) -> dict:
+        """Named questions over a state (text, JSON, or None with media alone) and its media (pictures, audio clips,
+        videos: `media`): `{"answers": {name: answer}, "usage": {"input_tokens": n, "output_tokens": 0}}`.
+        `score_readout`: a score's `median` level (the default), its expected level (`mean`) or its `argmax` level.
+        `set_size`: questions read together after the state, the model's trained size by default; larger is faster
+        and costs some accuracy on text."""
+        return self.engine.system_one(state, questions, media, score_readout, set_size=set_size)
+
+    def system_one_batch(self, requests: Sequence[tuple], score_readout: str = SCORE_READOUT,
+                         set_size: int | None = None) -> list[dict]:
+        """`(state, questions)` or `(state, questions, media)` requests, one response each."""
+        return self.engine.system_one_batch(requests, score_readout, set_size=set_size)

modeling_d1_lfm3_vl.py

@@ -0,0 +1,10 @@
+"""`AutoModel.from_pretrained(repo, trust_remote_code=True)`: LFM3-VL with the System One API (`modeling_d1`)."""
+
+from __future__ import annotations
+
+from .lfm3 import Lfm3SystemOne
+from .native_lfm3_vl import Lfm3VlForConditionalGeneration
+
+
+class D1Model(Lfm3SystemOne, Lfm3VlForConditionalGeneration):
+    compiled_parts = ("vision_tower", "multi_modal_projector")

modeling_lfm3.py

@@ -0,0 +1,881 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3/modeling_lfm3.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3/modular_lfm3.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import copy
+from collections.abc import Callable
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from transformers import initialization as init
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.configuration_utils import PreTrainedConfig
+from transformers.generation import GenerationMixin
+from transformers.integrations import use_kernel_forward_from_hub, use_kernel_func_from_hub_with_fallback, use_kernelized_func
+from transformers.integrations.accelerate import force_accelerate_hooks
+from .integrations_lfm3 import can_use_lfm3_conv, lfm3_causal_conv1d, lfm3_causal_conv1d_update
+from transformers.masking_utils import create_causal_mask, create_recurrent_attention_mask, create_sliding_window_causal_mask
+from transformers.modeling_layers import GradientCheckpointingLayer
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple, torch_compilable_check
+from transformers.utils.generic import merge_with_config_defaults
+from transformers.utils.import_utils import is_tracing
+from transformers.utils.output_capturing import capture_outputs
+from .configuration_lfm3 import Lfm3Config
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class Lfm3RMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps: float = 1e-6) -> None:
+        """
+        Lfm3RMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class Lfm3RotaryEmbedding(nn.Module):
+    """
+    Rotary embedding per attention layer type, like Gemma 3; `conv` layers register no buffers. Partial RoPE keeps the
+    frequency progression of the full head dimension and zeroes the tail instead of shrinking the rotated dimension.
+    """
+
+    def __init__(self, config: Lfm3Config, device=None):
+        super().__init__()
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+        self.config = config
+        self.layer_types = sorted(set(config.layer_types) & set(config.rope_parameters))
+        self.rope_type = {}
+        for layer_type in self.layer_types:
+            rope_params = self.config.rope_parameters[layer_type]
+            self.rope_type[layer_type] = rope_params["rope_type"]
+            curr_inv_freq, curr_attention_scaling = self.compute_rope_parameters(config, device, layer_type=layer_type)
+            setattr(self, f"{layer_type}_inv_freq", nn.Buffer(curr_inv_freq, persistent=False))
+            setattr(self, f"{layer_type}_original_inv_freq", nn.Buffer(curr_inv_freq.clone(), persistent=False))
+            setattr(self, f"{layer_type}_attention_scaling", curr_attention_scaling)
+
+    @staticmethod
+    def compute_default_rope_parameters(
+        config: Lfm3Config, device=None, layer_type: str | None = None, **kwargs
+    ) -> tuple[torch.Tensor, float]:
+        """
+        Inverse frequencies of the default RoPE with LFM3's zero-tail partial rotation: the first
+        `partial_rotary_factor * head_dim // 2` frequencies follow the progression of the full head dimension and the
+        rest are zero, leaving those dimensions unrotated.
+
+        Returns:
+            Tuple of (`torch.Tensor`, `float`): the inverse frequencies and the attention scaling factor (always 1).
+        """
+        base = config.rope_parameters[layer_type]["rope_theta"]
+        partial_rotary_factor = config.rope_parameters[layer_type].get("partial_rotary_factor", 1.0)
+        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
+        rope_angles = int(partial_rotary_factor * dim) // 2
+        inv_freq = 1.0 / (base ** (torch.arange(0, 2 * rope_angles, 2, dtype=torch.float) / dim))
+        inv_freq = torch.cat((inv_freq, torch.zeros(dim // 2 - rope_angles, dtype=torch.float)))
+        return inv_freq.to(device), 1.0
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids, layer_type):
+        inv_freq = getattr(self, f"{layer_type}_inv_freq")
+        attention_scaling = getattr(self, f"{layer_type}_attention_scaling")
+
+        freqs = position_ids[..., None].float() * inv_freq.to(device=x.device, dtype=torch.float)
+        emb = torch.cat((freqs, freqs), dim=-1)
+        cos = emb.cos() * attention_scaling
+        sin = emb.sin() * attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+    @classmethod
+    def compute_rope_parameters(
+        cls, config: Lfm3Config, device=None, layer_type: str | None = None
+    ) -> tuple[torch.Tensor, float]:
+        """Inverse frequencies of any RoPE type in the zero-tail layout: scaled types are evaluated over the full head
+        dimension and truncated to the rotated prefix."""
+        rope_params = config.rope_parameters[layer_type]
+        if rope_params["rope_type"] == "default":
+            return cls.compute_default_rope_parameters(config, device, layer_type=layer_type)
+        full_config = copy.copy(config)
+        full_config.rope_parameters = {
+            **config.rope_parameters,
+            layer_type: {**rope_params, "partial_rotary_factor": 1.0},
+        }
+        inv_freq, attention_factor = ROPE_INIT_FUNCTIONS[rope_params["rope_type"]](
+            full_config, device, layer_type=layer_type
+        )
+        dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
+        rope_angles = int(rope_params.get("partial_rotary_factor", 1.0) * dim) // 2
+        inv_freq = torch.cat((inv_freq[:rope_angles], inv_freq.new_zeros(dim // 2 - rope_angles)))
+        return inv_freq, attention_factor
+
+
+class Lfm3MLP(nn.Module):
+    def __init__(self, config: Lfm3Config):
+        super().__init__()
+        self.w1 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.w3 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.w2 = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def forward(self, x):
+        return self.w2(F.silu(self.w1(x)) * self.w3(x))
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+def apply_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
+    """Applies RoPE to the query and key tensors in float32, like the reference implementation."""
+    cos = cos.unsqueeze(unsqueeze_dim).float()
+    sin = sin.unsqueeze(unsqueeze_dim).float()
+    q_embed = (q.float() * cos) + (rotate_half(q.float()) * sin)
+    k_embed = (k.float() * cos) + (rotate_half(k.float()) * sin)
+    return q_embed.to(q.dtype), k_embed.to(k.dtype)
+
+
+@use_kernelized_func(apply_rotary_pos_emb)
+class Lfm3Attention(nn.Module):
+    """Llama attention with QK RMSNorm, a sigmoid output gate per head, a sliding window on `sliding_attention`
+    layers and RoPE in float32."""
+
+    def __init__(self, config: Lfm3Config, layer_idx: int):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.scaling = self.head_dim**-0.5
+        self.is_causal = True
+        self.q_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, config.num_key_value_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
+        self.q_norm = Lfm3RMSNorm(self.head_dim, eps=config.norm_eps)
+        self.k_norm = Lfm3RMSNorm(self.head_dim, eps=config.norm_eps)
+        self.gate_proj = nn.Linear(config.hidden_size, config.num_attention_heads, bias=False)
+        self.sliding_window = config.sliding_window if config.layer_types[layer_idx] == "sliding_attention" else None
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: torch.Tensor | None,
+        past_key_values: Cache | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_norm(self.q_proj(hidden_states).view(*hidden_shape)).transpose(1, 2)
+        key_states = self.k_norm(self.k_proj(hidden_states).view(*hidden_shape)).transpose(1, 2)
+        value_states = self.v_proj(hidden_states).view(*hidden_shape).transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
+
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            self.config._attn_implementation, eager_attention_forward
+        )
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0,
+            scaling=self.scaling,
+            sliding_window=self.sliding_window,
+            **kwargs,
+        )
+        gate = torch.sigmoid(self.gate_proj(hidden_states)).unsqueeze(-1)
+        attn_output = (attn_output * gate).reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+def apply_mask_to_padding_states(hidden_states, attention_mask):
+    """
+    Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66
+    """
+    # NOTE: attention mask is a 2D boolean tensor
+    if attention_mask is not None:
+        dtype = hidden_states.dtype
+        hidden_states = (hidden_states * attention_mask[:, :, None]).to(dtype)
+
+    return hidden_states
+
+
+@use_kernel_func_from_hub_with_fallback("causal_conv1d_update", "causal_conv1d")
+def causal_conv1d_update(
+    hidden_states: torch.Tensor,
+    conv_state: torch.Tensor,
+    weight: nn.Parameter,
+    bias: nn.Parameter | None = None,
+    activation: str | None = None,
+):
+    _, hidden_size, seq_len = hidden_states.shape
+    state_len = conv_state.shape[-1]
+
+    hidden_states_new = torch.cat([conv_state, hidden_states], dim=-1).to(weight.dtype)
+    conv_state.copy_(hidden_states_new[:, :, -state_len:])
+    out = F.conv1d(hidden_states_new, weight.unsqueeze(1), bias, padding=0, groups=hidden_size)
+    out = out[:, :, -seq_len:]
+    if activation is not None:
+        out = ACT2FN[activation](out)
+    return out.to(hidden_states.dtype)
+
+
+@use_kernel_func_from_hub_with_fallback("causal_conv1d_fn", "causal_conv1d")
+def causal_conv1d_fn(
+    hidden_states: torch.Tensor,
+    weight: nn.Parameter,
+    bias: nn.Parameter | None = None,
+    activation: str | None = None,
+    **kwargs,
+):
+    _, hidden_size, seq_len = hidden_states.shape
+    padding = weight.shape[-1] - 1
+
+    out = F.conv1d(
+        hidden_states.to(weight.dtype),
+        weight=weight.unsqueeze(1),
+        bias=bias,
+        padding=padding,
+        groups=hidden_size,
+    )[:, :, :seq_len]
+    if activation is not None:
+        out = ACT2FN[activation](out)
+    return out.to(hidden_states.dtype)
+
+
+@use_kernelized_func([causal_conv1d_update, causal_conv1d_fn])
+class Lfm3ShortConv(nn.Module):
+    """LFM2 gated short convolution with packed-document boundaries (`seq_idx`), the padding mask re-applied after a
+    biased `in_proj`, and width-7 Triton inference kernels with a PyTorch fallback."""
+
+    def __init__(
+        self,
+        config: Lfm3Config,
+        layer_idx: int,
+    ):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.conv_kernel_size = config.conv_L_cache
+        self.bias = config.conv_bias
+
+        self.conv = nn.Conv1d(
+            in_channels=config.hidden_size,
+            out_channels=config.hidden_size,
+            kernel_size=self.conv_kernel_size,
+            groups=config.hidden_size,
+            bias=self.bias,
+            padding=self.conv_kernel_size - 1,
+        )
+        self.in_proj = nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=self.bias)
+        self.out_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=self.bias)
+
+        self.layer_type = config.layer_types[layer_idx]
+
+    @force_accelerate_hooks("conv")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        past_key_values: Cache | None = None,
+        attention_mask: torch.Tensor | None = None,
+        seq_idx: torch.IntTensor | None = None,
+    ):
+        seq_len = hidden_states.shape[1]
+
+        hidden_states = apply_mask_to_padding_states(hidden_states, attention_mask)
+        BCx = self.in_proj(hidden_states).transpose(-1, -2)
+        B, C, x = BCx.chunk(3, dim=-2)
+        hidden_states = B * x
+        if attention_mask is not None:
+            # with `conv_bias` the projection makes the padded positions non-zero again
+            hidden_states = hidden_states * attention_mask[:, None, :].to(hidden_states.dtype)
+
+        use_precomputed_states = past_key_values is not None and past_key_values.has_previous_state(self.layer_idx)
+        use_kernels = 2 <= self.conv_kernel_size <= 4
+        use_wide_kernel = can_use_lfm3_conv(hidden_states, self.conv.weight)
+
+        if use_precomputed_states and seq_len == 1 and not past_key_values.layers[self.layer_idx].record_past:
+            conv_state = past_key_values.layers[self.layer_idx].conv_states[0]
+            if use_wide_kernel and conv_state.shape[-1] in (6, 7):
+                hidden_states = lfm3_causal_conv1d_update(
+                    hidden_states, conv_state, self.conv.weight.squeeze(1), self.conv.bias
+                )
+            elif use_kernels:
+                hidden_states = causal_conv1d_update(
+                    hidden_states, conv_state, self.conv.weight.squeeze(1), self.conv.bias
+                )
+            else:
+                hidden_states = torch.cat([conv_state, hidden_states], dim=-1)
+                conv_state.copy_(hidden_states[:, :, -conv_state.shape[-1] :])
+                hidden_states = self._conv(hidden_states)[:, :, -seq_len:]
+        else:
+            if past_key_values is not None and seq_idx is not None:
+                # `seq_idx` is only set on an empty cache, and the decode steps read the cached window without it:
+                # only the last document (after any positional left padding) may enter the window
+                last_document = (seq_idx == seq_idx[:, -1:])[:, None].to(hidden_states.dtype)
+                past_key_values.update_conv_state(
+                    hidden_states * last_document, self.layer_idx, conv_kernel_size=self.conv_kernel_size
+                )
+            elif past_key_values is not None:
+                hidden_states = past_key_values.update_conv_state(
+                    hidden_states, self.layer_idx, conv_kernel_size=self.conv_kernel_size
+                )
+
+            if use_wide_kernel:
+                hidden_states = lfm3_causal_conv1d(hidden_states, self.conv.weight.squeeze(1), self.conv.bias, seq_idx)
+            elif seq_idx is not None:
+                hidden_states = self._segmented_conv(hidden_states, seq_idx)
+            elif use_kernels:
+                hidden_states = causal_conv1d_fn(hidden_states, self.conv.weight.squeeze(1), self.conv.bias)
+            else:
+                hidden_states = self._conv(hidden_states)
+
+            # Drop the additional previous states
+            if past_key_values is not None:
+                hidden_states = hidden_states[:, :, -seq_len:]
+
+        y = C * hidden_states
+        y = y.transpose(-1, -2).contiguous()
+        y = self.out_proj(y)
+        return y
+
+    def _conv(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        """Causal depthwise conv in torch: the `causal_conv1d` kernels only accept widths 2 to 4."""
+        conv_out = self.conv(hidden_states.to(self.conv.weight.dtype))[..., : hidden_states.shape[-1]]
+        return conv_out.to(hidden_states.dtype)
+
+    def _segmented_conv(self, hidden_states: torch.Tensor, seq_idx: torch.Tensor) -> torch.Tensor:
+        """Causal depthwise conv that only reads positions sharing the current token's document index."""
+        kernel_size = self.conv_kernel_size
+        seq_idx = F.pad(seq_idx, (hidden_states.shape[-1] - seq_idx.shape[-1] + kernel_size - 1, 0), value=-1)
+        windows = F.pad(hidden_states, (kernel_size - 1, 0)).unfold(-1, kernel_size, 1)
+        seq_idx_windows = seq_idx.unfold(-1, kernel_size, 1)
+        same_document = (seq_idx_windows == seq_idx_windows[..., -1:]).unsqueeze(1)
+        conv_out = torch.einsum("bdtk,dk->bdt", windows * same_document.to(windows.dtype), self.conv.weight.squeeze(1))
+        if self.bias:
+            conv_out = conv_out + self.conv.bias[None, :, None]
+        return conv_out.to(hidden_states.dtype)
+
+
+class Lfm3DecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: Lfm3Config, layer_idx: int):
+        super().__init__()
+        self.is_attention_layer = config.layer_types[layer_idx] in config._attention_layer_types
+        self.use_sandwich_norm = config.use_sandwich_norm
+
+        if self.is_attention_layer:
+            self.self_attn = Lfm3Attention(config, layer_idx)
+        else:
+            self.conv = Lfm3ShortConv(config, layer_idx)
+        self.feed_forward = Lfm3MLP(config)
+        self.operator_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
+        self.ffn_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
+        if self.use_sandwich_norm:
+            self.post_operator_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
+            self.post_ffn_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        seq_idx: torch.IntTensor | None = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        residual = hidden_states
+        if self.is_attention_layer:
+            hidden_states, _ = self.self_attn(
+                hidden_states=self.operator_norm(hidden_states),
+                position_embeddings=position_embeddings,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                **kwargs,
+            )
+        else:
+            hidden_states = self.conv(
+                hidden_states=self.operator_norm(hidden_states),
+                past_key_values=past_key_values,
+                attention_mask=attention_mask,
+                seq_idx=seq_idx,
+            )
+        if self.use_sandwich_norm:
+            hidden_states = self.post_operator_norm(hidden_states)
+        hidden_states = hidden_states + residual
+
+        ffn_out = self.feed_forward(self.ffn_norm(hidden_states))
+        if self.use_sandwich_norm:
+            ffn_out = self.post_ffn_norm(ffn_out)
+        hidden_states = hidden_states + ffn_out
+
+        return hidden_states
+
+
+def create_masks_by_layer_type(
+    config: PreTrainedConfig,
+    inputs_embeds: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    past_key_values: Cache | None,
+    position_ids: torch.Tensor | None = None,
+) -> dict:
+    """The cache sizes each mask from the layer it is asked about, and the generic mask utilities ask about the first
+    non-sliding layer, which for a leading `conv` layer redirects to the first attention layer of any type: name the
+    first layer of each mask's own type instead."""
+    config = config.get_text_config()
+    mask_kwargs = {
+        "config": config,
+        "inputs_embeds": inputs_embeds,
+        "attention_mask": attention_mask,
+        "past_key_values": past_key_values,
+        "position_ids": position_ids,
+    }
+    layer_types, causal_mask_mapping = config.layer_types, {}
+    if "full_attention" in layer_types:
+        layer_idx = layer_types.index("full_attention")
+        causal_mask_mapping["full_attention"] = create_causal_mask(**mask_kwargs, layer_idx=layer_idx)
+    if "sliding_attention" in layer_types:
+        layer_idx = layer_types.index("sliding_attention")
+        causal_mask_mapping["sliding_attention"] = create_sliding_window_causal_mask(
+            **mask_kwargs, layer_idx=layer_idx
+        )
+    if "conv" in layer_types:
+        causal_mask_mapping["conv"] = create_recurrent_attention_mask(**mask_kwargs)
+    return causal_mask_mapping
+
+
+@auto_docstring
+class Lfm3PreTrainedModel(PreTrainedModel):
+    config: Lfm3Config
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["Lfm3DecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = True
+
+    _can_compile_fullgraph = True
+    _supports_attention_backend = True
+    _can_record_outputs = {
+        "hidden_states": Lfm3DecoderLayer,
+        "attentions": Lfm3Attention,
+    }
+
+    @torch.no_grad()
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, Lfm3RotaryEmbedding):
+            for layer_type in module.layer_types:
+                curr_inv_freq, _ = module.compute_rope_parameters(module.config, layer_type=layer_type)
+                init.copy_(getattr(module, f"{layer_type}_inv_freq"), curr_inv_freq)
+                init.copy_(getattr(module, f"{layer_type}_original_inv_freq"), curr_inv_freq)
+
+    @staticmethod
+    def create_masks_for_generate(
+        config: PreTrainedConfig,
+        inputs_embeds: torch.Tensor,
+        attention_mask: torch.Tensor | None,
+        past_key_values: Cache | None,
+        position_ids: torch.Tensor | None = None,
+        **kwargs,
+    ) -> dict:
+        return create_masks_by_layer_type(config, inputs_embeds, attention_mask, past_key_values, position_ids)
+
+
+def packed_under_padding(position_ids: torch.Tensor, padding_mask: torch.Tensor, filled) -> torch.Tensor:
+    """Padding repeats position 0, so packing is a restart between two kept tokens, or a kept position 0 on a filled
+    cache whose row holds a kept token; attention under a padding mask cannot isolate it."""
+    restarts = torch.diff(position_ids, prepend=position_ids[:, :1] - 1, dim=-1) != 1
+    valid = padding_mask[:, -restarts.shape[-1] :].bool()
+    packed = restarts[:, 1:] & valid[:, 1:] & valid[:, :-1]
+    if filled:
+        # a row whose cached tokens are all padding (chunked prefill of a left-padded batch) has no document to pack
+        # against; a mask covering only the chunk keeps the strict rule
+        cached = padding_mask[:, : padding_mask.shape[-1] - restarts.shape[-1]]
+        cached_valid = cached.bool().any(-1, keepdim=True) if cached.shape[-1] else torch.ones_like(valid[:, :1])
+        packed = packed | (valid[:, :1] & (position_ids[:, :1] == 0) & cached_valid)
+    return packed
+
+
+@auto_docstring
+class Lfm3Model(Lfm3PreTrainedModel):
+    """LFM2 model with sliding-attention layers as a third layer type: the mask and the rotary embedding are built per
+    layer type, and packed documents (`position_ids` restarting) reset the short-conv window through `seq_idx`."""
+
+    def __init__(self, config: Lfm3Config):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList(
+            [Lfm3DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.rotary_emb = Lfm3RotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+        self.embedding_norm = Lfm3RMSNorm(config.hidden_size, eps=config.norm_eps)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @merge_with_config_defaults
+    @capture_outputs
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        use_cache: bool | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> BaseModelOutputWithPast:
+        r"""
+        attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)` or `dict`, *optional*):
+            Mask to avoid performing attention on padding token indices (1 keeps a token, 0 masks it), or the mask
+            dict of `create_masks_for_generate`. A mask dict is used as given: `generate` refuses packed
+            `position_ids` before its prefill.
+        """
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+        # `seq_idx` is a shared kwarg (`DataCollatorWithFlattening(return_seq_idx=True)`, read by Lfm2); LFM3 derives
+        # the documents from the restarting `position_ids` that come with it
+        if kwargs.pop("seq_idx", None) is not None and position_ids is None:
+            raise ValueError("Pass packed documents as `position_ids` restarting at 0; LFM3 does not read `seq_idx`")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+        filled = past_key_values is not None and (
+            past_key_values.get_seq_length() > 0
+            if any(t != "conv" for t in self.config.layer_types)
+            else past_key_values.has_previous_state()
+        )
+        packed_cache = getattr(past_key_values, "lfm3_packed", None)
+        if isinstance(packed_cache, torch.Tensor) or packed_cache:
+            # Reset layers have no history to mix. Ordinary caches carry a Python `False` and skip this, so the decode
+            # steps do no tensor work here; only a compiled packed prefill leaves a tensor flag, and `filled` stays
+            # tensor-valued for static caches.
+            packed_message = (
+                "A cache filled by a packed-document forward cannot be continued: its attention spans the documents"
+            )
+            packed_cache = packed_cache & filled if isinstance(packed_cache, torch.Tensor) else filled
+            if isinstance(packed_cache, torch.Tensor):
+                torch_compilable_check(~packed_cache, packed_message)
+                if not is_tracing(packed_cache):
+                    past_key_values.lfm3_packed = bool(packed_cache)
+            elif packed_cache:
+                raise ValueError(packed_message)
+            else:
+                past_key_values.lfm3_packed = False
+
+        # The mask utilities detect packing only with `attention_mask=None`, so a multi-token forward with explicit
+        # positions drops the all-ones mask tokenizers emit (a data-dependent check, hence not when traced).
+        explicit_positions = position_ids is not None and inputs_embeds.shape[1] > 1
+        if (
+            explicit_positions
+            and isinstance(attention_mask, torch.Tensor)
+            and attention_mask.ndim == 2
+            and not is_tracing(position_ids)
+            and bool(attention_mask.all())
+        ):
+            attention_mask = None
+        seq_idx = None
+        if position_ids is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            position_ids = torch.arange(inputs_embeds.shape[1], device=inputs_embeds.device) + past_seen_tokens
+            position_ids = position_ids.unsqueeze(0)
+        elif explicit_positions and not isinstance(attention_mask, dict):
+            # a mask dict comes from `generate`, which refused packing before its prefill
+            if isinstance(attention_mask, torch.Tensor) and attention_mask.ndim == 2:
+                # data-dependent: a traced forward skips it (`generate` refuses packing eagerly before its prefill)
+                if not is_tracing(position_ids):
+                    torch_compilable_check(
+                        ~packed_under_padding(position_ids, attention_mask, filled),
+                        "Packed documents (`position_ids` restarting) cannot be combined with an attention mask: "
+                        "attention would not isolate the documents. Pass `attention_mask=None` without a static cache.",
+                    )
+            else:
+                restarts = torch.diff(position_ids, prepend=position_ids[:, :1] - 1, dim=-1) != 1
+                document_ids = restarts.cumsum(-1)
+                if is_tracing(position_ids) or bool((document_ids[..., -1] > 0).any()):
+                    seq_idx = document_ids.to(torch.int32)
+                # Packed documents are prefilled in one forward on an empty cache (cached tokens would join the first
+                # document), and their cache is never continued
+                if filled:
+                    torch_compilable_check(
+                        (~restarts[:, 1:]).all() & (position_ids[:, 0] != 0).all(),
+                        "Packed documents must be prefilled in one forward; the cache already holds tokens",
+                    )
+                    # Ordinary continuation must retain the convolution window from the cached document.
+                    seq_idx = None
+                if attention_mask is None:
+                    # without a mask only the cacheless packed-sequence mask isolates repeated zero positions (left
+                    # padding): a cache-sized mask and every later decode step would attend them, so they count too
+                    packed = (document_ids[..., -1] > 0).any()
+                else:
+                    # a prepared 4D mask is the caller's and carries the padding (repeated zero positions), but may
+                    # or may not isolate documents, so the cache records the packing and is never continued
+                    steps = torch.diff(position_ids, dim=-1)
+                    packed = ((steps != 1) & ~((steps == 0) & (position_ids[:, :-1] == 0))).any()
+                if past_key_values is not None:
+                    # resolved here, where the prefill already syncs, so the decode steps carry a Python flag: no
+                    # per-step host sync, data-dependent op or CUDA-graph output on the cache
+                    past_key_values.lfm3_packed = packed if is_tracing(position_ids) else bool(packed)
+
+        if not isinstance(causal_mask_mapping := attention_mask, dict):
+            mask_kwargs = {
+                "config": self.config,
+                "inputs_embeds": inputs_embeds,
+                "attention_mask": attention_mask,
+                "past_key_values": past_key_values,
+                "position_ids": position_ids,
+            }
+            # The mask utilities only isolate packed documents without a cache, and an empty cache adds nothing.
+            layer_types = self.config.layer_types
+            if seq_idx is not None and past_key_values is not None:
+                attention_layer = next((i for i, t in enumerate(layer_types) if t != "conv"), None)
+                if attention_layer is not None and past_key_values.get_seq_length(attention_layer) == 0:
+                    if past_key_values.is_compileable:
+                        # static layers always return their full length, so the masks are sized by the cache, which
+                        # isolates neither documents nor positional-only left padding
+                        torch_compilable_check(
+                            ~packed,
+                            "Packed documents, or left padding without an attention mask, cannot be prefilled into a "
+                            "static cache: its masks are sized by the cache. Pass the padding as `attention_mask`, or "
+                            "use a `DynamicCache` for packed documents",
+                        )
+                    else:
+                        mask_kwargs["past_key_values"] = None
+            causal_mask_mapping = create_masks_by_layer_type(**mask_kwargs)
+
+        hidden_states = inputs_embeds
+        position_embeddings = {
+            layer_type: self.rotary_emb(hidden_states, position_ids=position_ids, layer_type=layer_type)
+            for layer_type in self.rotary_emb.layer_types
+        }
+
+        for i, decoder_layer in enumerate(self.layers[: self.config.num_hidden_layers]):
+            layer_type = self.config.layer_types[i]
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask_mapping[layer_type],
+                position_embeddings=position_embeddings.get(layer_type),
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                seq_idx=seq_idx,
+                **kwargs,
+            )
+
+        hidden_states = self.embedding_norm(hidden_states)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+def reject_packed_generation(config: PreTrainedConfig, input_ids, generation_config, model_kwargs, is_first_iteration):
+    """`generate` decodes from the cache its prefill fills, and no mask it builds isolates packed documents
+    (`position_ids` restarting): the prompt is refused before the prefill, one forward or every chunk, writes to it."""
+    position_ids, attention_mask = model_kwargs.get("position_ids"), model_kwargs.get("attention_mask")
+    cache = model_kwargs.get("past_key_values")
+    if cache is None or position_ids is None or position_ids.ndim != 2:
+        return
+    if attention_mask is not None and attention_mask.ndim != 2:
+        return
+    has_attention = any(t != "conv" for t in config.get_text_config().layer_types)
+    filled = bool(cache.get_seq_length() > 0 if has_attention else cache.has_previous_state())
+    inputs_embeds = model_kwargs.get("inputs_embeds") if is_first_iteration else None
+    length = input_ids.shape[1] if inputs_embeds is None else inputs_embeds.shape[1]
+    if generation_config.prefill_chunk_size is not None:
+        # the chunks run from the start of the prompt
+        position_ids = position_ids[:, :length]
+    else:
+        if filled and has_attention:
+            # the slice `_prefill` feeds: a prompt that spans the cache is cut to its uncached part
+            mask_length = getattr(generation_config, "_mask_length", -1)
+            if attention_mask is not None:
+                mask_length = attention_mask.shape[1]
+            if inputs_embeds is not None or mask_length == length:
+                length -= int(cache.get_seq_length())
+        position_ids = position_ids[:, position_ids.shape[1] - length :]
+    if attention_mask is None:
+        attention_mask = torch.ones_like(position_ids)
+    if bool(packed_under_padding(position_ids, attention_mask, filled).any()):
+        raise ValueError(
+            "`generate` cannot decode packed documents (`position_ids` restarting, or starting at 0 on a cache that "
+            "holds tokens): attention would span them. Generate each document in its own row of a left-padded batch."
+        )
+
+
+@auto_docstring
+class Lfm3ForCausalLM(Lfm3PreTrainedModel, GenerationMixin):
+    _tied_weights_keys = {"lm_head.weight": "model.embed_tokens.weight"}
+    _tp_plan = {"lm_head": "colwise_gather_output"}
+    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
+    _fsdp_plan = {"lm_head": "keep_full_weight"}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = Lfm3Model(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        logits_to_keep: int | torch.Tensor = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> CausalLMOutputWithPast:
+        r"""
+        Example:
+
+        ```python
+        >>> from transformers import AutoTokenizer, Lfm3ForCausalLM
+
+        >>> model = Lfm3ForCausalLM.from_pretrained("LiquidAI/LFM3-1.7B")
+        >>> tokenizer = AutoTokenizer.from_pretrained("LiquidAI/LFM3-1.7B")
+
+        >>> inputs = tokenizer.apply_chat_template(
+        ...     [{"role": "user", "content": "What is C. elegans?"}], add_generation_prompt=True, return_tensors="pt"
+        ... )
+        >>> generate_ids = model.generate(**inputs, max_new_tokens=64)
+        >>> tokenizer.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)
+        ```"""
+        outputs: BaseModelOutputWithPast = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            **kwargs,
+        )
+
+        hidden_states = outputs.last_hidden_state
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+    def _prefill(self, input_ids, generation_config, model_kwargs, is_first_iteration=True):
+        # every prefill passes here once before its first forward: a check per chunk would leave the earlier chunks
+        # in the cache, and a dynamic cache would only refuse packing at the first decode step
+        reject_packed_generation(self.config, input_ids, generation_config, model_kwargs, is_first_iteration)
+        return super()._prefill(input_ids, generation_config, model_kwargs, is_first_iteration=is_first_iteration)
+
+
+__all__ = ["Lfm3ForCausalLM", "Lfm3Model", "Lfm3PreTrainedModel"]

modeling_lfm3_vl.py

@@ -0,0 +1,968 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3_vl/modeling_lfm3_vl.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3_vl/modular_lfm3_vl.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3_vl.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from collections.abc import Callable
+from dataclasses import dataclass
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache
+from transformers.configuration_utils import PreTrainedConfig
+from transformers.generation import GenerationMixin
+from transformers.integrations import use_kernel_forward_from_hub
+from transformers.masking_utils import create_causal_mask, create_recurrent_attention_mask, create_sliding_window_causal_mask
+from transformers.modeling_layers import GradientCheckpointingLayer
+from transformers.modeling_outputs import BaseModelOutputWithPast, BaseModelOutputWithPooling, CausalLMOutputWithPast
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.utils import (
+    TransformersKwargs,
+    auto_docstring,
+    can_return_tuple,
+    torch_compilable_check,
+)
+from transformers.utils.generic import (
+    accepts_precomputed_kwargs,
+    is_flash_attention_requested,
+    maybe_autocast,
+    merge_with_config_defaults,
+)
+from transformers.utils.output_capturing import OutputRecorder, capture_outputs
+from transformers.vision_utils import get_max_seqlen, get_vision_attention_seqlens, get_vision_position_ids
+from transformers.models.auto import AutoModel
+from .configuration_lfm3_vl import Lfm3VlConfig, Lfm3VlVisionConfig
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class Lfm3VlRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps: float = 1e-6) -> None:
+        """
+        Lfm3VlRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class Lfm3VlMLP(nn.Module):
+    def __init__(self, config: Lfm3VlConfig):
+        super().__init__()
+        self.w1 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.w3 = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.w2 = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def forward(self, x):
+        return self.w2(F.silu(self.w1(x)) * self.w3(x))
+
+
+class Lfm3VlVisionRotaryEmbedding(nn.Module):
+    """
+    Simple axial 2D rope with same freqs used for H and W grids. The freqs are
+    pre-computed using `head-dim//4` which is later used to concat H and W positions.
+    The final angles rotate over the whole head dim, no partial rotation involved.
+    """
+
+    def __init__(self, config: Lfm3VlVisionConfig):
+        super().__init__()
+        self.config = config
+
+        self.rope_type = self.config.rope_parameters["rope_type"]
+        rope_init_fn: Callable = self.compute_axial_rope_parameters
+        if self.rope_type != "axial":
+            raise ValueError(f"{self.__class__.__name__} supports only axial rope, but requested {self.rope_type}")
+        inv_freq, self.attention_scaling = rope_init_fn(self.config)
+
+        self.inv_freq = nn.Buffer(inv_freq, persistent=False)
+        self.original_inv_freq = nn.Buffer(inv_freq.clone(), persistent=False)
+
+    @staticmethod
+    def compute_axial_rope_parameters(config: Lfm3VlVisionConfig, **kwargs) -> tuple[torch.Tensor, float]:
+        """Inverse frequencies of the axial RoPE over half the head dimension per axis, with an attention scaling of 1."""
+        base = config.rope_parameters["rope_theta"]
+        spatial_dim = config.hidden_size // config.num_attention_heads // 2
+        inv_freq = 1.0 / (base ** (torch.arange(0, spatial_dim, 2, dtype=torch.float) / spatial_dim))
+        return inv_freq, 1.0
+
+    @torch.no_grad()
+    def forward(self, x, position_ids):
+        # position_ids: (2, N) — row 0 = h coords, row 1 = w coords
+        position_ids_expanded = position_ids[..., None].float()
+        device_type = x.device.type if isinstance(x.device.type, str) else "cpu"
+        with maybe_autocast(device_type=device_type, enabled=False):
+            freqs = position_ids_expanded * self.inv_freq.float()
+            cos = freqs.cos() * self.attention_scaling
+            sin = freqs.sin() * self.attention_scaling
+
+        cos = self.recomposition_frequencies(cos)
+        sin = self.recomposition_frequencies(sin)
+        return cos, sin
+
+    def recomposition_frequencies(self, freq):
+        """
+        Recompose the frequencies into the final spatial layout used per each grid.
+        """
+        freq_h, freq_w = freq[:, 0], freq[:, 1]
+        freq_hw = torch.cat([freq_h, freq_w], dim=-1)
+        return torch.cat([freq_hw, freq_hw], dim=-1)
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb_vision(
+    q: torch.Tensor, k: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor
+) -> tuple[torch.Tensor, torch.Tensor]:
+    orig_q_dtype = q.dtype
+    orig_k_dtype = k.dtype
+    q, k = q.float(), k.float()
+    cos, sin = cos.unsqueeze(-2).float(), sin.unsqueeze(-2).float()
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    q_embed = q_embed.to(orig_q_dtype)
+    k_embed = k_embed.to(orig_k_dtype)
+    return q_embed, k_embed
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+class Lfm3VlVisionAttention(nn.Module):
+    """Bidirectional grouped-query attention over the patches of one frame, with RMSNorm on the query and key heads."""
+
+    def __init__(self, config: Lfm3VlVisionConfig):
+        super().__init__()
+        self.config = config
+        self.num_heads = config.num_attention_heads
+        self.num_key_value_heads = config.num_key_value_heads
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.head_dim = config.hidden_size // config.num_attention_heads
+        self.scaling = self.head_dim**-0.5
+        self.attention_dropout = 0.0
+        self.is_causal = False
+        self.q_proj = nn.Linear(config.hidden_size, self.num_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(config.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(config.hidden_size, self.num_key_value_heads * self.head_dim, bias=False)
+        self.out_proj = nn.Linear(self.num_heads * self.head_dim, config.hidden_size, bias=False)
+        self.q_layernorm = Lfm3VlRMSNorm(self.head_dim, eps=config.norm_eps)
+        self.k_layernorm = Lfm3VlRMSNorm(self.head_dim, eps=config.norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        max_seqlen: int | None = None,
+        frame_shapes: list[list[int]] | None = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        seq_length = hidden_states.shape[0]
+        query_states = self.q_layernorm(self.q_proj(hidden_states).view(seq_length, self.num_heads, self.head_dim))
+        key_states = self.k_layernorm(
+            self.k_proj(hidden_states).view(seq_length, self.num_key_value_heads, self.head_dim)
+        )
+        value_states = self.v_proj(hidden_states).view(seq_length, self.num_key_value_heads, self.head_dim)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb_vision(query_states, key_states, cos, sin)
+
+        query_states = query_states.transpose(0, 1).unsqueeze(0)
+        key_states = key_states.transpose(0, 1).unsqueeze(0)
+        value_states = value_states.transpose(0, 1).unsqueeze(0)
+
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            self.config._attn_implementation, eager_attention_forward
+        )
+
+        if is_flash_attention_requested(self.config):
+            max_seqlen = get_max_seqlen(cu_seqlens, self.config, kwargs={"max_seqlen": max_seqlen})
+            attn_output, attn_weights = attention_interface(
+                self,
+                query_states,
+                key_states,
+                value_states,
+                attention_mask=None,
+                scaling=self.scaling,
+                dropout=0.0,
+                cu_seq_lens_q=cu_seqlens,
+                cu_seq_lens_k=cu_seqlens,
+                max_length_q=max_seqlen,
+                max_length_k=max_seqlen,
+                is_causal=False,
+                **kwargs,
+            )
+        else:
+            # Other implementations: frames of one shape (every video, equal-size images) attend as one batch, mixed
+            # shapes frame by frame. `frame_shapes` gives the lengths without a device sync.
+            if frame_shapes is not None:
+                lengths = [height * width for height, width in frame_shapes]
+            else:
+                lengths = (cu_seqlens[1:] - cu_seqlens[:-1]).tolist()
+            if len(set(lengths)) == 1:
+                query_states, key_states, value_states = (
+                    tensor.reshape(-1, len(lengths), lengths[0], self.head_dim).transpose(0, 1)
+                    for tensor in (query_states, key_states, value_states)
+                )
+                attn_output, attn_weights = attention_interface(
+                    self,
+                    query_states,
+                    key_states,
+                    value_states,
+                    attention_mask=None,
+                    scaling=self.scaling,
+                    dropout=0.0,
+                    is_causal=False,
+                    **kwargs,
+                )
+            else:
+                splits = [torch.split(tensor, lengths, dim=2) for tensor in (query_states, key_states, value_states)]
+                attn_outputs, attn_weights = [], []
+                for q, k, v in zip(*splits):
+                    output, weights = attention_interface(
+                        self,
+                        q,
+                        k,
+                        v,
+                        attention_mask=None,
+                        scaling=self.scaling,
+                        dropout=0.0,
+                        is_causal=False,
+                        **kwargs,
+                    )
+                    attn_outputs.append(output)
+                    attn_weights.append(weights)
+                attn_output = torch.cat(attn_outputs, dim=1)
+                # frames of different shapes have differently sized maps: one entry per frame, or None under sdpa
+                attn_weights = None if attn_weights[0] is None else tuple(attn_weights)
+
+        attn_output = attn_output.reshape(seq_length, -1).contiguous()
+        attn_output = self.out_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class Lfm3VlVisionConv2d(nn.Module):
+    """Gated depthwise 2D convolution over the patch grid of each frame."""
+
+    def __init__(self, config: Lfm3VlVisionConfig):
+        super().__init__()
+        self.spatial_merge_size = config.spatial_merge_size
+        self.in_proj = nn.Linear(config.hidden_size, 3 * config.hidden_size, bias=False)
+        self.spatial_conv = nn.Conv2d(
+            config.hidden_size,
+            config.hidden_size,
+            kernel_size=config.conv_kernel_size,
+            padding=config.conv_kernel_size // 2,
+            groups=config.hidden_size,
+            bias=False,
+        )
+        self.out_proj = nn.Linear(config.hidden_size, config.hidden_size, bias=False)
+
+    def forward(self, hidden_states: torch.Tensor, frame_shapes: list[list[int]]):
+        B, C, x = self.in_proj(hidden_states).chunk(3, dim=-1)
+        gated = B * x
+        merge = self.spatial_merge_size
+        num_frames = len(frame_shapes)
+        if len(set(map(tuple, frame_shapes))) == 1:
+            # frames of one shape (every video, equal-size images) convolve as one batch: merge-block order -> raster
+            # grid, convolve, back
+            height, width = frame_shapes[0]
+            frames = gated.view(num_frames, height // merge, width // merge, merge, merge, -1)
+            frames = frames.permute(0, 5, 1, 3, 2, 4).reshape(num_frames, -1, height, width)
+            frames = self.spatial_conv(frames)
+            frames = frames.view(num_frames, -1, height // merge, merge, width // merge, merge).permute(
+                0, 2, 4, 3, 5, 1
+            )
+            conv_output = frames.reshape(gated.shape[0], -1)
+        else:
+            conv_outputs = []
+            lengths = [height * width for height, width in frame_shapes]
+            for frame, (height, width) in zip(torch.split(gated, lengths, dim=0), frame_shapes):
+                frame = frame.view(height // merge, width // merge, merge, merge, -1)
+                frame = frame.permute(4, 0, 2, 1, 3).reshape(1, -1, height, width)
+                frame = self.spatial_conv(frame)
+                frame = frame.view(-1, height // merge, merge, width // merge, merge).permute(1, 3, 2, 4, 0)
+                conv_outputs.append(frame.reshape(height * width, -1))
+            conv_output = torch.cat(conv_outputs, dim=0)
+        return self.out_proj(C * conv_output)
+
+
+class Lfm3VlVisionLayer(GradientCheckpointingLayer):
+    def __init__(self, config: Lfm3VlVisionConfig, layer_idx: int):
+        super().__init__()
+        self.is_attention_layer = config.layer_types[layer_idx] == "full_attention"
+        if self.is_attention_layer:
+            self.self_attn = Lfm3VlVisionAttention(config)
+        else:
+            self.conv = Lfm3VlVisionConv2d(config)
+        self.feed_forward = Lfm3VlMLP(config)
+        self.operator_norm = Lfm3VlRMSNorm(config.hidden_size, eps=config.norm_eps)
+        self.ffn_norm = Lfm3VlRMSNorm(config.hidden_size, eps=config.norm_eps)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        cu_seqlens: torch.Tensor,
+        frame_shapes: list[list[int]],
+        position_embeddings: tuple[torch.Tensor, torch.Tensor] | None = None,
+        **kwargs,
+    ) -> torch.Tensor:
+        if self.is_attention_layer:
+            operator_output, _ = self.self_attn(
+                self.operator_norm(hidden_states),
+                cu_seqlens=cu_seqlens,
+                position_embeddings=position_embeddings,
+                frame_shapes=frame_shapes,
+                **kwargs,
+            )
+        else:
+            operator_output = self.conv(self.operator_norm(hidden_states), frame_shapes)
+        hidden_states = hidden_states + operator_output
+        hidden_states = hidden_states + self.feed_forward(self.ffn_norm(hidden_states))
+        return hidden_states
+
+
+def create_masks_by_layer_type(
+    config: PreTrainedConfig,
+    inputs_embeds: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    past_key_values: Cache | None,
+    position_ids: torch.Tensor | None = None,
+) -> dict:
+    """The cache sizes each mask from the layer it is asked about, and the generic mask utilities ask about the first
+    non-sliding layer, which for a leading `conv` layer redirects to the first attention layer of any type: name the
+    first layer of each mask's own type instead."""
+    config = config.get_text_config()
+    mask_kwargs = {
+        "config": config,
+        "inputs_embeds": inputs_embeds,
+        "attention_mask": attention_mask,
+        "past_key_values": past_key_values,
+        "position_ids": position_ids,
+    }
+    layer_types, causal_mask_mapping = config.layer_types, {}
+    if "full_attention" in layer_types:
+        layer_idx = layer_types.index("full_attention")
+        causal_mask_mapping["full_attention"] = create_causal_mask(**mask_kwargs, layer_idx=layer_idx)
+    if "sliding_attention" in layer_types:
+        layer_idx = layer_types.index("sliding_attention")
+        causal_mask_mapping["sliding_attention"] = create_sliding_window_causal_mask(
+            **mask_kwargs, layer_idx=layer_idx
+        )
+    if "conv" in layer_types:
+        causal_mask_mapping["conv"] = create_recurrent_attention_mask(**mask_kwargs)
+    return causal_mask_mapping
+
+
+@auto_docstring
+class Lfm3VlPreTrainedModel(PreTrainedModel):
+    config: Lfm3VlConfig
+    base_model_prefix = "model"
+    input_modalities = ("image", "video", "text")
+    supports_gradient_checkpointing = True
+    _skip_keys_device_placement = ["past_key_values"]
+
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    # the cached decode step traces without graph breaks, like the text model
+    _can_compile_fullgraph = True
+    _supports_flex_attn = True
+    _supports_attention_backend = True
+    _no_split_modules = ["Lfm3VlVisionLayer"]
+
+    @staticmethod
+    def create_masks_for_generate(
+        config: PreTrainedConfig,
+        inputs_embeds: torch.Tensor,
+        attention_mask: torch.Tensor | None,
+        past_key_values: Cache | None,
+        position_ids: torch.Tensor | None = None,
+        **kwargs,
+    ) -> dict:
+        # the static-cache masks of the text model are sized by layer type, not by the generic first attention layer
+        return create_masks_by_layer_type(config, inputs_embeds, attention_mask, past_key_values, position_ids)
+
+
+def get_vision_frame_shapes(grid_thw: torch.Tensor) -> list[list[int]]:
+    return grid_thw[:, 1:].repeat_interleave(grid_thw[:, 0], dim=0).tolist()
+
+
+@auto_docstring(
+    custom_intro="""
+    The LFM3-VL vision encoder: a stack of gated convolution and bidirectional attention layers over the patches of
+    each image or video frame, embedded by a linear patchifier.
+    """
+)
+class Lfm3VlVisionModel(Lfm3VlPreTrainedModel):
+    config: Lfm3VlVisionConfig
+    input_modalities = ("image", "video")
+    _input_embed_layer = "patchifier"
+    _can_record_outputs = {
+        "hidden_states": Lfm3VlVisionLayer,
+        "attentions": OutputRecorder(Lfm3VlVisionAttention, index=1),
+    }
+
+    def __init__(self, config: Lfm3VlVisionConfig):
+        super().__init__(config)
+        self.spatial_merge_size = config.spatial_merge_size
+        patch_dim = config.in_channels * config.temporal_patch_size * config.patch_size**2
+        self.patchifier = nn.Linear(patch_dim, config.hidden_size, bias=False)
+        self.embedding_norm = Lfm3VlRMSNorm(config.hidden_size, eps=config.norm_eps)
+        self.rotary_pos_emb = Lfm3VlVisionRotaryEmbedding(config)
+        self.layers = nn.ModuleList(
+            [Lfm3VlVisionLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.gradient_checkpointing = False
+        self.post_init()
+
+    @merge_with_config_defaults
+    @capture_outputs
+    @auto_docstring
+    def forward(
+        self, pixel_values: torch.Tensor, grid_thw: torch.Tensor, **kwargs: Unpack[TransformersKwargs]
+    ) -> BaseModelOutputWithPooling:
+        r"""
+        pixel_values (`torch.Tensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
+            The flattened patches of every image or video, grouped by merge block.
+        grid_thw (`torch.LongTensor` of shape `(num_images_or_videos, 3)`):
+            The temporal, height and width dimensions of the patch grid of each image or video.
+        """
+        position_ids = get_vision_position_ids(grid_thw, self.spatial_merge_size, kwargs=kwargs)
+        cu_seqlens, max_seqlen = get_vision_attention_seqlens(grid_thw, self.config, kwargs=kwargs)
+        frame_shapes = get_vision_frame_shapes(grid_thw)
+
+        hidden_states = self.patchifier(pixel_values.to(self.patchifier.weight.dtype))
+        hidden_states = self.embedding_norm(hidden_states)
+        position_embeddings = self.rotary_pos_emb(hidden_states, position_ids)
+
+        for layer in self.layers:
+            hidden_states = layer(
+                hidden_states,
+                cu_seqlens=cu_seqlens,
+                frame_shapes=frame_shapes,
+                max_seqlen=max_seqlen,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+
+        return BaseModelOutputWithPooling(last_hidden_state=hidden_states)
+
+
+class Lfm3VlMultiModalProjector(nn.Module):
+    """Merges each block of `downsample_factor**2` vision tokens into one language token."""
+
+    def __init__(self, config: Lfm3VlConfig):
+        super().__init__()
+        self.downsample_factor = config.downsample_factor
+        self.layer_norm = nn.LayerNorm(config.vision_config.hidden_size, eps=config.projector_layer_norm_eps)
+        self.linear_1 = nn.Linear(
+            config.vision_config.hidden_size * config.downsample_factor**2,
+            config.projector_hidden_size,
+            bias=config.projector_bias,
+        )
+        self.act = ACT2FN[config.projector_hidden_act]
+        self.linear_2 = nn.Linear(
+            config.projector_hidden_size, config.text_config.hidden_size, bias=config.projector_bias
+        )
+
+    def forward(self, image_features: torch.Tensor):
+        hidden_states = self.layer_norm(image_features).reshape(-1, self.linear_1.in_features)
+        hidden_states = self.linear_1(hidden_states)
+        hidden_states = self.act(hidden_states)
+        hidden_states = self.linear_2(hidden_states)
+        return hidden_states
+
+
+@auto_docstring(
+    custom_intro="""
+    Base class for Lfm3Vl outputs, with hidden states and attentions.
+    """
+)
+@dataclass
+class Lfm3VlModelOutputWithPast(BaseModelOutputWithPast):
+    r"""
+    past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`):
+        It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache).
+
+        Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see
+        `past_key_values` input) to speed up sequential decoding.
+    image_hidden_states (`torch.FloatTensor`, *optional*):
+        A `torch.FloatTensor` of size `(num_image_tokens, hidden_size)`.
+        image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
+    video_hidden_states (`torch.FloatTensor`, *optional*):
+        A `torch.FloatTensor` of size `(num_video_tokens, hidden_size)`.
+        video_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
+    """
+
+    image_hidden_states: torch.FloatTensor | None = None
+    video_hidden_states: torch.FloatTensor | None = None
+
+
+@auto_docstring(
+    custom_intro="""
+    Base class for Lfm3Vl causal language model (or autoregressive) outputs.
+    """
+)
+@dataclass
+class Lfm3VlCausalLMOutputWithPast(CausalLMOutputWithPast):
+    r"""
+    image_hidden_states (`torch.FloatTensor`, *optional*):
+        A `torch.FloatTensor` of size `(num_image_tokens, hidden_size)`.
+        image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
+    video_hidden_states (`torch.FloatTensor`, *optional*):
+        A `torch.FloatTensor` of size `(num_video_tokens, hidden_size)`.
+        video_hidden_states of the model produced by the vision encoder and after projecting the last hidden state.
+    """
+
+    image_hidden_states: torch.FloatTensor | None = None
+    video_hidden_states: torch.FloatTensor | None = None
+
+
+@auto_docstring(
+    custom_intro="""
+    The Lfm3Vl model which consists of a vision backbone and a language model, without a language modeling head.
+    """
+)
+class Lfm3VlModel(Lfm3VlPreTrainedModel):
+    def __init__(self, config: Lfm3VlConfig):
+        super().__init__(config)
+        self.vision_tower = AutoModel.from_config(config.vision_config)
+        self.multi_modal_projector = Lfm3VlMultiModalProjector(config)
+        self.language_model = AutoModel.from_config(config.text_config)
+        self.post_init()
+
+    @accepts_precomputed_kwargs(modality="image")
+    @can_return_tuple
+    @auto_docstring(
+        custom_intro="Obtains image last hidden states from the vision tower and apply multimodal projection."
+    )
+    def get_image_features(
+        self,
+        pixel_values: torch.FloatTensor,
+        image_grid_thw: torch.LongTensor,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        r"""
+        pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
+            The flattened patches of every image, grouped by merge block.
+        image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`):
+            The temporal, height and width dimensions of the patch grid of each image.
+        """
+        image_outputs = self.vision_tower(pixel_values, grid_thw=image_grid_thw, return_dict=True, **kwargs)
+        image_features = self.multi_modal_projector(image_outputs.last_hidden_state)
+        split_sizes = (image_grid_thw.prod(-1) // self.vision_tower.spatial_merge_size**2).tolist()
+        image_outputs.pooler_output = torch.split(image_features, split_sizes)
+        return image_outputs
+
+    def get_placeholder_mask(
+        self,
+        input_ids: torch.LongTensor,
+        inputs_embeds: torch.FloatTensor,
+        image_features: torch.FloatTensor | None = None,
+        video_features: torch.FloatTensor | None = None,
+    ):
+        """Placeholder masks from `input_ids` or `inputs_embeds`, checked against the feature counts."""
+        if input_ids is None:
+            special_image_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.full((), self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_image_mask = special_image_mask.all(-1)
+            special_video_mask = inputs_embeds == self.get_input_embeddings()(
+                torch.full((), self.config.video_token_id, dtype=torch.long, device=inputs_embeds.device)
+            )
+            special_video_mask = special_video_mask.all(-1)
+        else:
+            special_image_mask = input_ids == self.config.image_token_id
+            special_video_mask = input_ids == self.config.video_token_id
+
+        n_image_tokens = special_image_mask.sum()
+        special_image_mask = special_image_mask.unsqueeze(-1).to(inputs_embeds.device)
+        if image_features is not None:
+            torch_compilable_check(
+                n_image_tokens * inputs_embeds.shape[-1] == image_features.numel(),
+                f"Image features and image tokens do not match, tokens: {n_image_tokens}, features: {image_features.shape[0]}",
+            )
+
+        n_video_tokens = special_video_mask.sum()
+        special_video_mask = special_video_mask.unsqueeze(-1).to(inputs_embeds.device)
+        if video_features is not None:
+            torch_compilable_check(
+                n_video_tokens * inputs_embeds.shape[-1] == video_features.numel(),
+                f"Video features and video tokens do not match, tokens: {n_video_tokens}, features: {video_features.shape[0]}",
+            )
+        return special_image_mask, special_video_mask
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        pixel_values: torch.FloatTensor | None = None,
+        image_grid_thw: torch.LongTensor | None = None,
+        pixel_values_videos: torch.FloatTensor | None = None,
+        video_grid_thw: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        mm_encoder_outputs: dict[str, BaseModelOutputWithPooling] | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | Lfm3VlModelOutputWithPast:
+        r"""
+        pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
+            The flattened patches of every image, grouped by merge block.
+        image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
+            The temporal, height and width dimensions of the patch grid of each image.
+        pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
+            The flattened patches of every video, grouped by merge block.
+        video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
+            The temporal, height and width dimensions of the patch grid of each video.
+        """
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if (pixel_values is not None or pixel_values_videos is not None) and mm_encoder_outputs is not None:
+            raise ValueError(
+                "You cannot specify both pixel_values/pixel_values_videos and mm_encoder_outputs at the same time"
+            )
+
+        if inputs_embeds is None:
+            inputs_embeds = self.get_input_embeddings()(input_ids)
+
+        mm_encoder_outputs = mm_encoder_outputs if mm_encoder_outputs is not None else {}
+        if (pixel_values is None) != (image_grid_thw is None):
+            raise ValueError("pixel_values and image_grid_thw must be supplied together")
+        if (pixel_values_videos is None) != (video_grid_thw is None):
+            raise ValueError("pixel_values_videos and video_grid_thw must be supplied together")
+        if mm_encoder_outputs.get("image") is None and pixel_values is not None:
+            mm_encoder_outputs["image"] = self.get_image_features(
+                pixel_values, image_grid_thw, return_dict=True, **kwargs
+            )
+        if mm_encoder_outputs.get("video") is None and pixel_values_videos is not None:
+            mm_encoder_outputs["video"] = self.get_video_features(
+                pixel_values_videos, video_grid_thw, return_dict=True, **kwargs
+            )
+
+        image_features = None
+        if mm_encoder_outputs.get("image") is not None:
+            image_features = torch.cat(mm_encoder_outputs["image"].pooler_output, dim=0).to(
+                inputs_embeds.device, inputs_embeds.dtype
+            )
+            image_mask, _ = self.get_placeholder_mask(
+                input_ids, inputs_embeds=inputs_embeds, image_features=image_features
+            )
+            inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_features)
+
+        video_features = None
+        if mm_encoder_outputs.get("video") is not None:
+            video_features = torch.cat(mm_encoder_outputs["video"].pooler_output, dim=0).to(
+                inputs_embeds.device, inputs_embeds.dtype
+            )
+            _, video_mask = self.get_placeholder_mask(
+                input_ids, inputs_embeds=inputs_embeds, video_features=video_features
+            )
+            inputs_embeds = inputs_embeds.masked_scatter(video_mask, video_features)
+
+        outputs = self.language_model(
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            **kwargs,
+        )
+
+        return Lfm3VlModelOutputWithPast(
+            last_hidden_state=outputs.last_hidden_state,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+            image_hidden_states=image_features,
+            video_hidden_states=video_features,
+        )
+
+    @accepts_precomputed_kwargs(modality="video")
+    @can_return_tuple
+    @auto_docstring(
+        custom_intro="Obtains video last hidden states from the vision tower and apply multimodal projection."
+    )
+    def get_video_features(
+        self,
+        pixel_values_videos: torch.FloatTensor,
+        video_grid_thw: torch.LongTensor,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        r"""
+        pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
+            The flattened patches of every video, grouped by merge block.
+        video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`):
+            The temporal, height and width dimensions of the patch grid of each video.
+        """
+        return self.get_image_features(pixel_values_videos, video_grid_thw, **kwargs)
+
+
+def reject_packed_generation(config: PreTrainedConfig, input_ids, generation_config, model_kwargs, is_first_iteration):
+    """`generate` decodes from the cache its prefill fills, and no mask it builds isolates packed documents
+    (`position_ids` restarting): the prompt is refused before the prefill, one forward or every chunk, writes to it."""
+    position_ids, attention_mask = model_kwargs.get("position_ids"), model_kwargs.get("attention_mask")
+    cache = model_kwargs.get("past_key_values")
+    if cache is None or position_ids is None or position_ids.ndim != 2:
+        return
+    if attention_mask is not None and attention_mask.ndim != 2:
+        return
+    has_attention = any(t != "conv" for t in config.get_text_config().layer_types)
+    filled = bool(cache.get_seq_length() > 0 if has_attention else cache.has_previous_state())
+    inputs_embeds = model_kwargs.get("inputs_embeds") if is_first_iteration else None
+    length = input_ids.shape[1] if inputs_embeds is None else inputs_embeds.shape[1]
+    if generation_config.prefill_chunk_size is not None:
+        # the chunks run from the start of the prompt
+        position_ids = position_ids[:, :length]
+    else:
+        if filled and has_attention:
+            # the slice `_prefill` feeds: a prompt that spans the cache is cut to its uncached part
+            mask_length = getattr(generation_config, "_mask_length", -1)
+            if attention_mask is not None:
+                mask_length = attention_mask.shape[1]
+            if inputs_embeds is not None or mask_length == length:
+                length -= int(cache.get_seq_length())
+        position_ids = position_ids[:, position_ids.shape[1] - length :]
+    if attention_mask is None:
+        attention_mask = torch.ones_like(position_ids)
+    if bool(packed_under_padding(position_ids, attention_mask, filled).any()):
+        raise ValueError(
+            "`generate` cannot decode packed documents (`position_ids` restarting, or starting at 0 on a cache that "
+            "holds tokens): attention would span them. Generate each document in its own row of a left-padded batch."
+        )
+
+
+def packed_under_padding(position_ids: torch.Tensor, padding_mask: torch.Tensor, filled) -> torch.Tensor:
+    """Padding repeats position 0, so packing is a restart between two kept tokens, or a kept position 0 on a filled
+    cache whose row holds a kept token; attention under a padding mask cannot isolate it."""
+    restarts = torch.diff(position_ids, prepend=position_ids[:, :1] - 1, dim=-1) != 1
+    valid = padding_mask[:, -restarts.shape[-1] :].bool()
+    packed = restarts[:, 1:] & valid[:, 1:] & valid[:, :-1]
+    if filled:
+        # a row whose cached tokens are all padding (chunked prefill of a left-padded batch) has no document to pack
+        # against; a mask covering only the chunk keeps the strict rule
+        cached = padding_mask[:, : padding_mask.shape[-1] - restarts.shape[-1]]
+        cached_valid = cached.bool().any(-1, keepdim=True) if cached.shape[-1] else torch.ones_like(valid[:, :1])
+        packed = packed | (valid[:, :1] & (position_ids[:, :1] == 0) & cached_valid)
+    return packed
+
+
+@auto_docstring(
+    custom_intro="""
+    The LFM3_VL model which consists of a vision backbone and a language model.
+    """
+)
+class Lfm3VlForConditionalGeneration(Lfm3VlPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = {"lm_head.weight": "model.language_model.embed_tokens.weight"}
+
+    def __init__(self, config: Lfm3VlConfig):
+        super().__init__(config)
+        self.model = Lfm3VlModel(config)
+        self.lm_head = nn.Linear(config.text_config.hidden_size, config.text_config.vocab_size, bias=False)
+        self.post_init()
+
+    def get_output_embeddings(self) -> nn.Module:
+        return self.lm_head
+
+    @auto_docstring
+    def get_image_features(
+        self,
+        pixel_values: torch.FloatTensor,
+        image_grid_thw: torch.LongTensor,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        r"""
+        pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
+            The flattened patches of every image, grouped by merge block.
+        image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`):
+            The temporal, height and width dimensions of the patch grid of each image.
+        """
+        return self.model.get_image_features(pixel_values, image_grid_thw, **kwargs)
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        pixel_values: torch.FloatTensor | None = None,
+        image_grid_thw: torch.LongTensor | None = None,
+        pixel_values_videos: torch.FloatTensor | None = None,
+        video_grid_thw: torch.LongTensor | None = None,
+        labels: torch.LongTensor | None = None,
+        use_cache: bool | None = None,
+        logits_to_keep: int | torch.Tensor = 0,
+        mm_encoder_outputs: dict[str, BaseModelOutputWithPooling] | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | Lfm3VlCausalLMOutputWithPast:
+        r"""
+        pixel_values (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
+            The flattened patches of every image, grouped by merge block.
+        image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*):
+            The temporal, height and width dimensions of the patch grid of each image.
+        pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`, *optional*):
+            The flattened patches of every video, grouped by merge block.
+        video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*):
+            The temporal, height and width dimensions of the patch grid of each video.
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the masked language modeling loss. Indices should either be in `[0, ...,
+            config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored
+            (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+
+        Example:
+
+        ```python
+        >>> from transformers import AutoProcessor, AutoModelForImageTextToText
+
+        >>> model = AutoModelForImageTextToText.from_pretrained("LiquidAI/LFM3-VL-1.9B")
+        >>> processor = AutoProcessor.from_pretrained("LiquidAI/LFM3-VL-1.9B")
+
+        >>> conversation = [
+        ...     {
+        ...         "role": "user",
+        ...         "content": [
+        ...             {"type": "image", "url": "https://www.ilankelman.org/stopsigns/australia.jpg"},
+        ...             {"type": "text", "text": "What is in this image?"},
+        ...         ],
+        ...     },
+        ... ]
+
+        >>> inputs = processor.apply_chat_template(
+        ...     conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt"
+        ... )
+
+        >>> outputs = model.generate(**inputs, max_new_tokens=45)
+        >>> processor.batch_decode(outputs, skip_special_tokens=True)[0]
+        ```"""
+        outputs = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            pixel_values=pixel_values,
+            image_grid_thw=image_grid_thw,
+            pixel_values_videos=pixel_values_videos,
+            video_grid_thw=video_grid_thw,
+            use_cache=use_cache,
+            mm_encoder_outputs=mm_encoder_outputs,
+            **kwargs,
+        )
+
+        hidden_states = outputs[0]
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(
+                logits=logits, labels=labels, vocab_size=self.config.text_config.vocab_size, **kwargs
+            )
+
+        return Lfm3VlCausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+            image_hidden_states=outputs.image_hidden_states,
+            video_hidden_states=outputs.video_hidden_states,
+        )
+
+    def _prefill(self, input_ids, generation_config, model_kwargs, is_first_iteration=True):
+        # the text model's refusal, before the prefill writes to the cache
+        reject_packed_generation(self.config, input_ids, generation_config, model_kwargs, is_first_iteration)
+        return super()._prefill(input_ids, generation_config, model_kwargs, is_first_iteration=is_first_iteration)
+
+    @auto_docstring
+    def get_video_features(
+        self,
+        pixel_values_videos: torch.FloatTensor,
+        video_grid_thw: torch.LongTensor,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple | BaseModelOutputWithPooling:
+        r"""
+        pixel_values_videos (`torch.FloatTensor` of shape `(num_patches, in_channels * temporal_patch_size * patch_size**2)`):
+            The flattened patches of every video, grouped by merge block.
+        video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`):
+            The temporal, height and width dimensions of the patch grid of each video.
+        """
+        return self.model.get_video_features(pixel_values_videos, video_grid_thw, **kwargs)
+
+
+__all__ = ["Lfm3VlForConditionalGeneration", "Lfm3VlModel", "Lfm3VlPreTrainedModel", "Lfm3VlVisionModel"]

native_lfm3.py

@@ -0,0 +1,24 @@
+"""LFM3's transformers classes, one module a model type: `native_lfm3` (text), `native_lfm3_vl` (pictures),
+`native_lfm3_omni` (pictures and audio), each importing the one before it, so a Hub repository serves only its
+model's files. transformers' own classes once a release ships them; until then copies of transformers' LFM3 code
+beside these (each names its commit), registered with the Auto classes, so a checkpoint's config, tokenizer and
+processor load and the composite models build their parts.
+
+Shipping LFM3, transformers makes the fallbacks dead: delete them and the copies (the files headed "Vendored from"),
+nothing else."""
+
+from __future__ import annotations
+
+try:
+    from transformers import Lfm3Config, Lfm3ForCausalLM, Lfm3Model
+except ImportError:
+    from transformers import AutoConfig, AutoModel
+
+    from .configuration_lfm3 import Lfm3Config
+    from .modeling_lfm3 import Lfm3ForCausalLM, Lfm3Model
+
+    # exist_ok: a second copy of these files in one process (two snapshots) registers again
+    AutoConfig.register(Lfm3Config.model_type, Lfm3Config, exist_ok=True)
+    AutoModel.register(Lfm3Config, Lfm3Model, exist_ok=True)  # the composite models build their text model by config
+
+__all__ = ["Lfm3Config", "Lfm3ForCausalLM", "Lfm3Model"]

native_lfm3_vl.py

@@ -0,0 +1,25 @@
+"""LFM3-VL's transformers classes (`native_lfm3`): its config, vision tower, picture and video processors."""
+
+from __future__ import annotations
+
+from .native_lfm3 import Lfm3Model  # noqa: F401  (registered: the VL builds its text model by config)
+
+try:
+    from transformers import Lfm3VlConfig, Lfm3VlForConditionalGeneration, Lfm3VlProcessor
+except ImportError:
+    from transformers import AutoConfig, AutoImageProcessor, AutoModel, AutoProcessor, AutoVideoProcessor
+
+    from .configuration_lfm3_vl import Lfm3VlConfig, Lfm3VlVisionConfig
+    from .image_processing_lfm3_vl import Lfm3VlImageProcessor
+    from .modeling_lfm3_vl import Lfm3VlForConditionalGeneration, Lfm3VlVisionModel
+    from .processing_lfm3_vl import Lfm3VlProcessor
+    from .video_processing_lfm3_vl import Lfm3VlVideoProcessor
+
+    for config in (Lfm3VlConfig, Lfm3VlVisionConfig):
+        AutoConfig.register(config.model_type, config, exist_ok=True)
+    AutoModel.register(Lfm3VlVisionConfig, Lfm3VlVisionModel, exist_ok=True)  # the tower, built by config
+    AutoImageProcessor.register(Lfm3VlConfig, fast_image_processor_class=Lfm3VlImageProcessor, exist_ok=True)
+    AutoVideoProcessor.register(Lfm3VlConfig, Lfm3VlVideoProcessor, exist_ok=True)
+    AutoProcessor.register(Lfm3VlConfig, Lfm3VlProcessor, exist_ok=True)
+
+__all__ = ["Lfm3VlConfig", "Lfm3VlForConditionalGeneration", "Lfm3VlProcessor"]

parsing.py

@@ -0,0 +1,297 @@
+"""Strict readings of written values; no arithmetic, unit conversion or invented date components."""
+
+from __future__ import annotations
+
+import datetime as dt
+import math
+import re
+import unicodedata
+from decimal import Decimal
+from functools import cache
+
+# Month names a date is written with (accents folded; Polish and Russian in the genitive a date uses), and weekdays,
+# which a date may lead with.
+_MONTHS = (
+    "january janvier januar enero gennaio janeiro januari stycznia styczen января",
+    "february fevrier februar febrero febbraio fevereiro februari lutego luty февраля",
+    "march mars marz marzo marco maart marca марта",
+    "april avril abril aprile kwietnia kwiecien апреля",
+    "may mai mayo maggio maio mei maj maja мая",
+    "june juin juni junio giugno junho czerwca czerwiec июня",
+    "july juillet juli julio luglio julho lipca lipiec июля",
+    "august aout agosto augustus augusti sierpnia sierpien августа",
+    "september septembre septiembre settembre setembro wrzesnia wrzesien сентября",
+    "october octobre oktober octubre ottobre outubro pazdziernika pazdziernik октября",
+    "november novembre noviembre novembro listopada listopad ноября",
+    "december decembre dezember diciembre dicembre dezembro grudnia grudzien декабря",
+)
+MONTHS = {name: month for month, names in enumerate(_MONTHS, 1) for name in names.split()}
+_ABBREVIATIONS = {}
+for _name, _month in MONTHS.items():
+    _ABBREVIATIONS.setdefault(_name[:3], set()).add(_month)
+MONTHS.update({name: next(iter(months)) for name, months in _ABBREVIATIONS.items() if len(months) == 1})
+MONTHS.update(janv=1, fevr=2, juil=7, sept=9)
+WEEKDAYS = set("""monday tuesday wednesday thursday friday saturday sunday mon tue tues wed thu thur thurs fri sat sun
+lundi mardi mercredi jeudi vendredi samedi dimanche montag dienstag mittwoch donnerstag freitag samstag sonntag lunes
+martes miercoles jueves viernes sabado domingo lunedi martedi mercoledi giovedi venerdi sabato domenica segunda terca
+quarta quinta sexta""".split())
+CJK_DATE = re.compile(r"(\d{4})\s*[年년]\s*(\d{1,2})\s*[月월]\s*(\d{1,2})\s*[日일]")  # 2026年10月20日, 2026년 10월 20일
+NUMERIC_DATE = re.compile(r"(?<!\d)(\d{1,4})([/.-])(\d{1,2})\2(\d{2,4})(?!\d)")
+_MONTH_NAME = "|".join(sorted(MONTHS, key=len, reverse=True))
+WORD_DATE = re.compile(rf"\b(?:\d{{1,2}}\w*[ .]+(?:de\s+)?(?:{_MONTH_NAME})[ ,.]+(?:de\s+)?\d{{4}}"
+                       rf"|(?:{_MONTH_NAME})[ .]+\d{{1,2}}\w*[ ,.]+\d{{4}})\b")  # in folded text: 20 octobre 2026
+
+# Currencies by the decimal mark their amounts are written with: "" for none (whole units), None for both in use.
+CURRENCIES = {
+    ".": ("$", "£", "₹", "USD", "GBP", "CAD", "AUD", "NZD", "INR", "Rs.", "Rs", "MXN", "CNY", "RMB", "元", "HKD", "SGD"),
+    ",": ("R$", "BRL", "₽", "RUB", "zł", "PLN", "kr", "SEK", "NOK", "DKK", "₺", "TRY"),
+    "": ("¥", "￥", "円", "₩", "₫", "Rp.", "Rp", "IDR", "JPY", "KRW", "VND", "CLP"),
+    None: ("€", "EUR", "CHF"),
+}
+_NAMES = {c.casefold(): c for cs in CURRENCIES.values() for c in cs}
+_MARK = {c.casefold(): mark for mark, cs in CURRENCIES.items() for c in cs}
+_CURRENCY = "(?i:" + "|".join(rf"\b{re.escape(c)}(?![a-z])" if c[0].isascii() and c[0].isalpha() else re.escape(c)
+                              for c in sorted(_NAMES.values(), key=len, reverse=True)) + ")"
+# Scale words, written apart or attached: unambiguous, so no context needed (German "Billion" is 10^12: left out).
+SCALES = {**dict.fromkeys(("thousand", "mil", "tausend", "тыс", "千"), 10**3), **dict.fromkeys(("万", "萬"), 10**4),
+          **dict.fromkeys(("lakh", "lakhs"), 10**5), **dict.fromkeys(("crore", "crores"), 10**7),
+          **dict.fromkeys(("million", "millions", "mio", "mln", "millón", "millones", "milhão", "milhões", "milione",
+                           "milioni", "млн"), 10**6), **dict.fromkeys(("亿", "億"), 10**8),
+          **dict.fromkeys(("billion", "billions", "milliard", "milliards", "milliarde", "milliarden", "mrd", "miliardo",
+                           "miliardi", "млрд"), 10**9), "trillion": 10**12}
+_SCALE = re.compile(r"\s*(" + "|".join(sorted(map(re.escape, SCALES), key=len, reverse=True)) + r")\.?$", re.I)
+_SUFFIX = re.compile(r"(?<=\d)(k|m|bn|b)$", re.I)  # attached k, M, B: a magnitude only beside money or a count
+_COUNTED = re.compile(r"\b(?:money|price|invoice|revenue|cost|count|quantity|population|people|items|units|users|"
+                      r"downloads)\b", re.I)
+_GROUPED = re.compile(r"(?<![\d.,])\d{1,3}(?:([,. \u00a0\u202f'’])\d{3})+(?!\1)([.,])\d+(?![\d.,])")  # 1 234,50
+_SHORT = re.compile(r"(?<![\d.,])\d+([.,])\d{1,2}(?![\d.,])")  # 12,50: a decimal, never a group
+_REPEATED = re.compile(r"(?<![\d.,])\d{1,3}(?:([.,])\d{3}){2,}(?![\d.,])")  # 1.234.567: a group, never a decimal
+_LOCALE = re.compile(r'\b(?:locale|language|lang)["\']?\s*[=:]\s*["\']?(en|fr|de|es|it|pt|nl|pl|ru|sv|ja|zh|ko)'
+                     r"(?:[-_][a-z]{2})?\b")
+
+
+def _one(values: set):
+    """The one value evidence settles on; None when it holds none or disagrees with itself."""
+    return next(iter(values)) if len(values) == 1 else None
+
+
+def written_unit(text: str) -> str | None:
+    """Only an explicitly written percent or currency marker; a currency sign stays a sign."""
+    if text.rstrip().endswith("%"):
+        return "%"
+    markers = re.findall(_CURRENCY, text)
+    return _NAMES[markers[0].casefold()] if len(markers) == 1 else None
+
+
+def _currency_marks(text: str) -> set[str]:
+    """The decimal marks of the currencies `text` writes ("" for whole units); a currency of both says nothing."""
+    return {_MARK[c.casefold()] for c in re.findall(_CURRENCY, text)} - {None}
+
+
+def _decimal_mark(written: str, context: str) -> str | None:
+    """The decimal mark ("" when amounts have none), from the first evidence that speaks: the span's own currency,
+    the document's other numbers, the document's currencies, a declared locale. Evidence that disagrees with itself
+    decides nothing; a markup declaration's numbers (`<?xml version="1.0"?>`) are not the document's."""
+    context = re.sub(r"<\?[^>]*\?>", "", context)
+    numbers = {m[2] for m in _GROUPED.finditer(context)} | {m[1] for m in _SHORT.finditer(context)}
+    numbers |= {"," if m[1] == "." else "." for m in _REPEATED.finditer(context)}
+    locales = {"." if lang in ("en", "ja", "zh", "ko") else "," for lang in _LOCALE.findall(context.casefold())}
+    return _one(next((marks for marks in (_currency_marks(written), numbers, _currency_marks(context), locales) if marks),
+                     set()))
+
+
+def parse_number(text: str, context: str = "", *, unit: str | None = None) -> int | float | None:
+    """The written magnitude, percent points included: its currency and percent sign set apart, its scale word
+    applied; no implicit unit and no conversion. None when the writing does not settle one value."""
+    context += f"\n{unit}" if unit and re.fullmatch(_CURRENCY, unit.strip()) else ""
+    written = text
+    text = text.strip().replace("\u2212", "-").replace("\u00a0", " ").replace("\u202f", " ").removesuffix("%").rstrip()
+    negative = text.startswith("(") and text.endswith(")")
+    text = text[1:-1].strip() if negative else text
+    sign = currency = None
+    for _ in range(2):  # one currency and one sign, in either order
+        if currency is None and (m := re.fullmatch(rf"({_CURRENCY})\s*(.*)|(.*?)\s*({_CURRENCY})", text)):
+            currency, text = m[1] or m[4], (m[2] if m[1] else m[3]).strip()
+        if sign is None and text[:1] in ("-", "+"):
+            sign, text = text[0], text[1:].strip()
+    if negative and sign:
+        return None
+    sign = sign or ""
+    scale = 1
+    if word := _SCALE.search(text):
+        scale, text = SCALES[word[1].lower()] if word[1] != "Billion" else 0, text[: word.start()]
+    elif suffix := _SUFFIX.search(text):
+        if not (re.search(_CURRENCY, written + "\n" + context) or _COUNTED.search(context)):
+            return None
+        scale, text = {"k": 10**3, "m": 10**6, "b": 10**9, "bn": 10**9}[suffix[1].lower()], text[: suffix.start()]
+    digits = _digits(text, written, context)
+    if digits is None or not scale:
+        return None
+    value = Decimal(("-" if negative else sign) + digits) * scale
+    if value == value.to_integral_value():
+        return int(value)
+    result = float(value)
+    return result if math.isfinite(result) else None
+
+
+def _digits(text: str, written: str, context: str) -> str | None:
+    """`text`'s number as plain digits and a point: the decimal mark from the evidence, the other mark a complete
+    grouping (threes, or India's lakh grouping beside a rupee)."""
+    if not re.fullmatch(r"\d[\d., '’]*", text):
+        return None
+    if re.search(r"[ '’]", text):  # a space or an apostrophe only groups
+        grouping = re.fullmatch(r"(\d{1,3}(?:[ '’]\d{3})+)([.,]\d+)?", text)
+        return None if grouping is None else _digits(re.sub(r"[ '’]", "", grouping[1]) + (grouping[2] or ""), written,
+                                                       context)
+    if re.fullmatch(r"\d{1,2}(?:,\d{2})+,\d{3}(?:\.\d+)?", text) and re.search(r"₹|\bINR\b|\bRs\b|lakh|crore",
+                                                                              written + "\n" + context, re.I):
+        return text.replace(",", "")
+    if "." in text and "," in text:
+        point, group = (".", ",") if text.rfind(".") > text.rfind(",") else (",", ".")
+        if not re.fullmatch(rf"\d{{1,3}}(?:{re.escape(group)}\d{{3}})+{re.escape(point)}\d+", text):
+            return None
+        return text.replace(group, "").replace(point, ".")
+    if "." not in text and "," not in text:
+        return text
+    mark = "." if "." in text else ","
+    parts = text.split(mark)
+    if not all(parts):
+        return None
+    point = _decimal_mark(written, context)
+    if len(parts) > 2 or point is not None and point != mark:  # the mark groups: complete threes
+        return "".join(parts) if 1 <= len(parts[0]) <= 3 and all(len(p) == 3 for p in parts[1:]) else None
+    if point is None and len(parts[1]) == 3 and len(parts[0]) <= 3:  # 1,250: a group or a decimal, unsettled
+        return None
+    return ".".join(parts)
+
+
+def _fold(text: str) -> str:
+    return "".join(c for c in unicodedata.normalize("NFKD", text.casefold()) if not unicodedata.combining(c))
+
+
+# a declared numeric date format, in English, French (jj/mm/aaaa), German (tt.mm.jjjj), Spanish or Italian (gg/mm/aaaa)
+_DAY, _MONTH, _YEAR = r"(?:dd?|jj?|tt?|gg?|day|jour|tag|dia|giorno)", r"(?:mm?|month|mois|monat|mes|mese)", \
+    r"(?:yy(?:yy)?|aa(?:aa)?|jj(?:jj)?|year|annee|jahr|ano|anno)"
+DATE_FORMATS = ((rf"\b{_DAY}\s*[/.-]\s*{_MONTH}\s*[/.-]\s*{_YEAR}\b", "dmy"),
+                (rf"\b{_MONTH}\s*[/.-]\s*{_DAY}\s*[/.-]\s*{_YEAR}\b", "mdy"))
+
+
+def date_order(context: str, folded: str | None = None) -> str | None:
+    """Declared format/locale first, then the state's other unambiguous numeric dates; conflicts stay unresolved.
+    `folded`: `context` already folded."""
+    folded = _fold(context) if folded is None else folded
+    explicit = set()
+    for pattern, order in DATE_FORMATS:
+        if re.search(pattern, folded):
+            explicit.add(order)
+    if explicit:
+        return _one(explicit)
+    locales = re.findall(
+        r'\b(?:locale|language|lang|langue|sprache|idioma|lingua)["\']?\s*[=:]\s*["\']?'
+        r"(en(?:[-_](?:us|gb|au|ca))?|fr(?:[-_]fr)?|de(?:[-_]de)?|es(?:[-_]es)?|it(?:[-_]it)?)\b",
+        folded,
+    )
+    if locales:
+        return _one({"mdy" if locale.replace("_", "-") in ("en", "en-us") else "dmy" for locale in locales})
+    orders = set()
+    for match in NUMERIC_DATE.finditer(context):
+        a, b, year = int(match[1]), int(match[3]), int(match[4])
+        if len(match[4]) != 4 or a > 31 or b > 31 or a < 1 or b < 1:
+            continue
+        for day, month, order in ((a, b, "dmy"), (b, a, "mdy")):
+            try:
+                dt.date(year, month, day)
+            except ValueError:
+                continue
+            if (a > 12) != (b > 12):
+                orders.add(order)
+    return _one(orders)
+
+
+def _candidates(year: int, a: int, b: int, order: str | None) -> list[tuple[int, int, int]]:
+    """(year, month, day) for `a`/`b`/year: day first, month first, or the one the context's `order` says."""
+    dmy, mdy = (year, b, a), (year, a, b)
+    return [dmy, mdy] if order is None else [dmy if order == "dmy" else mdy]
+
+
+def parse_date(text: str, context: str = "") -> tuple[str | None, bool]:
+    """ISO value and unresolved-ambiguity flag; an invalid or unrecognised span is (None, False)."""
+    folded = cache(lambda: _fold(context))  # the context folded and its date order, once, when a date needs them
+    return _parse_date(text, context, folded, cache(lambda: date_order(context, folded())))
+
+
+def _parse_date(text: str, context: str, folded, order) -> tuple[str | None, bool]:
+    text = text.strip()
+    if not text:
+        return None, False
+    # Keep an explicitly written time and timezone, with no default time or timezone.
+    timed = re.fullmatch(r"(.+?)[T\s]+(\d{1,2}:\d{2}(?::\d{2}(?:\.\d+)?)?(?:Z|[+-]\d{2}:\d{2})?)", text)
+    if timed:
+        date, ambiguous = _parse_date(timed[1], context, folded, order)
+        if date is None:
+            return None, ambiguous
+        try:
+            clock = timed[2]
+            if clock[1] == ":":
+                clock = "0" + clock
+            return dt.datetime.fromisoformat(date + "T" + clock).isoformat(), False
+        except ValueError:
+            return None, False
+    numeric = NUMERIC_DATE.fullmatch(text)
+    if numeric:
+        a, b, c = int(numeric[1]), int(numeric[3]), int(numeric[4])
+        if len(numeric[1]) == 4:
+            candidates = [(a, b, c)]
+        elif len(numeric[4]) == 4:
+            candidates = _candidates(c, a, b, order())
+        else:
+            # A century is never invented: another unambiguous complete date must establish it.
+            if len(numeric[4]) != 2:
+                return None, False
+            centuries = set()
+            complete = [
+                match[0] for match in NUMERIC_DATE.finditer(context) if len(match[1]) == 4 or len(match[4]) == 4
+            ]
+            complete.extend(match[0] for match in WORD_DATE.finditer(folded()))
+            for candidate in complete:
+                value, ambiguous = _parse_date(candidate, context, folded, order)
+                if value is not None and not ambiguous:
+                    centuries.add(int(value[:4]) // 100 * 100)
+            if len(centuries) != 1:
+                return None, False
+            candidates = _candidates(centuries.pop() + c, a, b, order())
+        values = set()
+        for year, month, day in candidates:
+            try:
+                values.add(dt.date(year, month, day).isoformat())
+            except ValueError:
+                pass
+        if len(values) > 1:
+            return None, True
+        return next(iter(values), None), False
+    if cjk := CJK_DATE.fullmatch(text):
+        try:
+            return dt.date(*map(int, cjk.groups())).isoformat(), False
+        except ValueError:
+            return None, False
+    words = _fold(text)
+    words = re.sub(r"(?<=\d)(?:st|nd|rd|th|er)\b", "", words)
+    words = re.sub(r"(?<=\d)\.?(?:o|a|°)\b", "", words)
+    words = re.sub(r"\b(?:de|of|the|le|el|il|den)\b", " ", words)
+    words = re.sub(r"[,.]", " ", words).split()
+    if words and words[0].removesuffix("-feira") in WEEKDAYS:  # Tue, 20 Oct 2026; segunda-feira, 20 de ...
+        words = words[1:]
+    if len(words) != 3:
+        return None, False
+    if words[0] in MONTHS:
+        month, day, year = MONTHS[words[0]], words[1], words[2]
+    elif words[1] in MONTHS:
+        day, month, year = words[0], MONTHS[words[1]], words[2]
+    else:
+        return None, False
+    if not day.isdigit() or not year.isdigit() or len(year) != 4:
+        return None, False
+    try:
+        return dt.date(int(year), month, int(day)).isoformat(), False
+    except ValueError:
+        return None, False

pointer.py

@@ -0,0 +1,166 @@
+"""Parameter-free readouts over a row's state positions, with BOS0 for nothing."""
+
+from __future__ import annotations
+
+import math
+from bisect import bisect_right
+from dataclasses import dataclass
+
+import torch
+import torch.nn.functional as F
+
+from .json_spans import json_text, string_paths  # torch-free, so code without torch reads spans too
+
+SPAN_CAPS = {"text": 512, "number": 32, "date": 32}  # a read's longest span, in tokens
+
+
+@dataclass(frozen=True)
+class StatePositions:
+    """Token positions mapped to the caller's rendered state; None means not a text position."""
+
+    text: str
+    offsets: tuple[tuple[int, int] | None, ...]
+    paths: tuple[tuple[int, int, str], ...] = ()
+
+    @property
+    def valid(self) -> list[bool]:
+        return [i == 0 or span is not None for i, span in enumerate(self.offsets)]
+
+    @property
+    def pointer_mask(self) -> list[bool]:
+        """`valid` up to its last valid position: the keys a read's pointer scores."""
+        valid = self.valid
+        return valid[:max(i for i, v in enumerate(valid) if v) + 1]
+
+    def read(self, first: int, last: int, max_tokens: int = SPAN_CAPS["text"]) -> dict:
+        if first == last == 0:
+            return {"text": None, "span": None}
+        if not 0 < first <= last < len(self.offsets) or last - first >= max_tokens:
+            raise ValueError("invalid pointer endpoints")
+        a, b = self.offsets[first], self.offsets[last]
+        if a is None or b is None:
+            raise ValueError("pointer leaves the state text")
+        span = [a[0], b[1]]
+        # A token can carry what is around a value: the separating space, or a JSON string's quote (` €",`). Trim
+        # boundary space and clip to the one string the span starts or ends in; never alter a character inside.
+        while span[0] < span[1] and self.text[span[0]].isspace():
+            span[0] += 1
+        while span[1] > span[0] and self.text[span[1] - 1].isspace():
+            span[1] -= 1
+        held = [(start, end, p) for start, end, p in self.paths if start < span[1] and span[0] < end]
+        if len(held) == 1:
+            start, end, path = held[0]
+            span = [max(span[0], start), min(span[1], end)]
+            # inside one JSON string: the caller's characters, not their escapes
+            return {"text": json_text(self.text[slice(*span)]), "span": span, "path": path}
+        return {"text": self.text[slice(*span)], "span": span}
+
+
+def state_positions(
+    tokenizer, state, prefix: str, full_text: str, style: str = "json_only", *, encoded=None
+) -> StatePositions:
+    """Align one unchanged rendered prompt to text tokens; template, picture and question tokens stay invalid."""
+    from .prompt import plain_offsets, question_header, state_block, state_text
+
+    if not full_text.startswith(prefix):
+        raise ValueError("the tokenized prompt must start with the state prefix")
+    if state is None:
+        body, escaped, boundaries = "", "", [0]
+        start = len(prefix)
+    else:
+        body = state_block(state, style)
+        escaped, boundaries = plain_offsets(tokenizer, body)
+        tail = escaped + "\n" + question_header(tokenizer, state)
+        if not prefix.endswith(tail):
+            raise ValueError("state rendering differs from the prompt prefix")
+        start = len(prefix) - len(tail)
+    encoded = (
+        tokenizer(full_text, add_special_tokens=False, return_offsets_mapping=True) if encoded is None else encoded
+    )
+    text = state_text(state, style) if state is not None else ""  # the blank line after it is the template's
+    stop = start + bisect_right(boundaries, len(text)) - 1
+    offsets = []
+    for a, b in encoded["offset_mapping"]:
+        span = None
+        lo, hi = max(a, start), min(b, stop)
+        if lo < hi and not full_text[a:lo].strip() and not full_text[hi:b].strip():
+            left, right = boundaries[lo - start], boundaries[hi - start]
+            if left < right <= len(text) and text[left:right].strip():
+                span = (left, right)
+        offsets.append(span)
+    if not offsets or offsets[0] is not None:
+        raise ValueError("position 0 must be a template token for the nothing readout")
+    paths = (
+        string_paths(text)
+        if state is not None and not isinstance(state, str) and style.endswith("_only") and style.startswith("json")
+        else []
+    )
+    return StatePositions(text, tuple(offsets), tuple(paths))
+
+
+def position_logits(query: torch.Tensor, state: torch.Tensor, valid: torch.Tensor) -> torch.Tensor:
+    """[B, slots, D] against [B, T, D], sharing state keys between questions; valid [B, T], boolean, includes BOS0."""
+    dtype = torch.float64 if query.dtype == state.dtype == torch.float64 else torch.float32
+    logits = torch.bmm(query.to(dtype), state.to(dtype).transpose(1, 2)) / math.sqrt(query.shape[-1])
+    return logits.masked_fill(~valid[:, None], -torch.inf)
+
+
+def read_logits(hidden: torch.Tensor, states: torch.Tensor, plan: dict) -> torch.Tensor:
+    """The flat read slots of a mixed batch; queries share each retained state's keys."""
+    device = hidden.device
+    query = hidden[plan["queries"].to(device)]
+    logits = position_logits(query, states, plan["valid"].to(device))
+    return logits[plan["query_row"].to(device), plan["query_col"].to(device)]
+
+
+def query_plan(rows: list[int], masks: list[torch.Tensor], slots: list[int], count: int) -> dict:
+    """The read slots of a batch of `count` slots, grouped by their row's state: each state's keys shared by its
+    queries; `code_slots`, the other slots, which the LM head reads."""
+    unique = sorted(set(rows))
+    index = {r: i for i, r in enumerate(unique)}
+    width = max(map(len, masks))
+    valid = torch.zeros(len(unique), width, dtype=torch.bool)
+    groups = [[] for _ in unique]
+    row_index, column_index = [], []
+    for slot, row, keep in zip(slots, rows, masks):
+        r = index[row]
+        if groups[r] and not torch.equal(valid[r, : len(keep)], keep):
+            raise ValueError("read questions must share their state-position mask")
+        valid[r, : len(keep)] = keep
+        row_index.append(r)
+        column_index.append(len(groups[r]))
+        groups[r].append(slot)
+    queries = torch.zeros(len(unique), max(map(len, groups)), dtype=torch.long)
+    for r, group in enumerate(groups):
+        queries[r, : len(group)] = torch.tensor(group)
+    return dict(
+        rows=torch.tensor(unique),
+        width=width,
+        queries=queries,
+        valid=valid,
+        query_row=torch.tensor(row_index),
+        query_col=torch.tensor(column_index),
+        slots=torch.tensor(slots),
+        code_slots=torch.tensor(sorted(set(range(count)) - set(slots)), dtype=torch.long),
+    )
+
+
+def spans(logits: torch.Tensor, max_tokens: int = SPAN_CAPS["text"]) -> tuple[torch.Tensor, torch.Tensor]:
+    """Each read's span from its start and end scores [R, 2, T]: found when P(found) >= 0.5, then the likeliest span
+    within `max_tokens`; (0, 0) for nothing."""
+    logp = logits.log_softmax(-1)
+    start, end = logp.unbind(1)
+    found = -torch.expm1(start[:, 0])
+    if logits.shape[2] == 1:
+        return torch.zeros(logits.shape[0], 2, dtype=torch.long, device=logits.device), found
+    cap = min(max_tokens, logits.shape[2] - 1)
+    windows, indices = F.max_pool1d(
+        F.pad(end[:, None, 1:], (0, cap - 1), value=-torch.inf), cap, stride=1, return_indices=True
+    )
+    scores = start[:, 1:] + windows[:, 0]
+    index = scores.argmax(-1)
+    first = index + 1
+    last = indices[:, 0].gather(1, index[:, None])[:, 0] + 1
+    at = torch.stack((first, last), -1)
+    nothing = found < 0.5
+    return torch.where(nothing[:, None], torch.zeros_like(at), at), found

processing_lfm3_vl.py

@@ -0,0 +1,254 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3_vl/processing_lfm3_vl.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3_vl/modular_lfm3_vl.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3_vl.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import math
+
+import numpy as np
+
+from transformers.image_utils import is_pil_image, is_valid_image, make_flat_list_of_images
+from transformers.processing_utils import MultiModalData, ProcessingKwargs, ProcessorMixin
+from transformers.utils import auto_docstring, is_numpy_array, is_torch_tensor, logging
+from transformers.video_utils import VideoMetadata, is_valid_video, make_batched_videos
+
+
+logger = logging.get_logger(__name__)
+
+
+class Lfm3VlProcessorKwargs(ProcessingKwargs, total=False):
+    _defaults = {
+        "text_kwargs": {
+            "padding": False,
+            "return_token_type_ids": False,
+            "return_mm_token_type_ids": False,
+        },
+        "videos_kwargs": {"return_metadata": True},
+    }
+
+
+def _check_video_fps(metadata: VideoMetadata) -> None:
+    if metadata.fps is not None and not (math.isfinite(metadata.fps) and metadata.fps > 0):
+        raise ValueError(f"`video_metadata.fps` must be a positive number, got {metadata.fps}")
+
+
+def _media_per_prompt(media, is_item) -> list[int] | None:
+    # rows name the prompt that owns them while the expansion consumes media in flat order; a row is a list of whole
+    # items or one stacked batch, and a list of single frames is one video, not a row
+    if not isinstance(media, (list, tuple)):
+        return None
+    counts = []
+    for row in media:
+        if isinstance(row, (list, tuple)) and all(is_item(item) for item in row):
+            counts.append(len(row))
+        elif (is_numpy_array(row) or is_torch_tensor(row)) and row.ndim == 4 and (len(row) == 0 or is_item(row[0])):
+            counts.append(len(row))
+        else:
+            return None
+    return counts
+
+
+def _check_vision_placeholder_counts(processor, prompts, images, videos) -> None:
+    for name, token, media, flatten, is_item in (
+        (
+            "images",
+            processor.image_token,
+            images,
+            make_flat_list_of_images,
+            lambda item: is_valid_image(item) and (is_pil_image(item) or item.ndim == 3),
+        ),
+        (
+            "videos",
+            processor.video_token,
+            videos,
+            make_batched_videos,
+            lambda item: isinstance(item, str) or (not is_pil_image(item) and is_valid_video(item)),
+        ),
+    ):
+        num_placeholders = sum(prompt.count(token) for prompt in prompts)
+        num_media = 0 if media is None else len(flatten(media))
+        if num_placeholders != num_media:
+            raise ValueError(f"Found {num_placeholders} {token} tokens in the text but received {num_media} {name}")
+        # one prompt cannot take another's media
+        per_prompt = _media_per_prompt(media, is_item) if len(prompts) > 1 else None
+        in_text = [prompt.count(token) for prompt in prompts]
+        if per_prompt is not None and per_prompt != in_text:
+            raise ValueError(f"Found {in_text} {token} tokens per prompt but received {per_prompt} {name} per prompt")
+
+
+@auto_docstring
+class Lfm3VlProcessor(ProcessorMixin):
+    valid_processor_kwargs = Lfm3VlProcessorKwargs
+
+    def __init__(self, image_processor=None, tokenizer=None, video_processor=None, chat_template=None, **kwargs):
+        self.image_token = getattr(tokenizer, "image_token", "<|image|>")
+        self.video_token = getattr(tokenizer, "video_token", "<|video|>")
+        self.vision_start_token = getattr(tokenizer, "vision_start_token", "<|vision_start|>")
+        self.vision_end_token = getattr(tokenizer, "vision_end_token", "<|vision_end|>")
+        self.image_token_id = tokenizer.convert_tokens_to_ids(self.image_token)
+        self.video_token_id = tokenizer.convert_tokens_to_ids(self.video_token)
+        self.vision_start_token_id = tokenizer.convert_tokens_to_ids(self.vision_start_token)
+        self.vision_end_token_id = tokenizer.convert_tokens_to_ids(self.vision_end_token)
+        super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
+
+    def replace_image_token(self, image_inputs: dict, image_idx: int, **kwargs) -> str:
+        merge_length = self.image_processor.merge_size**2
+        num_image_tokens = image_inputs["image_grid_thw"][image_idx].prod() // merge_length
+        return self.image_token * num_image_tokens
+
+    def replace_video_token(self, video_inputs: dict, video_idx: int, **kwargs) -> str:
+        metadata = video_inputs["video_metadata"][video_idx]
+        if metadata.frames_indices is None:
+            raise ValueError("Frame timestamps need the sampled frame indices in `video_metadata`")
+        _check_video_fps(metadata)
+        merge_length = self.video_processor.merge_size**2
+        num_frames = video_inputs["video_grid_thw"][video_idx][0]
+        frame_seqlen = video_inputs["video_grid_thw"][video_idx][1:].prod() // merge_length
+        metadata = video_inputs["video_metadata"][video_idx]
+        video_placeholder = ""
+
+        if metadata.fps is None:
+            logger.warning_once(
+                "Lfm3Vl requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
+                "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
+                "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+            )
+        metadata.fps = 24 if metadata.fps is None else metadata.fps
+
+        # if timestamps are not provided, calculate them
+        curr_timestamp = self._calculate_timestamps(
+            metadata.frames_indices,
+            metadata.fps,
+            self.video_processor.temporal_patch_size,
+        )
+
+        for frame_idx in range(num_frames):
+            curr_time = curr_timestamp[frame_idx]
+            video_placeholder += f"<{curr_time:.1f} seconds>"
+            video_placeholder += self.vision_start_token + self.video_token * frame_seqlen + self.vision_end_token
+        return video_placeholder
+
+    def _get_num_multimodal_tokens(self, image_sizes=None, video_sizes=None, **kwargs):
+        """
+        Computes the number of placeholder tokens needed for multimodal inputs with the given sizes.
+
+        Args:
+            image_sizes (`list[list[int]]`, *optional*):
+                The input sizes formatted as (height, width) per each image.
+            video_sizes (`list[list[int]]`, *optional*):
+                The input sizes formatted as (num_frames, height, width) per each video.
+
+        Returns:
+            `MultiModalData`: the number of tokens per provided modality.
+        """
+        vision_data = {}
+        if image_sizes is not None:
+            # copy: `update` on the class-level defaults would leak kwargs into every later call
+            images_kwargs = dict(Lfm3VlProcessorKwargs._defaults.get("images_kwargs", {}))
+            images_kwargs.update(kwargs)
+            merge_size = images_kwargs.get("merge_size") or self.image_processor.merge_size
+
+            num_image_patches = [
+                self.image_processor.get_number_of_image_patches(*image_size, images_kwargs)
+                for image_size in image_sizes
+            ]
+            num_image_tokens = [(num_patches // merge_size**2) for num_patches in num_image_patches]
+            vision_data.update({"num_image_tokens": num_image_tokens, "num_image_patches": num_image_patches})
+
+        if video_sizes is not None:
+            videos_kwargs = dict(Lfm3VlProcessorKwargs._defaults.get("videos_kwargs", {}))
+            videos_kwargs.update(kwargs)
+            merge_size = videos_kwargs.get("merge_size") or self.video_processor.merge_size
+            num_video_patches = [
+                self.video_processor.get_num_of_video_patches(*video_size, videos_kwargs) for video_size in video_sizes
+            ]
+            num_video_tokens = [(num_patches // merge_size**2) for num_patches in num_video_patches]
+            vision_data["num_video_tokens"] = num_video_tokens
+
+        return MultiModalData(**vision_data)
+
+    def post_process_image_text_to_text(
+        self, generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False, **kwargs
+    ):
+        """
+        Post-process the output of the model to decode the text.
+
+        Args:
+            generated_outputs (`torch.Tensor` or `np.ndarray`):
+                The output of the model `generate` function. The output is expected to be a tensor of shape `(batch_size, sequence_length)`
+                or `(sequence_length,)`.
+            skip_special_tokens (`bool`, *optional*, defaults to `True`):
+                Whether or not to remove special tokens in the output. Argument passed to the tokenizer's `batch_decode` method.
+            clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
+                Whether or not to clean up the tokenization spaces. Argument passed to the tokenizer's `batch_decode` method.
+            **kwargs:
+                Additional arguments to be passed to the tokenizer's `batch_decode method`.
+
+        Returns:
+            `list[str]`: The decoded text.
+        """
+        return self.tokenizer.batch_decode(
+            generated_outputs,
+            skip_special_tokens=skip_special_tokens,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+
+    @property
+    def model_input_names(self):
+        return list(
+            dict.fromkeys(
+                self.tokenizer.model_input_names
+                + self.image_processor.model_input_names
+                + self.video_processor.model_input_names
+            )
+        )
+
+    def _calculate_timestamps(self, indices: list[int] | np.ndarray, video_fps: float, merge_size: int = 2):
+        if not isinstance(indices, list):
+            indices = indices.tolist()
+        if len(indices) % merge_size != 0:
+            indices.extend(indices[-1] for _ in range(merge_size - len(indices) % merge_size))
+        timestamps = [idx / video_fps for idx in indices]
+        # @JJJYmmm frames are merged by self.merge_size, \
+        # so we need to average the timestamps between the first/last frame within the temporal patch
+        timestamps = [
+            (timestamps[i] + timestamps[i + merge_size - 1]) / 2 for i in range(0, len(timestamps), merge_size)
+        ]
+        return timestamps
+
+    def prepare_inputs_layout(self, images=None, text=None, videos=None, audio=None, **kwargs):
+        for name, media in (("images", images), ("videos", videos)):
+            if isinstance(media, (list, tuple)) and len(media) == 0:
+                raise ValueError(f"`{name}` must hold at least one item; pass `{name}=None` for prompts without one")
+        return super().prepare_inputs_layout(images=images, text=text, videos=videos, audio=audio, **kwargs)
+
+    def validate_inputs(self, images=None, text=None, videos=None, audio=None, **kwargs):
+        super().validate_inputs(images=images, text=text, videos=videos, audio=audio, **kwargs)
+        if text is not None:
+            _check_vision_placeholder_counts(self, [text] if isinstance(text, str) else text, images, videos)
+
+    def _process_videos(self, videos, **kwargs):
+        # the frame timestamps need the metadata whatever the caller's `return_metadata`; `__call__` drops it from
+        # the output unless it was asked for
+        kwargs["return_metadata"] = True
+        return super()._process_videos(videos, **kwargs)
+
+
+__all__ = ["Lfm3VlProcessor"]

processor_config.json

@@ -0,0 +1,62 @@
+{
+  "image_processor": {
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_processor_type": "Lfm3VlImageProcessor",
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "merge_size": 2,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "longest_edge": 16777216,
+      "shortest_edge": 65536
+    },
+    "temporal_patch_size": 2
+  },
+  "processor_class": "Lfm3VlProcessor",
+  "video_processor": {
+    "cap_pixels_per_frame": false,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "do_sample_frames": true,
+    "fps": 2.0,
+    "image_mean": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "image_std": [
+      0.5,
+      0.5,
+      0.5
+    ],
+    "max_frames": 768,
+    "max_video_tokens": 768,
+    "merge_size": 2,
+    "min_frames": 4,
+    "patch_size": 16,
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "return_metadata": false,
+    "size": {
+      "longest_edge": 25165824,
+      "shortest_edge": 4096
+    },
+    "temporal_patch_size": 2,
+    "video_processor_type": "Lfm3VlVideoProcessor"
+  }
+}

prompt.py

@@ -0,0 +1,761 @@
+"""The prompt and the readout. A System One decision is a single forward pass that stops at the answer
+slot: everything the model sees is built here, and the answer is a softmax over its options' tokens.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+import re
+from dataclasses import dataclass
+from functools import cached_property, lru_cache
+from typing import Any, Mapping, Sequence
+
+import numpy as np
+
+from .media import SPOKEN, Media, as_media
+
+# The system turn: none by default.
+SYSTEMS: dict[str, str | None] = {"none": None, "d1": "You are d1, a decision model trained by Liquid AI."}
+DEFAULT_SYSTEM = "none"
+
+IM_START = "<|im_start|>"
+IM_END = "<|im_end|>"
+
+
+# --------------------------------------------------------------------------- #
+# question types
+# --------------------------------------------------------------------------- #
+
+
+@dataclass(frozen=True)
+class Choice:
+    instructions: str | Media  # a spoken question's clip (`media`)
+    criteria: Mapping[str, str | None]
+    type: str = "choice"
+    multiple: bool = False  # each option its own yes or no
+
+
+@dataclass(frozen=True)
+class Noul:
+    instructions: str | Media
+    criteria: Mapping[str, Any] | None = None
+    type: str = "noul"
+
+
+@dataclass(frozen=True)
+class Score:
+    instructions: str | Media
+    criteria: Sequence[str]
+    type: str = "score"
+
+
+@dataclass(frozen=True)
+class Read:
+    """A span read from the state: text, or a number or a date written there; `unit`, a number's."""
+
+    instructions: str | Media
+    type: str = "text"
+    unit: str | None = None
+
+
+@dataclass(frozen=True)
+class Estimate:
+    """A number estimated on a grid over the declared range: `step` apart, at most 128 values (a finer step
+    is read on 128)."""
+
+    instructions: str | Media
+    low: float
+    high: float
+    step: float | None = None
+    unit: str | None = None
+    type: str = "number"
+
+    @cached_property
+    def grid(self):
+        from .grid import Grid
+
+        return Grid(self.low, self.high, 128 if self.step is None else min(128, round((self.high - self.low) / self.step) + 1))
+
+
+@dataclass(frozen=True)
+class Box:
+    """Where something is in the picture: its four edges, or nothing."""
+
+    instructions: str | Media
+    type: str = "box"
+
+
+@dataclass(frozen=True)
+class Boxes:
+    """Every instance of something in the picture, in reading order, none if absent: `max` blocks read, else
+    `box.INSTANCES`. A readout choice: the question's text never says it."""
+
+    instructions: str | Media
+    type: str = "boxes"
+    max: int | None = None
+
+    @property
+    def blocks(self) -> int:
+        from .box import INSTANCES
+
+        return INSTANCES if self.max is None else self.max
+
+
+@dataclass(frozen=True)
+class Position:
+    name: str
+
+
+@dataclass(frozen=True)
+class Number:
+    """A slot over a grid's values as the model writes them: value i times `scale`, an integer, is one token of the
+    tokenizer's own (`number_ids`). A checkpoint that records `d1_numbers` reads its grids so."""
+
+    instructions: str
+    grid: Any
+    scale: int = 1
+    type: str = "grid"
+
+
+Question = Choice | Noul | Score | Read | Estimate | Box | Boxes
+
+
+TYPED = ("text", "number", "date", "box", "boxes")  # the typed answers' kinds; a choice is one when `multiple`
+
+
+def typed(q: Question) -> bool:
+    """A typed answer, read at its own slots or state positions and answered in its own shape (api.py):
+    a read, an estimate, a box, a multiple choice."""
+    return q.type in TYPED or isinstance(q, Choice) and q.multiple
+
+
+def cardinality(q: Question) -> int:
+    if isinstance(q, (Estimate, Number)):
+        return q.grid.size
+    return 2 if isinstance(q, Noul) else len(q.criteria)
+
+
+# --------------------------------------------------------------------------- #
+# verbalizer
+# --------------------------------------------------------------------------- #
+
+
+PROMPTS = (1, 2)
+
+
+def version(tokenizer) -> int:
+    """The prompt version the checkpoint behind `tokenizer` trained with (`PROMPTS`)."""
+    return getattr(tokenizer, "d1_prompt", 1)
+
+
+def most_blocks(tokenizer) -> int:
+    """The most blocks a `boxes` question's `max` asks of this checkpoint: the 16 prompt 1 trained, else
+    `box.MAX_INSTANCES`."""
+    from .box import INSTANCES, MAX_INSTANCES
+
+    return INSTANCES if version(tokenizer) == 1 else MAX_INSTANCES
+
+
+LETTERS = tuple(chr(ord("A") + i) for i in range(26))
+
+
+def option_codes(labels: Sequence[str], scheme: int = 1) -> list[str]:
+    """Native letters when the labels already are letters, else A..Z, else the `scheme`'s codes: 00.. (1), A..Z then
+    AA, AB.. (2); one rule for every cardinality."""
+    labs = [str(x).strip() for x in labels]
+    if labs and all(len(k) == 1 and k.isalpha() for k in labs):
+        return labs
+    if len(labs) <= 26:
+        return list(LETTERS[: len(labs)])
+    if scheme == 2:
+        return (list(LETTERS) + [a + b for a in LETTERS for b in LETTERS])[: len(labs)]
+    return [f"{i:02d}" for i in range(len(labs))]
+
+
+_FALLBACK_POOL = (
+    [chr(c) for c in range(ord("A"), ord("Z") + 1)]
+    + [f"{i:02d}" for i in range(100)]
+    + [chr(c) for c in range(ord("a"), ord("z") + 1)]
+    + [f"#{i}" for i in range(200)]
+    # Where a tokenizer splits digits, "00".."99" and "#i" are two tokens, and two capital letters are
+    # often one. Last in the pool, so a tokenizer with digit pairs never reaches it.
+    + [chr(a) + chr(b) for a in range(ord("A"), ord("Z") + 1) for b in range(ord("A"), ord("Z") + 1)]
+)
+
+
+@lru_cache(maxsize=4096)
+def _aliases_cached(tokenizer_key: int, codes: tuple[str, ...]) -> tuple[tuple[str, int], ...]:
+    tokenizer = _TOKENIZERS[tokenizer_key]
+    used: set[int] = set()
+    out: list[tuple[str, int]] = []
+
+    def take(raw: str) -> bool:
+        enc = tokenizer.encode(raw, add_special_tokens=False)
+        if len(enc) != 1 or enc[0] in used:
+            return False
+        out.append((raw, enc[0]))
+        used.add(enc[0])
+        return True
+
+    for code in codes:
+        if take(code):
+            continue
+        if not any(take(raw) for raw in _FALLBACK_POOL):
+            raise RuntimeError(f"no single-token alias left for {len(codes)} options")
+    return tuple(out)
+
+
+_TOKENIZERS: dict[int, object] = {}
+
+
+def aliases(tokenizer, labels: Sequence[str]) -> list[tuple[str, int]]:
+    """Every label's distinct single-token code: [(code, token_id)]. Memoised on the codes, which are positional unless
+    the labels are letters, so every ordering of one option list shares an entry."""
+    _TOKENIZERS.setdefault(id(tokenizer), tokenizer)
+    codes = tuple(option_codes(labels, getattr(tokenizer, "d1_codes", 1)))
+    return list(_aliases_cached(id(tokenizer), codes))
+
+
+@lru_cache(maxsize=8192)
+def _ids_cached(tokenizer_key: int, texts: tuple[str, ...]) -> tuple[int, ...]:
+    tokenizer = _TOKENIZERS[tokenizer_key]
+    out, seen = [], set()
+    for t in texts:
+        enc = tokenizer.encode(t, add_special_tokens=False)
+        if len(enc) == 1 and enc[0] not in seen:
+            out.append(enc[0])
+            seen.add(enc[0])
+    return tuple(out)
+
+
+def _ids(tokenizer, texts: Sequence[str]) -> list[int]:
+    _TOKENIZERS.setdefault(id(tokenizer), tokenizer)
+    return list(_ids_cached(id(tokenizer), tuple(texts)))
+
+
+QUESTION_TYPES = ("noul", "choice", "score", "text", "number", "date", "box", "boxes")
+CHOICE = '"criteria" names the options: {"billing": "Charges and refunds", "shipping": "Delivery"}'
+SCORE = '"criteria" lists 2 to 10 levels, lowest first: ["Can wait", "Today", "Now"]'
+
+
+def check_question(q: Any, tokenizer=None) -> None:
+    """A caller's question in the API's JSON, refused with what to change; a class passes. `tokenizer` carries the
+    checkpoint's prompt version, which bounds a `boxes` question's `max` (`most_blocks`; prompt 1 without it)."""
+    if isinstance(q, (Choice, Noul, Score, Read, Estimate, Box, Boxes)):
+        return
+    if not isinstance(q, Mapping):
+        raise ValueError(f'a question is a dict such as {{"type": "noul", "instructions": "Is it urgent?"}}, not '
+                         f"{type(q).__name__}")
+    kind, said, criteria = q.get("type"), q.get("instructions"), q.get("criteria")
+    if kind not in QUESTION_TYPES:
+        raise ValueError(f'"type" is one of {", ".join(QUESTION_TYPES)}, not {kind!r}')
+    if not (isinstance(said, str) and said.strip() or isinstance(said, Mapping)):
+        raise ValueError('"instructions" is the question as text (or a spoken one, {"type": "audio", "path": ...})')
+    if kind == "choice" and not (isinstance(criteria, Mapping) and criteria
+                                 and all(isinstance(k, str) and (v is None or isinstance(v, str))
+                                         for k, v in criteria.items())):
+        raise ValueError(f"a choice's {CHOICE}" + ("; a list of levels is a score" if isinstance(criteria, list) else ""))
+    if kind == "score" and not (isinstance(criteria, (list, tuple)) and 2 <= len(criteria) <= 10
+                                and all(isinstance(c, str) for c in criteria)):
+        raise ValueError(f"a score's {SCORE}")
+    if kind == "noul" and criteria is not None and not isinstance(criteria, Mapping):
+        raise ValueError('a noul\'s "criteria", when given, is {"true": "what yes means", "false": "what no means"}')
+    if q.get("multiple") and kind != "choice":
+        raise ValueError('"multiple" belongs to a choice: {"type": "choice", "multiple": true, "criteria": {...}}')
+    if "max" in q and not (kind == "boxes" and isinstance(q["max"], int) and not isinstance(q["max"], bool)
+                           and 1 <= q["max"] <= most_blocks(tokenizer)):
+        raise ValueError(f'"max" belongs to boxes, a whole number from 1 to {most_blocks(tokenizer)}: '
+                         '{"type": "boxes", "instructions": "Every cup.", "max": 4}')
+    if "step" in q and "range" not in q:
+        raise ValueError('"step" needs a "range": {"type": "number", "range": [0, 10], "step": 1}')
+    if "range" in q:
+        bounds = q["range"]
+        if kind != "number" or not (isinstance(bounds, (list, tuple)) and len(bounds) == 2
+                                    and all(isinstance(b, (int, float)) and not isinstance(b, bool) for b in bounds)):
+            raise ValueError('an estimate is {"type": "number", "range": [low, high]}, with an optional "step"')
+
+
+def as_question(q: Mapping | Question) -> Question:
+    """A question in the API's JSON (`type`, `instructions`, `criteria`; `multiple`, `range`, `step`, `unit` for the
+    typed answers; `instructions` a media dict for a spoken question), as one of the classes above; a class passes
+    through."""
+    if not isinstance(q, Mapping):
+        return q
+    if isinstance(q.get("instructions"), Mapping):
+        q = {**q, "instructions": as_media(q["instructions"])}
+    kind = q.get("type", "choice")
+    if kind == "number" and "range" in q:
+        low, high = q["range"]
+        step = q.get("step")
+        if not low < high or step is not None and not (step > 0 and math.isclose((high - low) / step, round((high - low) / step))):
+            raise ValueError(f"an estimate's range runs low to high and its step, above 0, divides it: not "
+                             f"{[low, high]} in steps of {step}")
+        return Estimate(q["instructions"], low, high, step, q.get("unit"))
+    if kind in ("text", "number", "date"):
+        return Read(q["instructions"], kind, q.get("unit"))
+    if kind == "box":
+        return Box(q["instructions"])
+    if kind == "boxes":
+        return Boxes(q["instructions"], max=q.get("max"))
+    if kind == "noul":
+        return Noul(q["instructions"], q.get("criteria"))
+    if kind == "score":
+        return Score(q["instructions"], list(q["criteria"]))
+    return Choice(q["instructions"], q["criteria"], multiple=bool(q.get("multiple", False)))
+
+
+def question_slots(q: Question, tokenizer=None) -> tuple[Question, ...]:
+    """Each answer position's readout, in question order; a grid's through the tokenizer's numbers where the
+    checkpoint reads them so (`grid_slot`)."""
+    if isinstance(q, Read):
+        return Position("start"), Position("end")
+    if isinstance(q, Estimate):
+        return (grid_slot(tokenizer, q.instructions, q.grid),)
+    if isinstance(q, Box):
+        from .box import EDGES, edge_grid
+
+        grid, scale = edge_grid(tokenizer)
+        return (Noul(q.instructions), *(grid_slot(tokenizer, q.instructions, grid, scale) for _ in EDGES))
+    if isinstance(q, Boxes):
+        return question_slots(Box(q.instructions), tokenizer) * q.blocks
+    if isinstance(q, Choice) and q.multiple:
+        if not q.criteria:
+            raise ValueError("choice with no options")
+        return tuple(Noul(q.instructions) for _ in q.criteria)
+    return (q,)
+
+
+YES_FORMS = ("yes", "Yes", "YES")
+NO_FORMS = ("no", "No", "NO")
+
+
+def readout_ids(tokenizer, q: Question) -> list[list[int]]:
+    """Token ids to score, one group per option, max-pooled: the answer is a softmax over these and
+    nothing else."""
+    if isinstance(q, Estimate):
+        return readout_ids(tokenizer, question_slots(q, tokenizer)[0])
+    if isinstance(q, Number):
+        return [list(group) for group in number_ids(tokenizer, q.grid, q.scale)]
+    if isinstance(q, (Read, Position)):
+        raise ValueError("a pointer reads state positions, not vocabulary codes")
+    if isinstance(q, Choice) and q.multiple:
+        raise ValueError("a multiple choice reads one yes/no distribution per option")
+    if isinstance(q, Noul):
+        yes, no = _ids(tokenizer, YES_FORMS), _ids(tokenizer, NO_FORMS)
+        if not yes or not no:
+            raise RuntimeError("tokenizer has no single-token yes/no")
+        return [yes, no]
+    if isinstance(q, Score):
+        groups = [_ids(tokenizer, [str(i)]) for i in range(len(q.criteria))]
+        if any(not g for g in groups):
+            raise RuntimeError(f"score with {len(q.criteria)} levels needs single-token digits; the primitive is defined "
+                               "for 2 to 10")
+        return groups
+    groups = []
+    for code, tid in aliases(tokenizer, list(q.criteria.keys())):
+        extra = _ids(tokenizer, [f" {code}"])
+        groups.append([tid] + [i for i in extra if i != tid])
+    if not groups:
+        raise RuntimeError("choice with no options")
+    return groups
+
+
+def readout(tokenizer, q: Question, logz, calibration=None) -> list[float]:
+    """Option probabilities from the log-probabilities at the answer slot, `logz` (a vocabulary tensor, or a dict
+    holding at least the option tokens). Each option scores its best form."""
+    if not hasattr(logz, "tolist"):
+        return probabilities(q, [max(float(logz[i]) for i in g) for g in readout_ids(tokenizer, q)], calibration)
+    tokens, forms, _ = readout_table(tokenizer, q)
+    values = np.asarray(logz[list(tokens)].tolist())  # one device transfer
+    return probabilities(q, values[forms].max(-1).tolist(), calibration)
+
+
+def readouts(tokenizer, qs: Sequence[Question], logz, calibrations: Sequence) -> list[list[float]]:
+    """`readout` for each row of `logz` ([slots, vocabulary], a torch tensor) and its question, as one gather for
+    the rows sharing a readout table (a box's edges) and one transfer for all."""
+    import torch
+
+    tables = [readout_table(tokenizer, q) for q in qs]
+    shared: dict[int, list[int]] = {}
+    for j, table in enumerate(tables):
+        shared.setdefault(id(table), []).append(j)
+    picked = [logz[rows][:, torch.from_numpy(tables[rows[0]][2]).to(logz.device)] for rows in shared.values()]
+    host = torch.cat([p.flatten() for p in picked]).cpu().numpy() if picked else None
+    out, at = [None] * len(qs), 0
+    for rows, part in zip(shared.values(), picked):
+        values = host[at:at + part.numel()].reshape(part.shape)
+        at += part.numel()
+        scores = values[:, tables[rows[0]][1]].max(-1).astype(np.float64)
+        scores /= np.array([1.0 if calibrations[j] is None else calibrations[j].temperature(qs[j]) for j in rows])[:, None]
+        exps = np.exp(scores - scores.max(-1, keepdims=True))  # the softmax of `probabilities`, a block at once
+        for j, p in zip(rows, (exps / exps.sum(-1, keepdims=True)).tolist()):
+            out[j] = p
+    return out
+
+
+def probabilities(q: Question, scores: list[float], calibration=None) -> list[float]:
+    """Each option's best score, through the calibration's temperature, as a softmax."""
+    if calibration is not None:
+        scores = calibration.apply(q, scores)
+    m = max(scores)
+    exps = [math.exp(s - m) for s in scores]
+    total = sum(exps)
+    return [e / total for e in exps]
+
+
+def readout_table(tokenizer, q: Question) -> tuple[tuple[int, ...], np.ndarray, np.ndarray]:
+    """`readout_ids` as one gather: every token once, each group's forms as positions among them (a short group
+    repeats its first form, which leaves its best as it is), and the tokens as an array. Cached: a box edge scores
+    1,000 numbers."""
+    if isinstance(q, Number):
+        _TOKENIZERS.setdefault(id(tokenizer), tokenizer)
+        return _number_table(id(tokenizer), q.grid.low, q.grid.high, q.grid.size, q.scale)
+    return _table(tuple(map(tuple, readout_ids(tokenizer, q))))
+
+
+@lru_cache(maxsize=4096)
+def _table(groups: tuple[tuple[int, ...], ...]) -> tuple[tuple[int, ...], np.ndarray, np.ndarray]:
+    tokens = tuple(dict.fromkeys(i for g in groups for i in g))
+    at = {t: k for k, t in enumerate(tokens)}
+    width = max(map(len, groups))
+    return tokens, np.array([[at[i] for i in g] + [at[g[0]]] * (width - len(g)) for g in groups]), np.array(tokens)
+
+
+@lru_cache(maxsize=256)
+def _number_table(tokenizer_key: int, low: float, high: float, size: int, scale: int):
+    from .grid import Grid
+
+    return _table(tuple(map(tuple, readout_ids(_TOKENIZERS[tokenizer_key], Number("", Grid(low, high, size), scale)))))
+
+
+# --------------------------------------------------------------------------- #
+# state and question rendering
+# --------------------------------------------------------------------------- #
+
+
+# How a state is rendered: `json_only`, the default, writes every state as the object it is; `json` keeps
+# three shortcuts (`Message:`, `Passage:` / `Asked:`, a lone question's text); `sections` writes nested
+# states as labelled blocks.
+DEFAULT_STATE_STYLE = "json_only"
+
+
+def _is_scalar(v: Any) -> bool:
+    return v is None or isinstance(v, (str, int, float, bool)) and "\n" not in str(v)
+
+
+def _sections(obj: Any, path: str, out: list[str]) -> None:
+    """Flatten a nested state into labelled blocks, keeping the real newlines `json.dumps` would escape."""
+    head = f"[{path}]\n" if path else ""
+    if isinstance(obj, dict):
+        scalars = [(k, v) for k, v in obj.items() if _is_scalar(v)]
+        rest = [(k, v) for k, v in obj.items() if not _is_scalar(v)]
+        if scalars:
+            body = "\n".join(f"{k}: {'' if v is None else v}" for k, v in scalars)
+            out.append(f"{head}{body}")
+        for k, v in rest:
+            _sections(v, f"{path}.{k}" if path else str(k), out)
+        return
+    if isinstance(obj, (list, tuple)):
+        if obj and all(_is_scalar(v) for v in obj):
+            body = "\n".join(f"- {'' if v is None else v}" for v in obj)
+            out.append(f"{head}{body}")
+            return
+        for i, v in enumerate(obj, start=1):
+            _sections(v, f"{path} {i}/{len(obj)}" if path else f"{i}/{len(obj)}", out)
+        return
+    out.append(f"{head}{'' if obj is None else obj}")
+
+
+def render_state(state: Any) -> str:
+    if isinstance(state, str):
+        return state
+    out: list[str] = []
+    _sections(state, "", out)
+    return "\n\n".join(out)
+
+
+def state_text(state: Any, style: str = DEFAULT_STATE_STYLE) -> str:
+    """The state as the prompt renders it, without the blank line after it: what a read's span offsets index."""
+    return state_block(state, style)[:-2]
+
+
+def state_block(state: Any, style: str = DEFAULT_STATE_STYLE) -> str:
+    """The state as the block before `QUESTION:`. The `_only` styles (the default) write every state as the object
+    it is; the others give three shapes a shortcut (`Message:`, `Passage:` / `Asked:`, a lone question's text)."""
+    if isinstance(state, dict) and not style.endswith("_only"):
+        keys = set(state.keys())
+        if keys == {"text"}:
+            return f"Message: {state['text']}\n\n"
+        if {"passage", "question"} <= keys and len(keys) == 2:
+            return f"Passage: {state['passage']}\n\nAsked: {state['question']}\n\n"
+        if keys == {"question"}:
+            return f"{state['question']}\n\n"
+    if style.startswith("json"):
+        if isinstance(state, str):
+            return f"{state}\n\n"
+        return json.dumps(state, ensure_ascii=False, indent=2) + "\n\n"
+    return f"{render_state(state)}\n\n"
+
+
+# A zero-width space after the first character of a special token's string: it reads the same and is no longer the
+# token. Without it a state holding `<|im_end|>` closed the user turn, and `<image>` became an image token
+# with no picture behind it (the processor then refused the row).
+BREAK = "\u200b"
+
+
+@lru_cache(maxsize=64)
+def _specials_cached(tokenizer_key: int) -> re.Pattern | None:
+    tok = _TOKENIZERS[tokenizer_key]
+    tok = getattr(tok, "tokenizer", tok)  # a processor's own
+    names = set(getattr(tok, "all_special_tokens", ()))
+    names |= {t.content for t in getattr(tok, "added_tokens_decoder", {}).values() if t.special}
+    names = sorted((n for n in names if len(n) > 1 and BREAK not in n), key=len, reverse=True)
+    return re.compile("|".join(map(re.escape, names))) if names else None
+
+
+def plain(tokenizer, text: str) -> str:
+    """`text` as the caller's text: no special token of `tokenizer` in it once encoded. Text without one is returned
+    unchanged."""
+    return _plain(tokenizer, text, False)[0]
+
+
+def plain_offsets(tokenizer, text: str) -> tuple[str, list[int]]:
+    """The same escaped text, with every character boundary mapped to the original text."""
+    return _plain(tokenizer, text, True)
+
+
+def _plain(tokenizer, text: str, offsets: bool) -> tuple[str, list[int] | None]:
+    _TOKENIZERS.setdefault(id(tokenizer), tokenizer)
+    boundaries = list(range(len(text) + 1)) if offsets else None
+    pattern = _specials_cached(id(tokenizer))
+    if pattern is None:
+        return text, boundaries
+    n = 1
+    while n:  # again for a match that overlapped the one broken before it
+        if offsets:
+            insertions = [m.start() + 1 for m in pattern.finditer(text)]
+            for at in reversed(insertions):
+                boundaries.insert(at, boundaries[at])
+        text, n = pattern.subn(lambda m: m[0][0] + BREAK + m[0][1:], text)
+    return text, boundaries
+
+
+_PLACEHOLDER = re.compile(r"^opt\d+$")
+
+
+def _option_line(code: str, label: str, desc: str | None, style: str) -> str:
+    text = desc or label.replace("_", " ")
+    if style == "name_desc" and not _PLACEHOLDER.match(label) and label != text:
+        return f"{code} {label}: {text}"
+    return f"{code} {text}"
+
+
+def said(q: Question) -> str:
+    """A question's instructions as its text writes them: their place (`media.SPOKEN`) when they are spoken."""
+    return SPOKEN if isinstance(q.instructions, Media) else q.instructions
+
+
+def edge_legend(tokenizer) -> str:
+    """How a box's edges are written: number tokens on a checkpoint that reads numbers, else codes."""
+    from .box import edge_grid
+
+    grid, scale = edge_grid(tokenizer)
+    if scale > 1:
+        return ("Each edge is a number from 0 to 999, in thousandths of the picture: top and bottom from its top edge, "
+                "left and right from its left edge.")
+    codes = [code for code, _ in aliases(tokenizer, list(grid_question("", grid).criteria))]
+    return (f"Each edge is a code on a {grid.size}-value scale: {codes[0]} is 0 (top or left), {codes[-1]} is 1 "
+            "(bottom or right). Values are equally spaced.")
+
+
+def question_block(tokenizer, q: Question, option_style: str = "desc") -> str:
+    if isinstance(q, Box):
+        one = "one number" if isinstance(question_slots(q, tokenizer)[-1], Number) else "one code"
+        return (f"{said(q)}\n\nPresence is yes or no. {edge_legend(tokenizer)} Reply with yes or no at present, "
+                f"then {one} at top, bottom, left and right.")
+    if isinstance(q, Boxes):
+        from .box import INSTANCES
+
+        most = f"{INSTANCES} at most, " if version(tokenizer) == 1 else ""
+        return (f"{said(q)}\n\nEvery one of them, {most}in reading order (left edge, then top). "
+                f"Present k is yes when there are at least k. {edge_legend(tokenizer)} Reply with yes or no at each "
+                "present, then its top, bottom, left and right.")
+    if isinstance(q, Estimate) and isinstance(question_slots(q, tokenizer)[0], Number):
+        unit = f" {q.unit}" if q.unit is not None else ""
+        return (f"{said(q)}\n\nEstimate from {q.grid.low:g} to {q.grid.high:g}{unit}, in steps of "
+                f"{q.grid.step:g}. Reply with the number only.")
+    if isinstance(q, Estimate):
+        bins = grid_question("", q.grid)
+        codes = ([str(i) for i in range(q.grid.size)] if isinstance(bins, Score)
+                 else [code for code, _ in aliases(tokenizer, list(bins.criteria))])
+        unit = f" {q.unit}" if q.unit is not None else ""
+        return (f"{said(q)}\n\nEstimate from {q.grid.low:g} to {q.grid.high:g}{unit} on "
+                f"{q.grid.size} equally spaced values, step {q.grid.step:g}: "
+                f"{codes[0]} is {q.grid.low:g}, {codes[-1]} is {q.grid.high:g}. Reply with the code only.")
+    if isinstance(q, Read):
+        unit = f" (unit: {q.unit})" if q.type == "number" and q.unit else ""
+        return f"{said(q)}\n\nRead the {q.type}{unit} from the state, or nothing if it is absent."
+    if isinstance(q, Choice):
+        labels = list(q.criteria.keys())
+        if q.multiple:
+            lines = "\n".join(f"{json.dumps(name, ensure_ascii=False)}: {q.criteria[name] or name}" for name in labels)
+            return f"{said(q)}\n\nOptions:\n{lines}\n\nReply with yes or no for each option."
+        lines = "\n".join(_option_line(code, lab, q.criteria[lab], option_style)
+                          for (code, _), lab in zip(aliases(tokenizer, labels), labels))
+        return f"{said(q)}\n\nOptions:\n{lines}\n\nReply with the option code only."
+    if isinstance(q, Noul):
+        extra = ""
+        if q.criteria:
+            extra = f"\nYes: {q.criteria.get('true')}\nNo: {q.criteria.get('false')}"
+        return f"{said(q)}{extra}\n\nReply with yes or no only."
+    if isinstance(q, Score):
+        legend = "\n".join(f"{i} {name}" for i, name in enumerate(q.criteria))
+        return f"{said(q)}\n\n{legend}\n\nReply with a single digit 0-{len(q.criteria) - 1} only."
+    raise TypeError(f"unknown question type {type(q)}")
+
+
+def prefix_text(tokenizer, state: Any, bos: str = "", style: str = DEFAULT_STATE_STYLE, system: str = DEFAULT_SYSTEM,
+                media: str = "") -> str:
+    """Everything before the question, shared by all questions on one state: the media's markup (`media`, as
+    `media.Reader` writes it) at the head of the user turn, then the state. With no state (`None`) the question
+    follows the media directly. The state is the caller's text (`plain`)."""
+    text = SYSTEMS[system]
+    turn = "" if text is None else f"{IM_START}system\n{text}{IM_END}\n"
+    if state is None:
+        body = ""
+    else:
+        body = f"{plain(tokenizer, state_block(state, style))}\n{question_header(tokenizer, state)}"
+    return f"{bos}{turn}{IM_START}user\n{media}{body}"
+
+
+def question_header(tokenizer, state: Any) -> str:
+    """What `prefix_text` ends with to open a question. `set` attention repeats it before each further question
+    packed after the state (`attention.pack_set`)."""
+    return "" if state is None else "QUESTION:\n"
+
+
+# What sits between the assistant header and the answer slot. A family module registers its measured one
+# (`LEADS`); any other family's is read off its chat template (`template_lead`).
+DEFAULT_LEAD = ""
+LEADS: dict[str, str] = {}
+
+
+def default_lead(model_type: str | None, tokenizer=None) -> str:
+    """What a checkpoint's own template writes before a non-thinking answer: `LEADS` for the families measured,
+    else read off the tokenizer's chat template when one is given."""
+    if model_type in LEADS or tokenizer is None:
+        return LEADS.get(model_type, DEFAULT_LEAD)
+    return template_lead(tokenizer)
+
+
+def template_lead(tokenizer) -> str:
+    """What a ChatML template writes after the assistant header for a non-thinking reply. The prompt is ChatML
+    (`IM_START`, `IM_END`), so a model with another template is refused rather than mis-prompted."""
+    text = tokenizer.apply_chat_template([{"role": "user", "content": "x"}], add_generation_prompt=True,
+                                         tokenize=False, enable_thinking=False)
+    head = f"{IM_START}assistant\n"
+    if head not in text:
+        raise ValueError("not a ChatML chat template: d1 renders ChatML prompts only")
+    return text[text.rindex(head) + len(head):]
+
+
+def suffix_text(tokenizer, q: Question, lead: str = DEFAULT_LEAD, option_style: str = "desc") -> str:
+    """The question and the assistant header, up to the answer slot. The question is the caller's text (`plain`)."""
+    return suffix_layout(tokenizer, q, lead, option_style)[0]
+
+
+def suffix_layout(tokenizer, q: Question, lead: str = DEFAULT_LEAD,
+                  option_style: str = "desc") -> tuple[str, list[int]]:
+    """The suffix and each slot's token offset from its last token; old questions have one slot at zero."""
+    body = question_block(tokenizer, q, option_style)
+    if isinstance(q, Read):
+        return marked_suffix(tokenizer, body, ["start", "end"], lead)
+    if isinstance(q, Box):
+        from .box import EDGES
+
+        return marked_suffix(tokenizer, body, ["present", *EDGES], lead)
+    if isinstance(q, Boxes):
+        from .box import EDGES, INSTANCES
+
+        # the last block ends on the newline a next block would follow, where the colon and it are one token, so every
+        # slot is the token the whole read has there: prompt 1 when `max` reads fewer than its 16, prompt 2 always
+        return marked_suffix(tokenizer, body, [f"{name} {k}" for k in range(1, q.blocks + 1)
+                                               for name in ("present", *EDGES)], lead,
+                             "\n" if q.blocks < INSTANCES or version(tokenizer) > 1 else "")
+    if isinstance(q, Choice) and q.multiple:
+        names = [json.dumps(name, ensure_ascii=False) for name in q.criteria]
+        return marked_suffix(tokenizer, body, names, lead)
+    return f"{plain(tokenizer, body)}{IM_END}\n{IM_START}assistant\n{lead}", [0]
+
+
+def marked_suffix(tokenizer, body: str, names: Sequence[str], lead: str = DEFAULT_LEAD,
+                  after: str = "") -> tuple[str, list[int]]:
+    """A question's named answer markers and their token offsets from its last token; `after`, whitespace the last
+    marker's colon token takes where the tokenizer writes the two as one token."""
+    if not names or len(set(names)) != len(names) or any(not isinstance(n, str) or not n or "\n" in n for n in names):
+        raise ValueError("slot names must be distinct nonempty single lines")
+    text = f"{plain(tokenizer, body)}{IM_END}\n{IM_START}assistant\n{lead}"
+    ends = []
+    for i, name in enumerate(names):
+        text += ("\n" if i else "") + plain(tokenizer, name) + ":"
+        ends.append(len(text))
+    if after and tokenizer.decode(tokenizer.encode(text + after, add_special_tokens=False)[-1:]).rstrip().endswith(":"):
+        text += after
+    encoded = tokenizer(text, add_special_tokens=False, return_offsets_mapping=True)
+    ids = encoded["input_ids"]
+    slots = []
+    for chars in ends:
+        matching = [(i, stop) for i, (start, stop) in enumerate(encoded["offset_mapping"]) if start <= chars - 1 < stop]
+        if len(matching) != 1:
+            raise ValueError("a slot marker must have an exact colon token position")
+        end, stop = matching[0]
+        # The colon may share its token with following whitespace, never with the next marker's text.
+        if text[chars:stop].strip() or not tokenizer.decode([ids[end]]).rstrip().endswith(":"):
+            raise ValueError("a slot marker token must end with its colon and optional whitespace")
+        slots.append(end - len(ids) + 1)
+    return text, slots
+
+
+def number_ids(tokenizer, grid, scale: int = 1) -> tuple[tuple[int, ...], ...] | None:
+    """Each grid value times `scale` as the tokenizer writes it, plain and after a space, one token a form: None when
+    a value is no integer or takes more than one token (a tokenizer of digits)."""
+    _TOKENIZERS.setdefault(id(tokenizer), tokenizer)
+    return _number_ids_cached(id(tokenizer), grid.low, grid.high, grid.size, scale)
+
+
+@lru_cache(maxsize=256)
+def _number_ids_cached(tokenizer_key: int, low: float, high: float, size: int, scale: int):
+    from .grid import Grid
+
+    groups = []
+    for value in Grid(low, high, size).values:
+        n = round(value * scale)
+        if not math.isclose(n, value * scale, abs_tol=1e-6):
+            return None
+        group = _ids_cached(tokenizer_key, (str(n), f" {n}"))
+        if not group or len(_TOKENIZERS[tokenizer_key].encode(str(n), add_special_tokens=False)) != 1:
+            return None
+        groups.append(group)
+    return tuple(groups)
+
+
+def grid_slot(tokenizer, instructions: str, grid, scale: int = 1) -> Number | Choice | Score:
+    """A grid's slot: the tokenizer's own numbers where the checkpoint reads them so (`d1_numbers`) and every value is
+    one (`number_ids`), else the grid's codes (`grid_question`)."""
+    if getattr(tokenizer, "d1_numbers", False) and number_ids(tokenizer, grid, scale) is not None:
+        return Number(instructions, grid, scale)
+    return grid_question(instructions, grid)
+
+
+def grid_question(instructions: str, grid) -> Choice | Score:
+    """The grid's ordered bins on the existing digit or option-code readout."""
+    labels = [f"{v:g}" for v in grid.values]
+    return (Score(instructions, labels) if grid.size <= 10
+            else Choice(instructions, {str(i): v for i, v in enumerate(labels)}))
+
+
+def render(tokenizer, state: Any, q: Question, bos: str = "", lead: str = DEFAULT_LEAD,
+           style: str = DEFAULT_STATE_STYLE, system: str = DEFAULT_SYSTEM, option_style: str = "desc",
+           media: str = "") -> str:
+    return prefix_text(tokenizer, state, bos, style, system, media) + suffix_text(tokenizer, q, lead, option_style)

runner.py

@@ -0,0 +1,379 @@
+"""One-pass typed decisions on any backbone.
+
+The model sees the state and the question once, and the answer is read off the
+logits at the answer slot. Nothing is decoded, so a schema violation is
+impossible and `tokens generated per decision` is zero.
+
+- Questions are read at their answer slots, a whole prompt or packed after their state (`SlotReader._padded`):
+  every typed answer, and every question on a backbone with no faster path of its own (`SystemOne.run`).
+- A request is `(state, questions, media)`: its pictures, audio clips and videos head the state, a spoken question's
+  clip sits in its own text (`media`).
+"""
+
+from __future__ import annotations
+
+import os
+from typing import Any, Sequence
+
+import torch
+
+from .api import SystemOneApi
+from .attention import SET_SIZE, TRAINED_SET_SIZE, both_ways, language_model, pack_set
+from .box import EDGE_READS
+from .media import Media, Reader, media_list, spoken
+from .pointer import SPAN_CAPS, query_plan, spans, state_positions
+from .prompt import (
+    DEFAULT_STATE_STYLE,
+    DEFAULT_SYSTEM,
+    PROMPTS,
+    Box,
+    Boxes,
+    Choice,
+    Position,
+    Question,
+    Read,
+    default_lead,
+    prefix_text,
+    question_header,
+    question_slots,
+    readout,
+    readouts,
+    render,
+    suffix_layout,
+    suffix_text,
+    typed,
+)
+from .slots import slot_logits, text_backbone
+
+# Every picture is bounded at this many pixels before the processor (`media.cap_pixels`), in training and serving alike. A checkpoint
+# records its own (`d1_vision_max_pixels` in its config); D1_VISION_MAX_PIXELS, when set, overrides it for a run or a
+# check (reading a checkpoint at another resolution).
+PIXELS_SET = os.environ.get("D1_VISION_MAX_PIXELS")
+VISION_MAX_PIXELS = int(PIXELS_SET or 1024 * 1024)
+
+
+def group_size(n: int | None) -> int | None:
+    """A group size (`set_size`): a whole number from 1 (each question alone), or None for the engine's own."""
+    if n is None:
+        return None
+    if isinstance(n, bool) or not isinstance(n, int):
+        raise ValueError(f"set_size must be a whole number, not {n!r}")
+    if n < 1:
+        raise ValueError(f"set_size must be at least 1, not {n}")
+    return n
+
+
+def remote_ok(model_id) -> bool:
+    """Whether a checkpoint's own code may run: a local folder or a LiquidAI repo.
+    Any other Hub id loads with transformers' classes only, and one that needs its own code fails."""
+    return os.path.isdir(str(model_id)) or str(model_id).startswith("LiquidAI/")
+
+
+def auto_processor(path: str, revision: str | None = None):
+    """The media processor a checkpoint holds."""
+    from transformers import AutoProcessor
+
+    return AutoProcessor.from_pretrained(path, revision=revision, trust_remote_code=remote_ok(path))
+
+
+def _under(config: dict | None, compiled):
+    """`compiled` run under inductor's `config` (`SystemOne.compile`)."""
+    if not config:
+        return compiled
+    import torch._inductor.config as inductor
+
+    def run(*args, **kwargs):
+        with inductor.patch(config):
+            return compiled(*args, **kwargs)
+
+    return run
+
+
+class SlotReader:
+    """Prompts read at their answer slots, whatever runs the backbone: a request's sequences (`_sequences`), each
+    question's slots, and the answers read off them (`_padded`). An engine sets what `SystemOne.__init__` sets (the
+    checkpoint's record through `_serve`, its `processor` or `_load_processor`) and gives `_batches` and
+    `_slot_logits`."""
+
+    def _serve(self, recorded, codes: int | None = None, max_pixels: int | None = None, attention: str | None = None,
+               set_size: int | None = None, numbers: bool | None = None, box_edges: str | None = None,
+               prompt: int | None = None) -> None:
+        """The option codes, number readout, picture cap, attention mode, set size, box-edge read and prompt version
+        `recorded` (a checkpoint's config) holds (`d1_codes`, `d1_numbers`, `d1_vision_max_pixels`, `d1_attention`,
+        `d1_set_size`, `d1_box_edges`: the median without it, `d1_prompt`: 1 without it), unless given. A recorded set
+        size is 2 to `TRAINED_SET_SIZE`, the default; one given is any whole number from 1 (each question alone):
+        larger groups read faster, in packs training never saw."""
+        self.tokenizer.d1_codes = codes if codes is not None else getattr(recorded, "d1_codes", 1)
+        self.tokenizer.d1_numbers = bool(numbers if numbers is not None else getattr(recorded, "d1_numbers", False))
+        self.tokenizer.d1_box_edges = box_edges or getattr(recorded, "d1_box_edges", None) or "median"
+        if self.tokenizer.d1_box_edges not in EDGE_READS:
+            raise ValueError(f"d1_box_edges is one of {EDGE_READS}, not {self.tokenizer.d1_box_edges!r}")
+        self.tokenizer.d1_prompt = prompt or getattr(recorded, "d1_prompt", 1)
+        if self.tokenizer.d1_prompt not in PROMPTS:
+            raise ValueError(f"d1_prompt is one of {PROMPTS}, not {self.tokenizer.d1_prompt!r}")
+        self.max_pixels = (VISION_MAX_PIXELS if PIXELS_SET else
+                           max_pixels or getattr(recorded, "d1_vision_max_pixels", None) or VISION_MAX_PIXELS)
+        self.attention = attention or getattr(recorded, "d1_attention", None) or "causal"
+        self.set_size = group_size(getattr(recorded, "d1_set_size", SET_SIZE) if set_size is None else set_size)
+        if set_size is None and not 2 <= self.set_size <= TRAINED_SET_SIZE:
+            raise ValueError(f"d1_set_size must be between 2 and {TRAINED_SET_SIZE}")
+
+    def _readout(self, q: Question, logz, tempered: bool = True) -> list[float]:
+        """An option answer through the served temperature; a typed answer's slot raw (`tempered=False`): its own
+        table calibrates it (`calibration.json` `tables`, fitted on raw reads)."""
+        return readout(self.tokenizer, q, logz, self.calibration if tempered else None)
+
+    def _readouts(self, qs: Sequence[Question], logz, tempered: Sequence[bool]) -> list[list[float]]:
+        """`_readout` for each code slot of a batch, `logz` as `_slot_logits` gives them."""
+        return [self._readout(q, z, t) for q, z, t in zip(qs, logz, tempered)]
+
+    @property
+    def media(self) -> Reader:
+        """The checkpoint's media reader, its processor loaded on first use."""
+        if getattr(self, "_reader", None) is None or self._reader.processor is not self.processor:
+            if self.processor is None:
+                self.processor = self._load_processor()
+            self._reader = Reader(self.processor, self.max_pixels)
+        return self._reader
+
+    def _prefix(self, state: Any, media: Sequence[Media]) -> tuple[str, list[dict]]:
+        """`prompt.prefix_text` headed by `media`'s markup, and their processor outputs."""
+        markup, parts = self.media.encode(media) if media else ("", [])
+        return prefix_text(self.tokenizer, state, self.bos, self.state_style, self.system, markup), parts
+
+    def _suffixes(self, qs: Sequence[Question]) -> tuple[list[str], list[list[int]], list[list[dict]]]:
+        """Each question's suffix (a spoken one's clip in place), its slots' offsets and its clip's outputs."""
+        layouts = [suffix_layout(self.tokenizer, q, self.lead, self.option_style) for q in qs]
+        placed = [self.media.place(text, q) if spoken(q) else (text, []) for (text, _), q in zip(layouts, qs)]
+        return [text for text, _ in placed], [ends for _, ends in layouts], [parts for _, parts in placed]
+
+    def _sequences(self, requests: Sequence[tuple[Any, list[Question], Sequence[Media]]], *,
+                   position_maps=None, set_size: int | None = None) -> list[tuple]:
+        """Every request's sequences, `(request, first question, ids, state length, each token's question or None,
+        answer slots, media inputs)`: under `set` up to `set_size` questions after their state (`attention.pack_set`),
+        else one whole prompt a question. A request's media head its state, a spoken question's clip sits in its
+        suffix, each as its processor expands it, so the text is the sequence. `position_maps` gets each read's state
+        positions; `set_size`, this call's group size (the engine's by default)."""
+        seqs = []
+        for i, (state, qs, media) in enumerate(requests):
+            media = media_list(media)
+            if any(isinstance(q, (Box, Boxes)) for q in qs) and (n := sum(m.type == "image" for m in media)) != 1:
+                raise ValueError(f"a box or boxes question reads exactly one picture in `media`; this request has {n}")
+            prefix, state_parts = self._prefix(state, media)
+            suffixes, offsets, voices = self._suffixes(qs)
+            step = (self.set_size if set_size is None else set_size) if self.attention == "set" else 1
+            for k in range(0, len(qs), step):
+                text = prefix + suffixes[k]
+                encoded = None
+                if position_maps is not None and any(isinstance(q, Read) for q in qs[k:k + step]):
+                    encoded = self.tokenizer(text, add_special_tokens=False, return_offsets_mapping=True)
+                ids = self.tokenizer.encode(text, add_special_tokens=False) if encoded is None else encoded["input_ids"]
+                if self.attention == "set":
+                    ids, state_len, question, ends = pack_set(self.tokenizer, prefix, suffixes[k:k + step],
+                                                              question_header(self.tokenizer, state),
+                                                              offsets[k:k + step], first_ids=ids)
+                else:
+                    state_len, question = both_ways(self.attention, self.tokenizer, ids, suffixes[k]), None
+                    ends = [[len(ids) - 1 + offset for offset in offsets[k]]]
+                parts = state_parts + [part for own in voices[k:k + step] for part in own]
+                inputs = self.media.inputs(parts, ids) if parts else {}
+                if encoded is not None:
+                    position_maps[len(seqs)] = state_positions(self.tokenizer, state, prefix, text, self.state_style,
+                                                               encoded=encoded)
+                seqs.append((i, k, ids, state_len, question, ends, inputs))
+        return seqs
+
+    def _padded(self, requests: Sequence[tuple[Any, list[Question], Sequence]],
+                set_size: int | None = None) -> list[tuple[list[list[float]], int]]:
+        """`_sequences` in the engine's batches (`_batches`), each read at its answer slots as training reads them
+        (`_slot_logits`): option codes through the readout, reads through the pointer."""
+        position_maps = {}
+        seqs = self._sequences(requests, position_maps=position_maps, set_size=set_size)
+        read = [0] * len(requests)
+        for i, _, ids, *_ in seqs:
+            read[i] += len(ids)
+
+        def asked(key: tuple[int, int]) -> Question:
+            return requests[key[0]][1][key[1]]
+
+        probs: dict[tuple[int, int], list[float]] = {}
+        for batch in self._batches(seqs):
+            slots = []  # (row in the batch, (request, question), the slot's readout, its position)
+            for r, b in enumerate(batch):
+                i, first = seqs[b][:2]
+                for k, ends in enumerate(seqs[b][5]):
+                    key = (i, first + k)
+                    slots.extend((r, key, q, end) for q, end in zip(question_slots(asked(key), self.tokenizer), ends))
+            reads = [j for j, s in enumerate(slots) if isinstance(s[2], Position)]  # a read's start then its end
+            pointer = None
+            if reads:
+                masks = [torch.tensor(position_maps[batch[slots[j][0]]].pointer_mask) for j in reads]
+                pointer = query_plan([slots[j][0] for j in reads], masks, reads, len(slots))
+            logz, positions = self._slot_logits(seqs, batch, slots, pointer)
+            if pointer is not None:
+                logits = positions.reshape(-1, 2, positions.shape[-1])
+                capped = {}  # the reads of each span cap
+                for index, j in enumerate(reads[::2]):
+                    capped.setdefault(SPAN_CAPS[asked(slots[j][1]).type], []).append((index, j))
+                for cap, own in capped.items():
+                    at, found = spans(logits[[index for index, _ in own]], max_tokens=cap)
+                    for (_, j), span, p in zip(own, at.tolist(), found.tolist()):
+                        r, key = slots[j][:2]
+                        probs[key] = {**position_maps[batch[r]].read(*span, max_tokens=cap), "found": p}
+            codes = pointer["code_slots"].tolist() if pointer else range(len(slots))
+            scored = self._readouts([slots[j][2] for j in codes], logz, [not typed(asked(slots[j][1])) for j in codes])
+            for j, p in zip(codes, scored):
+                key = slots[j][1]
+                q = asked(key)
+                if isinstance(q, Choice) and q.multiple:
+                    probs.setdefault(key, []).append(p[0])
+                elif isinstance(q, (Box, Boxes)):
+                    probs.setdefault(key, []).append(p)
+                else:
+                    probs[key] = p
+        return [([probs[(i, k)] for k in range(len(qs))], read[i]) for i, (_, qs, _) in enumerate(requests)]
+
+
+class SystemOne(SlotReader, SystemOneApi):
+    """State in, calibrated distribution out, one forward pass."""
+
+    def __init__(
+        self,
+        model=None,
+        tokenizer=None,
+        device: str | None = None,
+        calibration=None,
+        lead: str | None = None,
+        state_style: str = DEFAULT_STATE_STYLE,
+        system: str = DEFAULT_SYSTEM,
+        option_style: str = "desc",
+        codes: int | None = None,
+        max_pixels: int | None = None,
+        attention: str | None = None,
+        set_size: int | None = None,
+        numbers: bool | None = None,
+        box_edges: str | None = None,
+        prompt: int | None = None,
+        compile: bool = False,
+        token_budget: int = 65536,
+        revision: str | None = None,
+    ):
+        """`model` and `tokenizer`: a backbone already loaded (the Hub's class passes itself, and the `revision` its
+        processor loads at); it stays on its device unless `device` says otherwise. `codes` and `max_pixels` default
+        to what the checkpoint's config records (`d1_codes`, `d1_vision_max_pixels`), else scheme 1 and the cap;
+        `attention` likewise (`d1_attention`, else causal; `attention`)."""
+        if model is None or tokenizer is None or isinstance(model, str):
+            raise TypeError("SystemOne reads a loaded backbone: pass `model=` and `tokenizer=` (a checkpoint's own "
+                            "class serves it: AutoModel.from_pretrained(repo, trust_remote_code=True).system_one)")
+        self.model, self.tokenizer = model, tokenizer
+        self.model_id, device = model.config._name_or_path, device or next(model.parameters()).device
+        self.device = torch.device(device or ("cuda" if torch.cuda.is_available() else "cpu"))
+        if getattr(self.model, "hf_device_map", None):  # spread over GPUs at load: inputs go to the embeddings
+            self.device = self.model.get_input_embeddings().weight.device
+        else:
+            self.model.to(self.device)
+        self.model.eval()
+        if hasattr(self.model, "fuse_projections"):  # one matmul for the linears on one input
+            self.model.fuse_projections()
+        bos = getattr(self.tokenizer, "bos_token", None)
+        self.bos = bos if isinstance(bos, str) else ""
+        self.lead = default_lead(model.config.model_type, tokenizer) if lead is None else lead
+        self.calibration, self.state_style, self.system, self.option_style = calibration, state_style, system, option_style
+        self._serve(self.model.config, codes, max_pixels, attention, set_size, numbers, box_edges, prompt)
+        self.token_budget = token_budget
+        self.processor, self.revision = None, revision
+        if compile:
+            self.compile()
+
+    def compile(self, mode: str | None = None) -> None:
+        """torch.compile what the passes run: the language model's forward, shapes dynamic; not its hooks, so
+        `state_attention`'s plan is made outside the graph. Also the media encoders the backbone names
+        (`compiled_parts`; 20 to 15 ms on a 384 px picture), under the inductor config it names (`compile_options`),
+        set while they run: dynamo reads `freezing` as it traces, where `torch.compile`'s own `options` do not reach
+        (d1-2B on an MI325X: 4.5 ms a question with `options`, 3.5 set). CUDA graphs measured no faster there, and
+        inductor's graph pool refuses the packed varlen calls."""
+        if mode in ("reduce-overhead", "max-autotune") and self.attention != "causal":
+            raise ValueError(f"mode={mode!r} records CUDA graphs, which `{self.attention}` attention does not take: "
+                             "model.compile() fuses without them")
+        body, parts = text_backbone(self.model), getattr(self.model, "compiled_parts", ())
+        config = getattr(self.model, "compile_options", None)
+        for part in (language_model(self.model), *(getattr(body, name) for name in parts)):
+            part.forward = _under(config, torch.compile(part.forward, dynamic=True, mode=mode))
+
+    # ---------------------------------------------------------------- prompt
+
+    def render(self, state: Any, q: Question) -> str:
+        return render(self.tokenizer, state, q, self.bos, self.lead, self.state_style, self.system, self.option_style)
+
+    # ------------------------------------------------------------------- api
+
+    @torch.inference_mode()
+    def run(self, requests: Sequence[tuple[Any, list[Question], Sequence[Media]]],
+            set_size: int | None = None) -> list[tuple[list[list[float]], int]]:
+        """Each `(state, questions, media)` request's probabilities and the tokens it read, at its answer slots
+        (`_padded`), every request in the same batches. `set_size`: this call's questions a group under `set` (the
+        engine's by default)."""
+        set_size = group_size(set_size)
+        requests = [(state, qs, media_list(media)) for state, qs, media in requests]
+        return self._padded(requests, set_size)
+
+    def _batches(self, seqs: list[tuple]) -> list[list[int]]:
+        """`_sequences` right-padded into batches of at most `token_budget` padded tokens, whatever their media: a
+        request's sequences in one batch (one forward) unless they alone pass the budget, requests of similar length
+        together. Every row is whole, its media given with it (`_rows`)."""
+        own: dict[int, list[int]] = {}
+        for j in sorted(range(len(seqs)), key=lambda j: len(seqs[j][2])):
+            own.setdefault(seqs[j][0], []).append(j)
+        batches: list[list[int]] = [[]]
+        for rows in sorted(own.values(), key=lambda rows: len(seqs[rows[-1]][2])):
+            if batches[-1] and len(seqs[rows[-1]][2]) * (len(batches[-1]) + len(rows)) > self.token_budget:
+                batches.append([])
+            for j in rows:  # shortest first, so a batch's width is its last sequence's length
+                if batches[-1] and len(seqs[j][2]) * (len(batches[-1]) + 1) > self.token_budget:
+                    batches.append([])
+                batches[-1].append(j)
+        return list(filter(None, batches))
+
+    def _rows(self, seqs: list[tuple], batch: list[int]) -> tuple[torch.Tensor, list[int], dict]:
+        """A batch's right-padded ids, each row's length and its media inputs: every row whole, its own ids and its
+        media's inputs, joined in row order (`media.Reader.batch`) for the one forward."""
+        pad = self.tokenizer.pad_token_id or 0
+        lengths = [len(seqs[b][2]) for b in batch]
+        ids = torch.tensor([seqs[b][2] + [pad] * (max(lengths) - n) for b, n in zip(batch, lengths)], device=self.device)
+        inputs = [seqs[b][6] for b in batch]
+        return ids, lengths, self.media.batch(inputs, ids) if any(inputs) else {}
+
+    def _slot_logits(self, seqs: list[tuple], batch: list[int], slots: list[tuple], pointer: dict | None):
+        """A batch's code slots' log-probabilities over the vocabulary, and its read slots' position logits
+        (`slots.slot_logits`), in one forward (`_rows`)."""
+        ids, lengths, media = self._rows(seqs, batch)
+        width = ids.shape[1]
+        mask = (torch.arange(width, device=self.device) < torch.tensor(lengths, device=self.device)[:, None]).long()
+        state = None if self.attention == "causal" else torch.tensor([seqs[b][3] for b in batch])
+        question = None if self.attention != "set" else torch.tensor(
+            [seqs[b][4] + [-1] * (width - len(seqs[b][4])) for b in batch])
+        rows = slot_logits(self.model, ids, mask, torch.tensor([s[3] for s in slots]), state, question,
+                           torch.tensor([s[0] for s in slots]), pointer=pointer, **self._on_device(media))
+        codes = rows.codes.float()
+        return codes - torch.logsumexp(codes, dim=-1, keepdim=True), rows.positions
+
+    def _readouts(self, qs: Sequence[Question], logz: torch.Tensor, tempered: Sequence[bool]) -> list[list[float]]:
+        """Every code slot of a batch read at once (`prompt.readouts`): one transfer, not one a slot."""
+        return readouts(self.tokenizer, qs, logz, [self.calibration if t else None for t in tempered])
+
+    def tokens(self, state: Any, questions: Sequence[Question]) -> int:
+        """The longest prompt one of `questions` makes over a text state: its state's tokens and its own."""
+        prefix, _ = self._prefix(state, ())
+        return len(self.tokenizer.encode(prefix, add_special_tokens=False)) + max(
+            len(self.tokenizer.encode(suffix_text(self.tokenizer, q, self.lead, self.option_style),
+                                      add_special_tokens=False)) for q in questions)
+
+    # ----------------------------------------------------------------- media
+
+    def _load_processor(self):
+        return auto_processor(self.model.config._name_or_path, self.revision)
+
+    def _on_device(self, inputs: dict) -> dict:
+        """The media inputs on the model's device, but what the backbone reads on the host (`host_inputs`)."""
+        host = getattr(self.model, "host_inputs", ())
+        return {k: (v.to(self.device) if hasattr(v, "to") and k not in host else v) for k, v in inputs.items()}

slots.py

@@ -0,0 +1,61 @@
+"""Answer positions of a right-padded batch, read through whatever wraps the model."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import torch
+
+from .attention import state_attention, text_backbone
+
+
+@dataclass(frozen=True)
+class SlotLogits:
+    """A batch's answer slots: the code slots' logits, in slot order, and the read slots' over their rows' state
+    positions, in the pointer plan's order (pointer.py `query_plan`)."""
+
+    codes: torch.Tensor
+    positions: torch.Tensor | None = None
+
+
+def split(slots: torch.Tensor, states: torch.Tensor | None, plan: dict | None, project) -> SlotLogits:
+    """The hidden states at the slots [S, D]: the code slots through `project` (the LM head, or its option rows),
+    the read slots against their rows' state positions `states` [R, T, D]."""
+    if plan is None:
+        return SlotLogits(project(slots))
+    from .pointer import read_logits
+
+    return SlotLogits(project(slots[plan["code_slots"].to(slots.device)]), read_logits(slots, states, plan))
+
+
+def output_head(model):
+    """The LM head, through whatever wrappers are in the way."""
+    head = getattr(model, "module", model).get_output_embeddings()
+    if head is None:
+        raise RuntimeError("no output embedding on this model")
+    return head
+
+
+def slot_logits(
+    model,
+    ids: torch.Tensor,
+    mask: torch.Tensor,
+    last: torch.Tensor,
+    state_len: torch.Tensor | None = None,
+    question: torch.Tensor | None = None,
+    row: torch.Tensor | None = None,
+    pointer: dict | None = None,
+    **vision,
+):
+    """Logits at one position per row (or at each `row`, `last` pair: a question may have several slots in a row),
+    the backbone read directly and only the slots projected, never `B x T x vocab`. `vision`: the processor's picture
+    inputs, for a vision-language model."""
+    with state_attention(model, state_len, question):  # None: the model's own causal mask
+        out = text_backbone(model)(input_ids=ids, attention_mask=mask, use_cache=False, **vision)
+    hidden = getattr(out, "last_hidden_state", None)
+    if hidden is None:
+        hidden = out[0]
+    row = torch.arange(hidden.size(0)) if row is None else row
+    slots = hidden[row.to(hidden.device), last.to(hidden.device)]  # a model over GPUs ends elsewhere
+    states = hidden[pointer["rows"].to(hidden.device), : pointer["width"]] if pointer is not None else None
+    return split(slots, states, pointer, output_head(model))

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a39eed58008cbcbc09e5b629b81338a654269c64cd13348aaa6090f4db75c7e7
+size 17908258

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "eos_token": "<|im_end|>",
+  "extra_special_tokens": [],
+  "image_token": "<|image|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "model_specific_special_tokens": {
+    "image_token": "<|image|>",
+    "video_token": "<|video|>",
+    "vision_end_token": "<|vision_end|>",
+    "vision_start_token": "<|vision_start|>"
+  },
+  "pad_token": "<|pad|>",
+  "processor_class": "Lfm3VlProcessor",
+  "tokenizer_class": "TokenizersBackend",
+  "video_token": "<|video|>",
+  "vision_end_token": "<|vision_end|>",
+  "vision_start_token": "<|vision_start|>"
+}

video_processing_lfm3_vl.py

@@ -0,0 +1,404 @@
+# Vendored from transformers' LFM3 code (not in a release yet) at 69da1b47a11e362fbb15bfe5d5c8da0790c688c5, src/transformers/models/lfm3_vl/video_processing_lfm3_vl.py:
+# its relative imports made absolute, nothing else changed.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+#           This file was automatically generated from src/transformers/models/lfm3_vl/modular_lfm3_vl.py.
+#               Do NOT edit this file manually as any edits will be overwritten by the generation of
+#             the file from the modular. If any change should be done, please apply the change to the
+#                          modular_lfm3_vl.py file directly. One of our CI enforces this.
+#                🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨🚨
+# Copyright 2026 The HuggingFace Team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import math
+
+import numpy as np
+import torch
+from torchvision.transforms.v2 import functional as tvF
+
+from transformers.image_processing_utils import BatchFeature
+from transformers.image_utils import PILImageResampling, SizeDict
+from transformers.processing_utils import Unpack, VideosKwargs
+from transformers.utils import TensorType, auto_docstring, logging
+from transformers.video_processing_utils import BaseVideoProcessor
+from transformers.video_utils import (
+    VideoMetadata,
+    group_videos_by_shape,
+    make_batched_metadata,
+    make_batched_videos,
+    reorder_videos,
+)
+
+
+logger = logging.get_logger(__name__)
+
+
+class Lfm3VlVideoProcessorInitKwargs(VideosKwargs, total=False):
+    r"""
+    patch_size (`int`, *optional*, defaults to 16):
+        The spatial patch size of the vision encoder.
+    temporal_patch_size (`int`, *optional*, defaults to 2):
+        The temporal patch size of the vision encoder.
+    merge_size (`int`, *optional*, defaults to 2):
+        The merge size of the vision encoder to llm encoder.
+    min_frames (`int`, *optional*, defaults to 4):
+        The minimum number of frames to sample from video.
+    max_frames (`int`, *optional*, defaults to 768):
+        The maximum number of frames to sample from video.
+    cap_pixels_per_frame (`bool`, *optional*):
+        Whether to cap each frame's pixel cost the way the reference implementation (qwen-vl-utils) does:
+        per-frame pixels are limited to `min(max_video_tokens * factor**2, size["longest_edge"] / num_frames)`
+        and floored at `1.05 * size["shortest_edge"]`, so token cost scales with clip duration. Without the
+        cap, videos that sample few frames spend the whole `size["longest_edge"]` budget on those frames and
+        keep near-native per-frame resolution, so a short clip can cost almost as many tokens as a long
+        video. If unset, the current uncapped behavior is kept and a warning is emitted: the default will
+        change to `True` in v5.22, after which the argument will be removed.
+    max_video_tokens (`int`, *optional*, defaults to 768):
+        The per-frame token ceiling applied by `cap_pixels_per_frame`, in vision tokens per frame
+        (qwen-vl-utils' `VIDEO_MAX_TOKEN_NUM`).
+    """
+
+    patch_size: int
+    temporal_patch_size: int
+    merge_size: int
+    min_frames: int
+    max_frames: int
+    cap_pixels_per_frame: bool
+    max_video_tokens: int
+
+
+def smart_resize(
+    num_frames: int,
+    height: int,
+    width: int,
+    temporal_factor: int = 2,
+    factor: int = 28,
+    min_pixels: int = 112 * 112,
+    max_pixels: int = 14 * 14 * 2 * 2 * 2 * 6144,
+):
+    if num_frames < temporal_factor:
+        raise ValueError(f"t:{num_frames} must be larger than temporal_factor:{temporal_factor}")
+    if height < factor or width < factor:
+        scale = max(factor / height, factor / width)
+        height = int(height * scale)
+        width = int(width * scale)
+
+    if max(height, width) / min(height, width) > 200:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
+        )
+    h_bar = round(height / factor) * factor
+    w_bar = round(width / factor) * factor
+    t_bar = round(num_frames / temporal_factor) * temporal_factor
+
+    if t_bar * h_bar * w_bar > max_pixels:
+        beta = math.sqrt((num_frames * height * width) / max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+    elif t_bar * h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (num_frames * height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+
+    return h_bar, w_bar
+
+
+def _check_video_fps(metadata: VideoMetadata) -> None:
+    if metadata.fps is not None and not (math.isfinite(metadata.fps) and metadata.fps > 0):
+        raise ValueError(f"`video_metadata.fps` must be a positive number, got {metadata.fps}")
+
+
+def _check_video_metadata_count(videos, video_metadata) -> None:
+    # the shared sampler pairs clips with metadata by `zip`, which drops the clips past the last entry
+    if video_metadata is None:
+        return
+    videos = make_batched_videos(videos)
+    entries = make_batched_metadata(videos, video_metadata) if video_metadata != [] else []
+    if len(entries) != len(videos):
+        raise ValueError(
+            f"Received {len(videos)} videos but {len(entries)} `video_metadata` entries; pass one entry per video"
+        )
+
+
+@auto_docstring
+class Lfm3VlVideoProcessor(BaseVideoProcessor):
+    resample = PILImageResampling.BICUBIC
+    size = {"shortest_edge": 4_096, "longest_edge": 25_165_824}
+    image_mean = [0.5, 0.5, 0.5]
+    image_std = [0.5, 0.5, 0.5]
+    do_resize = True
+    do_rescale = True
+    do_normalize = True
+    do_convert_rgb = True
+    patch_size = 16
+    temporal_patch_size = 2
+    merge_size = 2
+    min_frames = 4
+    max_frames = 768
+    do_sample_frames = True
+    cap_pixels_per_frame = False
+    max_video_tokens = 768
+    valid_kwargs = Lfm3VlVideoProcessorInitKwargs
+    model_input_names = ["pixel_values_videos", "video_grid_thw"]
+    max_image_size = {"longest_edge": 28 * 28 * 2 * 30000}
+    max_duration = None
+    num_frames = None
+    fps = 2.0
+
+    def __init__(self, **kwargs: Unpack[Lfm3VlVideoProcessorInitKwargs]):
+        super().__init__(**kwargs)
+
+    def sample_frames(
+        self, metadata: VideoMetadata, num_frames: int | None = None, fps: int | float | None = None, **kwargs
+    ):
+        """Decoded frames whose metadata carries no fps are sampled and timestamped at the 24 fps the shared sampler
+        assumes (with a warning); pass `video_metadata` with the clip's fps for accurate timestamps."""
+        _check_video_fps(metadata)
+        if fps is not None and not (math.isfinite(fps) and fps > 0):
+            raise ValueError(f"`fps` must be a positive number, got {fps}")
+        if fps is not None and num_frames is not None:
+            raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
+
+        total_num_frames = metadata.total_num_frames
+        fps = fps if fps is not None else self.fps
+
+        # If num_frames is not given but fps is, calculate num_frames from fps
+        if num_frames is None and fps is not None:
+            if metadata.fps is None:
+                metadata.fps = 24
+                logger.warning_once(
+                    "Asked to sample `fps` frames per second but no video metadata was provided which is required when sampling with `fps`. "
+                    "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+                )
+            num_frames = int(total_num_frames / metadata.fps * fps)
+            num_frames = min(max(num_frames, self.min_frames), self.max_frames, total_num_frames)
+
+        if num_frames is None:
+            num_frames = min(max(total_num_frames, self.min_frames), self.max_frames)
+
+        indices = np.linspace(0, total_num_frames - 1, num_frames).round().astype(int)
+
+        return indices
+
+    def resize(
+        self,
+        videos: "torch.Tensor",
+        size: SizeDict,
+        resample: "PILImageResampling | tvF.InterpolationMode | int | None",
+        factor: int,
+        temporal_factor: int,
+        cap_pixels_per_frame: bool | None = None,
+        **kwargs,
+    ) -> "torch.Tensor":
+        """Resize dynamically based on input video aspect ratio."""
+        if not size.shortest_edge or not size.longest_edge:
+            raise ValueError(f"`size` dict must contain 'shortest_edge' and 'longest_edge' keys but got {size}.")
+
+        num_frames = videos.shape[1]
+        max_pixels = size.longest_edge
+        if cap_pixels_per_frame:
+            # per-frame pixels are capped at `max_video_tokens` patches or the budget's even share per frame
+            frame_cap = self.max_video_tokens * factor * factor
+            pixels_per_frame = max(min(frame_cap, size.longest_edge // num_frames), int(size.shortest_edge * 1.05))
+            max_pixels = pixels_per_frame * num_frames
+
+        height, width = videos.shape[-2:]
+        resized_height, resized_width = smart_resize(
+            height=height,
+            width=width,
+            num_frames=num_frames,
+            factor=factor,
+            temporal_factor=temporal_factor,
+            min_pixels=size.shortest_edge,
+            max_pixels=max_pixels,
+        )
+        return super().resize(
+            image=videos,
+            size=SizeDict(height=resized_height, width=resized_width),
+            resample=resample,
+        )
+
+    def patchify(
+        self,
+        videos: "torch.Tensor",
+        patch_size: int,
+        merge_size: int,
+        temporal_patch_size: int,
+    ) -> tuple["torch.Tensor", int, int]:
+        "Patchifies each video into flat layout of shape (`seq_len`, `patch_dim`) so we can concat dynamically shaped pixels."
+        batch_size, num_frames, channel, resized_height, resized_width = videos.shape
+
+        # Check that videos have `num_frames` divisible by `temporal_patch_size`
+        if pad := -num_frames % temporal_patch_size:
+            repeats = videos[:, -1:].expand(-1, pad, -1, -1, -1)
+            videos = torch.cat((videos, repeats), dim=1)
+            num_frames += pad
+
+        grid_t = num_frames // temporal_patch_size
+        grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+        patches = videos.view(
+            batch_size,
+            grid_t,
+            temporal_patch_size,
+            channel,
+            grid_h // merge_size,
+            merge_size,
+            patch_size,
+            grid_w // merge_size,
+            merge_size,
+            patch_size,
+        )
+        patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+        flatten_patches = patches.reshape(
+            batch_size,
+            grid_t * grid_h * grid_w,
+            channel * temporal_patch_size * patch_size * patch_size,
+        )
+
+        return flatten_patches, grid_t, grid_h, grid_w
+
+    def _preprocess(
+        self,
+        videos: list["torch.Tensor"],
+        do_resize: bool,
+        size: SizeDict,
+        resample: "PILImageResampling | tvF.InterpolationMode | int | None",
+        do_rescale: bool,
+        rescale_factor: float,
+        do_normalize: bool,
+        image_mean: float | list[float] | None,
+        image_std: float | list[float] | None,
+        patch_size: int | None = None,
+        temporal_patch_size: int | None = None,
+        merge_size: int | None = None,
+        cap_pixels_per_frame: bool | None = None,
+        return_tensors: str | TensorType | None = None,
+        **kwargs,
+    ):
+        if cap_pixels_per_frame is None:
+            logger.warning_once(
+                "Lfm3Vl video processing does not apply the per-frame pixel cap the reference "
+                "implementation (qwen-vl-utils) applies, so some videos cost far more tokens than they "
+                "would there. In v5.22 the capped behavior will become the default and "
+                "`cap_pixels_per_frame` will be removed. Pass `cap_pixels_per_frame=True` to adopt the "
+                "reference behavior now, or `False` to keep the current behavior and silence this "
+                "warning."
+            )
+            cap_pixels_per_frame = False
+        # Group videos by size for batched resizing
+        grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
+        resized_videos_grouped = {}
+        for shape, stacked_videos in grouped_videos.items():
+            if do_resize:
+                stacked_videos = self.resize(
+                    videos=stacked_videos,
+                    size=size,
+                    resample=resample,
+                    factor=patch_size * merge_size,
+                    temporal_factor=temporal_patch_size,
+                    cap_pixels_per_frame=cap_pixels_per_frame,
+                )
+            resized_videos_grouped[shape] = stacked_videos
+        resized_videos = reorder_videos(resized_videos_grouped, grouped_videos_index)
+
+        # Group videos by size for further processing
+        # Needed in case do_resize is False, or resize returns videos with different sizes
+        grouped_videos, grouped_videos_index = group_videos_by_shape(resized_videos)
+        processed_videos_grouped = {}
+        processed_grids = {}
+        for shape, stacked_videos in grouped_videos.items():
+            # Fused rescale and normalize
+            stacked_videos = self.rescale_and_normalize(
+                stacked_videos, do_rescale, rescale_factor, do_normalize, image_mean, image_std
+            )
+            patches, grid_t, grid_h, grid_w = self.patchify(
+                stacked_videos,
+                patch_size=patch_size,
+                merge_size=merge_size,
+                temporal_patch_size=temporal_patch_size,
+            )
+
+            processed_videos_grouped[shape] = patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * len(stacked_videos)
+
+        processed_videos = reorder_videos(processed_videos_grouped, grouped_videos_index)
+        processed_grids = reorder_videos(processed_grids, grouped_videos_index)
+        pixel_values_videos = torch.cat(processed_videos, dim=0)
+        video_grid_thw = torch.tensor(processed_grids)
+
+        return BatchFeature(
+            data={"pixel_values_videos": pixel_values_videos, "video_grid_thw": video_grid_thw},
+            tensor_type=return_tensors,
+        )
+
+    def get_num_of_video_patches(self, num_frames: int, height: int, width: int, videos_kwargs=None):
+        """
+        A utility that returns number of video patches a given video size.
+
+        Args:
+            num_frames (`int`):
+                Number of frames in the input video.
+            height (`int`):
+                Height of the input video.
+            width (`int`):
+                Width of the input video.
+            videos_kwargs (`dict`, *optional*)
+                Any kwargs to override defaults of the video processor.
+        Returns:
+            `int`: Number of video patches per video.
+        """
+        videos_kwargs = videos_kwargs if videos_kwargs is not None else {}
+        size = videos_kwargs.get("size", None) or self.size
+        min_pixels = size["shortest_edge"]
+        max_pixels = size["longest_edge"]
+        patch_size = videos_kwargs.get("patch_size", None) or self.patch_size
+        merge_size = videos_kwargs.get("merge_size", None) or self.merge_size
+        temporal_patch_size = videos_kwargs.get("temporal_patch_size", None) or self.temporal_patch_size
+        cap_pixels_per_frame = videos_kwargs.get("cap_pixels_per_frame", None) or self.cap_pixels_per_frame
+
+        factor = patch_size * merge_size
+        if cap_pixels_per_frame:
+            # Keep the count in sync with `resize` when the per-frame cap is active.
+            frame_cap = self.max_video_tokens * factor * factor
+            pixels_per_frame = max(min(frame_cap, max_pixels // num_frames), int(min_pixels * 1.05))
+            max_pixels = pixels_per_frame * num_frames
+
+        resized_height, resized_width = smart_resize(
+            num_frames,
+            height,
+            width,
+            temporal_factor=temporal_patch_size,
+            factor=factor,
+            min_pixels=min_pixels,
+            max_pixels=max_pixels,
+        )
+        grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+        grid_t = (num_frames + -num_frames % temporal_patch_size) // temporal_patch_size
+        return grid_t * grid_h * grid_w
+
+    def preprocess(self, videos, **kwargs):
+        _check_video_metadata_count(videos, kwargs.get("video_metadata"))
+        # the shared `preprocess` fills every unset option from the instance before `sample_frames` runs, so the
+        # caller's options are told apart from the instance ones here: a caller option beats an instance one, and a
+        # configured `num_frames` beats the default fps
+        num_frames, fps = kwargs.get("num_frames"), kwargs.get("fps")
+        if num_frames is not None and fps is not None:
+            raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
+        if num_frames is not None or (fps is None and self.num_frames is not None):
+            kwargs["fps"] = None
+        elif fps is not None:
+            kwargs["num_frames"] = None
+        return super().preprocess(videos, **kwargs)
+
+
+__all__ = ["Lfm3VlVideoProcessor"]