TwIL-LM3 LiteRT-LM card

README.md

@@ -0,0 +1,78 @@
+---
+license: other
+license_name: webai-non-commercial-license-ver.-1.0
+license_link: LICENSE.md
+base_model: webAI-Official/TwIL-LM3
+tags:
+  - litert
+  - litert-lm
+  - litertlm
+  - on-device
+  - edge
+pipeline_tag: text-generation
+library_name: litert-lm
+---
+
+# TwIL-LM3 — LiteRT-LM
+
+[webAI-Official/TwIL-LM3](https://huggingface.co/webAI-Official/TwIL-LM3) converted to the **LiteRT-LM** (`.litertlm`) format for on-device inference with Google's [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime. **Requires litert-lm ≥ 0.16.**
+
+**⚠ Non-commercial.** The upstream model ships under the webAI Non-Commercial License v1.0, and this conversion inherits it unchanged: **non-commercial research and educational use only**. The full license text is in `LICENSE.md` in this repository — §3.1 requires that any redistribution carries it, so keep it with the file.
+
+| File | Recipe | Size |
+|---|---|---|
+| `TwIL-LM3_int4.litertlm` | int4 blockwise-32 + OCTAV on linears, int8 embedding (externalised) | 2.00 GB |
+
+## Correctness
+
+**8-question sanity gate: 8/8 on every backend measured** — Mac GPU, Mac CPU, and on an iPhone 17 Pro on both Metal and CPU. Non-degenerate in all four. A 3B model answering all eight on a phone at the bundle's own 4096-token context, with no ladder reduction, is the useful part.
+
+Graded with `--max-tokens 2048`: this is a reasoning model and emits a `<think>` block before answering, so a smaller budget scores it as wrong when it has simply not finished thinking.
+
+**Not measured: the formal-logic claim.** Upstream describes TwIL-LM3 as specialised for formal logic. The eight-question gate is general knowledge, so it cannot see that claim in either direction — a model could be better or worse at logic than its base and score identically here. This card therefore makes **no claim about logical reasoning ability**, and none should be read into the 8/8.
+
+## Performance
+
+`litert-lm benchmark` (litert-lm 0.16.0), Apple M4 Max, `-p 256 -d 256 --cache no`, quiet machine, serialized, measured twice (run-to-run spread under 1%):
+
+| Backend | Prefill (256) | Decode | TTFT | Init |
+|---|---|---|---|---|
+| **GPU (Metal)** | **1341 tok/s** | **93.0 tok/s** | 0.206 s | 2.50 s |
+| CPU | 141.5 tok/s | 25.2 tok/s | 2.23 s | 1.97 s |
+
+iPhone 17 Pro, same file, own harness driving the LiteRT-LM Swift API:
+
+| Backend | 8Q | Context | Init |
+|---|---|---|---|
+| **GPU (Metal)** | **8/8** | bundle default (4096) | 6.34 s |
+| CPU | **8/8** | bundle default (4096) | 3.52 s |
+
+The phone harness measures correctness and load time only — **it does not measure throughput, so no phone tok/s figure is quoted here** rather than one being estimated from the desktop.
+
+## Usage
+
+```bash
+litert-lm run ./TwIL-LM3_int4.litertlm --prompt "If it is raining then the ground is wet. The ground is not wet. Is it raining?"
+
+# GPU
+litert-lm run ./TwIL-LM3_int4.litertlm --backend gpu --cache no --prompt "..."
+```
+
+The bundle carries the tokenizer and the SmolLM3 chat template with its thinking controls (`/think`, `/no_think`) — not plain ChatML. A 4096-token KV budget.
+
+## Conversion notes
+
+Converted with [`litert-torch`](https://github.com/google-ai-edge/litert-torch) 0.9.3 / litert-converter 0.3.1 / ai-edge-quantizer 0.8.0 / litert-lm-builder 0.16.0 — a pristine released stack, no patched checkout.
+
+- **Same rail as SmolLM3-3B, and the file size proves it.** TwIL-LM3 is a SmolLM3-3B finetune, so it converts with the identical one-command recipe. The result is **2,002,241,456 bytes** against the shipped SmolLM3-3B's 2,002,257,840 — a 16 KB difference on the same architecture and recipe, which is the strongest single check that the rail applied correctly.
+- **`EXTERNALIZE_EMBEDDER=1` is required, not cosmetic.** The model ties its embedding and `lm_head`, so a recipe asking for int4 linears and an int8 embedder describes one tensor two ways; the quantizer resolves it by copying the 128256-row table once per prefill signature. Externalising the table removes the conflict. Verified duplication-free after export (0.65 bytes/param).
+- **No `start_token` in the bundle**, which is correct here: upstream sets `bos_token: None`. A bundle that prepends a BOS the model was never trained to see is a quiet quality killer, so it is worth confirming rather than assuming.
+- **Quantization**: int4 blockwise-32 with OCTAV clipping on the linears, int8 on the embedding.
+
+## License and changes
+
+Distributed under the **webAI Non-Commercial License v1.0**, inherited unchanged from the base model; the complete license text is included as `LICENSE.md` as its §3.1 requires. **Use is limited to non-commercial research and educational purposes (§3.3).** 
+
+**Changes from the original work:** weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged into the `.litertlm` bundle. No weights were retrained or altered in value beyond quantization.
+
+This repository is a community conversion and is not affiliated with webAI.