Mordant-3B-Think — LiteRT-LM (on-device)
On-device conversion of Kezmark/Mordant-3B-Think —
a full fine-tune of ibm-granite/granite-4.1-3b
for AI image-generation prompt composition with chain-of-thought reasoning — to a .litertlm
bundle for the LiteRT-LM runtime. All credit for
the model itself goes to its author; this repo only packages it for phones and desktops.
Requires litert-lm ≥ 0.16 to run.
| file | quant | size |
|---|---|---|
Mordant-3B-Think_int8.litertlm |
dynamic int8 (linears + embedding) | 3.76 GB |
Conversion & verification
Converted with one command by hf-to-litertlm
(python scripts/convert.py Kezmark/Mordant-3B-Think, 2026-08-25):
- The finetune's own chat template — a thinking-form template that opens the assistant turn
with
<think>— is embedded verbatim (byte-equal to the checkpoint'schat_template.jinja, 1474/1474). - The spurious metadata start token is dropped: this family declares
bos == eos == <|end_of_text|>and its template never renders a leading BOS, so an engine-prepended start token reads as "this document already ended" — measured on this checkpoint, it flips HF bf16 greedy output into a code-fence loop. - Reduced 7-signature prefill ladder + externalized embedder (the ≥3B ship shape; the full 11-signature ladder is killed by iOS at Metal init on this family).
- Quality gate: 8/8 on the 8-question sanity gate (think-aware budget), non-degenerate.
Performance (Apple M4 Max, litert-lm 0.16.0, -p 256 -d 256 --runs 3 --cache no)
| backend | prefill tok/s | decode tok/s | TTFT |
|---|---|---|---|
| CPU | 97.3 | 20.2 | 2.68 s |
| GPU | 1129 | 71.9 | 0.24 s |
Usage
pip install litert-lm
litert-lm run Mordant-3B-Think_int8.litertlm \
--prompt "A cat sitting on a windowsill at sunset" --max-num-tokens 4096
The model answers with a <think>…</think> block followed by the composed image prompt —
budget generation length accordingly. On Android, load the bundle in an app embedding the
LiteRT-LM engine (e.g. Google AI Edge Gallery-style hosts).
License
apache-2.0, inherited from the source model and its granite-4.1 base.
- Downloads last month
- -