README.md

---
license: apache-2.0
base_model: Kezmark/Mordant-3B-Think
base_model_relation: quantized
tags:
  - litert
  - litert-lm
  - on-device
  - granite
  - reasoning
  - image-prompt
language:
  - en
pipeline_tag: text-generation
---

# Mordant-3B-Think — LiteRT-LM (on-device)

On-device conversion of [Kezmark/Mordant-3B-Think](https://huggingface.co/Kezmark/Mordant-3B-Think) —
a full fine-tune of [ibm-granite/granite-4.1-3b](https://huggingface.co/ibm-granite/granite-4.1-3b)
for AI image-generation prompt composition with chain-of-thought reasoning — to a `.litertlm`
bundle for the [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) runtime. All credit for
the model itself goes to its author; this repo only packages it for phones and desktops.

**Requires litert-lm ≥ 0.16 to run.**

| file | quant | size |
|---|---|---|
| `Mordant-3B-Think_int8.litertlm` | dynamic int8 (linears + embedding) | 3.76 GB |

## Conversion & verification

Converted with one command by [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm)
(`python scripts/convert.py Kezmark/Mordant-3B-Think`, 2026-08-25):

- The finetune's own chat template — a thinking-form template that opens the assistant turn
  with `<think>` — is embedded verbatim (byte-equal to the checkpoint's
  `chat_template.jinja`, 1474/1474).
- The spurious metadata start token is dropped: this family declares `bos == eos ==
  <|end_of_text|>` and its template never renders a leading BOS, so an engine-prepended
  start token reads as "this document already ended" — measured on this checkpoint, it flips
  HF bf16 greedy output into a code-fence loop.
- Reduced 7-signature prefill ladder + externalized embedder (the ≥3B ship shape; the full
  11-signature ladder is killed by iOS at Metal init on this family).
- Quality gate: **8/8** on the 8-question sanity gate (think-aware budget), non-degenerate.

## Performance (Apple M4 Max, litert-lm 0.16.0, `-p 256 -d 256 --runs 3 --cache no`)

| backend | prefill tok/s | decode tok/s | TTFT |
|---|---:|---:|---:|
| CPU | 97.3 | 20.2 | 2.68 s |
| GPU | 1129 | 71.9 | 0.24 s |

## Usage

```bash
pip install litert-lm
litert-lm run Mordant-3B-Think_int8.litertlm \
  --prompt "A cat sitting on a windowsill at sunset" --max-num-tokens 4096
```

The model answers with a `<think>…</think>` block followed by the composed image prompt —
budget generation length accordingly. On Android, load the bundle in an app embedding the
LiteRT-LM engine (e.g. Google AI Edge Gallery-style hosts).

## License

apache-2.0, inherited from the source model and its granite-4.1 base.