README.md

---
license: other
license_name: falcon-llm-license
license_link: https://falconllm.tii.ae/falcon-terms-and-conditions.html
base_model: tiiuae/Falcon-H1-3B-Instruct
tags:
  - litert
  - litert-lm
  - litertlm
  - on-device
  - edge
  - hybrid
  - mamba2
  - falcon-h1
pipeline_tag: text-generation
library_name: litert-lm
---

# Falcon-H1-3B-Instruct — LiteRT-LM

[tiiuae/Falcon-H1-3B-Instruct](https://huggingface.co/tiiuae/Falcon-H1-3B-Instruct) converted to the **LiteRT-LM** (`.litertlm`) format for on-device inference with Google's [LiteRT-LM](https://github.com/google-ai-edge/litert-lm) runtime. **Requires litert-lm ≥ 0.15.** Sibling of [litert-community/Falcon-H1-0.5B-Instruct](https://huggingface.co/litert-community/Falcon-H1-0.5B-Instruct) and [litert-community/Falcon-H1-1.5B-Instruct](https://huggingface.co/litert-community/Falcon-H1-1.5B-Instruct) — same conversion, same patch.

Falcon-H1 is TII's fully-hybrid design: every one of the 32 layers runs a grouped-query attention branch and a Mamba2 selective-scan branch **in parallel** on the same input and sums them. Each layer therefore carries both a KV cache and constant-size conv + SSM recurrent state.

| File | Recipe | Size |
|---|---|---|
| `Falcon-H1-3B-Instruct_int8.litertlm` | int8 dynamic on linears + embedding (convs and the scan stay float); fp32 activations declared for GPU | 3.15 GB |

## Correctness

- **Logits parity vs PyTorch**: the float export matches the HF model teacher-forced across 48 decode positions — top-1 and top-5 identical at every position, mean per-position logit correlation 1.0000, mean KL ≈ 0.
- **8-question sanity gate**: **8/8 on every lane** — GPU and CPU, litert-lm 0.15.0 and 0.16.0. No degeneration, no greedy flips (the first Falcon-H1 size where int8 drops nothing).
- **Prompt-length robustness**: hermetic prefill-chunk sweep (fresh engine per length) — CPU fills 12–51 and GPU fills 12–31 all clean.
- **iPhone 17 Pro (Metal)**: the 8-item composite quality probe answers **8/8 on GPU and 8/8 on CPU**, identical answers on both backends.

## Usage

```bash
litert-lm run ./Falcon-H1-3B-Instruct_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Falcon-H1-3B-Instruct_int8.litertlm --backend gpu --cache no --prompt "..."
```

Multi-length prefill signatures (1–1024) are exported so the runtime picks tight chunks. The bundle carries the tokenizer and the stock ChatML-style Falcon-H1 chat template.

## Performance

`litert-lm benchmark` (litert-lm 0.16.0), Apple M4 Max, `-p 256 -d 256 --runs 3 --cache no`, quiet machine:

| Backend | Prefill (256) | Decode | TTFT |
|---|---|---|---|
| GPU | 979 tok/s | 65.3 tok/s | 0.28 s |
| CPU | 121 tok/s | 20.9 tok/s | 2.17 s |

On device (cold start, single runs, 146-token composite prompt, quality harness):

| Device | Backend | Prefill | Decode | TTFT | Peak memory |
|---|---|---|---|---|---|
| iPhone 17 Pro | **GPU (Metal)** | 111.5 tok/s | **14.0 tok/s** | **1.49 s** | 3.03 GB |
| iPhone 17 Pro | CPU | 48.7 tok/s | 7.8 tok/s | 3.14 s | 1.46 GB |

Honest notes:
- **GPU runs with fp32 activations** (declared in the bundle) — expect a corresponding memory multiple over CPU.

## Conversion notes

Converted with [`litert-torch`](https://github.com/google-ai-edge/litert-torch) plus a hybrid-cache patch (reproduction script + patch: [hf-to-litertlm `falcon_h1_work/`](https://github.com/john-rocky/hf-to-litertlm)):

- **Composite hybrid cache layer**: every layer holds KV + conv + recurrent state at ONE layer index — a cache layer class that is full-attention and Mamba2 at the same time (the runtime binds states by tensor name, so co-residency is just packaging).
- **Folded selective scan**: the Mamba2 scan is re-expressed as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no `BROADCAST_TO`, no int64 index math) — this is what makes the graph fully delegable on GPU.
- **Falcon-specific wiring**: the µP multiplier vector (`mup_vector`, a non-persistent model-level buffer) and `ssm_in_multiplier` are preserved in the traced scan; the exporter's timestamp-index kwargs are re-injected at the attention layer (FalconH1's layer loop drops kwargs).
- **Prefill-pad guard**: the runtime runs partially-filled prefill chunks; pad positions are made exact identity steps for the SSM and the stored conv window is gathered at the last valid column.
- **Quantization**: post-hoc dynamic int8 over linears + embedding only; convs and the scan stay float.

## License and changes

Distributed under the **Falcon LLM License** (inherited from the base model — see the license link). **Changes from the original work:** weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above; tokenizer and chat template repackaged unmodified. This repository is a community conversion and is not affiliated with TII.