Falcon-H1R-7B — LiteRT-LM

tiiuae/Falcon-H1R-7B converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime: an int8 file and an int4 file. Sibling of the Falcon-H1-0.5B/1.5B/1.5B-Deep/3B-Instruct and Falcon-H1-Tiny-R-0.6B conversions (the same hybrid-cache patch and reasoning packaging). Converted with litert-torch (commit 115a136 with the Falcon-H1 patch) and tested on litert-lm 0.18.0 on a Mac Studio M4 Max, where the int4 file decodes at 39.7 tok/s on the GPU (measured 2026-10-15); the two phone tries are described under Performance. On litert-lm 0.17.1 the int8 file answered a plain run with a closed thought and then "Paris." (one GPU process; the int4 file was not run on 0.17.1).

TII describes Falcon-H1R-7B as a reasoning-specialized model built on Falcon-H1-7B-Base with cold-start supervised fine-tuning on long reasoning traces and then reinforcement learning (GRPO), a causal decoder with a hybrid Transformer + Mamba2 architecture, English and multilingual. The checkpoint has 7,585,648,736 parameters in 44 layers; every layer runs a grouped-query attention branch (12 heads, 2 KV heads) and a Mamba2 selective-scan branch (24 heads, state size 256) in parallel on the same input; hidden size 3072, a 130,048-token vocabulary with untied embeddings, a 256K context (4096 in these files). TII recommends temperature 0.6 and top-p 0.95 with up to 65,536 new tokens.

Falcon-H1R is a reasoning model: it opens a <think>…</think> block on its own before answering. The bundle declares the thinking channel (thought, <think> / </think>), so the runtime separates the reasoning from the answer (the CLI prints it as [thought] … [/thought]) and --thinking-budget works: with a 64-token budget the reasoning stopped mid-sentence and the model answered 17 × 23 with 481, so a cut reasoning does not finish, it gets cut.

File	Recipe	Size	sha256
Falcon-H1R-7B_int8.litertlm	int8 dynamic on every linear, int8 embedding; the Mamba conv and scan float; fp32 GPU activations declared	7.81 GB (7,810,102,560 bytes)	9bde28ab1e3026dfec0b16fb5d21e035e8fcb82b7e6c7cd9c679f898ef0a726e
Falcon-H1R-7B_int4.litertlm	int4 blockwise-32 on the linears (block scales floored at 2^-14, Conversion notes), int8 output projection and embedding; fp32 GPU activations declared	4.69 GB (4,689,037,856 bytes)	7a8820ae4f38bc82b40b06f181ed013c012ec38dbf5329b866d68183080a8ea1

On GSM8K with a 2,048-token reasoning budget (answers scored after </think>, table in Accuracy) the int8 file scored 96 and the int4 file 98 on the Mac GPU; the bf16 reference ran on the first 21 questions only and got all 21 (the int8 file 20 and the int4 file 21 of those). The int4 file is 40 % smaller. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int4 GPU	8.26	GPU engine does not build (Performance)	stopped by our memory rule (Performance)
int8 GPU	8.95	not run (Performance)	not run (Performance)
int8 CPU	29.65	not run (Performance)	not run (Performance)
int4 CPU	37.72	not run (Performance)	not run (Performance)

On the Mac int4 is the faster file on the GPU (7.7 % less time, decode 39.7 vs 35.9 tok/s at the same prefill rate) and int8 on the CPU (int4's blockwise prefill runs at 44 % of int8's). Not measured on iOS: each file is one TFLite section (7.27 GiB int8, 4.36 GiB int4, the embedding table inside it), over the ~2 GiB single-section limit the Ministral-3-3B-Instruct-2512 card describes; the 0.6B sibling is the family member that runs on iPhone.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, one fresh engine each, greedy, up to 2,048 output tokens, answers scored after </think>): int8 8/8 on CPU and GPU, int4 8/8 on CPU and GPU, every thought closed; bf16 (transformers) 8/8 with the int8 file's eight GPU answers word for word.

All 7 signatures of both files are fully delegated on the Mac GPU (6865 / 6865 / 7039 / 7039 / 7039 / 4058 / 4109 ops, no validation error), and the Galaxy S26 logged the same full delegation to its OpenCL delegate before it dropped (Performance). No start token: the engine's prefill count equals transformers' on three prompts, and TII's tokenizer adds no BOS. A three-turn conversation on both files and both backends ("My name is Kenji." / "What is my name?" / "What is 17 times 23?") answered Kenji and 391 with every thought closed; past turns are fed back as their answer without the thought, as TII's template renders turns before the last question, and the engine's KV count before each turn equalled transformers' render of the whole conversation. A prompt-length check (17 prompt lengths between 28 and 60 tokens, a fresh engine each): int8 CPU 17/17 with the engine's prefill count equal to transformers', int8 GPU 16/17, int4 CPU 16/17, int4 GPU 17/17; the misses are a thought that quotes the filler words of the prompt ("well well …") and repeats "well" to the 512-token cap (length 41 on the int4 file in 3 of 3 runs, length 43 once on the int8 file). Six multiplication questions × 3 runs: 18/18 on both files, every thought closed.

Long-output check (four prompts, greedy, up to 2,048 tokens, Mac CPU): the date extraction closes on both files (172 / 186 tokens), the story closes on int8 (1,358) and the ten-tips list on int4 (1,652); the refrigerator prompt reaches the cap inside the thought on every form, the bf16 model included (it keeps adding "We can also mention …" lines). One form-specific failure: the int8 file on the CPU, on that prompt, reasons for about 150 words, quotes its own system message and then emits the six-character text \u200b (the characters backslash, u, 2, 0, 0, b) to the cap, byte-identical in 3 of 3 runs.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, the bundle's template (its default system turn, the model opens <think> itself), greedy, at most 2,048 output tokens, a 4096-token KV budget, a fresh process per question (litert-lm 0.18.0 Python API, Mac GPU). Scoring rule for a reasoner: a reply that closed its thought is scored on the text after </think>; a reply that never closed it (every one of them at the 2,048-token cap) is scored on its whole text and counted as unclosed. The bf16 reference (transformers 5.14.1 on MPS, the same prompts) covers the first 21 questions only: its two cold model loads took 981 and 664 s and the run was stopped there. IFEval was not run; TII's own table reports other benchmarks (AIME, MATH500, GPQA-D …) and no GSM8K.

Configuration	GSM8K	unclosed thoughts
bf16 reference (transformers, MPS), questions 0–20	21 / 21	3 (all right on their whole text)
int8, litert-lm Mac GPU, questions 0–99	96 / 100	4 (3 right on their whole text)
int4, litert-lm Mac GPU, questions 0–99	98 / 100	10 (9 right on their whole text)

On the closed replies alone: int8 93 of 96, int4 89 of 90; counting every unclosed reply as wrong: 93 and 89. On the 21 reference questions: int8 20 (question 3), int4 21. Both files pass our bars (int8 within 5 points of bf16, int4 within 8) for any bf16 score on the hundred. The files declare fp32 activations for the GPU: with the runtime's default fp16 activations the int8 file answered none of the 8 check questions (<|pad|> from the first token to the cap), and an fp16-declared copy was only 1 % faster on the Mac GPU (9.17 vs 9.27 s in the same windows).

Usage

litert-lm run ./Falcon-H1R-7B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./Falcon-H1R-7B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./Falcon-H1R-7B_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./Falcon-H1R-7B_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm benchmark ./Falcon-H1R-7B_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

All four run commands printed a [thought] … [/thought] block of about 110 words and then the dates ("May 3, May 17" from the int8 file on both backends; "May 3, May 17" and "May 3 and May 17." from the int4 file, whose replies use narrow no-break spaces, U+202F, inside the dates), with no control token in the output. --thinking-budget N caps the reasoning (0 disables it, -1 is unlimited; --thinking true|false switches it); the CLI has no output-token cap, a reply ends at a stop token or when the KV budget is full (4096 in these bundles, --max-num-tokens to change it). Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (7.22 GB for int8, 4.24 GB for int4). The bundle sets no sampler; run passes one only when you give --top-k, --top-p, --temperature or --seed. For Android, see android-gpu.md.

The chat template is TII's rewritten for the runtime's template engine: content parts as well as strings, the messages[::-1] pass and .get(…) calls replaced, and every past assistant turn rendered as its answer without the thought, the form TII's template uses for turns before the last question; everything else is verbatim, including TII's default system message ("You are Falcon, a helpful AI assistant created by Technology Innovation Institute (TII) …", replaced when you pass a system message) and the tools block. Its renders equal transformers' render of TII's template byte for byte in 24 of 24 cells (eight conversation shapes × thinking unset / true / false); TII's own template does not run on the 0.18.0 runtime's engine (errors in 66 of 72 cells). The prompt ends with a bare assistant header and the model opens <think> itself; no start token; stop tokens <|end_of_text|> (id 11) and <|im_end|> (id 228) from TII's generation config, plus the string \n<|im_end|>.

Tool calling: TII's format (the function signatures as JSON inside <tools></tools> in the system turn, a call as <tool_call>{…}</tool_call>, results as a user turn of <tool_response>…</tool_response>) is in the template; with a get_weather tool the int8 file wrote a <tool_call> block holding {"name": "get_weather", "arguments":{"location": "Paris"}} as text after its thought, which the app has to parse (the bundle's generic model type has no tool-call parser).

• One Engine per conversation. On litert-lm 0.18.0 the runtime starts a new Conversation on an Engine that already ran another conversation from the earlier conversation's recurrent state: it reuses the prompt tokens the two conversations share and does not reset the state (the open report). The probe (three questions first in a pirate-persona conversation, then in a conversation told to answer in one word, on the same Engine against a fresh Engine; int8 file, CPU, greedy, one run each) gave identical replies on both Engines for this file, where the sibling 3B and Tiny-R files showed the carry-over; the state is there all the same, so give each conversation its own Engine.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two processes per file (the second pass in reverse order), the median of the two; GPU is WebGPU on Metal. Init is the CLI's printed init time of the first iteration; in litert-lm 0.18.0 it sums overlapping init phases and reads about twice the engine build measured from the log's timestamps (on two other files of this series: a 9B Mixed INT4 GPU run printed 89.2 s for a 44.5 s build, a granite-4.1-8b int4 GPU run 7.38 s for 3.66 s; the Python API built the 9B GPU engines in 43–45 s).

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	565.05	35.88	1.840	18.22
int4 GPU	565.78	39.71	1.835	19.00
int8 CPU	135.64	11.59	7.638	54.14
int4 CPU	60.33	12.35	17.066	19.95

A three-signature variant of the int4 file (prefill 1024 / 256 / 1, the same weights) took 2.5 % less time on this workload, inside the spread between runs, with a shorter engine init (11.8 vs 19.2 s) and a slower first token on a 15-token prompt (0.38 vs 0.25 s); the six-signature files ship. Without a weight cache the CPU benchmark process of the int8 file reached 49 GB.

Phones (Android 16, the prebuilt litert_lm_advanced_main of 2026-09-18 from LiteRT-LM's public binary bucket, sha256 adac974b…, verified on the phone before each run):

• Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno 840, MemTotal 11,389,756 kB): one try of the int4 file on the GPU on 2026-10-15. The bundle was pushed and verified (sha256 = the Mac's); the phone was at thermal status 1 (skin 39.9 °C) with 7,555,180 kB available; the benchmark process started at 22:46:19, the OpenCL delegate took every node of all 7 signatures (the same counts as on the Mac), and 69 s after the start, during the warm-up, the phone rebooted (its uptime, read when it came back on 2026-10-17, dates the boot to 22:47:45, 17 s after it left USB; its boot log, written to the system's crash store when the phone was next unlocked, gives the boot reason as reboot and holds no kernel-panic text) and stayed locked until its owner unlocked it. We did not run it again, did not try the int8 file (after a sibling 8B int8 file dropped the same phone during its GPU warm-up on 2026-10-14 we start no 8B-class int8 engine on it), and did not try the CPU (1.5–2.0 × the file by the earlier granite-4.0-micro measurements, above the phone's memory). The Falcon-H1 0.5B–3B int8 files passed this phone's GPU gate with litert-lm 0.16.0 on 2026-08-24, and Falcon-H1-Tiny-R-0.6B has a Galaxy S26 row, so the drop is a property of this 7B-class file on a phone of this memory class; Zamba2-2.7B, another Mamba2 hybrid, rebooted the same phone in its GPU gate on 2026-08-25. On a platform device-lab Galaxy S26 of the same model line (SM-S942U1, MemTotal 11,389,764 kB; 2026-10-16) the same int4 file showed what the drop looks like from inside: the OpenCL delegate took all 7 signatures, then the low-memory killer removed 92 apps in 27 seconds (the platform's own agent last) and the system services died 11 s after the last runtime line, with no output; the GPU engine of this file does not build on a 12 GB Galaxy S26.
• Pixel 8a (Tensor G3, Mali-G715, MemTotal 7,754,700 kB): the int4 file's one GPU try was stopped by our memory rule (a run ends at the first MemAvailable reading under 400,000 kB; here 307,456 kB, 50 s after the start, from 3,888,840 kB before it, after prefill_1024 had been delegated 6865/6865 to the OpenCL delegate; the runtime prepares weights on the CPU side for Mali GPUs; the phone did not restart). The int8 file was not tried: its 7.81 GB is twice the memory the phone had available at the start of the int4 try and almost the phone's total RAM.
• Not run on an NPU: these dynamic-range int8 and blockwise int4 graphs have no HTP path (the NPU path takes static-range or float graphs; the Mamba2 scan's ops come on top).

Conversion notes

• Source and stack: tiiuae/Falcon-H1R-7B at revision a6f74bf1 (every file checked against the Hub's record); litert-torch at commit 115a136 with the Falcon-H1 patch of hf-to-litertlm falcon_h1_work/falcon_h1_litert_torch.patch (the Mamba cache layer, no BOS, the prefill pad guard), torch 2.12.1, transformers 5.14.1, ai-edge-quantizer 0.8.0, litert-converter 0.3.0; packed by the litert-lm 0.18.0 CLI. falcon_h1_work/convert_falcon_h1r_7b.sh in hf-to-litertlm is the one command; run as published on 2026-10-16 (one command, the environment of its header, 74 minutes on a 128 GB Mac, peak 60.1 GB), it rebuilt all four sections of both files byte for byte from the pinned checkpoint, as the conversion run of 2026-10-14 behind the shipped files had (only the packer's uuid and timestamp differ, so compare sections); one earlier run of the same recipe failed on a random symbol-name collision in litert-torch's JAX bridge and passed when run again.
• Export: a float export with 6 prefill signatures (1024, 256, 64, 16, 4, 1) plus decode over a 4096-position KV budget and 176 state buffers (44 KV pairs, 44 conv windows, 44 SSM states), through litert-torch's lightweight conversion with a 1e8-byte lazy-constant threshold; 83 minutes, 61 of them in the runtime constant folding of ten small scan masks (the pass copies all 611 composite decompositions); peak 61 GB on a 128 GiB Mac.
• Quantization after the export: int8 = dynamic int8 on every FULLY_CONNECTED plus an int8 embedding, the Mamba conv and scan left in float; int4 = int4 blockwise-32 (min-max) on the linears, the output projection and the embedding int8. The plain int4 build left 96 zero and 748 subnormal fp16 block scales (dead 32-weight blocks in 7 Mamba-mixer tensors), and XNNPACK prepares a blockwise int4 layer only when every scale is a normal float, so the shipped int4 file floors its block scales at 2^-14 (847 of 212,044,800 scales; the weights are otherwise the plain build's).
• Metadata: the bundle template above; no start token; stop ids 11 and 228; the thought channel; fp32 GPU activations declared (with the runtime's fp16 default the int8 file emits only <|pad|>); max_num_tokens 4096. The tokenizer is the repo's tokenizer.json repackaged with an identity post-processor in place of none, ids unchanged (15 test sentences in English, code, Japanese, Hindi, Thai, Tamil, Arabic and emoji, with the template's control strings and digits, encode to the same ids with and without special tokens).

License and changes

Distributed under the Falcon LLM License (inherited from the base model: TII's README names the "Falcon-LLM License" and links the terms page, which holds the license text; the source repo carries no LICENSE file and neither does this one). Use is subject to the Acceptable Use Policy those terms reference. Falcon-H1R-7B (LiteRT-LM) is built using artificial intelligence technology from the Technology Innovation Institute. Changes from the original work: weights converted from safetensors bf16 to LiteRT flatbuffers and quantized as described above, the int4 block scales floored at 2^-14; tokenizer repackaged with its ids unchanged; chat template rewritten for the runtime's template engine with TII's default system message verbatim; a thought channel, fp32 GPU activations and a 4096-token KV budget declared. This repository is a community conversion and is not affiliated with TII.