Nanbeige4.1-3B — LiteRT-LM (blockwise int4)

Nanbeige/Nanbeige4.1-3B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime (the engine behind the official litert-community/* models).

Nanbeige4.1-3B is a fresh (Dec 2025) phone-size reasoning model on a plain dense Llama architecture (Apache-2.0), reported to be competitive with much larger models. It works the problem inside a <think>…</think> block before giving the final answer.

File	model.litertlm (~2.2 GB; embedding externalized so every section is <2 GiB → loads on iOS)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8
Compute	integer
Context (KV cache)	4096
Base model	Nanbeige/Nanbeige4.1-3B (Apache-2.0)
Decode speed	~89 tok/s (Mac M4 Max, Metal GPU, greedy)

Usage

# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
  --model_path model.litertlm \
  --backend gpu \
  --input_prompt "A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts total?"

The .litertlm bundle carries the tokenizer and the prompt template (ChatML — <|im_start|>role\n … <|im_end|>), so no separate tokenizer files are needed. This is a reasoning model: it emits a <think>…</think> chain then the final answer (best evaluated with a generous token budget), and stops cleanly at <|im_end|>.

Run on Android

Install a recent Google AI Edge Gallery (1.0.16+ can import .litertlm directly from Hugging Face), download model.litertlm (or import this repo in-app), pick the GPU backend (CPU also works), and chat. Give it a high max-tokens — it's a reasoning model with long chains of thought.

Quality — GSM8K

Measured on GSM8K (n=50, greedy, 0-shot chain-of-thought, max-tokens 2048 — a reasoning model needs the budget to finish; scoring it at 512 tokens falsely penalises it):

Configuration	GSM8K
This model — LiteRT int4 (block32 + OCTAV)	84.0%

84% is a strong on-device GSM8K for a 3B, non-degenerate; the model also passes the local quality gate 8/8 with a clean stop at <|im_end|>. Blockwise-32 + OCTAV optimal-clipping (data-free) preserves the accuracy versus a naive min-max int4.

Conversion

Converted with litert-torch using a blockwise int4 recipe (INT4 weights, block size 32, symmetric, OCTAV optimal-clipping) with the embedding at INT8, KV cache 4096, and a ChatML prompt template. Nanbeige4.1 is a standard dense LlamaForCausalLM, so it rides the existing converter and runtime with no custom graph code.

externalize_embedder=True (required for iPhone). The large 166k-token vocab makes the weights a >2 GiB single TFLite section, which exceeds the ~2 GiB single-section mmap limit on iOS. Externalizing the embedding drops the main section under 2 GiB so the model loads on iPhone (Metal GPU) as well as Android/desktop. Same weights, so GSM8K is unchanged.

Added-tokens tokenizer fix. Nanbeige's 10 special tokens (<|im_start|>, <|im_end|>, <think>, </think>, <tool_call>, …) live at vocab ids 166100–166109, above the base SentencePiece vocab (166100). The base SP conversion drops them, so the reasoning model would generate <think> (id 166103) and the runtime would crash with "Token id out of range." The converted tokenizer here appends those added tokens as USER_DEFINED SentencePiece pieces at their exact ids (padded to the model vocab), so <think> and friends decode correctly.

License

Apache-2.0, inherited from the base model Nanbeige/Nanbeige4.1-3B.