G9v3-3B — LiteRT-LM

ai9stars/G9v3-3B converted to the LiteRT-LM (.litertlm) format for Google's LiteRT-LM runtime. The int8 file carries the GPU-composite graph since 2026-10-18 (Conversion notes); its weights are those of the 2026-10-13 file. The int4 file is the 2026-10-13 file. Tested on litert-lm 0.18.0 on a Mac; the Pixel 8a rows ran the prebuilt Android binary named below on our own phone, the Galaxy S26 rows of the int8 file ran the same binary on Google's Developer Device Platform and the int4 rows on our own phone. On litert-lm 0.17.1 both files answered a plain run with "The capital of France is Paris." (one CPU process each; the re-shipped int8 file re-run on 2026-10-18).

AI9Stars describes G9v3 as a dense 3B causal language model built for local deployment and resource-constrained use: everyday assistant tasks, coding, tool-use workflows and reasoning where a compact model is preferred; English and Chinese. It is a standard LlamaForCausalLM: 2,988,656,640 parameters, 52 layers, hidden size 2048, 16 attention heads with 2 KV heads, a 130,560-token vocabulary with a separate output head, a 131,072-token context (4096 in these files). The model can reason before it answers; in these files thinking is off by default and switched on per request (Usage).

File	Recipe	Size	sha256
G9v3-3B_int8.litertlm	int8 dynamic on linears, float activations; embedding table in its own section; GPU activations declared fp32; GPU-composite graph	3.10 GB	8d9f5e8522e553458ab3fb7f4a461359367934d0741c5e82cd777da55ce69dc5
G9v3-3B_int4.litertlm	GPTQ-calibrated int4 (group 128) carried as blockwise-128 int4 on linears, int8 lm_head + embedding; GPU activations declared fp16	1.83 GB	ea2c773ec678d44ffca43c0fca21a46622fd812fb9b53706f82d966d02e3c468

On GSM8K (n=100, thinking off) bf16 scored 82, the int8 file 85 on the CPU and 84 on the GPU, the int4 file 81 on the GPU (table in Accuracy); the int4 file is 59.1 % of the int8 file's size. Time for a 1024-token prefill plus 256 decoded tokens at context 2048, from the rates in Performance:

Seconds	Mac M4 Max	Galaxy S26	Pixel 8a
int8 GPU	3.81	22.27 (platform session)	no row (Performance)
int4 GPU	4.84	16.41	36.76
int8 CPU	14.18	29.32 (platform session)	84.38
int4 CPU	14.48	35.63	97.55

The two files carry different GPU activation declarations because the accuracy measurements differ: the int8 file loses on the GPU with the runtime's default fp16 activations (GSM8K 69 against 85 on the CPU), so it declares fp32 and runs slower on the GPU; the int4 file keeps its score with fp16 (81, against 78 with fp32) and runs fastest that way. The composite graph shortens the int8 file's GPU time by 21 % on the Mac and by 20 % on the Galaxy S26 against the 2026-10-13 file, and lengthens its CPU time on the platform's Galaxy S26 by 4–10 % (Performance); the same rewrite of the int4 file was measured and not shipped (Conversion notes). The Pixel 8a CPU rows are one hold of 2026-10-18 for both files (Performance). The int8 file has no Pixel 8a GPU row (Performance). Not measured on iOS: the int8 file's decoder weights section is 2,823,168,928 bytes, over 2 GiB.

Correctness

8-question check on the Mac M4 Max (the questions with "Answer briefly." appended, --temperature 0, one litert-lm run process each, thinking off): int8 8/8 on CPU and 7/8 on GPU, int4 7/8 on CPU and GPU; every miss is the rhyme question ("white" for "blue"); the int8 file's answers are those of the 2026-10-13 file on both backends.

All 7 decoder signatures of both files were fully delegated on the Mac GPU, and those of the int4 file on the phone GPUs measured (the embedding section runs on the CPU); the Galaxy S26 GPU session of the int8 file loaded the OpenCL accelerator and initialized all 7 decoder graphs from their serialized programs. On the Pixel 8a, every leg below answered "What is the capital of France? Answer in one sentence." with "The capital of France is Paris."; on our own Galaxy S26 the int4 file did so in every leg, and on the platform's Galaxy S26 the int8 file answered it with the same sentence in one prompt session per backend (two processes each, the same binary as the timing sessions). The runtime prefilled exactly the tokens of the Hugging Face template, with one <s> at the start (three prompts, both files); 39 prompt lengths from 22 to 60 tokens were answered by both files; and a three-turn conversation (word → sentence → question) passed on both files with thinking off on the CPU and the GPU — on the Mac GPU the int8 file's third turn ends with one stray character after the question, as the 2026-10-13 file's does — and on the int8 file with thinking on (2026-10-13). With thinking on, the int4 file's third turn returned no answer: its trace ran to the turn's 1536-token output limit without closing.

Accuracy

GSM8K: test questions 0–99 with a step-by-step instruction and a final #### <number> line, thinking off, greedy, at most 1024 output tokens, a fresh process per question (litert-lm 0.18.0 Python API); bf16 in transformers on MPS with the same prompts. No accuracy was measured with thinking on; IFEval-lite was not run. Scores are comparable only within this table.

Configuration	GSM8K (n=100)
bf16 reference (transformers, MPS)	82
int8, litert-lm CPU	85
int8, litert-lm Mac GPU (fp32 activations, as shipped)	84
int8, Mac GPU with fp16 activations (not shipped)	69
int4, litert-lm Mac GPU (fp16 activations, as shipped)	81
int4, Mac GPU with fp32 activations (not shipped)	78
int4, litert-lm CPU	78

Against bf16, int8 on the CPU missed 1 question bf16 got right and got 4 the other way; int8 on the GPU 4 / 6; int4 on the GPU 4 / 3; int4 on the CPU 7 / 3. The re-shipped int8 file got the same 84 questions right on the GPU as the 2026-10-13 file (0 discordant); the CPU row is the 2026-10-13 file's (the CPU path runs the same arithmetic). The int4 weights are GPTQ-calibrated: a data-free blockwise int4 export of this checkpoint scored 58 on the same questions and is not shipped.

Usage

litert-lm run ./G9v3-3B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./G9v3-3B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./G9v3-3B_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17."
litert-lm run ./G9v3-3B_int4.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --backend gpu --cache no
litert-lm run ./G9v3-3B_int8.litertlm --prompt "Extract the dates from: The meeting moved from May 3 to May 17." --thinking true
litert-lm benchmark ./G9v3-3B_int8.litertlm -p 1024 -d 256 --max-num-tokens 2048 --runs 3 --cache no --backend cpu

The four plain run commands answered with the two dates (int8 CPU: "The dates mentioned in the text are May 3 and May 17."; int8 GPU: "The dates mentioned are May 3 and May 17."; int4: a two-item list), with no chat marker and no thought block. With --thinking true the reply begins with a [thought] block before the answer "The dates mentioned are May 3 and May 17.". Without --backend, run uses the CPU; without --cache, it writes a compiled-weights cache next to the model (2.73 GB for int8, 1.54 GB for int4). The bundle sets no sampler, so run is greedy unless you pass sampling flags, and on the 0.18.0 CLI --temperature and --top-p take effect only together with --top-k (AI9Stars recommends temperature 0.7 / top_p 0.95 with thinking off and 0.9 / 0.95 with thinking on). For Android, see android-gpu.md.

The embedded chat template is built on LiteRT-LM's models/qwen3 template, reading message content as parts; it renders the same text as AI9Stars's template through transformers for 10 of 11 conversation shapes, the difference being a request that does not set the thinking switch: the bundle treats it as thinking off, the vendor template leaves it to the model. Stops are the model's two end tokens (ids 1 and 130073), the start token is <s>. Thinking on puts the trace in the bundle's thought channel (<think> … </think>), printed by the CLI between [thought] and [/thought]: on three questions the trace closed and the answer followed in 3 of 3 runs for both files (2026-10-13).

Tool calling is not declared: with a get_weather tool both files wrote the vendor's XML form as text, <function name="get_weather"><param name="city">Paris</param></function>, for the app to parse; the rendered tool list matched the vendor template's token count. Engine reuse (int8, Python API, CPU, greedy): a second conversation on an Engine that had served another gave the same replies as on a fresh Engine, including with the shared-prefix condition of LiteRT-LM #3165; this model keeps only a KV cache, no recurrent state. The template, tool-call and engine-reuse probes ran on the 2026-10-13 files; the re-shipped int8 file carries the same metadata sections (template included), tokenizer section and embedding section byte for byte.

Performance

Mac Studio M4 Max, litert-lm 0.18.0: the benchmark command above with --backend cpu or gpu (1024-token prefill, 256-token decode, context 2048), one warmup, then the mean of 3 iterations, two passes per file, the median of the two; GPU is WebGPU on Metal. Init is the first iteration's engine init. The int8 rows are the re-shipped file's, measured in alternating order with the 2026-10-13 file in the same windows (that file: GPU 1428.3 / 62.76 tok/s, CPU 252.4 / 24.10); the int4 rows are the 2026-10-13 measurements. The int4 CPU row was measured on the same weights with the fp32 declaration (the CPU path does not read it); the int4 weights declared fp32 on the Mac GPU took 5.29 s for the workload against 4.84 s as shipped.

Mac M4 Max	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init (s)
int8 GPU	1340.1	84.17	0.78	4.1
int4 GPU	1506.9	61.49	0.70	4.4
int8 CPU	263.8	24.86	3.93	21.5
int4 CPU	174.6	29.73	5.93	3.7

Phones, Android 16, the same workload through litert_lm_advanced_main with --benchmark=true --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --max_num_tokens=2048 --num_iterations=3 --report_peak_memory_footprint=true. Galaxy S26, int4 rows: our own phone (SM-S942Q, Qualcomm SM8850, Adreno 840, MemTotal 10.9 GiB), the prebuilt binary (sha256 adac974b…, 40,382,152 bytes) published 2026-09-18 in LiteRT-LM's public binary bucket (binaries/latest/android_arm64/litert_lm/); rates are the mean of iterations 2 and 3 (GPU: LITERT_CL), Init cold = the first process after the compiled-program caches were removed, warm = the benchmark process, peak = the runtime's Peak system ram usage line; before each leg the phone rested until its clocks were at their top and the thermal status was 0 (up to 300 s). Galaxy S26, int8 rows: the platform's device m1q-36 on Google's Developer Device Platform (Android 16; the farm's logs do not name the SoC), one session per row through the litert CLI (litert benchmark <file> --ddp --device m1q-36 --cpu|--gpu --prefill-tokens 1024 --decode-tokens 256 --max-num-tokens 2048 --num-iterations 3, 2026-10-17 and 18), which runs the same binary from the project's bucket: a warm-up process writes the compiled-program caches, then the measured process runs three prefill-and-decode cycles; rates and TTFT are the CLI's summary, the median of cycles 2 and 3, Init cold = the warm-up process's engine init, warm = the measured process's, peak = the measured process's Peak system ram usage line. The farm's phone is allocated per session, so the int8 rows are not same-hold pairs with the int4 rows.

Galaxy S26	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU (platform)	137.0	17.3	7.54	9.7 / 6.3	1119
int4 GPU	402.9	18.46	2.6	7.5 / 5.0	1182
int8 CPU (platform)	134.3	11.8	7.71	5.2 / 0.6	4095
int4 CPU	63.4	13.14	16.3	3.0 / 1.0	2939

On the platform's Galaxy S26 the 2026-10-13 int8 file, run in the same way just before, took 27.89 s for the workload on the GPU (prefill 122.6 tok/s, decode 13.1, TTFT 8.43 s) against the re-shipped file's 22.27 s: the composite graph raised decode by 32 % and prefill by 12 % on this fp32-activation file, and the first token arrives 0.9 s sooner. On the CPU the composite graph costs time: in two session pairs, one in each order, the 2026-10-13 file took 30.14 and 26.89 s against the re-shipped file's 31.58 and 29.32 s (prefill 133.2 / 164.1 against 120.3 / 134.3 tok/s, decode 11.4 / 12.4 against 11.1 / 11.8); the int8 CPU row above is the second pair's session, and on the Mac CPU the two files are within 4 % (the re-shipped file faster). On our own phone (int4 rows) the clocks lowered during every run even after the rest: the GPU clock ceiling was lowered in 18 of 26 samples, the CPU clock in 45 of 47, read every 2 s. The int4 weights declared fp32 took 25.09 s on this GPU against 16.41 s as shipped.

Pixel 8a (Tensor G3, Mali-G715, MemTotal 7.4 GiB), our own phone, the prebuilt binary named above, the same rest before each leg as on our own Galaxy S26; rates are the mean of iterations 2 and 3, Init cold / warm and peak as for our own Galaxy S26.

Pixel 8a	Prefill 1024 (tok/s)	Decode 256 (tok/s)	TTFT (s)	Init cold / warm (s)	Peak (MB)
int8 GPU	—	—	—	—	—
int4 GPU	125.1	8.96	8.3	20.6 / 6.7	1639
int8 CPU	34.5	4.68	29.9	23.9 / 1.5	3766
int4 CPU	19.2	5.79	53.5	10.2 / 2.3	2644

The int8 file has no Pixel 8a GPU row: the harness stops a process when the phone's MemAvailable falls below 400,000 kB, and it did so for the 2026-10-13 int8 file with fp32 activations (lowest 300,700 kB in the first process, 0 in the benchmark) and with fp16 activations (393,560 kB); the re-shipped file has the same weights and nearly the same size (1,163,264 bytes more), and the try was not repeated. The int4 file's GPU run stayed at 2,179,860 kB or more. The two CPU rows are one hold of 2026-10-18 (the int8 file first, then the int4 file; the CPU clock was lowered in 67 of 80 and 88 of 93 samples); with the same rest before each leg, this phone's CPU results move by up to 9 % between holds (the int4 file read 107.36 s for the workload in a hold nine hours earlier), so the 2026-10-13 int8 file was not paired with the re-shipped one on this phone, and the earlier card rows of the 2026-10-13 files (int8 121.63 s, int4 130.37 s) are not comparable with these. The Mali GPU clock is not readable on this phone. Not run on the NPU.

Conversion notes

• Source and stack: ai9stars/G9v3-3B at revision d9553445 (every file checked against the Hub's record); litert-torch 0.9.4, litert-converter 0.4.0, ai-edge-quantizer 0.9.0, transformers 5.14.1 (export); gptqmodel 7.3.5 on torch 2.14.1 (MPS) for the calibration; packed by the litert-lm 0.18.0 CLI. g9v3_work/convert_g9v3_3b.sh in hf-to-litertlm rebuilt the 2026-10-13 files, all five sections byte-identical (only the packer's uuid and timestamp differ); the GPTQ step reproduces bit for bit on the Mac.
• GPU-composite graph (2026-10-18, int8 file): the int8 file was rebuilt from the 2026-10-13 int8 file without re-exporting, by tools/gpu_graph/gpu_graph_retrofit.py --decomp exporter --mask add_bcast in the same repository: the two KV-cache writes per layer become one odml.cache_update composite, each attention matmul an odml.runtime_bmm composite, and a param_tensor input is added to every signature — the shape litert-torch's --apply_gpu_composites exports. All 470 weight buffers, the metadata, tokenizer and embedding sections and the fp32 activation declaration are byte-identical to the 2026-10-13 file; the decoder section grew by 1,163,264 bytes. The attention mask stays a float input added by broadcast. The same rewrite of the int4 file was measured and not shipped: it ran the Mac GPU 21 % and the Galaxy S26 GPU 6.6 % faster but the Pixel 8a GPU 10 % slower, and with the fp16 activations that file declares it scored 79 on GSM8K against 81, 8 questions differing (both files reproduce their own answers run to run), so the int4 file stays the 2026-10-13 one.
• int4: a data-free blockwise int4 export scored 58 on GSM8K and its zero and subnormal block scales did not load on the CPU, so the int4 file is calibrated: gptqmodel (4 bits, group 128, symmetric, 256 chat samples from ultrachat_200k, alpaca and the GSM8K training split at pinned revisions, rendered with thinking off; the script rebuilds the set, which is not redistributed because of alpaca's licence), dequantized, then exported with the GPTQ scales recovered as blockwise-128 int4; 781 block scales below 2^-14 were floored to it.
• Order: export (6 prefill signatures, 1024 down to 1, plus decode, over a 4096-position KV cache; the embedding table as its own section; the bundle template embedded) with the int8 recipe or the GPTQ-recovered int4 → metadata edit (stops ids 1 and 130073, thought channel <think> / </think>, thinking declared, max_num_tokens 4096) with every other section proven unchanged → the GPU activation type declared on each file (int8 fp32, int4 fp16) → for the int8 file, the GPU-composite retrofit.

License and changes

Distributed under the Apache License 2.0 inherited from the base model (the model repo's README metadata and closing line). Changes from the original work: exported to the LiteRT-LM format and quantized to int8 (dynamic) and int4 (GPTQ-calibrated, blockwise recovery) as described above; the chat template replaced by the bundle template (thinking off by default, content as parts), stops and the thought channel set in the bundle metadata, the embedding table externalized, a GPU activation type declared per file, and the int8 file's prefill and decode graphs rewritten into LiteRT-LM's GPU-composite form (2026-10-18, weights unchanged). This conversion was made and is published independently of AI9Stars.