litert-community/TIPSv2-B14-LiteRT

TIPSv2-B/14 image encoder for LiteRT

Standalone image encoder from Google TIPSv2-B/14, producing two global embeddings and spatial patch features. Text encoders and task heads are not included. 448 is the native resolution; 224 offers lower compute and memory use.

Files

Profile	File	Size (MiB)
224 FP32	fp32/224/model.tflite	327.0
224 A16W8	a16w8/224/model.tflite	86.3
224 A16W8 Qualcomm NPU	a16w8/224/model_qualcomm.tflite	86.3
448 FP32	fp32/448/model.tflite	329.3
448 A16W8	a16w8/448/model.tflite	88.6
448 A16W8 Qualcomm NPU	a16w8/448/model_qualcomm.tflite	88.6

Use FP32 as the floating baseline and for CPU inference. A16W8 is the compact NPU option, with 8-bit FC weights and 16-bit transformer activations. Public input and outputs remain FLOAT32.

model_qualcomm.tflite files are dedicated NPU compilation inputs; CPU fallback is unsupported. They preserve the portable model's calibrated values using unsigned 16-bit activation storage.

Usage

Signature: main. Input: pixel_values, FLOAT32 NCHW [1,3,S,S], where S is 224 or 448. Convert to RGB, resize directly to the square with PIL bilinear interpolation, and divide by 255. Do not crop or apply ImageNet mean/std normalization.

Output	Shape	Meaning
cls_token	[1,768]	Global token trained with web captions
secondary_cls_token	[1,768]	Global token trained with synthetic captions
patch_features	[1,256,768] or [1,1024,768]	Spatial patch tokens

Outputs are post-LayerNorm and are not L2-normalized.

pip install ai-edge-litert==2.3.0 numpy Pillow
python run_tipsv2_tflite.py --model fp32/224/model.tflite --image image.jpg --output features.npz

This example runs on CPU. All portable files passed CPU checks with public LiteRT 2.3.0. A16W8 can be slow on CPU and is unsupported by the tested GPU runtime. NPU use requires a compatible vendor compiler and runtime; see the compatibility note below.

Evaluation

Results use the same 5,000 ImageNet validation images (five per class, excluding calibration and earlier smoke-test images) and all 1,449 VOC2012 validation images. Zero-shot classification uses cls_token, the matching text encoder, and the prompt a photo of a {class name}.. VOC scores patch features against 21 text classes, including background, with void pixels ignored. These are our fixed zero-shot protocols, not reproductions of Google's published benchmarks.

Resolution	Backend / source profile	ImageNet top-1	VOC mIoU
224	PyTorch FP32	65.78%	24.026%
224	LiteRT CPU / FP32	65.78%	24.026%
224	MT6897 NPU / FP32	65.72%	24.027%
224	MT6897 NPU / A16W8	65.56%	24.023%
224	SM8750 NPU / FP32	65.80%	24.034%
224	SM8750 NPU / A16W8	65.56%	23.998%
448	PyTorch FP32	68.84%	24.567%
448	LiteRT CPU / FP32	68.84%	24.567%
448	MT6897 NPU / FP32	68.88%	24.562%
448	MT6897 NPU / A16W8	68.84%	24.529%
448	SM8750 NPU / FP32	68.96%	24.567%
448	SM8750 NPU / A16W8	68.82%	24.537%

A16W8 preserves primary-token accuracy closely on both tested NPUs. At 224, secondary-token top-1 losses are larger: 0.38–0.88 percentage points versus PyTorch FP32. Detailed results include top-5, secondary-token scores, paired 95% confidence intervals, per-class IoU, and feature-error distributions. Evaluation protocol and image selections document the measurements.

On-device performance

These are representative measurements from retail devices to give a rough sense of on-device runtime behavior, not a direct comparison between hardware platforms. Numbers were collected using LiteRT's benchmark_model from the command line; they are not app-level measurements from an integrated application.

All runs were fully delegated to the target accelerator (NPU or WebGPU). NPU timings are medians across three five-minute trials, with five warmups and a one-minute cooldown per trial; WebGPU timings use benchmark_model --use_gpu=true --gpu_backend=webgpu on macOS (Dawn Metal backend). Values are rounded to integer milliseconds, and invocation excludes preprocessing and output copies. FP32 describes the source model; accelerator compilation may use lower-precision floating arithmetic.

Device model ID	SoC	Backend	Profile	p50 (ms)	p95 (ms)
LJ9	MT6897	NPU	224 FP32	88	91
LJ9	MT6897	NPU	224 A16W8	60	62
LJ9	MT6897	NPU	448 FP32	379	385
LJ9	MT6897	NPU	448 A16W8	307	313
SM-S937U1	SM8750	NPU	224 FP32	15	16
SM-S937U1	SM8750	NPU	224 A16W8	14	15
SM-S937U1	SM8750	NPU	448 FP32	265	267
SM-S937U1	SM8750	NPU	448 A16W8	112	113
Apple M4 Pro 48GB	Apple M4 Pro	WebGPU	224 FP32	11	11
Apple M4 Pro 48GB	Apple M4 Pro	WebGPU	448 FP32	44	44

Compatibility: These NPU measurements use the compiler fixes recorded in evaluation_results.json. Unmodified LiteRT 2.3.0 compiler paths do not correctly handle the portable A16W8 files on the tested NPUs. The dedicated NPU variants passed with the public compiler; their SDK and validation records are in the same file. Device/SDK versions, initialization, memory and per-trial measurements are included in the same results file.

Hardware benchmark disclosure: Results were measured by us on retail devices purchased through normal channels. These results are not affiliated with, sponsored by, endorsed by, or verified by Apple, Samsung, vivo, TECNO, Qualcomm, MediaTek, Google, MLCommons, or Hugging Face. Results depend on device SKU, OS build, thermal state, battery mode, backend, model quantization, runtime version, and benchmark settings.

Quantization

A16W8 uses static post-training quantization with TorchAO 0.18.0 and 500 calibration images per resolution. LayerScale is folded into FP32 projection weights and biases before calibration. All 72 FCs use symmetric per-output-channel INT8 weights, INT16 activations and INT64 biases. Min/max activation ranges are adjusted where needed for 32-bit FC accumulation; attention, GELU, residual and normalization regions keep 16-bit interfaces. The three public outputs are dequantized to FLOAT32.

Quantization metadata records versions and calibration identities. Exact TFLite reproduction also requires export support for the integer regions and LayerNorm composites.

Source and license

Source revision: ed1e4dc6b74bf3935ae099e9d5eb30fa96528454. Model identities and preprocessing are in model_info.json. The original model and implementation are by Google, under Apache 2.0. See the TIPS repository, TIPSv2 paper, LICENSE and NOTICE.