Hugging Face's logo

litert-community
/
vit-base-patch16-224

Image Classification
LiteRT
LiteRT
vit
vision
Model card Files
xet
 Community
vit-base-patch16-224 / README.md
snnn001's picture
snnn001
Update README.md
f10f70a verified
preview code
|
Raw History Blame Contribute Delete
11.7 kB
---
library_name: litert
pipeline_tag: image-classification
license: apache-2.0
base_model: google/vit-base-patch16-224
tags:
- vision
- image-classification
- vit
- tflite
datasets:
- imagenet-1k
- imagenet-21k
---
# Google ViT Base Patch16 224 — LiteRT
LiteRT/TFLite variants of
[`google/vit-base-patch16-224`](https://huggingface.co/google/vit-base-patch16-224),
pinned to source revision `3f49326eb077187dfe1c2a2bb15fbd74e6ab91e3`.
The package provides FP32, A16W8 and mixed A8W8 models for image classification.
All three have FP32 input and output and require external image preprocessing.
The `.tflite` files are portable source artifacts for target-specific NPU
compilation; they do not contain device-specific compiled NPU programs.
## Model files
| File | Description |
| --- | --- |
| `model_fp32.tflite` | Floating-point reference with FP32 weights. |
| `model_a16w8.tflite` | INT16 transformer activations and INT8 fully connected weights; the primary quantized option for compatible NPUs. |
| `model_a8w8.tflite` | Experimental mixed precision: INT8 fully connected layers with INT16 attention, residual and LayerNorm boundaries. |
| `config.json` | Original source configuration and class mapping. |
| `preprocessor_config.json` | Source image preprocessing configuration. |
| `labels.json` | Class index to label mapping. |
| `model_metadata.json` | Source identity, tensor interfaces, quantization scope and model hashes. |
| `run_image.py` | CPU image-classification example. |
| `requirements.txt` | Runtime and preprocessing package versions. |
## Choosing a variant
- **FP32** provides the floating-point reference and is the example's default.
- **A16W8** offers the best observed accuracy/speed balance on the tested NPUs.
Its fully connected layers use INT16 inputs and outputs, INT8 weights and
INT64 biases. CPU fallback can be slow, and the tested LiteRT GPU IR importer
does not support its INT16 quantization.
- **Mixed A8W8** is retained as a reference that may suit devices limited to
32-bit integer accumulation in their fully connected kernels. It has a
substantial accuracy tradeoff and retains INT16 regions, so backend support
must be verified.
Extra notes: Apple ANE and QC NPUs support FP16 natively and the performances are
usually very good. Deloying a FP16 model is much easier since it does not require
calibration steps. Note that the A16W8 and A8W8 assets provided here were calibrated
from a particular dataset and should not used in production directly. A FP16 model
file is not provided here is because it is easy to be generated from the fp32 asset.
It is actually a top choice.
## TorchAO quantization recipe
Both variants use post-training static quantization, without QAT or fine-tuning.
The run used PyTorch `2.14.1+cu130`, TorchAO `0.18.0+cu130` and Transformers
`5.21.0`. To reproduce the quantization approach:
- Load the pinned source checkpoint in FP32 evaluation mode. Calibrate with
500 ImageNet-1k validation images, batch size one, using the included PIL
preprocessing configuration. Keep calibration separate from evaluation.
- Observe inputs to all 73 `torch.nn.Linear` layers, including the classifier,
with `AffineQuantizedMinMaxObserver`: asymmetric INT8 `[-128, 127]`,
`PerTensor()` granularity, FP32 scales and INT32 zero points.
- Quantize the original FP32 weights with symmetric `PerRow()` granularity
(one scale per output channel), zero point zero and the explicit narrow
range `[-127, 127]`. Use TorchAO's `choose_qparams_affine` and
`quantize_affine` with these bounds; this weight-range customization is
additional to the activation configuration below.
After calibrating each layer's input observer, its static configuration is:
```python
from torchao.quantization import Int8StaticActivationInt8WeightConfig
from torchao.quantization.granularity import PerRow, PerTensor
from torchao.quantization.quant_primitives import MappingType
scale, zero_point = observer.calculate_qparams()
config = Int8StaticActivationInt8WeightConfig(
act_quant_scale=scale,
act_quant_zero_point=zero_point,
granularity=[PerTensor(), PerRow()],
act_mapping_type=MappingType.ASYMMETRIC,
set_inductor_config=False,
)
```
The final TFLite export also needs calibrated linear-output and transformer
activation ranges, collected using the same 500 images. **A16W8** uses symmetric
INT16 activations with zero point zero and INT64 fully connected biases.
**Mixed A8W8** keeps asymmetric INT8 fully connected inputs/outputs and INT32
biases, with INT16 attention, residual and LayerNorm boundaries. Both retain
the same INT8 weight codes and scales; activation scales are per-tensor.
The transformer graphs keep activations quantized across attention, residual,
reshape, transpose, softmax and GELU operations. LayerNorm uses composite
operators with quantized interfaces and floating-point decomposition subgraphs
for fallback execution. The patch embedding and public input/output remain
floating-point, so these models do not use integer arithmetic throughout.
A16W8 retains input-scale coarsening for fully connected layers that needed it
under a 32-bit accumulator profile, but omits explicit accumulator input
clipping. Validate accumulator behavior and accuracy on another backend.
TorchAO provides the linear quantization stage; recreating the complete TFLite
graph requires export support for the activation encodings above. The exact
calibration image list is not distributed, so this recipe does not guarantee
identical scales or files. See the
[TorchAO static quantization guide](https://docs.pytorch.org/ao/stable/eager_tutorials/static_quantization.html)
for the observer/calibration workflow.
## Accuracy evaluation
The results below use the same 5,000-image subset of ImageNet-1k validation:
five images per class, selected with seed `20261031`. The selection excludes
the 500 calibration images and 100 earlier diagnostic images. All runs use
the same input tensors, produced with the included Hugging Face PIL
preprocessing configuration.
The original PyTorch FP32 checkpoint scored **79.68% top-1** and **95.40%
top-5**, with TF32 and autocast disabled. The TFLite results below were
measured with AOT-compiled models fully delegated to each device's NPU; all
outputs were finite. FP32 denotes the source model, with FP16 execution
permitted by the NPU compiler.
| Model file | `SM-S937U1` top-1 (%) | `SM-S937U1` top-5 (%) | `LJ9` top-1 (%) | `LJ9` top-5 (%) |
| --- | ---: | ---: | ---: | ---: |
| `model_fp32.tflite` | 79.92 | 95.40 | 79.68 | 95.38 |
| `model_a16w8.tflite` | 79.72 | 95.32 | 79.66 | 95.32 |
| `model_a8w8.tflite` | 75.14 | 92.50 | 75.10 | 92.76 |
A16W8 stays close to the PyTorch baseline on this subset. Mixed A8W8 reduces
top-1 accuracy by 4.54 percentage points on `SM-S937U1` and 4.58 points on
`LJ9` relative to that baseline.
This subset was reused while comparing quantization recipes. It is not an
independent final test set or a full 50,000-image ImageNet validation run.
Small score differences should not be interpreted as general improvements
over the original checkpoint. Raw outputs, input images and diagnostic
reports are not included in this package.
The A16W8 scores were collected before unused tensor records were removed.
That cleanup preserved all live computation, tensor encodings and weights,
and left the Qualcomm compiled file identical. A fresh 100-image MTK check
reproduced the earlier logits exactly; the full 5,000-image MTK run was not
repeated after this metadata cleanup.
## NPU performance
These are representative measurements from retail devices to give a rough
sense of on-device runtime behavior, not a direct comparison between hardware
platforms. All latency numbers were collected with LiteRT's compiled-model
`benchmark_model` launched through `adb shell` on the connected device; they
are not app-level measurements from an integrated Android application.
Warm batch-one inference latency, rounded to the nearest millisecond. Each
value is the median of three process means, with five warmup invocations and
30 measured invocations per process. All configurations are AOT compiled and
fully delegated to the NPU, with no residual CPU operations.
| Model file | `SM-S937U1`(ms) | `LJ9`(ms) |
| --- | ---: | ---: |
| `model_fp32.tflite` | 6 | 129 |
| `model_a16w8.tflite` | 4 | 71 |
| `model_a8w8.tflite` | 10 | 52 |
FP32 identifies the source model. Qualcomm compiles it to FP16-based NPU
execution, and MediaTek enables FP32-to-FP16 relaxation. The floating-source
timings therefore do not imply FP32 arithmetic throughout the NPU program.
Measurements use LiteRT's compiled-model benchmark with persistent buffers
and deterministic random input. They exclude model initialization, image
preprocessing, file I/O and explicit host input/output buffer transfers.
Some variants were measured in separate sessions using the same benchmark
and dispatch runtime for each device.
The `SM-S937U1` ran Android 17 with Qualcomm QNN/QAIRT 2.50 in burst performance
mode. The LJ9 ran Android 15 with the MediaTek Neuron runtime in sustained
speed mode. Power settings were not normalized, and energy consumption was
not measured. These results apply to the listed device/compiler/runtime
combinations and do not characterize JIT performance.
**Hardware benchmark disclosure:** Results were measured by us on retail
devices purchased through normal channels. These results are not affiliated
with, sponsored by, endorsed by, or verified by Samsung, TECNO, vivo, Qualcomm,
MediaTek, Google, MLCommons, or Hugging Face. Results depend on device SKU,
OS build, thermal state, battery mode, backend, model quantization, runtime
version, and benchmark settings.
## Input and output
- Input: FP32 image tensor, NCHW `[1, 3, 224, 224]`.
- Output: FP32 classifier logits, `[1, 1000]`, ordered by `labels.json`.
- Preprocessing: convert to RGB, resize to 224 × 224 with the included
Transformers PIL processor, rescale by 1/255, then normalize each channel
with mean and standard deviation 0.5. The example selects `backend="pil"`.
The models have static batch and spatial dimensions. All 25 LayerNorm
operations use TFLite `STABLEHLO_COMPOSITE` boundaries named `odml.group_norm`,
with FP32 decomposition subgraphs.
## Run a local image
Download the listed files into one directory and run the following from that
directory. The CPU example and dependencies were checked with Python 3.13 on
Linux:
```bash
uv venv --python 3.13
source .venv/bin/activate
uv pip install -r requirements.txt
python run_image.py --image image.jpg
python run_image.py --image image.jpg --model model_a8w8.tflite
python run_image.py --image image.jpg --model model_a16w8.tflite
```
The helper loads the included processor configuration and TFLite model, then
prints the top classes with their raw logits. A16W8 CPU execution may be slow;
the example is an interface check for that variant.
For NPU deployment, compile the selected `.tflite` for the target SoC using the
matching LiteRT compiler plugin, then load the compiled model with the matching
device dispatch runtime and select the NPU accelerator. Verify full delegation
on that device. The CPU example above does not select the NPU.
## Source and attribution
This artifact derives from the pinned Google checkpoint, whose model card
declares the Apache-2.0 license. Refer to that card for training details and
intended use. The architecture was introduced in
[An Image Is Worth 16x16 Words](https://arxiv.org/abs/2010.11929).
The accompanying `config.json` describes the original Transformers model;
inference here uses the `.tflite` file.