| --- |
| license: apache-2.0 |
| tags: |
| - executorch |
| - xnnpack |
| - pte |
| - on-device |
| - image-classification |
| base_model: |
| - google/vit-base-patch16-224 |
| --- |
| # ViT-Base/16 ImageNet-1k — ExecuTorch |
|
|
| The original Vision Transformer, fine-tuned on ImageNet-1k: an image in, 1000 logits |
| out. The reference classifier most tutorials reach for, as four `.pte` files. |
|
|
| - **Source**: google/vit-base-patch16-224 — 86M parameters, ViT-Base/16, 12 layers, |
| hidden 768, 224x224 |
| - **License**: apache-2.0 |
| - **Input**: `pixel_values` `[1, 3, 224, 224]` fp32 — RGB resized to 224x224, scaled to |
| [0,1], then normalised with mean=(0.5, 0.5, 0.5) std=(0.5, 0.5, 0.5). **Not** the |
| ImageNet statistics; read off this model's own `preprocessor_config.json`. |
| - **Output**: logits `[1, 1000]`, in the order of the repo's `config.json` `id2label`. |
| Softmax is left to the caller. |
|
|
| ## Variants |
|
|
| | build | file | size (MB) | Mac median (ms)* | labels kept | margin shift at the boundary (logits) | |
| |---|---|---|---|---|---| |
| | fp32 | `imgcls_vit_base_in1k_xnnpack_fp32.pte` | 346.4 | 34.9 | 24 of 24 | 0.0000 | |
| | fp16 | `imgcls_vit_base_in1k_xnnpack_fp16.pte` | 174.6 | 71.0 | 24 of 24 | 0.0117 | |
| | Core ML (fp16, iOS) | `imgcls_vit_base_in1k_coreml_all.pte` | 173.6 | **4.0** | 24 of 24 | 0.0547 | |
|
|
| \*Mac arm64, single process, median of 10. PyTorch eager fp32 on the same machine is |
| **27.5 ms**, so the Core ML build is **6.9x eager**, 100% delegated in one subgraph. |
| fp16 is slower than fp32 — XNNPACK emulates it — and is listed only because it halves |
| the file. |
| |
| ## The int8 build is not published, and correlation would have shipped it |
| |
| Dynamic int8 converts, comes out at **89.7 MB**, runs at 31.7 ms, and reads |
| **correlation 0.999814** against fp32 eager. It keeps the fp32 label on all 24 |
| photographs. Every number a conversion normally reports says ship it. |
| |
| The number that decides says otherwise: |
| |
| ``` |
| worst margin shift at the decision boundary 0.1856 logits |
| closest photograph's distance to a decision 0.0900 logits |
| ``` |
| |
| The build's error is **twice the distance that separates the closest of these |
| photographs from being called something else**. It cannot be trusted to agree with fp32 |
| on an image the model is not already sure about. |
| |
| **What changed is the number of classes, not the quantisation.** The same recipe on |
| this shelf's two-class ViT head shifts the margin by 1.4795 logits and passes |
| comfortably — because with two classes the closest photograph sits **7.88 logits** from |
| the decision. Put a thousand classes in competition and the runner-up is usually a near |
| tie: median distance **1.09 logits**, minimum **0.09**. The error budget collapses by |
| roughly two orders of magnitude while the error itself does not. |
| |
| So the honest form of the rule is not *int8 is fine on ViTs*. It is: **an error is |
| small or large only relative to the decision it has to survive, and a 1000-way decision |
| has almost no room.** |
| |
| ## How these were checked |
| |
| Label agreement alone would not have separated any of these builds — all four keep |
| 24 of 24. So each photograph is also walked along the gradient of its class margin |
| (the winner's lead over its closest rival) until the margin crosses zero, bisected onto |
| it, and the builds compared *there*. All 6 boundary inputs were reached for every build; |
| a walk that never leaves the class is reported as not measured rather than as a zero. |
| |
| The boundary inputs are perturbed photographs, not natural ones. They exercise the |
| arithmetic where it decides something; they are not a claim about accuracy on real data. |
| |
| ## What is not tested here |
| |
| The 24 photographs are this shelf's calibration set — street scenes, animals, people, |
| general views — not an ImageNet validation split. What is verified is **fidelity to the |
| fp32 model**, not the model's own top-1 accuracy. These files reproduce whatever the |
| upstream checkpoint does. |
| |
| Worth knowing before you build on it: on these images the fp32 model's own confidence |
| runs from **0.022 to 0.501** (median). A 1000-class classifier is rarely sure, and a |
| caller that thresholds on probability should read that row before picking a threshold. |
| |
| ## Conversion |
| |
| ```bash |
| python convert/export_imgcls.py vit_base_in1k |
| python convert/check_imgcls.py vit_base_in1k int8 |
| ``` |
| |
| (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models)) |
| |
