--- license: mit tags: - executorch - xnnpack - pte - on-device - image-text-to-text - image-to-text - object-detection base_model: - microsoft/Florence-2-base --- # Florence-2-base — ExecuTorch (vision + text encoder + text decoder) One set of weights that captions, detects, reads text and grounds phrases, with the task chosen by the prompt you send. 0.23 B parameters. Three `.pte` files, split the way Whisper is on this shelf and for the same reason: the vision tower and the text encoder run once per image, the decoder runs once per generated token. ``` vision : pixel_values (1,3,768,768) -> image_features (1,577,768) encoder: (image_features, input_ids (1,32), mask (1,32)) -> hidden (1,609,768) decoder: (hidden, mask (1,32), decoder_input_ids (1,128)) -> logits (1,128,51328) ``` Every file takes and returns fp32 tensors, so a precision is a file swap, and the three parts can be mixed — an fp16 vision tower pairs with an fp32 encoder. | part | build | file | MB | corr vs fp32 eager | Mac ms* | |---|---|---|---|---|---| | vision | XNNPACK fp32 | `florence2_base_vision_xnnpack_fp32.pte` | 365.7 | 1.000000 | 360.0 | | vision | XNNPACK int8 | `florence2_base_vision_xnnpack_int8.pte` | **112.5** | 0.998901 | 334.7 | | vision | XNNPACK fp16 | `florence2_base_vision_xnnpack_fp16.pte` | 196.2 | 0.999946 | 1034.5 | | vision | Core ML (iOS) | `florence2_base_vision_coreml_all.pte` | 185.1 | 0.998304 | 59.7 | | encoder | XNNPACK fp32 | `florence2_base_encoder_xnnpack_fp32.pte` | 331.0 | 1.000000 | 51.0 | | encoder | XNNPACK int8 | `florence2_base_encoder_xnnpack_int8.pte` | **203.8** | 0.999534 | 44.3 | | encoder | Core ML (iOS) | `florence2_base_encoder_coreml_all.pte` | 165.1 | 0.999993 | 13.9 | | decoder | XNNPACK fp32 | `florence2_base_decoder_xnnpack_fp32.pte` | 545.5 | 1.000000 | 35.7 | | decoder | XNNPACK int8 | `florence2_base_decoder_xnnpack_int8.pte` | **257.9** | 0.999470 | 33.1 | | decoder | Core ML (iOS) | `florence2_base_decoder_coreml_all.pte` | 193.0 | 0.999995 | 4.7 | Three sets: **1242 MB** all-fp32, **574 MB** all-int8, **543 MB** Core ML. The int8 set is the one to reach for on Android — it is smaller than fp32 by more than half and no slower, and it returns the same captions (below). The int8 recipe is dynamic quantisation, which reaches the linear layers and leaves the 51328×768 token embedding table in fp32; that table is 158 MB of the decoder's file, which is why the decoder halves rather than quarters. \*Mac arm64, single process, median of 10 — a reference point for relative cost, not a device number. Torch eager fp32 on the same machine: vision 466.0 ms, encoder 30.8 ms, decoder 23.5 ms. A caption of *n* tokens costs one vision pass, one encoder pass and *n* decoder passes: about 0.95 s for a 15-token caption on the fp32 XNNPACK set, 0.88 s on the int8 set, 0.14 s on Core ML, on this Mac. The decoder graph is a fixed 128-token window, so every step costs the same whether it is the first token or the fiftieth. ## Running it **1. The image.** RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std .229/.224/.225), bicubic resize to 768×768. No crop. Run the vision `.pte`. **2. The prompt.** Florence-2's task tokens are shorthand the processor expands into a sentence before tokenising — the model never sees ``: | task | the sentence that is actually tokenised | |---|---| | `` | What does the image describe? | | `` | Describe in detail what is shown in the image. | | `` | Describe with a paragraph what is shown in the image. | | `` | Locate the objects with category name in the image. | | `` | Locate the objects in the image, with their descriptions. | | `` | Locate the region proposals in the image. | | `` | What is the text in the image? | | `` | What is the text in the image, with regions? | | `` | Locate the phrases in the caption: {your caption} | | `` | Locate {your phrase} in the image. | Tokenise that sentence as ` sentence ` with the repo's tokenizer, right-pad to 32 with the pad id (1), and build an `attention_mask` that is 1 on the real tokens and 0 on the padding. Run the encoder `.pte` with `(image_features, input_ids, mask)`. The 577 image tokens are handled inside the graphs. The prompt sequence the original model sees is `×577 + prompt `, and because the image tokens are a contiguous prefix, the encoder here concatenates the vision features in front of the text embeddings instead of scattering them into placeholder positions. Same arithmetic, no data-dependent mask, and nothing for the caller to line up. **3. Greedy decoding.** Fill a `(1,128)` int64 window with the pad id, write the decoder start token (2) at position 0, then for step `t = 0, 1, 2, ...`: ``` logits = decoder(hidden, mask, window) # mask is the same one the encoder took ban every token that would repeat a 3-gram already in the output # <- see below next = argmax(logits[0, t]) if next == 2: stop # window[0, t + 1] = next ``` Detokenise the collected ids with the repo's tokenizer. There is no KV cache: the decoder is a plain static graph over the window, which is what makes it a single `.pte` with no state to carry between calls. **Implement the 3-gram ban.** `no_repeat_ngram_size: 3` is in the model's own `generation_config.json`, and it is not decoration: Florence-2-**large**'s decoder returns `` as its argmax three times in a row and only that ban moves it on to the caption — without it the loop emits `` forever and returns an empty string. This base model happens to move on by itself on every image tested here, so a plain argmax loop gives the same five captions. That is luck, not a contract. **4. Reading a detection answer.** Grounded tasks answer with `` tokens, `N` in 0..999. Four in a row are a box, and each coordinate is `(N + 0.5) × side / 1000` in the original image's pixels — `side` being the image's width for x and its height for y, not 768. That is the same arithmetic the processor's own parser does. ## Verification The three wrappers reproduce `Florence2ForConditionalGeneration` **exactly**: composition `max_abs_diff` 0.000e+00 against the full model's logits on the same image and prompt. End to end through the three `.pte` files, greedy `` on five photographs against the same decoding in eager PyTorch: **5/5 captions identical**, character for character. Correlation is measured per part in the table above; the caption test is what says the split, the prompt layout and the decode agree. ``` A black and white photo of a person playing a piano. A wooden walkway leading to the ocean on a sunny day. A group of dead trees in a forest under a cloudy sky. A long empty road in the middle of a forest. A couple of wooden benches sitting on top of a field of leaves. ``` The **int8 set answers with the same five sentences**, character for character, with all three parts quantised at once — which is how they would be used together. ```bash python convert/check_florence2.py fp32 # or int8 ``` ## Not shipped, and why **fp16 for the encoder and the decoder does not export.** BART clamps its activations when, and only when, they are half precision: ```python if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all(): ``` In fp32 that line short-circuits and never reaches the graph. Halve the model and it becomes a question about values `torch.export` cannot answer, and export stops with `GuardOnDataDependentSymNode`. The clamp could be shimmed away, but then the shipped graph would lack the overflow guard the original has, which is not a trade worth making silently. Nothing is lost by it: int8 is both smaller and faster than fp16 here, and the Core ML builds compute in fp16 internally anyway, holding at correlation 0.999993 and 0.999995. **The fp16 vision tower ships and is probably not what you want.** It passes both gates — 196.2 MB at correlation 0.999946 — but the int8 tower is 112.5 MB and three times faster (334.7 ms against 1034.5 ms), because XNNPACK has no fp16 kernels here and inserts casts instead. It is in the repo for anyone who needs more fidelity than int8 without carrying the fp32 file. ## Conversion notes Converted from [florence-community/Florence-2-base](https://huggingface.co/florence-community/Florence-2-base), the transformers-format mirror of [microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) — same MIT weights. The original repo predates the in-tree implementation and its weight names do not match it: loading it into `Florence2ForConditionalGeneration` prints a load report where every key is unexpected and hands back a randomly initialised model without raising. The checkpoint declares `torch_dtype: float16` and transformers honours it, so `from_pretrained` must be given `dtype=torch.float32` explicitly. Without it the "fp32" build serialises fp16 weights — 196 MB for a 90 M parameter tower — and the lowered vision graph returns NaN. - **Source**: microsoft/Florence-2-base (via florence-community mirror) - **License**: MIT torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))