Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,178 @@
+---
+license: mit
+tags:
+- executorch
+- xnnpack
+- pte
+- on-device
+- image-text-to-text
+- image-to-text
+- object-detection
+base_model:
+- microsoft/Florence-2-base
+---
+# Florence-2-base — ExecuTorch (vision + text encoder + text decoder)
+
+One set of weights that captions, detects, reads text and grounds phrases, with the
+task chosen by the prompt you send. 0.23 B parameters. Three `.pte` files, split the
+way Whisper is on this shelf and for the same reason: the vision tower and the text
+encoder run once per image, the decoder runs once per generated token.
+
+```
+vision : pixel_values (1,3,768,768)                        -> image_features (1,577,768)
+encoder: (image_features, input_ids (1,32), mask (1,32))   -> hidden (1,609,768)
+decoder: (hidden, mask (1,32), decoder_input_ids (1,128))  -> logits (1,128,51328)
+```
+
+Every file takes and returns fp32 tensors, so a precision is a file swap, and the
+three parts can be mixed — an fp16 vision tower pairs with an fp32 encoder.
+
+| part | build | file | MB | corr vs fp32 eager | Mac ms* |
+|---|---|---|---|---|---|
+| vision | XNNPACK fp32 | `florence2_base_vision_xnnpack_fp32.pte` | 365.7 | 1.000000 | 360.0 |
+| vision | XNNPACK int8 | `florence2_base_vision_xnnpack_int8.pte` | **112.5** | 0.998901 | 334.7 |
+| vision | XNNPACK fp16 | `florence2_base_vision_xnnpack_fp16.pte` | 196.2 | 0.999946 | 1034.5 |
+| vision | Core ML (iOS) | `florence2_base_vision_coreml_all.pte` | 185.1 | 0.998304 | 59.7 |
+| encoder | XNNPACK fp32 | `florence2_base_encoder_xnnpack_fp32.pte` | 331.0 | 1.000000 | 51.0 |
+| encoder | XNNPACK int8 | `florence2_base_encoder_xnnpack_int8.pte` | **203.8** | 0.999534 | 44.3 |
+| encoder | Core ML (iOS) | `florence2_base_encoder_coreml_all.pte` | 165.1 | 0.999993 | 13.9 |
+| decoder | XNNPACK fp32 | `florence2_base_decoder_xnnpack_fp32.pte` | 545.5 | 1.000000 | 35.7 |
+| decoder | XNNPACK int8 | `florence2_base_decoder_xnnpack_int8.pte` | **257.9** | 0.999470 | 33.1 |
+| decoder | Core ML (iOS) | `florence2_base_decoder_coreml_all.pte` | 193.0 | 0.999995 | 4.7 |
+
+Three sets: **1242 MB** all-fp32, **574 MB** all-int8, **543 MB** Core ML. The int8
+set is the one to reach for on Android — it is smaller than fp32 by more than half
+and no slower, and it returns the same captions (below). The int8 recipe is dynamic
+quantisation, which reaches the linear layers and leaves the 51328×768 token
+embedding table in fp32; that table is 158 MB of the decoder's file, which is why the
+decoder halves rather than quarters.
+
+\*Mac arm64, single process, median of 10 — a reference point for relative cost, not
+a device number. Torch eager fp32 on the same machine: vision 466.0 ms, encoder
+30.8 ms, decoder 23.5 ms.
+
+A caption of *n* tokens costs one vision pass, one encoder pass and *n* decoder
+passes: about 0.95 s for a 15-token caption on the fp32 XNNPACK set, 0.88 s on the
+int8 set, 0.14 s on Core ML, on this Mac. The decoder graph is a fixed 128-token
+window, so every step costs the same whether it is the first token or the fiftieth.
+
+## Running it
+
+**1. The image.** RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std
+.229/.224/.225), bicubic resize to 768×768. No crop. Run the vision `.pte`.
+
+**2. The prompt.** Florence-2's task tokens are shorthand the processor expands into
+a sentence before tokenising — the model never sees `<CAPTION>`:
+
+| task | the sentence that is actually tokenised |
+|---|---|
+| `<CAPTION>` | What does the image describe? |
+| `<DETAILED_CAPTION>` | Describe in detail what is shown in the image. |
+| `<MORE_DETAILED_CAPTION>` | Describe with a paragraph what is shown in the image. |
+| `<OD>` | Locate the objects with category name in the image. |
+| `<DENSE_REGION_CAPTION>` | Locate the objects in the image, with their descriptions. |
+| `<REGION_PROPOSAL>` | Locate the region proposals in the image. |
+| `<OCR>` | What is the text in the image? |
+| `<OCR_WITH_REGION>` | What is the text in the image, with regions? |
+| `<CAPTION_TO_PHRASE_GROUNDING>` | Locate the phrases in the caption: {your caption} |
+| `<OPEN_VOCABULARY_DETECTION>` | Locate {your phrase} in the image. |
+
+Tokenise that sentence as `<s> sentence </s>` with the repo's tokenizer, right-pad to
+32 with the pad id (1), and build an `attention_mask` that is 1 on the real tokens and
+0 on the padding. Run the encoder `.pte` with `(image_features, input_ids, mask)`.
+
+The 577 image tokens are handled inside the graphs. The prompt sequence the original
+model sees is `<image>×577 + <s> prompt </s>`, and because the image tokens are a
+contiguous prefix, the encoder here concatenates the vision features in front of the
+text embeddings instead of scattering them into placeholder positions. Same
+arithmetic, no data-dependent mask, and nothing for the caller to line up.
+
+**3. Greedy decoding.** Fill a `(1,128)` int64 window with the pad id, write the
+decoder start token (2) at position 0, then for step `t = 0, 1, 2, ...`:
+
+```
+logits = decoder(hidden, mask, window)      # mask is the same one the encoder took
+next   = argmax(logits[0, t])
+if next == 2: stop                          # </s>
+window[0, t + 1] = next
+```
+
+Detokenise the collected ids with the repo's tokenizer. There is no KV cache: the
+decoder is a plain static graph over the window, which is what makes it a single
+`.pte` with no state to carry between calls.
+
+**4. Reading a detection answer.** Grounded tasks answer with `<loc_N>` tokens,
+`N` in 0..999. Four in a row are a box, and each coordinate is `(N + 0.5) × side /
+1000` in the original image's pixels — `side` being the image's width for x and its
+height for y, not 768. That is the same arithmetic the processor's own parser does.
+
+## Verification
+
+The three wrappers reproduce `Florence2ForConditionalGeneration` **exactly**:
+composition `max_abs_diff` 0.000e+00 against the full model's logits on the same
+image and prompt.
+
+End to end through the three `.pte` files, greedy `<CAPTION>` on five photographs
+against the same decoding in eager PyTorch: **5/5 captions identical**, character for
+character. Correlation is measured per part in the table above; the caption test is
+what says the split, the prompt layout and the decode agree.
+
+```
+A black and white photo of a person playing a piano.
+A wooden walkway leading to the ocean on a sunny day.
+A group of dead trees in a forest under a cloudy sky.
+A long empty road in the middle of a forest.
+A couple of wooden benches sitting on top of a field of leaves.
+```
+
+The **int8 set answers with the same five sentences**, character for character, with
+all three parts quantised at once — which is how they would be used together.
+
+```bash
+python convert/check_florence2.py fp32     # or int8
+```
+
+## Not shipped, and why
+
+**fp16 for the encoder and the decoder does not export.** BART clamps its activations
+when, and only when, they are half precision:
+
+```python
+if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():
+```
+
+In fp32 that line short-circuits and never reaches the graph. Halve the model and it
+becomes a question about values `torch.export` cannot answer, and export stops with
+`GuardOnDataDependentSymNode`. The clamp could be shimmed away, but then the shipped
+graph would lack the overflow guard the original has, which is not a trade worth
+making silently. Nothing is lost by it: int8 is both smaller and faster than fp16
+here, and the Core ML builds compute in fp16 internally anyway, holding at
+correlation 0.999993 and 0.999995.
+
+**The fp16 vision tower ships and is probably not what you want.** It passes both
+gates — 196.2 MB at correlation 0.999946 — but the int8 tower is 112.5 MB and three
+times faster (334.7 ms against 1034.5 ms), because XNNPACK has no fp16 kernels here
+and inserts casts instead. It is in the repo for anyone who needs more fidelity than
+int8 without carrying the fp32 file.
+
+## Conversion notes
+
+Converted from
+[florence-community/Florence-2-base](https://huggingface.co/florence-community/Florence-2-base),
+the transformers-format mirror of
+[microsoft/Florence-2-base](https://huggingface.co/microsoft/Florence-2-base) — same
+MIT weights. The original repo predates the in-tree implementation and its weight
+names do not match it: loading it into `Florence2ForConditionalGeneration` prints a
+load report where every key is unexpected and hands back a randomly initialised model
+without raising.
+
+The checkpoint declares `torch_dtype: float16` and transformers honours it, so
+`from_pretrained` must be given `dtype=torch.float32` explicitly. Without it the
+"fp32" build serialises fp16 weights — 196 MB for a 90 M parameter tower — and the
+lowered vision graph returns NaN.
+
+- **Source**: microsoft/Florence-2-base (via florence-community mirror)
+- **License**: MIT
+
+torch.export -> to_edge_transform_and_lower(partitioner) -> .pte
+(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))