mlboydaisuke/Florence-2-large-ExecuTorch

Florence-2-large — ExecuTorch (vision + text encoder + text decoder)

One set of weights that captions, detects, reads text and grounds phrases, with the task chosen by the prompt you send. 0.77 B parameters — the same interface as Florence-2-base at roughly three times the size, and it notices more: on the same photograph base says "A black and white photo of a person playing a piano" where large reads the brand off the instrument.

Three .pte files, split the way Whisper is on this shelf and for the same reason: the vision tower and the text encoder run once per image, the decoder runs once per generated token.

vision : pixel_values (1,3,768,768)                        -> image_features (1,577,1024)
encoder: (image_features, input_ids (1,32), mask (1,32))   -> hidden (1,609,1024)
decoder: (hidden, mask (1,32), decoder_input_ids (1,128))  -> logits (1,128,51328)

Every file takes and returns fp32 tensors, so a precision is a file swap and the three parts can be mixed.

part	build	file	MB	corr vs fp32 eager	Mac ms*
vision	XNNPACK fp32	florence2_large_vision_xnnpack_fp32.pte	1452.6	1.000000	803.9
vision	Core ML (iOS)	florence2_large_vision_coreml_all.pte	729.5	0.999953	238.2
encoder	XNNPACK fp32	florence2_large_encoder_xnnpack_fp32.pte	831.8	1.000000	144.4
encoder	Core ML (iOS)	florence2_large_encoder_coreml_all.pte	409.2	0.999944	43.1
decoder	XNNPACK fp32	florence2_large_decoder_xnnpack_fp32.pte	1243.8	1.000000	85.8
decoder	Core ML (iOS)	florence2_large_decoder_coreml_all.pte	509.5	0.999980	13.8

Two sets: 3528 MB all-fp32, 1648 MB Core ML. Torch eager fp32 on the same machine: vision 1288.2 ms, encoder 96.8 ms, decoder 60.6 ms.

*Mac arm64, single process, median of 5 — a reference point for relative cost, not a device number. A caption of n tokens costs one vision pass, one encoder pass and n decoder passes: about 2.2 s for a 15-token caption on the fp32 set and 0.49 s on Core ML, on this Mac. The decoder graph is a fixed 128-token window, so every step costs the same whether it is the first token or the fiftieth.

Running it

Identical to the base model's contract except for the width (1024 instead of 768).

1. The image. RGB, divide by 255, ImageNet normalise (mean .485/.456/.406, std .229/.224/.225), bicubic resize to 768×768. No crop. Run the vision .pte.

2. The prompt. Florence-2's task tokens are shorthand the processor expands into a sentence before tokenising — the model never sees <CAPTION>:

task	the sentence that is actually tokenised
<CAPTION>	What does the image describe?
<DETAILED_CAPTION>	Describe in detail what is shown in the image.
<MORE_DETAILED_CAPTION>	Describe with a paragraph what is shown in the image.
<OD>	Locate the objects with category name in the image.
<DENSE_REGION_CAPTION>	Locate the objects in the image, with their descriptions.
<REGION_PROPOSAL>	Locate the region proposals in the image.
<OCR>	What is the text in the image?
<OCR_WITH_REGION>	What is the text in the image, with regions?
<CAPTION_TO_PHRASE_GROUNDING>	Locate the phrases in the caption: {your caption}
<OPEN_VOCABULARY_DETECTION>	Locate {your phrase} in the image.

Tokenise that sentence as <s> sentence </s> with the repo's tokenizer, right-pad to 32 with the pad id (1), and build an attention_mask that is 1 on the real tokens and 0 on the padding. Run the encoder .pte with (image_features, input_ids, mask).

The 577 image tokens are handled inside the graphs. The prompt sequence the original model sees is <image>×577 + <s> prompt </s>, and because the image tokens are a contiguous prefix, the encoder here concatenates the vision features in front of the text embeddings instead of scattering them into placeholder positions.

3. Greedy decoding, and the rule you cannot skip. Fill a (1,128) int64 window with the pad id, write the decoder start token (2) at position 0, then for step t:

logits = decoder(hidden, mask, window)      # mask is the same one the encoder took
ban every token that would repeat a 3-gram already in the output   # <- see below
next = argmax(logits[0, t])
if next == 2: stop                          # </s>
window[0, t + 1] = next

no_repeat_ngram_size: 3 from the model's generation_config.json is load-bearing on this size. Large's decoder returns <s> as its argmax three times in a row on most images. The ban on repeating that 3-gram is the only thing that moves it on to the caption — a plain argmax loop emits <s> forever and returns an empty string. On five test photographs it did so every time, before and after conversion, in eager PyTorch as well as through the .pte. Base tolerates the omission and large does not, so implement the rule.

4. Reading a detection answer. Grounded tasks answer with <loc_N> tokens, N in 0..999. Four in a row are a box, and each coordinate is (N + 0.5) × side / 1000 in the original image's pixels — side being the image's width for x and its height for y, not 768.

Verification

The three wrappers reproduce Florence2ForConditionalGeneration exactly: composition max_abs_diff 0.000e+00 against the full model's logits on the same image and prompt.

End to end through the three .pte files, greedy <CAPTION> on five photographs against the same decoding in eager PyTorch: fp32 5/5 and Core ML 5/5 captions identical, character for character.

A person's hands playing a piano with the words Lauberger and Gloss written on it.
A long wooden pier stretching out into the ocean on a sunny day.
A forest of dead trees in the middle of a forest.
A road in the middle of a pine forest lined with tall trees.
A couple of wooden benches sitting on top of a park bench covered in leaves.

python convert/check_florence2.py large fp32     # or coreml_all

Not shipped, and why

int8 converts and is not published. Dynamic int8 brings the set from 3528 MB to 1304 MB at correlations of 0.999, 0.9995 and 0.9893 — numbers that would pass any gate on this shelf. The captions do not: 2 of 5 match eager, and the other three are plausible but different sentences, one of them reworded from the first word. The same recipe on Florence-2-base keeps all five, so this is a property of the larger decoder rather than of the recipe. On iOS the Core ML set is the small one (1648 MB, 5/5); on Android it is fp32 or nothing until a better recipe is found.

fp16 for the encoder and the decoder does not export. BART clamps its activations when, and only when, they are half precision:

if hidden_states.dtype == torch.float16 and not torch.isfinite(hidden_states).all():

In fp32 that line short-circuits and never reaches the graph. Halve the model and it becomes a question about values torch.export cannot answer, and export stops with GuardOnDataDependentSymNode.

Conversion notes

Converted from florence-community/Florence-2-large, the transformers-format mirror of microsoft/Florence-2-large — same MIT weights. The original repo predates the in-tree implementation and its weight names do not match it: loading it into Florence2ForConditionalGeneration prints a load report where every key is unexpected and hands back a randomly initialised model without raising.

The checkpoint declares torch_dtype: float16 and transformers honours it, so from_pretrained must be given dtype=torch.float32 explicitly.

• Source: microsoft/Florence-2-large (via florence-community mirror)
• License: MIT

torch.export -> to_edge_transform_and_lower(partitioner) -> .pte (conversion scripts: executorch-models)