Instructions to use microsoft/Florence-2-large with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Florence-2-large with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="microsoft/Florence-2-large", trust_remote_code=True)

# Load model directly
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/Florence-2-large with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Florence-2-large"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Florence-2-large",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/microsoft/Florence-2-large

SGLang

How to use microsoft/Florence-2-large with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Florence-2-large" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Florence-2-large",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Florence-2-large" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Florence-2-large",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use microsoft/Florence-2-large with Docker Model Runner:
```
docker model run hf.co/microsoft/Florence-2-large
```

leoxiaobin

haipingwu commited on Dec 8, 2024

Commit

00d2f15

verified ·

1 Parent(s): f0acedb

initial tuned 4k context length model (#84)

Browse files

- add 4k context length model (43013a8f5eccffac1f83ec7d00e94b9ecb218380)

Co-authored-by: Haiping Wu <haipingwu@users.noreply.huggingface.co>

Files changed (3) hide show

README.md +3 -1
config.json +1 -1
pytorch_model.bin +2 -2

README.md CHANGED Viewed

@@ -10,6 +10,8 @@ tags:
 ## Model Summary
 This Hub repository contains a HuggingFace's `transformers` implementation of Florence-2 model from Microsoft.
 Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks.  Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.
@@ -53,7 +55,7 @@ inputs = processor(text=prompt, images=image, return_tensors="pt").to(device, to
 generated_ids = model.generate(
     input_ids=inputs["input_ids"],
     pixel_values=inputs["pixel_values"],
-    max_new_tokens=1024,
     num_beams=3,
     do_sample=False
 )

 ## Model Summary
+**This is a continued pretrained version of Florence-2-large model with 4k context length, only 0.1B samples are used for continue pretraining, thus it might not be trained well. In addition, OCR task has been updated with line separator ('\n'). COCO OD AP 39.8**
 This Hub repository contains a HuggingFace's `transformers` implementation of Florence-2 model from Microsoft.
 Florence-2 is an advanced vision foundation model that uses a prompt-based approach to handle a wide range of vision and vision-language tasks.  Florence-2 can interpret simple text prompts to perform tasks like captioning, object detection, and segmentation. It leverages our FLD-5B dataset, containing 5.4 billion annotations across 126 million images, to master multi-task learning. The model's sequence-to-sequence architecture enables it to excel in both zero-shot and fine-tuned settings, proving to be a competitive vision foundation model.
 generated_ids = model.generate(
     input_ids=inputs["input_ids"],
     pixel_values=inputs["pixel_values"],
+    max_new_tokens=4096,
     num_beams=3,
     do_sample=False
 )

config.json CHANGED Viewed

@@ -46,7 +46,7 @@
         "LABEL_1": 1,
         "LABEL_2": 2
       },
-      "max_position_embeddings": 1024,
       "no_repeat_ngram_size": 3,
       "normalize_before": false,
       "num_hidden_layers": 12,

         "LABEL_1": 1,
         "LABEL_2": 2
       },
+      "max_position_embeddings": 4096,
       "no_repeat_ngram_size": 3,
       "normalize_before": false,
       "num_hidden_layers": 12,

pytorch_model.bin CHANGED Viewed

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a8b6ee6144f20a57200a0e3fab21f067d6cc77036262b72d9f6f7f4e556c8f15
-size 1543107459

 version https://git-lfs.github.com/spec/v1
+oid sha256:8b7d99c2ca930af3bcc4625df55c82b6bb372456280310b5189c519d6083a270
+size 1555689792