Instructions to use microsoft/Florence-2-large with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/Florence-2-large with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="microsoft/Florence-2-large", trust_remote_code=True)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)
model = AutoModelForMultimodalLM.from_pretrained("microsoft/Florence-2-large", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use microsoft/Florence-2-large with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/Florence-2-large"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Florence-2-large",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/microsoft/Florence-2-large

SGLang

How to use microsoft/Florence-2-large with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/Florence-2-large" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Florence-2-large",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/Florence-2-large" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/Florence-2-large",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use microsoft/Florence-2-large with Docker Model Runner:
```
docker model run hf.co/microsoft/Florence-2-large
```

Assert config.vision_config.model_type == 'davit', 'only DaViT is supported for now'

#44

by Glider95 - opened Jul 11, 2024

Discussion

Glider95

Jul 11, 2024

Hello,

I facing an issue that I can't solve, similar to the one happening to this space: https://huggingface.co/spaces/mihirinamdar/DocVQA-Florence-2.
Checking config.json, I can see that it it there:
"vision_config": {
"model_type": "davit",

However, the error keep happening.
It might be related to PR : https://huggingface.co/microsoft/Florence-2-large/commit/f0a0996b38848cb40f47db06d4735652bbd7af4d

Does anyone is facing the same issue ?

Best regards,

3Darkons

Jul 12, 2024

I fixed this by adding this to my run script iirc
if 'vision_config' not in config_dict:
config_dict['vision_config'] = {}
config_dict['vision_config']['model_type'] = 'davit'

dutta18

Sep 12, 2024

I fixed this by adding this to my run script iirc
if 'vision_config' not in config_dict:
config_dict['vision_config'] = {}
config_dict['vision_config']['model_type'] = 'davit'

Can u explain in some detail how to do this ? I was trying to import AutoConfig and then to do this, but its not working

JMog

Nov 14, 2024

@dutta18 did you ever find out?

shetumohanto

Dec 23, 2024

•

edited Dec 23, 2024

in config.json file search for 'model_type' inside 'vision_config', and replace its value with 'davit'

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment