it only took me half a day bro, using lmdeploy is faster than vllm and it worked

sglang works too

please guide me i did it with sglang but the result is not correct. i think i have to adjust the chat template but after n times still failed the result is not the same as vllm did.

docker:
#!/bin/bash

Verificar si el contenedor está en ejecución

if docker ps --filter "name=ocr_server" --format "{{.ID}}"; then
echo "El contenedor ocr_server está en ejecución. Deteniendo..."
docker stop "ocr_server"
else
echo "El contenedor ocr_server no está en ejecución."
fi

docker run --rm --name "ocr_server" --user root --runtime nvidia --gpus '"device=0"' --shm-size 32g --ipc=host
-e HF_HUB_ENABLE_HF_TRANSFER="true"
-e HF_HOME="/data"
-p 8001:8001 -v "$(pwd)/data_aphro/cache:/app/aphrodite-engine/.cache" -v "$(pwd)/data_aphro:/home/workspace" -v "$(pwd)/data_aphro:/home/workspace" -v "$(pwd)/datafolder:/data"
--entrypoint /bin/sh
lmsysorg/sglang:latest
-c "python3 -m sglang.launch_server --port 8000 --download-dir '/data' --host 0.0.0.0
--port 8001 --download-dir '/data' --tp-size 1 --model-path 'ChatDOC/OCRFlux-3B'
--mem-fraction-static 0.7 --context-length 8192 --disable-cuda-graph
--api-key 123 --chat-template /data/flux.jinja
--served-model-name 'gpt-4o' --show-time-cost --enable-metrics
--max-running-requests 50 --chunked-prefill-size 2048 --enable-hierarchical-cache --hicache-size 24

to use the model I have used the official library/examples from the git of flux, is a lot of plumbing code to paste here

the chat template in the params is on the model repo, anyway the content is:

{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first an
d message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'
] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% i
f content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{%
if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] =
= 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.valu
e }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}
<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}