GLM-OCR-FP8

Model Details

This model is a FP8 model with group_size 128 of zai-org/GLM-OCR generated by vastai modelzoo. Please follow the license of the original model.

vllm Inference

• vllm >= v0.17.0
• VVI >= 26.04

vllm serve vastai-ais/GLM-OCR-FP8  --served-model-name glm-ocr --max-model-len 16384

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm-ocr",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "请帮我提取图片中的所有文字，并按原始排版输出。"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "https://cdn.bigmodel.cn/static/logo/introduction.png"
            }
          }
        ]
      }
    ],
    "max_tokens": 4096
  }'