openbmb

/

MiniCPM-SALA

like 489

Follow

OpenBMB 2.93k

The minicpm-sala model supports tool calls.
To use it, please configure your sglang environment according to the readme:

# 0. Ensure uv is installed
pip install uv

# 1. Create venv
uv venv --python 3.12 sglang_minicpm_sala_env
source sglang_minicpm_sala_env/bin/activate

# 2. Install SGLang
uv pip install --upgrade pip setuptools wheel
uv pip install -e ./python[all]

# 3. Compile CUDA Extensions
# (Ensure dependencies are cloned to 3rdparty/)
cd 3rdparty/infllmv2_cuda_impl && python setup.py install && cd ../..
cd 3rdparty/sparse_kernel && python setup.py install && cd ../..

# 4. Install extra deps
uv pip install tilelang flash-linear-attention

Then, add the corresponding tool-call-parser during deployment: --tool-call-parser minicpm4_xml

MODEL_PATH=/path/to/your/MiniCPM-SALA

python3 -m sglang.launch_server \
    --model ${MODEL_PATH} \
    --trust-remote-code \
    --disable-radix-cache \
    --attention-backend minicpm_flashinfer \
    --chunked-prefill-size 8192 \
    --max-running-requests 32 \
    --skip-server-warmup \
    --port 31111 \
    --dense-as-sparse \
    --tool-call-parser minicpm4_xml

Simply pass the list of tools and the list of messages when making a request:

curl -X POST "http://localhost:31111/v1/chat/completions" \
  -H "Content-Type: application/json" \
   -d '{
    "model": "minicpm-sala",
    "messages": [{"role": "user", "content": "北京天气怎么样"}],
     "tools": [
      {
        "type": "function",
        "function": {
          "name": "get_weather",
          "description": "查询天气",
          "parameters": {
            "type": "object",
            "properties": {
              "city": {"type": "string"}
            }
          }
        }
      }
    ]
  }'