vllm-sr

/

d3

Like 286

Follow

vLLM Semantic Router 613

Summary

vllm-sr serve -e <engine> --platform rocm fatally fails because the CLI writes
global.model_catalog.deployments.<name>.profile into the materialized runtime
config, but the current vllm-sr-rocm:latest image removed that field and
rejects it. The newest published CLI still emits profile, and serve
re-materializes it on every run, so there is no config/file-level workaround.

Environment

• CLI: 0.5.0.dev20261014021837 (newest dev build available; built 2026-10-14 ~02:18 UTC)
• Image: ghcr.io/vllm-project/semantic-router/vllm-sr-rocm:latest
  seen as sha256:06cd2145… (created 2026-10-14 20:28 UTC) and, after a later
  pull, sha256:b4684f61… — i.e. :latest keeps moving ahead with pullPolicy Always.
• Platform rocm, runtime docker, engine vllm-sr/d3

Error

runtime_config_load_failed  config_path=/app/.vllm-sr/runtime-config.yaml
error: global.model_catalog.deployments.primary.profile: profile was removed;
batching is on by default and set by the deployment's engine arguments
(max_num_seqs, max_num_batched_tokens, batch_window_ms, enable_shared_context,
dtype); run `vllm-sr config migrate --config old-config.yaml`

Why the suggested fixes don't work

• Source config.yaml has no profile; the CLI injects profile: exact when
  it materializes the runtime config (source: 0 matches, runtime: 1 match).
• vllm-sr config migrate does not strip profile from the generated runtime
  config, and it literally suggests a placeholder path:
  vllm-sr config migrate --config old-config.yaml →
  Error: Config file not found: old-config.yaml.
• Upgrading to the newest dev CLI, clearing ~/.local/state/vllm-sr/engine/<engine>,
  and manually deleting the profile: line all fail — serve re-materializes the
  runtime config with profile before the container reads it.

Root cause

Dev-channel CLIs track vllm-sr-rocm:latest (pullPolicy Always). A breaking
schema change (removal of deployment profile, replaced by explicit engine args:
max_num_seqs / max_num_batched_tokens / batch_window_ms /
enable_shared_context / dtype) shipped in the image before any CLI build
stopped writing profile. This breaks every dev-channel serve on rocm during
the gap.

Expected / asks

1. Don't publish an image that rejects config the latest dev/released CLI
   generates — gate the profile removal behind a CLI that already stopped
   emitting it, or have the CLI omit removed fields per image/schema version.
2. Make the router tolerant: treat a now-defaulted profile as ignorable rather
   than fatal (the message already says batching is on by default).
3. Fix the error text: reference the real generated file path, not the
   placeholder old-config.yaml, and make config migrate actually strip removed
   keys from the materialized runtime config.

Impact

All dev-channel serve --platform rocm runs are currently blocked; there is no
user-side workaround until a CLI that stops emitting profile is published (or a
matching older image tag is made pinnable, which serve currently doesn't expose).