mistralai

/

Leanstral-2603

like 157

Follow

Mistral AI_ 17.5k

Fixing the run command as (on 8xH100 node):

docker run -it --gpus all mistralllm/vllm-ms4:latest --model mistralai/Leanstral-2603 \
  --max-model-len 200000 \
  --tensor-parallel-size 4 \
  --attention-backend FLASH_ATTN_MLA \
  --tool-call-parser mistral \
  --enable-auto-tool-choice \
  --reasoning-parser mistral

gives the following bad-looking message from vllm:

(APIServer pid=1) INFO 04-03 15:33:35 [model.py:533] Resolved architecture: PixtralForConditionalGeneration
(APIServer pid=1) ERROR 04-03 15:33:35 [repo_utils.py:47] Error retrieving safetensors: 'mistralai/Leanstral-2603' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files., retrying 1 of 2
(APIServer pid=1) ERROR 04-03 15:33:38 [repo_utils.py:45] Error retrieving safetensors: 'mistralai/Leanstral-2603' is not a safetensors repo. Couldn't find 'model.safetensors.index.json' or 'model.safetensors' files.

But then it still proceeds

The repo indeed does not have model.safetensors.index.json (but it does have consolidated.safetensors.index.json - should it be renamed in the repo to model.safetensors.index.json?)

I had to add --config-format mistral --load-format mistral to the docker run command, but then it complains with:

[2026-04-03 16:11:53] WARNING _http.py:916: Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
params.json: 1.51kB [00:00, 7.08MB/s]
Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'apply_yarn_scaling'}
`rope_parameters`'s factor field must be a float >= 1, got 128
`rope_parameters`'s beta_fast field must be a float, got 32
`rope_parameters`'s beta_slow field must be a float, got 1
Unrecognized keys in `rope_parameters` for 'rope_type'='yarn': {'apply_yarn_scaling'}
`rope_parameters`'s factor field must be a float >= 1, got 128
`rope_parameters`'s beta_fast field must be a float, got 32
`rope_parameters`'s beta_slow field must be a float, got 1

which also is quite suspicious

To make use of all 8 GPUs, should I run two copies of the docker (each using 4 GPUs)? Or do you also provide any sort of proxy, allowing to multiplex requests to several vllm copies running inside a single docker?

сс @juliendenize

I'm also a bit worried for Unrecognized keys in rope_parameters for 'rope_type'='yarn': {'apply_yarn_scaling'}

apply_yarn_scaling is actually not found in params.json, then why would loader complain about it?

Probably related: https://discuss.huggingface.co/t/hf-transformers-config-issue-for-mistral-large-3-models/171369/4

Maybe some sort of yarn/transformers version mismatch in Docker after key remapping?

hey sorry for the late response. So basically there is nothing to worry about that which sounds counterintuitive. These errors are due to the fact that our config is mapped to Transformers internally of vLLM which sometimes don't have the right type from Transformers perspective. The errors you're seeing are not detrimental to the model usage as the model behaves properly, they're more like a warning than an error. In more recent releases than the docker has they are mostly gone except for this one:

Unrecognized keys in rope_parameters for 'rope_type'='yarn': {'apply_yarn_scaling'}

but it will be gone after a PR is merged.

oh the last one was actually merged an hour ago thanks to Harry from 🤗
https://github.com/vllm-project/vllm/commit/edcc37a8cee26813fe868b9fc267c3cba5818ff7

@juliendenize Could the container be also updated to the most recent versions?

HF Transformers is also fixing autocasting rope ints to floats :https://github.com/huggingface/transformers/pull/45289

Looking forward to having the Mistral-4/Leanstral fixes in HF/vllm released versions

we're actually not looking to update the container but fixing everything upstream. The last PR to ensure guidance is in will be submitted very shortly and merged soon !

@juliendenize The vllm PRs are merged. Are there any known vllm / transformers mainline commit hashes that should be fine? Thanks!

@vadimkantorov updated the model card, thanks for hanging in there :)