Does not work with vllm 0.17.1
#1
by jtvino - opened
Model does not work with vllm 0.17.1, or 0.10.0 while original model does
/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/chat_completion/protocol.py:346: SyntaxWarning: invalid escape sequence '\e'
"(e.g. 'abcdabcdabcd...' or '\emoji \emoji \emoji ...'). This feature "
/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/protocol.py:176: SyntaxWarning: invalid escape sequence '\e'
"(e.g. 'abcdabcdabcd...' or '\emoji \emoji \emoji ...'). This feature "
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:302]
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:302] █ █ █▄ ▄█
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:302] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.17.1
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:302] █▄█▀ █ █ █ █ model RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:302] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:302]
(APIServer pid=50) INFO 04-20 19:15:09 [utils.py:238] non-default args: {'model_tag': 'RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4', 'enable_auto_tool_choice': True, 'tool_call_parser': 'mistral', 'model': 'RedHatAI/Mistral-Small-3.2-24B-Instruct-2506-NVFP4', 'tokenizer_mode': 'mistral', 'trust_remote_code': True, 'load_format': 'mistral', 'tensor_parallel_size': 2, 'limit_mm_per_prompt': {'prompt': 10}, 'enable_chunked_prefill': True}
(APIServer pid=50) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=50) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=50) INFO 04-20 19:15:16 [model.py:531] Resolved architecture: Mistral3ForConditionalGeneration
(APIServer pid=50) INFO 04-20 19:15:16 [model.py:1554] Using max model len 131072
(APIServer pid=50) INFO 04-20 19:15:16 [scheduler.py:231] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=50) INFO 04-20 19:15:16 [vllm.py:747] Asynchronous scheduling is enabled.
(APIServer pid=50) [2026-04-20 19:15:16] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(APIServer pid=50) [2026-04-20 19:15:16] INFO tekken.py:572: Cutting non special vocabulary to first 130072 tokens.
(APIServer pid=50) [2026-04-20 19:15:17] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(APIServer pid=50) [2026-04-20 19:15:17] INFO tekken.py:572: Cutting non special vocabulary to first 130072 tokens.
(APIServer pid=50) [2026-04-20 19:15:21] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(APIServer pid=50) [2026-04-20 19:15:21] INFO tekken.py:572: Cutting non special vocabulary to first 130072 tokens.
(APIServer pid=50) [2026-04-20 19:15:22] INFO tekken.py:195: Non special vocabulary size is 130072 with 1000 special tokens.
(APIServer pid=50) [2026-04-20 19:15:22] INFO tekken.py:572: Cutting non special vocabulary to first 130072 tokens.
(APIServer pid=50) Traceback (most recent call last):
(APIServer pid=50) File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=50) sys.exit(main())
(APIServer pid=50) ^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 73, in main
(APIServer pid=50) args.dispatch_function(args)
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 112, in cmd
(APIServer pid=50) uvloop.run(run_server(args))
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=50) return __asyncio.run(
(APIServer pid=50) ^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=50) return runner.run(main)
(APIServer pid=50) ^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=50) return self._loop.run_until_complete(task)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=50) return await main
(APIServer pid=50) ^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 471, in run_server
(APIServer pid=50) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 490, in run_server_worker
(APIServer pid=50) async with build_async_engine_client(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=50) return await anext(self.gen)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 96, in build_async_engine_client
(APIServer pid=50) async with build_async_engine_client_from_engine_args(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=50) return await anext(self.gen)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 137, in build_async_engine_client_from_engine_args
(APIServer pid=50) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 225, in from_vllm_config
(APIServer pid=50) return cls(
(APIServer pid=50) ^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 143, in __init__
(APIServer pid=50) self.input_processor = InputProcessor(self.vllm_config, renderer)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/input_processor.py", line 63, in __init__
(APIServer pid=50) mm_budget = MultiModalBudget(vllm_config, mm_registry)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/encoder_budget.py", line 87, in __init__
(APIServer pid=50) all_mm_max_toks_per_item = get_mm_max_toks_per_item(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/encoder_budget.py", line 32, in get_mm_max_toks_per_item
(APIServer pid=50) mm_inputs = mm_registry.get_dummy_mm_inputs(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/registry.py", line 235, in get_dummy_mm_inputs
(APIServer pid=50) processor_inputs = processor.dummy_inputs.get_dummy_processor_inputs(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/processing/dummy_inputs.py", line 82, in get_dummy_processor_inputs
(APIServer pid=50) dummy_text = self.get_dummy_text(mm_counts)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mistral3.py", line 230, in get_dummy_text
(APIServer pid=50) processor = self.info.get_hf_processor()
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/mistral3.py", line 259, in get_hf_processor
(APIServer pid=50) return self.ctx.get_hf_processor(PixtralProcessor, **kwargs)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/multimodal/processing/context.py", line 203, in get_hf_processor
(APIServer pid=50) return cached_processor_from_config(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/processor.py", line 315, in cached_processor_from_config
(APIServer pid=50) return cached_get_processor_without_dynamic_kwargs(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/processor.py", line 288, in cached_get_processor_without_dynamic_kwargs
(APIServer pid=50) final_processor = cached_get_processor(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/processor.py", line 164, in get_processor
(APIServer pid=50) processor = processor_cls.from_pretrained(
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1396, in from_pretrained
(APIServer pid=50) return cls.from_args_and_dict(args, processor_dict, **kwargs)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/transformers/processing_utils.py", line 1197, in from_args_and_dict
(APIServer pid=50) processor = cls(*args, **valid_kwargs)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/transformers/models/pixtral/processing_pixtral.py", line 110, in __init__
(APIServer pid=50) self.image_token_id = tokenizer.convert_tokens_to_ids(self.image_token)
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_mistral_common.py", line 591, in convert_tokens_to_ids
(APIServer pid=50) ids.append(self._piece_to_id(token))
(APIServer pid=50) ^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=50) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_mistral_common.py", line 566, in _piece_to_id
(APIServer pid=50) assert len(pieces) == 1, f"Expected to decode 1 token, got {len(pieces)}"
(APIServer pid=50) ^^^^^^^^^^^^^^^^
(APIServer pid=50) AssertionError: Expected to decode 1 token, got 3