Backend tokenizer issue due to "transformers_version": "5.16.0.dev0"

#2
by qenme - opened

Hey, me again. I got around to trying the image but it fails with a bad tokenizer config. After reviewing the tokenizer, it's the same as other working repos. The only difference seems to be the "transformers_version": "5.16.0.dev0" is too new for the vllm nightly container. Whereas other repos like https://huggingface.co/lued/Qwen3.8-27B-INT8-W8A16-MTP/blob/main/config.json works fine. Have you seen this issue?

Thanks for the report. Unfortunately, the answer is no, and I couldn't reproduce it at all. I don't think the transformers_version is the issue here. Neither transformers nor vLLM ever reads it.
Can you give me:

  1. the full traceback
  2. the branch/revision you pulled
  3. the image digest and python3 -c "import vllm, transformers, tokenizers; print(vllm.version, transformers.version, tokenizers.version)" from inside the container
  4. your full vllm serve command

What I tried to reproduce it:

  • Pulled the stock vllm/vllm-openai:nightly (transformers 5.14.1, tokenizers 0.22.2, vllm 0.26.1rc1.dev542+gb22afe45a) and loaded this repo's tokenizer directory. AutoTokenizer → Qwen2Tokenizer, vocab 248077, chat template picked up from chat_template.jinja, apply_chat_template() and encoding both fine.
  • Same files load clean on transformers 4.57.6, 5.10.1, 5.12.1, 5.13.0, 5.14.1 and 5.16.0.dev0. AutoProcessor resolves to Qwen3VLProcessor and AutoConfig to Qwen3_5Config on 5.14.1 too.
  • EngineArgs(...).create_engine_config() on that stock nightly resolves Qwen3_5ForConditionalGeneration without any tokenizer error.
    One difference against lued's repo is that lued ships the chat template both embedded in tokenizer_config.json and as chat_template.jinja, whereas I only ship the .jinja (which is the transformers v5 default layout), but that's the only difference.

Hmm thanks for looking. Very strange. Here are the logs I have. This is with the bigger revision model. I tried redownloading the model and updating vLLM too.

The command:

sudo docker rm -f qwen3.8

sudo docker run --runtime nvidia -d --gpus '"device=0,1"' --ipc=host
--name qwen3.8 --restart always -p 8000:8000
-v vllm-hf-cache:/root/.cache/huggingface
-v vllm-cache:/root/.cache/vllm
--env "PYTORCH_ALLOC_CONF=expandable_segments:True"
--env "HUGGING_FACE_HUB_TOKEN=REDACTED"
--env "CUDA_DEVICE_ORDER=PCI_BUS_ID"
vllm/vllm-openai:nightly
Minachist/Qwen3.8-27B-INT8-AutoRound --revision linear-attn-bf16-gs128
--served-model-name Qwen3.8-27B
--mm-processor-cache-type shm
--max-model-len auto
--kv-cache-dtype bfloat16
--gpu-memory-utilization 0.9
--performance-mode interactivity
--enable-prefix-caching
--enable-chunked-prefill
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--enable-auto-tool-choice
--trust-remote-code
--async-scheduling
--generation-config auto

sudo docker logs -f qwen3.8

(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:333]
(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:333] █ █ █▄ ▄█
(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:333] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.27.2rc1.dev122+g8efa13b70
(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:333] █▄█▀ █ █ █ █ model Minachist/Qwen3.8-27B-INT8-AutoRound
(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:333] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:333]
(APIServer pid=1) INFO 08-16 14:41:14 [api_utils.py:272] non-default args: {'model_tag': 'Minachist/Qwen3.8-27B-INT8-AutoRound', 'enable_auto_tool_choice': True, 'tool_call_parser': 'qwen3_coder', 'model': 'Minachist/Qwen3.8-27B-INT8-AutoRound', 'trust_remote_code': True, 'revision': 'linear-attn-bf16-gs128', 'max_model_len': -1, 'served_model_name': ['Qwen3.8-27B'], 'reasoning_parser': 'qwen3', 'gpu_memory_utilization': 0.9, 'kv_cache_dtype': 'bfloat16', 'enable_prefix_caching': True, 'mm_processor_cache_type': 'shm', 'enable_chunked_prefill': True, 'async_scheduling': True, 'performance_mode': 'interactivity'}
(APIServer pid=1) INFO 08-16 14:41:14 [model.py:672] Resolved architecture: Qwen3_5ForConditionalGeneration
(APIServer pid=1) INFO 08-16 14:41:14 [model.py:1965] Using max model len 262144
(APIServer pid=1) [ERROR] min_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(APIServer pid=1) [ERROR] max_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(APIServer pid=1) INFO 08-16 14:41:19 [vllm.py:1042] Performance mode set to 'interactivity'.
(APIServer pid=1) INFO 08-16 14:41:19 [config.py:605] Mamba cache mode is set to 'align' for Qwen3_5ForConditionalGeneration by default when prefix caching is enabled
(APIServer pid=1) INFO 08-16 14:41:19 [kernel.py:308] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=1) [transformers] The use_fast parameter is deprecated and will be removed in a future version. Use backend="torchvision" instead of use_fast=True, or backend="pil" instead of use_fast=False.
(EngineCore pid=128) INFO 08-16 14:41:40 [core.py:122] Initializing a V1 LLM engine (v0.27.2rc1.dev122+g8efa13b70) with config: model='Minachist/Qwen3.8-27B-INT8-AutoRound', speculative_config=None, tokenizer='Minachist/Qwen3.8-27B-INT8-AutoRound', skip_tokenizer_init=False, tokenizer_mode=auto, revision=main, tokenizer_revision=main, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=inc, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=bfloat16, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='qwen3', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen3.8-27B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::qwen_gdn_attention_core_fused_norm_packed', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=128) [ERROR] min_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(EngineCore pid=128) [ERROR] max_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(EngineCore pid=128) ERROR 08-16 14:41:41 [repo_utils.py:106] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound: Unrecognized model in Minachist/Qwen3.8-27B-INT8-AutoRound. Should have a model_type key in its config.json., retrying 1 of 2
(EngineCore pid=128) ERROR 08-16 14:41:43 [repo_utils.py:104] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound: Unrecognized model in Minachist/Qwen3.8-27B-INT8-AutoRound. Should have a model_type key in its config.json.
(EngineCore pid=128) WARNING 08-16 14:41:43 [registry.py:124] Model Minachist/Qwen3.8-27B-INT8-AutoRound is treated as multimodal but has no registered multimodal processor; running in text-only mode.
(EngineCore pid=128) INFO 08-16 14:41:43 [parallel_state.py:1638] world_size=1 rank=0 local_rank=0 distributed_init_method=file:///tmp/vllm_dist_ec729b9bbedc47878e770a70d6c8e4f4 backend=nccl
(EngineCore pid=128) INFO 08-16 14:41:44 [parallel_state.py:1982] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=128) ERROR 08-16 14:41:44 [repo_utils.py:106] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound: Unrecognized model in Minachist/Qwen3.8-27B-INT8-AutoRound. Should have a model_type key in its config.json., retrying 1 of 2
(EngineCore pid=128) ERROR 08-16 14:41:46 [repo_utils.py:104] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound: Unrecognized model in Minachist/Qwen3.8-27B-INT8-AutoRound. Should have a model_type key in its config.json.
(EngineCore pid=128) INFO 08-16 14:41:46 [topk_topp_sampler.py:62] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=128) INFO 08-16 14:41:46 [gpu_model_runner.py:5419] Starting to load model Minachist/Qwen3.8-27B-INT8-AutoRound...
(EngineCore pid=128) ERROR 08-16 14:41:47 [repo_utils.py:106] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound: Unrecognized model in Minachist/Qwen3.8-27B-INT8-AutoRound. Should have a model_type key in its config.json., retrying 1 of 2
(EngineCore pid=128) ERROR 08-16 14:41:49 [repo_utils.py:104] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound: Unrecognized model in Minachist/Qwen3.8-27B-INT8-AutoRound. Should have a model_type key in its config.json.
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] EngineCore failed to start.
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] Traceback (most recent call last):
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1315, in run_engine_core
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) Process EngineCore:
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return func(*args, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1073, in init
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] super().init(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 133, in init
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] self.model_executor = executor_class(vllm_config)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return func(*args, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 110, in init
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] self._init_executor()
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 74, in _init_executor
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] self.driver_worker.load_model()
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 457, in load_model
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return func(*args, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5435, in load_model
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] self.model = model_loader.load_model(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return func(*args, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] model = initialize_model(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return func(*args, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 58, in initialize_model
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] model = model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 483, in init
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] self.tokenizer = cached_tokenizer_from_config(vllm_config.model_config)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/registry.py", line 276, in cached_tokenizer_from_config
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return cached_get_tokenizer(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/registry.py", line 253, in get_tokenizer
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] tokenizer = tokenizer_cls
.from_pretrained(tokenizer_name, *args, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 238, in from_pretrained
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] raise e
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 210, in from_pretrained
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] tokenizer = AutoTokenizer.from_pretrained(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 954, in from_pretrained
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1736, in from_pretrained
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] return cls._from_pretrained(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1932, in _from_pretrained
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] tokenizer = cls(*init_inputs, **init_kwargs)
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_tokenizers.py", line 376, in init
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] raise ValueError(
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] ValueError: Couldn't instantiate the backend tokenizer from one of:
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] (1) a tokenizers library serialization file,
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] (2) a slow tokenizer instance to convert or
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] (3) an equivalent slow tokenizer class to instantiate and convert.
(EngineCore pid=128) ERROR 08-16 14:41:50 [core.py:1346] You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.
(EngineCore pid=128) Traceback (most recent call last):
(EngineCore pid=128) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=128) self.run()
(EngineCore pid=128) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=128) self._target(*self._args, **self._kwargs)
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1350, in run_engine_core
(EngineCore pid=128) raise e
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1315, in run_engine_core
(EngineCore pid=128) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) return func(*args, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1073, in init
(EngineCore pid=128) super().init(
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 133, in init
(EngineCore pid=128) self.model_executor = executor_class(vllm_config)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) return func(*args, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 110, in init
(EngineCore pid=128) self._init_executor()
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 74, in _init_executor
(EngineCore pid=128) self.driver_worker.load_model()
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 457, in load_model
(EngineCore pid=128) self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) return func(*args, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5435, in load_model
(EngineCore pid=128) self.model = model_loader.load_model(
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) return func(*args, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 55, in load_model
(EngineCore pid=128) model = initialize_model(
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=128) return func(*args, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/utils.py", line 58, in initialize_model
(EngineCore pid=128) model = model_class(vllm_config=vllm_config, prefix=prefix)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 483, in init
(EngineCore pid=128) self.tokenizer = cached_tokenizer_from_config(vllm_config.model_config)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/registry.py", line 276, in cached_tokenizer_from_config
(EngineCore pid=128) return cached_get_tokenizer(
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/registry.py", line 253, in get_tokenizer
(EngineCore pid=128) tokenizer = tokenizer_cls
.from_pretrained(tokenizer_name, *args, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 238, in from_pretrained
(EngineCore pid=128) raise e
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/vllm/tokenizers/hf.py", line 210, in from_pretrained
(EngineCore pid=128) tokenizer = AutoTokenizer.from_pretrained(
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 954, in from_pretrained
(EngineCore pid=128) return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1736, in from_pretrained
(EngineCore pid=128) return cls._from_pretrained(
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_base.py", line 1932, in _from_pretrained
(EngineCore pid=128) tokenizer = cls(*init_inputs, **init_kwargs)
(EngineCore pid=128) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=128) File "/usr/local/lib/python3.12/dist-packages/transformers/tokenization_utils_tokenizers.py", line 376, in init
(EngineCore pid=128) raise ValueError(
(EngineCore pid=128) ValueError: Couldn't instantiate the backend tokenizer from one of:
(EngineCore pid=128) (1) a tokenizers library serialization file,
(EngineCore pid=128) (2) a slow tokenizer instance to convert or
(EngineCore pid=128) (3) an equivalent slow tokenizer class to instantiate and convert.
(EngineCore pid=128) You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one.
[rank0]:[W816 14:41:51.990265779 ProcessGroupNCCL.cpp:1624] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=1) INFO 08-16 14:41:52 [utils.py:615] [shutdown] Process manager: send sigterm to process EngineCore
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1) File "/usr/local/bin/vllm", line 10, in
(APIServer pid=1) sys.exit(main())
(APIServer pid=1) ^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 97, in main
(APIServer pid=1) args.dispatch_function(args)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 152, in cmd
(APIServer pid=1) uvloop.run(run_server(args))
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 96, in run
(APIServer pid=1) return __asyncio.run(
(APIServer pid=1) ^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=1) return runner.run(main)
(APIServer pid=1) ^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1) return self._loop.run_until_complete(task)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=1) return await main
(APIServer pid=1) ^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 739, in run_server
(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 753, in run_server_worker
(APIServer pid=1) async with build_async_engine_client(
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 127, in build_async_engine_client
(APIServer pid=1) async with build_async_engine_client_from_engine_args(
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 163, in build_async_engine_client_from_engine_args
(APIServer pid=1) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config
(APIServer pid=1) return cls(
(APIServer pid=1) ^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 149, in init
(APIServer pid=1) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1) return func(*args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client
(APIServer pid=1) return AsyncMPClient(*client_args)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1) return func(*args, **kwargs)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 990, in init
(APIServer pid=1) super().init(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 609, in init
(APIServer pid=1) with launch_core_engines(
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=1) next(self.gen)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1206, in launch_core_engines
(APIServer pid=1) wait_for_engine_startup(
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1286, in wait_for_engine_startup
(APIServer pid=1) raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Ok I've narrowed the issue down to the linear-attn-bf16-gs128 branch. The main branch loads fine, but this branch (linear-attn-bf16-gs128) does not. The logs above were from the linear attn branch.

ERROR [repo_utils.py:106] Error parsing config for Minachist/Qwen3.8-27B-INT8-AutoRound:
Unrecognized model in ... Should have a model_type key in its config.json.

I checked what produces that error. In transformers, a config whose model_type is present but unknown gives a different error ("The checkpoint you are trying to load has model type X but Transformers does not recognize this architecture"). The message you got is only raised when the parsed config dict has no model_type key at all. So the config.json transformers read on your machine is not the one in the repo. The repo's has "model_type": "qwen3_5".

vllm/tokenizers/registry.py:227 wraps that get_config() call in contextlib.suppress(ValueError, OSError), so it becomes config = None. That's also why you see treated as multimodal but has no registered multimodal processor. On a normal load that line instead reads "All limits of multimodal modalities supported by the model are set to 0".

Looking at transformers/tokenization_utils_tokenizers.py (lines 328–382), the "Couldn't instantiate the backend tokenizer branch" is reachable in one situation. Tokenizer_file is None, or os.path.isfile(tokenizer_file) returns False. That is, a path that resolves to nothing, like a dangling cache symlink. Having vocab.json + merges.txt in the repo doesn't help the situation, because that fallback branch reads an already-parsed vocab dict, not those files.

So I think config.json and tokenizer.json from the same snapshot are both unreadable on your side. That's a corrupted or partial HF cache snapshot for that commit. Your vllm-hf-cache named volume persists across docker rm -f, and huggingface_hub trusts a cached entry when the etag matches, which is why redownloading and updating vLLM didn't change anything.

To fix this, you need to either delete my model from the HF cache entirely, or remove just the broken config/tokenizer files so they get fetched again:

To delete the model entirely:

 sudo docker run --rm -v vllm-hf-cache:/root/.cache/huggingface alpine \
 rm -rf /root/.cache/huggingface/hub/models--Minachist--Qwen3.8-27B-INT8-AutoRound

To remove only the broken files (they will be re-downloaded on the next start):

 sudo docker run --rm -v vllm-hf-cache:/root/.cache/huggingface alpine sh -c '
   cd /root/.cache/huggingface/hub/models--Minachist--Qwen3.8-27B-INT8-AutoRound || exit 1
   for s in snapshots/*/*; do
     case "$s" in *.safetensors) continue;; esac
     [ -e "$s" ] || [ -L "$s" ] || continue
     b=$(readlink -f "$s" 2>/dev/null)
     case "$b" in */blobs/*) rm -f "$b";; esac
     rm -f "$s"
   done
   rm -rf .no_exist
   echo done' 

(Note: I'm guessing the file paths here, so please check them before you run the command.)

I am honestly perplexed, I've deleted all of the volumes multiple times yet this one branch doesn't work. I even validated there are no remaining files. When I check the actual files downloaded they are the same as the ones in the repo. I will keep looking, thanks for the advice.

Wow I think I got it (with LLM (and your) help lol)

add:
--language-model-only

remove:
--mm-processor-cache-type shm

And the issue goes away. Removing the mm-processor-cache-type is not enough.

More interesting news... The order of operations seems to matter here.

The --language-model-only actually worked because I previously used:
--tokenizer Minachist/Qwen3.8-27B-INT8-AutoRound
--tokenizer-revision main \

Taking out the parameters without clearing the cache seems to leave some type of links to the previous commands.
So it's loading the model weights from the revision but the tokenizer from main. I confirmed it's actually loading the correct weights.

Final working config that includes multimodal.

sudo docker run --runtime nvidia -d --gpus '"device=0,1"' --ipc=host
--name qwen3.8 --restart always -p 8000:8000
-v vllm-hf-cache:/root/.cache/huggingface
-v vllm-cache:/root/.cache/vllm
--env "PYTORCH_ALLOC_CONF=expandable_segments:True"
--env "CUDA_DEVICE_ORDER=PCI_BUS_ID"
vllm/vllm-openai:nightly
Minachist/Qwen3.8-27B-INT8-AutoRound
--revision linear-attn-bf16-gs128
--tokenizer Minachist/Qwen3.8-27B-INT8-AutoRound
--tokenizer-revision main
--served-model-name Qwen3.8-27B
--mm-processor-cache-type shm
--max-model-len auto
--kv-cache-dtype bfloat16
--gpu-memory-utilization 0.9
--max-num-seqs 8
--pipeline-parallel-size 2
--performance-mode interactivity
--enable-prefix-caching
--enable-chunked-prefill
--reasoning-parser qwen3
--tool-call-parser qwen3_coder
--enable-auto-tool-choice
--trust-remote-code
--async-scheduling
--enable-prompt-tokens-details
--enable-per-request-metrics
--generation-config auto
--default-chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}'
--override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0,"presence_penalty":0.0}'

Closing issue thanks for your help!

qenme changed discussion status to closed

Sign up or log in to comment