Instructions to use hotdogs/Qwen35B-Agent-R2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use hotdogs/Qwen35B-Agent-R2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="hotdogs/Qwen35B-Agent-R2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("hotdogs/Qwen35B-Agent-R2")
model = AutoModelForCausalLM.from_pretrained("hotdogs/Qwen35B-Agent-R2", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use hotdogs/Qwen35B-Agent-R2 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Use Docker

docker model run hf.co/hotdogs/Qwen35B-Agent-R2:Q4_K_M

LM Studio
Jan

vLLM

How to use hotdogs/Qwen35B-Agent-R2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "hotdogs/Qwen35B-Agent-R2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hotdogs/Qwen35B-Agent-R2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/hotdogs/Qwen35B-Agent-R2:Q4_K_M

SGLang

How to use hotdogs/Qwen35B-Agent-R2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "hotdogs/Qwen35B-Agent-R2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hotdogs/Qwen35B-Agent-R2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "hotdogs/Qwen35B-Agent-R2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "hotdogs/Qwen35B-Agent-R2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use hotdogs/Qwen35B-Agent-R2 with Ollama:
```
ollama run hf.co/hotdogs/Qwen35B-Agent-R2:Q4_K_M
```

Unsloth Studio

How to use hotdogs/Qwen35B-Agent-R2 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for hotdogs/Qwen35B-Agent-R2 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for hotdogs/Qwen35B-Agent-R2 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for hotdogs/Qwen35B-Agent-R2 to start chatting

How to use hotdogs/Qwen35B-Agent-R2 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "hotdogs/Qwen35B-Agent-R2:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use hotdogs/Qwen35B-Agent-R2 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "hotdogs/Qwen35B-Agent-R2:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use hotdogs/Qwen35B-Agent-R2 with Docker Model Runner:
```
docker model run hf.co/hotdogs/Qwen35B-Agent-R2:Q4_K_M
```

Lemonade

How to use hotdogs/Qwen35B-Agent-R2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull hotdogs/Qwen35B-Agent-R2:Q4_K_M

Run and chat with the model

lemonade run user.Qwen35B-Agent-R2-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use hotdogs/Qwen35B-Agent-R2 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf hotdogs/Qwen35B-Agent-R2:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default hotdogs/Qwen35B-Agent-R2:Q4_K_M

Run Hermes

hermes

Atomic Chat

ValueError: no module or parameter named 'language_model

by Ronnaphon - opened 4 days ago

Discussion

Ronnaphon

4 days ago

(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:345]
(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:345] █ █ █▄ ▄█
(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.27.1.dev0+g4bdc8a788.d20260811
(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:345] █▄█▀ █ █ █ █ model hotdogs/Qwen35B-Agent-R2
(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:345]
(APIServer pid=54) INFO 08-11 06:04:32 [api_utils.py:273] non-default args: {'model_tag': 'hotdogs/Qwen35B-Agent-R2', 'host': '0.0.0.0', 'model': 'hotdogs/Qwen35B-Agent-R2', 'trust_remote_code': True, 'max_model_len': 8192, 'gpu_memory_utilization': 0.8, 'language_model_only': True}
(APIServer pid=54) WARNING 08-11 06:04:32 [envs.py:2128] Unknown vLLM environment variable detected: VLLM_BASE_DIR
(APIServer pid=54) INFO 08-11 06:04:34 [model.py:645] Resolved architecture: Qwen3_5MoeForCausalLM
(APIServer pid=54) INFO 08-11 06:04:34 [model.py:1883] Using max model len 8192
(APIServer pid=54) INFO 08-11 06:04:34 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=104) INFO 08-11 06:04:43 [core.py:121] Initializing a V1 LLM engine (v0.27.1.dev0+g4bdc8a788.d20260811) with config: model='hotdogs/Qwen35B-Agent-R2', speculative_config=None, tokenizer='hotdogs/Qwen35B-Agent-R2', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=8192, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=hotdogs/Qwen35B-Agent-R2, enable_prefix_caching=False, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'enable_qk_norm_rope_fusion': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False, 'fuse_qk_norm_rope_kvcache': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend='auto', linear_backend='auto')
(EngineCore pid=104) INFO 08-11 06:04:45 [parallel_state.py:1640] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:53089 backend=nccl
(EngineCore pid=104) INFO 08-11 06:04:45 [parallel_state.py:1977] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=104) INFO 08-11 06:04:47 [topk_topp_sampler.py:62] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=104) [ERROR] min_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(EngineCore pid=104) [ERROR] max_frames is part of Qwen3VLVideoProcessorInitKwargs, but not documented. Make sure to add it to the docstring of the function in /usr/local/lib/python3.12/dist-packages/transformers/models/qwen3_vl/video_processing_qwen3_vl.py.
(EngineCore pid=104) INFO 08-11 06:04:48 [gpu_model_runner.py:5308] Starting to load model hotdogs/Qwen35B-Agent-R2...
(EngineCore pid=104) INFO 08-11 06:04:48 [qwen_gdn_linear_attn.py:150] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=104) INFO 08-11 06:04:48 [unquantized.py:304] Using FlashInfer CUTLASS Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
(EngineCore pid=104) INFO 08-11 06:04:48 [cuda.py:482] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=104) INFO 08-11 06:04:48 [flash_attn.py:789] Using FlashAttention version 2
(EngineCore pid=104) INFO 08-11 06:04:53 [weight_utils.py:867] Filesystem type for checkpoints: EXT4. Checkpoint size: 64.56 GiB. Available RAM: 48.16 GiB.
(EngineCore pid=104) INFO 08-11 06:04:53 [weight_utils.py:897] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (64.56 GiB) exceeds 90% of available RAM (48.16 GiB).
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] EngineCore failed to start.
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] Traceback (most recent call last):
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1318, in run_engine_core
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1074, in __init__
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] super().__init__(
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] self.model_executor = executor_class(vllm_config)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] self._init_executor()
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 68, in _init_executor
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] self.driver_worker.load_model()
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 442, in load_model
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5324, in load_model
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] self.model = model_loader.load_model(
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] self.load_weights(model, model_config)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return func(*args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 419, in load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return loader.load_weights(weights)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return original_load_weights(self, weights, *args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 424, in load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 358, in _load_module
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] yield from self._load_module(
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 331, in _load_module
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] loaded_params = module_load_weights(weights)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 283, in load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] return original_load_weights(self, weights, *args, **kwargs)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 424, in load_weights
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 395, in _load_module
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] raise ValueError(msg)
(EngineCore pid=104) ERROR 08-11 06:05:00 [core.py:1349] ValueError: There is no module or parameter named 'language_model' in Qwen3_5Model. The available parameters belonging to (Qwen3_5Model) are: {'layers.5.linear_attn.conv1d.weight', 'layers.15.post_attention_layernorm.weight', ........
(EngineCore pid=104) Process EngineCore:
(EngineCore pid=104) Traceback (most recent call last):
(EngineCore pid=104) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=104) self.run()
(EngineCore pid=104) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=104) self._target(*self._args, **self._kwargs)
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1353, in run_engine_core
(EngineCore pid=104) raise e
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1318, in run_engine_core
(EngineCore pid=104) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) return func(*args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1074, in __init__
(EngineCore pid=104) super().__init__(
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 132, in __init__
(EngineCore pid=104) self.model_executor = executor_class(vllm_config)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) return func(*args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=104) self._init_executor()
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/uniproc_executor.py", line 68, in _init_executor
(EngineCore pid=104) self.driver_worker.load_model()
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 442, in load_model
(EngineCore pid=104) self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) return func(*args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5324, in load_model
(EngineCore pid=104) self.model = model_loader.load_model(
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) return func(*args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(EngineCore pid=104) self.load_weights(model, model_config)
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=104) return func(*args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 427, in load_weights
(EngineCore pid=104) loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 419, in load_weights
(EngineCore pid=104) return loader.load_weights(weights)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(EngineCore pid=104) return original_load_weights(self, weights, *args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 424, in load_weights
(EngineCore pid=104) autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 358, in _load_module
(EngineCore pid=104) yield from self._load_module(
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 331, in _load_module
(EngineCore pid=104) loaded_params = module_load_weights(weights)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py", line 283, in load_weights
(EngineCore pid=104) return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(EngineCore pid=104) return original_load_weights(self, weights, *args, **kwargs)
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 424, in load_weights
(EngineCore pid=104) autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore pid=104) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=104) File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 395, in _load_module
(EngineCore pid=104) raise ValueError(msg)
(EngineCore pid=104) ValueError: There is no module or parameter named 'language_model' in Qwen3_5Model. The available parameters belonging to (Qwen3_5Model) are: {'layers.5.linear_attn.conv1d.weight', 'layers.15.post_attention_layernorm.weight', 'layers.2.mlp.shared_expert.down_proj.weight', 'layers.14.linear_attn.norm.weight', 'layers.31.mlp.shared_expert.down_proj.weight', 'layers.11.mlp.shared_expert_gate.weight', ..................'layers.38.linear_attn.in_proj_qkvz.weight', 'layers.28.mlp.experts.routed_experts.w2_weight', 'layers.2.input_layernorm.weight',
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:07<?, ?it/s]
(EngineCore pid=104)
[rank0]:[W811 06:05:01.567059544 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=54) INFO 08-11 06:05:01 [utils.py:612] [shutdown] Process manager: send sigterm to process EngineCore
(APIServer pid=54) Traceback (most recent call last):
(APIServer pid=54) File "/usr/local/bin/vllm", line 10, in
(APIServer pid=54) sys.exit(main())
(APIServer pid=54) ^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 97, in main
(APIServer pid=54) args.dispatch_function(args)
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 152, in cmd
(APIServer pid=54) uvloop.run(run_server(args))
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 96, in run
(APIServer pid=54) return __asyncio.run(
(APIServer pid=54) ^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=54) return runner.run(main)
(APIServer pid=54) ^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=54) return self._loop.run_until_complete(task)
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=54) return await main
(APIServer pid=54) ^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 764, in run_server
(APIServer pid=54) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 778, in run_server_worker
(APIServer pid=54) async with build_async_engine_client(
(APIServer pid=54) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=54) return await anext(self.gen)
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 132, in build_async_engine_client
(APIServer pid=54) async with build_async_engine_client_from_engine_args(
(APIServer pid=54) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=54) return await anext(self.gen)
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 168, in build_async_engine_client_from_engine_args
(APIServer pid=54) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 220, in from_vllm_config
(APIServer pid=54) return cls(
(APIServer pid=54) ^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 149, in init
(APIServer pid=54) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=54) return func(*args, **kwargs)
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 139, in make_async_mp_client
(APIServer pid=54) return AsyncMPClient(*client_args)
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=54) return func(*args, **kwargs)
(APIServer pid=54) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 987, in init
(APIServer pid=54) super().init(
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 609, in init
(APIServer pid=54) with launch_core_engines(
(APIServer pid=54) File "/usr/lib/python3.12/contextlib.py", line 144, in exit
(APIServer pid=54) next(self.gen)
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1194, in launch_core_engines
(APIServer pid=54) wait_for_engine_startup(
(APIServer pid=54) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1253, in wait_for_engine_startup
(APIServer pid=54) raise RuntimeError(
(APIServer pid=54) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

Ronnaphon changed discussion title from alueError: no module or parameter named 'language_model to ValueError: no module or parameter named 'language_model 4 days ago

hotdogs

Owner 4 days ago

I recommend using GGUF with llama.cpp instead, as the model I developed often has issues with vLLM, and I'm sometimes unable to fix them. I sincerely apologize for this.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment