Instructions to use CohereLabs/North-Mini-Code-1.0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use CohereLabs/North-Mini-Code-1.0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="CohereLabs/North-Mini-Code-1.0")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("CohereLabs/North-Mini-Code-1.0")
model = AutoModelForCausalLM.from_pretrained("CohereLabs/North-Mini-Code-1.0", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use CohereLabs/North-Mini-Code-1.0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CohereLabs/North-Mini-Code-1.0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/North-Mini-Code-1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CohereLabs/North-Mini-Code-1.0

SGLang

How to use CohereLabs/North-Mini-Code-1.0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "CohereLabs/North-Mini-Code-1.0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/North-Mini-Code-1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "CohereLabs/North-Mini-Code-1.0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CohereLabs/North-Mini-Code-1.0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use CohereLabs/North-Mini-Code-1.0 with Docker Model Runner:
```
docker model run hf.co/CohereLabs/North-Mini-Code-1.0
```

vLLM 0.23.0 fails to load

#10

by virtualroadside - opened Jun 15

Discussion

virtualroadside

Jun 15

Trying to run this in docker, here's the dockerfile:

FROM vllm/vllm-openai:v0.23.0-cu129

RUN pip install cohere_melody>=0.9.0

Using the following docker compose snippet:

  vllm-server-north:
    build:
      dockerfile: north.dockerfile

    container_name: vllm-server-north
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: ["gpu"]
    volumes:
      - ./model-north-mini-code-1:/root/model
    ipc: host
    environment:
      VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: "1"
      CUDA_VISIBLE_DEVICES: "4,5"
      HF_HUB_OFFLINE: "1"
  
    command:
      - /root/model
      - --served-model-name
      - CohereLabs/North-Mini-Code-1.0
      - --tool-call-parser
      - cohere_command4
      - --reasoning-parser
      - cohere_command4
      - --enable-auto-tool-choice
      - --gpu-memory-utilization
      - "0.95"
      - --tensor-parallel-size
      - "2"

vLLM fails to load with the following error:

$ docker logs -f vllm-server-north
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:339] 
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:339]        █     █     █▄   ▄█
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.23.0
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:339]   █▄█▀ █     █     █     █  model   /root/model
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:339] 
(APIServer pid=1) INFO 06-15 16:08:48 [api_utils.py:273] non-default args: {'model_tag': '/root/model', 'enable_auto_tool_choice': True, 'tool_call_parser': 'cohere_command4', 'model': '/root/model', 'served_model_name': ['CohereLabs/North-Mini-Code-1.0'], 'reasoning_parser': 'cohere_command4', 'tensor_parallel_size': 2, 'gpu_memory_utilization': 0.95}
(APIServer pid=1) WARNING 06-15 16:08:48 [envs.py:2088] Unknown vLLM environment variable detected: VLLM_BUILD_COMMIT
(APIServer pid=1) WARNING 06-15 16:08:48 [envs.py:2088] Unknown vLLM environment variable detected: VLLM_BUILD_PIPELINE
(APIServer pid=1) WARNING 06-15 16:08:48 [envs.py:2088] Unknown vLLM environment variable detected: VLLM_BUILD_URL
(APIServer pid=1) WARNING 06-15 16:08:48 [envs.py:2088] Unknown vLLM environment variable detected: VLLM_IMAGE_TAG
(APIServer pid=1) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
(APIServer pid=1) <frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(APIServer pid=1) INFO 06-15 16:09:28 [model.py:611] Resolved architecture: Cohere2MoeForCausalLM
(APIServer pid=1) INFO 06-15 16:09:28 [model.py:1745] Using max model len 500000
(APIServer pid=1) INFO 06-15 16:09:28 [vllm.py:999] Asynchronous scheduling is enabled.
(APIServer pid=1) INFO 06-15 16:09:28 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=1632) INFO 06-15 16:10:04 [core.py:113] Initializing a V1 LLM engine (v0.23.0) with config: model='/root/model', speculative_config=None, tokenizer='/root/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=500000, download_dir=None, load_format=auto, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='cohere_command4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=CohereLabs/North-Mini-Code-1.0, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=1632) WARNING 06-15 16:10:04 [multiproc_executor.py:1054] Reducing Torch parallelism from 112 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=1632) /usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py:139: UserWarning: Failed to get the IP address, using 0.0.0.0 by default. The value can be set by the environment variable VLLM_HOST_IP or HOST_IP.
(EngineCore pid=1632)   mq_connect_ip = get_ip()
(EngineCore pid=1632) INFO 06-15 16:10:04 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=0.0.0.0 (local), world_size=2, local_world_size=2
<frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1301: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
(Worker pid=1866) INFO 06-15 16:10:32 [parallel_state.py:1568] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:44235 backend=nccl
[W615 16:10:32.382360185 socket.cpp:207] [c10d] The hostname of the client socket cannot be retrieved. err=-3
(Worker pid=1867) INFO 06-15 16:10:32 [parallel_state.py:1568] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:44235 backend=nccl
[W615 16:10:32.675867214 socket.cpp:207] [c10d] The hostname of the client socket cannot be retrieved. err=-3
(Worker pid=1866) INFO 06-15 16:10:32 [pynccl.py:113] vLLM is using nccl==2.28.9
(Worker pid=1866) WARNING 06-15 16:10:33 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=1867) WARNING 06-15 16:10:33 [symm_mem.py:66] SymmMemCommunicator: Device capability 8.9 not supported, communicator is not available.
(Worker pid=1866) INFO 06-15 16:10:33 [cuda_communicator.py:237] Using ['CUSTOM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=1866) INFO 06-15 16:10:34 [cuda_communicator.py:237] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=1866) INFO 06-15 16:10:34 [parallel_state.py:1903] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=1866) INFO 06-15 16:10:35 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(Worker_TP0 pid=1866) INFO 06-15 16:10:36 [gpu_model_runner.py:5092] Starting to load model /root/model...
(Worker_TP0 pid=1866) INFO 06-15 16:10:37 [cuda.py:378] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(Worker_TP0 pid=1866) INFO 06-15 16:10:37 [flash_attn.py:636] Using FlashAttention version 2
(Worker_TP0 pid=1866) INFO 06-15 16:10:37 [unquantized.py:291] Using TRITON Unquantized MoE backend out of potential backends: ['FlashInfer TRTLLM', 'FlashInfer CUTLASS', 'TRITON', 'BATCHED_TRITON'].
(Worker_TP0 pid=1866) INFO 06-15 16:10:38 [weight_utils.py:922] Filesystem type for checkpoints: EXT4. Checkpoint size: 56.78 GiB. Available RAM: 684.67 GiB.
(Worker_TP0 pid=1866) INFO 06-15 16:10:38 [weight_utils.py:945] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards:   0% Completed | 0/49 [00:00<?, ?it/s]
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888] WorkerProc failed to start.
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888] Traceback (most recent call last):
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 855, in worker_main
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     worker = WorkerProc(*args, **kwargs)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 634, in __init__
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.worker.load_model()
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 356, in load_model
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5108, in load_model
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.model = model_loader.load_model(
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.load_weights(model, model_config)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 394, in load_weights
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/cohere2_moe.py", line 581, in load_weights
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return loader.load_weights(weights)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return original_load_weights(self, weights, *args, **kwargs)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 368, in load_weights
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 301, in _load_module
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     yield from self._load_module(
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 274, in _load_module
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     loaded_params = module_load_weights(weights)
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/cohere2_moe.py", line 513, in load_weights
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     param = params_dict[name]
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888]             ~~~~~~~~~~~^^^^^^
(Worker_TP1 pid=1867) ERROR 06-15 16:10:39 [multiproc_executor.py:888] KeyError: 'layers.0.mlp.down_proj.weight'
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888] WorkerProc failed to start.
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888] Traceback (most recent call last):
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 855, in worker_main
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     worker = WorkerProc(*args, **kwargs)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
Loading safetensors checkpoint shards:   0% Completed | 0/49 [00:01<?, ?it/s]
(Worker_TP0 pid=1866) 
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 634, in __init__
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.worker.load_model()
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 356, in load_model
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.model_runner.load_model(load_dummy_weights=load_dummy_weights)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 5108, in load_model
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.model = model_loader.load_model(
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                  ^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 64, in load_model
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     self.load_weights(model, model_config)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return func(*args, **kwargs)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 394, in load_weights
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     loaded_weights = model.load_weights(self.get_all_weights(model_config, model))
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/cohere2_moe.py", line 581, in load_weights
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return loader.load_weights(weights)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/reload/torchao_decorator.py", line 50, in patched_model_load_weights
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     return original_load_weights(self, weights, *args, **kwargs)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 368, in load_weights
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     autoloaded_weights = set(self._load_module("", self.module, weights))
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 301, in _load_module
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     yield from self._load_module(
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/utils.py", line 274, in _load_module
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     loaded_params = module_load_weights(weights)
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/cohere2_moe.py", line 513, in load_weights
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]     param = params_dict[name]
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888]             ~~~~~~~~~~~^^^^^^
(Worker_TP0 pid=1866) ERROR 06-15 16:10:39 [multiproc_executor.py:888] KeyError: 'layers.0.mlp.down_proj.weight'
(EngineCore pid=1632) INFO 06-15 16:10:39 [multiproc_executor.py:428] [shutdown] Executor: waiting for worker exit count=2
[rank0]:[W615 16:10:41.329572365 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(EngineCore pid=1632) WARNING 06-15 16:10:43 [multiproc_executor.py:438] [shutdown] Executor: workers still running after grace period; sending SIGTERM count=2
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195] EngineCore failed to start.
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195] Traceback (most recent call last):
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1164, in run_engine_core
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     return func(*args, **kwargs)
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 930, in __init__
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     super().__init__(
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     self.model_executor = executor_class(vllm_config)
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     super().__init__(vllm_config)
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     return func(*args, **kwargs)
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     self._init_executor()
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 762, in wait_for_ready
(EngineCore pid=1632) Process EngineCore:
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195]     raise e from None
(EngineCore pid=1632) ERROR 06-15 16:10:44 [core.py:1195] Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(EngineCore pid=1632) Traceback (most recent call last):
(EngineCore pid=1632)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=1632)     self.run()
(EngineCore pid=1632)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=1632)     self._target(*self._args, **self._kwargs)
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1199, in run_engine_core
(EngineCore pid=1632)     raise e
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1164, in run_engine_core
(EngineCore pid=1632)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=1632)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1632)     return func(*args, **kwargs)
(EngineCore pid=1632)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 930, in __init__
(EngineCore pid=1632)     super().__init__(
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 122, in __init__
(EngineCore pid=1632)     self.model_executor = executor_class(vllm_config)
(EngineCore pid=1632)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 108, in __init__
(EngineCore pid=1632)     super().__init__(vllm_config)
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=1632)     return func(*args, **kwargs)
(EngineCore pid=1632)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 109, in __init__
(EngineCore pid=1632)     self._init_executor()
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 201, in _init_executor
(EngineCore pid=1632)     self.workers = WorkerProc.wait_for_ready(unready_workers)
(EngineCore pid=1632)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=1632)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 762, in wait_for_ready
(EngineCore pid=1632)     raise e from None
(EngineCore pid=1632) Exception: WorkerProc initialization failed due to an exception in a background process. See stack trace for root cause.
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1)   File "/usr/local/bin/vllm", line 10, in <module>
(APIServer pid=1)     sys.exit(main())
(APIServer pid=1)              ^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 95, in main
(APIServer pid=1)     args.dispatch_function(args)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd
(APIServer pid=1)     uvloop.run(run_server(args))
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=1)     return __asyncio.run(
(APIServer pid=1)            ^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1)     return runner.run(main)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1)     return self._loop.run_until_complete(task)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=1)     return await main
(APIServer pid=1)            ^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 665, in run_server
(APIServer pid=1)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 679, in run_server_worker
(APIServer pid=1)     async with build_async_engine_client(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 99, in build_async_engine_client
(APIServer pid=1)     async with build_async_engine_client_from_engine_args(
(APIServer pid=1)                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=1)     return await anext(self.gen)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 135, in build_async_engine_client_from_engine_args
(APIServer pid=1)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=1)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=1)     return cls(
(APIServer pid=1)            ^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=1)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 131, in make_async_mp_client
(APIServer pid=1)     return AsyncMPClient(*client_args)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1)     return func(*args, **kwargs)
(APIServer pid=1)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 948, in __init__
(APIServer pid=1)     super().__init__(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 570, in __init__
(APIServer pid=1)     with launch_core_engines(
(APIServer pid=1)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1)     next(self.gen)
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1190, in launch_core_engines
(APIServer pid=1)     wait_for_engine_startup(
(APIServer pid=1)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1249, in wait_for_engine_startup
(APIServer pid=1)     raise RuntimeError(
(APIServer pid=1) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
/usr/lib/python3.12/multiprocessing/resource_tracker.py:279: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

Any thoughts as to why I'm seeing this error?

virtualroadside

Jun 15

Looks like there were fixes after 0.23.0 even though it just came out. vllm/vllm-openai:cu129-nightly-b8336c3c7c298e0878f22a7bf70f4e295b2f4e01 seems to load correctly.

virtualroadside changed discussion status to closed Jun 15

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment