Unable to load LANSG/GEM model in vLLM Docker container
Hello! I tried to run the LANSG/GEM model using the latest vllm/vllm-openai Docker container (vLLM also tried). The container starts, but fails to load the model with the following error:
Status: Downloaded newer image for vllm/vllm-openai:latest
INFO 09-29 06:02:08 [init.py:241] Automatically detected platform cuda.
(APIServer pid=1) INFO 09-29 06:02:10 [api_server.py:1805] vLLM API server version 0.10.1.1
(APIServer pid=1) INFO 09-29 06:02:10 [utils.py:326] non-default args: {'model': 'LANSG/GEM'}
(APIServer pid=1) Traceback (most recent call last):
(APIServer pid=1) File "", line 198, in _run_module_as_main
(APIServer pid=1) File "", line 88, in _run_code
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1920, in
(APIServer pid=1) uvloop.run(run_server(args))
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 109, in run
(APIServer pid=1) return __asyncio.run(
(APIServer pid=1) ^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
(APIServer pid=1) return runner.run(main)
(APIServer pid=1) ^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=1) return self._loop.run_until_complete(task)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/uvloop/init.py", line 61, in wrapper
(APIServer pid=1) return await main
(APIServer pid=1) ^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1850, in run_server
(APIServer pid=1) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1870, in run_server_worker
(APIServer pid=1) async with build_async_engine_client(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 178, in build_async_engine_client
(APIServer pid=1) async with build_async_engine_client_from_engine_args(
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/lib/python3.12/contextlib.py", line 210, in aenter
(APIServer pid=1) return await anext(self.gen)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 204, in build_async_engine_client_from_engine_args
(APIServer pid=1) vllm_config = engine_args.create_engine_config(usage_context=usage_context)
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 1057, in create_engine_config
(APIServer pid=1) model_config = self.create_model_config()
(APIServer pid=1) ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 904, in create_model_config
(APIServer pid=1) return ModelConfig(
(APIServer pid=1) ^^^^^^^^^^^^
(APIServer pid=1) File "/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py", line 123, in init
(APIServer pid=1) s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
(APIServer pid=1) pydantic_core._pydantic_core.ValidationError: 1 validation error for ModelConfig
(APIServer pid=1) Value error, The checkpoint you are trying to load has model type llava_llama but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
(APIServer pid=1)
(APIServer pid=1) You can update Transformers with the command pip install --upgrade transformers. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command pip install git+https://github.com/huggingface/transformers.git [type=value_error, input_value=ArgsKwargs((), {'model': ...gits_processors': None}), input_type=ArgsKwargs]
(APIServer pid=1) For further information visit https://errors.pydantic.dev/2.11/v/value_error
It seems that the Transformers library included in the container does not support the llava_llama model type yet. Updating Transformers inside the container or installing it from source might resolve this issue. Thank you!
Same error here, is impossible to use from transformer library
GEM Model - Quick Fix for Common Errors
Hey everyone! 👋 Seeing lots of people struggling with GEM, so here's a quick fix:
The Problem
Getting these errors?
'NoneType' object has no attribute 'ndim'cache_positionerrorsUnknown ecg towerwarnings
Quick Solution
1. Use the right transformers version:
pip install transformers==4.37.2
Newer versions break stuff!
2. The ECG checkpoint is missing - that's normal. The model still works for ECG images, just not raw ECG signals.
3. Can't use standard HF pipelines - GEM needs its custom LLaVA setup.
Working Code
Clone the repo and use this:
import torch
from llava.model.builder import load_pretrained_model
from llava.mm_utils import process_images, tokenizer_image_token
from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
# Load model
tokenizer, model, image_processor, _ = load_pretrained_model(
"LANSG/GEM", None, "llava_llama",
device_map="auto", torch_dtype=torch.float16
)
# Use it
def analyze_ecg(image):
image_tensor = process_images([image], image_processor, model.config)
prompt = DEFAULT_IMAGE_TOKEN + "\nAnalyze this ECG"
input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt')
with torch.inference_mode():
output = model.generate(
input_ids.unsqueeze(0).to(model.device),
images=image_tensor.to(model.device, dtype=torch.float16),
ecgs=None, # Important!
max_new_tokens=300
)
return tokenizer.decode(output[0, input_ids.shape[0]:]).strip()
What Works
✅ ECG image analysis
✅ Clinical interpretation
❌ Raw ECG signal processing (missing checkpoint)
That's it! Model works great for ECG images. If you want something simpler, try PULSE-7B instead.
Hope this helps! 🩺