Inference Endpoints throwing an error
Hi,
I'm trying to run the model through HF Inference Endpoints for a quick POC. I'm running into this particular issue:
2023/10/19 17:56:21 ~ INFO | No custom pipeline found at /repository/handler.py
2023/10/19 17:56:21 ~ INFO | Using device GPU
2023/10/19 17:56:21 ~ 2023-10-19 15:56:21,563 | INFO | Initializing model from directory:/repository
2023/10/19 17:56:21 ~ KeyError: 'llava_mistral'
2023/10/19 17:56:21 ~ self.pipeline = get_pipeline(model_dir=model_dir, task=task)
2023/10/19 17:56:21 ~ inference_handler = get_inference_handler_either_custom_or_default_handler(HF_MODEL_DIR, task=HF_TASK)
2023/10/19 17:56:21 ~ config = AutoConfig.from_pretrained(model, _from_pipeline=task, **hub_kwargs, **model_kwargs)
2023/10/19 17:56:21 ~ File "/app/huggingface_inference_toolkit/handler.py", line 17, in init
2023/10/19 17:56:21 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 710, in getitem
2023/10/19 17:56:21 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/models/auto/configuration_auto.py", line 998, in from_pretrained
2023/10/19 17:56:21 ~ File "/app/huggingface_inference_toolkit/handler.py", line 45, in get_inference_handler_either_custom_or_default_handler
2023/10/19 17:56:21 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 584, in aenter
2023/10/19 17:56:21 ~ async with self.lifespan_context(app) as maybe_state:
2023/10/19 17:56:21 ~ File "/app/huggingface_inference_toolkit/utils.py", line 261, in get_pipeline
2023/10/19 17:56:21 ~ await handler()
2023/10/19 17:56:21 ~ File "/opt/conda/lib/python3.9/site-packages/transformers/pipelines/init.py", line 705, in pipeline
2023/10/19 17:56:21 ~ await self._router.startup()
2023/10/19 17:56:21 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 705, in lifespan
2023/10/19 17:56:21 ~ Traceback (most recent call last):
2023/10/19 17:56:21 ~ File "/app/webservice_starlette.py", line 57, in some_startup_task
2023/10/19 17:56:21 ~ return HuggingFaceHandler(model_dir=model_dir, task=task)
2023/10/19 17:56:21 ~ File "/opt/conda/lib/python3.9/site-packages/starlette/routing.py", line 682, in startup
2023/10/19 17:56:21 ~ Application startup failed. Exiting.
Can you let me know if there's anything going wrong with my setup?
Thanks!
@kk-envision i made a notebook which you can use for the inference.
I have no idea what is "Inference Endpoints", but you can make your own API with this.
Let me know if this was helpful.
@kk-envision simply put transformers doesnt have mistral_llava support. You have to use the bakllava repository