tool calls sometimes fail on vllm
Really fast model, but periodically tool calling seems to fail, the model may need its own vllm parser?
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] Error in extracting tool call from response.
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] Traceback (most recent call last):
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] File "/usr/local/lib/python3.12/dist-packages/vllm/tool_parsers/hermes_tool_parser.py", line 146, in extract_tool_calls
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] json.loads(match[0] if match[0] else match[1])
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] File "/usr/lib/python3.12/json/init.py", line 346, in loads
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] return _default_decoder.decode(s)
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] File "/usr/lib/python3.12/json/decoder.py", line 338, in decode
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] obj, end = self.raw_decode(s, idx=_w(s, 0).end())
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] File "/usr/lib/python3.12/json/decoder.py", line 354, in raw_decode
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] obj, end = self.scan_once(s, idx)
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] ^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1) ERROR 01-24 04:23:59 [hermes_tool_parser.py:171] json.decoder.JSONDecodeError: Unterminated string starting at: line 2 column 73 (char 73)