Return invalid_question for malformed span thresholds and instructions

#3
by subin - opened
vLLM Semantic Router org

Most malformed span questions get a per-question invalid_question answer while the rest of the request is answered. Two did not: "threshold": null reached float(None), and instructions given as an object reached the prompt builder's "\n".join. Both raised TypeError out of system_one, so the whole request failed, including the valid questions next to it.

span_answer now checks both before running the model. Instructions must be a string or absent, and a threshold must be a number or a string, since numeric strings were already accepted through float() and a non-numeric one already got invalid_question. Anything else gets invalid_question with "span instructions must be a string" or "threshold must be a number".

Tested on vela3-edge on CPU with a yes/no question next to each malformed span question. A null threshold and object instructions now return invalid_question and the yes/no question is still answered (0.962, as with the other malformed cases). Thresholds of 0.3, 1 and "0.3" and empty instructions give the same answers as before. The change is the same in all six sizes.

vLLM Semantic Router org

Thanks, merged.

adaamko changed pull request status to merged

Sign up or log in to comment