JSON output not supported?
#57
by hfuserash - opened
I am using vLLM and Nemo Instruct and playing around with the new model (fp8) and even with vLLMs guided_json and including the schema in prompt, nemo almost never returns a output if JSON is involved however it does finish generating a output that is usually a letter or a symbol at the most.
Does Mistral Nemo not support JSON output at all?
This works as expected with instruct-v0.3
This is what I get using the inference API
write a sample JSON response for a user info query,
{
"status": "success",
"data": {
"user_id": 1,
"name": "Julien",
"email": "julien@example.com",
"created_at": "2022-01-01T00:00:00Z",
"updated_at": "2022-01-02T00:00:00Z"
}
}
is it valid to assume this is a vLLM issue?
This is an issue with guided_json extra_body param with vLLM, everything else just works fine.
hfuserash changed discussion status to closed