How to control Hallucinations?
#1
by rallapallinagarjun - opened
I am trying to use this GGUF file for processing some documents and get the output in JSON, but the model is responding with non-JSON data!
Llama 3.1 GGUF responds with JSON output!
Can you please tell me why this is happening?
Try using using a grammar file to constrain the output to JSON . llama.cpp supports this
I am trying to use this GGUF file for processing some documents and get the output in JSON, but the model is responding with non-JSON data!
Llama 3.1 GGUF responds with JSON output!
Can you please tell me why this is happening?
As well as using a gramma file . It might be better to use the instruct version of this model . This repo is the base version . Try https://huggingface.co/Mungert/Foundation-Sec-8B-Instruct-GGUF