Need sample format for custom function with context.
Hello,
I am able to download the model and run the model successfully with the below command.
"""
model=Trelis/Mistral-7B-Instruct-v0.1-function-calling-v2
docker run -d --gpus "device=1" --shm-size 1g -e HUGGING_FACE_HUB_TOKEN=$token -p 6691:80 -v $volume:/data ghcr.io/huggingface/text-generation-inference:latest --model-id $model --max-input-length 4000 --max-total-tokens 4096
"""
Inference code :
"""
from text_generation import Client
client = Client("http://127.0.0.1:6691", timeout=100)
text = ""
data = client.generate(text, max_new_tokens=700).generated_text
"""
I am expecting the response in json format with below conversation, prompt and system prompt. Could you please send me the sample function calling format to the TGI inference API.?
Input context :
customer : Hello hello is anyone there this is amanda here can you hear me. ,agent : Yes you are audible amanda first of all good morning to you my name is robert how may i help you. ,customer : Yeah hi robert so i have actually lost my phone and and this is pretty urgent i have a very important meeting to go to can you help me with this i am calling from a payphone here i don't know what exactly do so yeah tell me tell me what to do help me with. ,agent : Certainly amanda so first of all i would need to check some details so first of all you have to tell me the number that you are calling from is it the same number. ,customer : Yeah no this is this is a pay phone i just told you right the payphone this is not my number. ,agent : Oh i am so sorry i think i missed out on that so just tell me the name and some more details like how you lost it. ,customer : Last i remember i i left my house and i joke to my office and i stopped trying between at the grocery store to get the job after coming back to my car that i don't have it i check in the grocery store they are saying they don't know anything about it they are not ready to take any kind of responsibilities about this and i feel like it has been it has been stoll and there itself but i do not know how to go forward with this and. ,customer : Yeah it is pretty much about how i lost me. ,agent : I am so sorry for this just tell me your number that was on the last phone. ,customer : Yeah so the number is if you can note it down it is. ,customer : One one one. ,customer : Nine seven eight three. ,customer : One zero. ,customer : One zero two. ,agent : Great i have not in the number can you just repeat the number again. ,customer : It is one one one nine seven eight three one zero two. ,agent : Alright please hold the line for a while let me check the details for you. ,agent : Thanks for the long hold as i can check in our system i think i need to know a couple of things from your side could you please confirm. ,agent : Your name and the last dial number on your phone. ,customer : Okay that's going to be hard so my full name is amanda rama krishnan and the last number i dialed. ,customer : Last number that i did was i need. ,customer : Yeah my house me but i don't know her number how i know in these days you don't you never remember your number how do i. ,agent : Alright let me help you with this. ,agent : Do you remember the last person you called maybe your mother or your friend. ,
prompt:
how empathetic the agent is in the given conversation on a score of 0 to 10 with explanation.
system_prompt:
You are a call quality analyst. You should assess the agent's performance.
Expected sample output:
{
"score": 8,
"explanation": "The agent is empathetic towards the customer and is actively trying to help the customer find their lost phone. The agent is patient and understanding with the customer, and is taking the time to gather all the necessary information to assist the customer. The agent is also apologetic for the inconvenience caused to the customer and is doing their best to provide a solution."
}
Thanks,
Santhosh.
Hi Santhosh.
I’m confused and maybe I’m misunderstanding your question…
A) the custom function format is on the model card for this repo if you scroll down
B) there is a link for tgi that brings you to a GitHub repo with instructions for detailed setup.
Are you asking something else?
Hello,
I tried with 20 samples for getting sentiment of the conversation but there is no consistency on json response.
Below is the code for it. Please asses if there are any changes required in the format.
sentiment_meta_data = {
"function": "get_sentiment",
"description": "what is the sentiment of the overall call conversation",
"arguments": [
{
"name": "score",
"type": "float",
"description": "score on sentiment on a scale of 0 to 100"
},
{
"name": "explanation",
"type": "string",
"description": "explanation on sentiment in the given conversation"
}
]
}
functionList = ''
functionList += json.dumps(search_bing_metadata, indent=4, separators=(',', ': '))
print(functionList)
def mistral_json_format(context):
# Define the roles and markers
B_INST, E_INST = "[INST]", "[/INST]"
B_FUNC, E_FUNC = "", "\n\n"
B_SYS, E_SYS = "<>\n", "\n<>\n\n"
# assuming functionList is defined as above
system_prompt = 'You are a call qulaity analyst. Responsible for assess the agent performance'
user_prompt = 'what is the overall sentiment score in the given conversation on score of range of 0 to 100 with explanation'
# Format your prompt template
prompt = f"{B_FUNC}{functionList.strip()}{E_FUNC}{B_INST} {B_SYS}{system_prompt.strip()}{E_SYS}{user_prompt.strip()} {E_INST} context: {context}\n\n"
return prompt
context = "sample conversation ...."
Results:
1.
{'function': 'get_sentiment', 'arguments': {'score': 75, 'explanation': "The sentiment of the conversation is positive. The agent was able to resolve the customer's issue and provide helpful information. The customer was also satisfied with the resolution."}}
2.{'sentiment_score': 75, 'explanation': 'The sentiment of the conversation is positive. The customer was able to get the information they needed and the agent was able to provide helpful information. The customer was also able to express their frustration with the wait time, but the agent was able to address their concerns and provide a solution. Overall, the conversation was productive and the customer was satisfied with the outcome.'}
3. some are failed ...
The above results shows no consistency in the response.
Please check and resolve it.
Thanks,
santhosh
1.
# Format your prompt template
prompt = f"{B_FUNC}{functionList.strip()}{E_FUNC}{B_INST} {B_SYS}{system_prompt.strip()}{E_SYS}{user_prompt.strip()} {E_INST} context: {context}\n\n"
Any user input needs to be within B_INST and E_INST, not afterwards. i.e. this should be:
prompt = f"{B_FUNC}{functionList.strip()}{E_FUNC}{B_INST} {B_SYS}{system_prompt.strip()}{E_SYS}{user_prompt.strip()}\n\ncontext: {context} {E_INST}\n\n"
2.
I don't know why there is this line referring to search bing:
functionList += json.dumps(search_bing_metadata, indent=4, separators=(',', ': '))
I'm unclear what your function metadata is and I don't see where function list is correctly set.
3.
Thanks for your patience with any further queries as I have limited bandwidth for support. If you have further questions, please provide complete code and an example for replication.
On a separate note, perhaps what you are doing here requires significantly more than function calling (although you should indeed be getting back correct json objects).
Function calling trains a model to return a structured response. Here, the model is additionally required to grade the inputs and then put that info in the function call. To achieve this you may need a combination of:
- A very strong model (maybe 70B Llama or Falcon 180B)
- Much better prompting. You need to give some examples (maybe 3-5) to the language model of how to grade different inputs. It's very unlikely to be able to grade without this background.
- If #2 doesn't work, you may even need to go beyond prompting with some fine-tuning, although ideally you want to avoid this.
Hello,
Please check the below sample code.
https://colab.research.google.com/drive/1wbTP0pNOut_eHbFrdQWDgZZ95BEzv025?usp=sharing
Thanks,
Santhosh
howdy, i'm on holidays for 10 days and didn't get to read in detail.
Your colab is very helpful though.
It seems that most calls are working, and the ones that aren't are pre-pending [FUNCTION].
My thoughts:
I've never seen that [FUNCTION] pre-pend before. One tweak worth trying is to remove the space here in the prompt '{E_INST} \n\n'. There should be no space before \n\n if you match syntax exactly. Small differences like this can have an effect.
Fix typos in your system message and instruction.
You can consider just extracting the json from the response, which will cut out anything prepended like [FUNCTION]. This is a bit hacky but may work. Alternately, yes, if you fine-tune with your specific function I believe there is a strong chance that will fix the issue.
All of what I say above is about getting the right syntax. If the contents of the json are not good quality enough, that indicates a strong base model is needed.