How to properly return function call return values
Hello,
I have a simple program working with this model but realized that I wasn't sure how to send the function's output back to the model. Do you just put the raw value back as part of the user message or does it need to be wrapped in a special tag so the LLM knows that it is a response from the last function call?
Thanks,
Jason O
Two options (depending on what kind of function you're using):
A. Replace the json in the assistant's message with the function call response. (if your function returns a clean answer that is useful for the user).
B. Add the function response as a new user message, prepended by "Here is the response to that function call:\n\n{function_response}", then let the llm generate a new assistant response now that it has the required data. After getting back this new assistant response, you can optionally drop the function call message and the function response message from the array of messages (so that the user is blind to the entire process).
Thank you for getting back with me!
I tried approach B and it seems to kind of work. But I just realized that I may not be formatting my prompts correctly and wanted to confirm with you if I am doing it correctly.
So far, I'm able to get the model to send function calls and I'm able to execute them and return the value. (How the model is responding is another story that I'll take a look at next). My first question is about the use of the tags and their placement.
Example with 1 user prompt:
<FUNCTIONS>
</FUNCTIONS>
[INST]<<SYS>>
System Prompt here
<</SYS>>
User Prompt 1 [/INST]
Example with two user prompts:
<FUNCTIONS>
</FUNCTIONS>
<s>[INST]<<SYS>>
System Prompt here
<</SYS>>
User Prompt 1 [/INST] Assistant response here</s>
[INST]User Prompt 2 [/INST]
Example with three user prompts:
<FUNCTIONS>
</FUNCTIONS>
<s>[INST]<<SYS>>
System Prompt here
<</SYS>>
User Prompt 1 [/INST] Assistant response 1</s>
<s>[INST]User Prompt 2 [/INST] Assistant response 2</s>
[INST]User Prompt 3 [/INST]
Example with four user prompts:
<FUNCTIONS>
</FUNCTIONS>
<s>[INST]<<SYS>>
System Prompt here
<</SYS>>
User Prompt 1 [/INST] Assistant response 1</s>
<s>[INST]User Prompt 2 [/INST] Assistant response 2</s>
<s>[INST]User Prompt 3 [/INST] Assistant response 3</s>
[INST]User Prompt 4 [/INST]
Does this look correct?
Yeah, that looks about right.
BTW, with Llama it's common to have a space after [INST] and before [/INST], so it would be:
[INST] {prompt} [/INST]
You can see that in the model card.