add function call example
Browse files
README.md
CHANGED
|
@@ -55,6 +55,8 @@ huggingface-cli download internlm/internlm2_5-7b-chat-gguf internlm2_5-7b-chat-f
|
|
| 55 |
|
| 56 |
You can use `llama-cli` for conducting inference. For a detailed explanation of `llama-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
|
| 57 |
|
|
|
|
|
|
|
| 58 |
```shell
|
| 59 |
build/bin/llama-cli \
|
| 60 |
--model internlm2_5-7b-chat-fp16.gguf \
|
|
@@ -76,6 +78,49 @@ build/bin/llama-cli \
|
|
| 76 |
--in-suffix "<|im_end|>\n<|im_start|>assistant\n"
|
| 77 |
```
|
| 78 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 79 |
## Serving
|
| 80 |
|
| 81 |
`llama.cpp` provides an OpenAI API compatible server - `llama-server`. You can deploy `internlm2_5-7b-chat-fp16.gguf` into a service like this:
|
|
|
|
| 55 |
|
| 56 |
You can use `llama-cli` for conducting inference. For a detailed explanation of `llama-cli`, please refer to [this guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/main/README.md)
|
| 57 |
|
| 58 |
+
### chat example
|
| 59 |
+
|
| 60 |
```shell
|
| 61 |
build/bin/llama-cli \
|
| 62 |
--model internlm2_5-7b-chat-fp16.gguf \
|
|
|
|
| 78 |
--in-suffix "<|im_end|>\n<|im_start|>assistant\n"
|
| 79 |
```
|
| 80 |
|
| 81 |
+
### Function call example
|
| 82 |
+
|
| 83 |
+
`llama-cli` example:
|
| 84 |
+
|
| 85 |
+
```shell
|
| 86 |
+
build/bin/llama-cli \
|
| 87 |
+
--model internlm2_5-7b-chat-fp16.gguf \
|
| 88 |
+
--predict 512 \
|
| 89 |
+
--ctx-size 4096 \
|
| 90 |
+
--gpu-layers 32 \
|
| 91 |
+
--temp 0.8 \
|
| 92 |
+
--top-p 0.8 \
|
| 93 |
+
--top-k 50 \
|
| 94 |
+
--seed 1024 \
|
| 95 |
+
--color \
|
| 96 |
+
--prompt '<|im_start|>system\nYou are InternLM2-Chat, a harmless AI assistant.<|im_end|>\n<|im_start|>system name=<|plugin|>[{"name": "get_current_weather", "parameters": {"required": ["location"], "type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string"}}}, "description": "Get the current weather in a given location"}]<|im_end|>\n<|im_start|>user\n' \
|
| 97 |
+
--interactive \
|
| 98 |
+
--multiline-input \
|
| 99 |
+
--conversation \
|
| 100 |
+
--verbose \
|
| 101 |
+
--in-suffix "<|im_end|>\n<|im_start|>assistant\n" \
|
| 102 |
+
--special
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
Conversation results:
|
| 106 |
+
|
| 107 |
+
```text
|
| 108 |
+
<s><|im_start|>system
|
| 109 |
+
You are InternLM2-Chat, a harmless AI assistant.<|im_end|>
|
| 110 |
+
<|im_start|>system name=<|plugin|>[{"name": "get_current_weather", "parameters": {"required": ["location"], "type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string"}}}, "description": "Get the current weather in a given location"}]<|im_end|>
|
| 111 |
+
<|im_start|>user
|
| 112 |
+
|
| 113 |
+
> I want to know today's weather in Shanghai
|
| 114 |
+
I need to use the get_current_weather function to get the current weather in Shanghai.<|action_start|><|plugin|>
|
| 115 |
+
{"name": "get_current_weather", "parameters": {"location": "Shanghai"}}<|action_end|>
|
| 116 |
+
<|im_end|>
|
| 117 |
+
|
| 118 |
+
> <|im_start|>environment name=<|plugin|>\n{"temperature": 22}
|
| 119 |
+
The current temperature in Shanghai is 22 degrees Celsius.<|im_end|>
|
| 120 |
+
|
| 121 |
+
>
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
## Serving
|
| 125 |
|
| 126 |
`llama.cpp` provides an OpenAI API compatible server - `llama-server`. You can deploy `internlm2_5-7b-chat-fp16.gguf` into a service like this:
|