How to use the inference API correctly?
Based on the docs, the following JSON input (tested via Postman) should work:
Post Endpoint: https://api-inference.huggingface.co/models/CohereForAI/c4ai-command-r-plus
{
"inputs": {
"role": "user",
"content": "Who is Elon Musk?"
}
}
But I get the error Failed to deserialize the JSON body into the target type: inputs: invalid type: map, expected a string at line 2 column 12
What's the expected format?
good
Sorry for unrelated question, but is it possible to use command-r-+ with hf inference api with pro subscription although it is 103B?
Okay I got this to work after I read the docs properly and did more research.
Out of the box it's just a model. What I'm after is a chat-like template. Hugging Face offers something called TGI (Text Generation Inference): https://huggingface.co/docs/text-generation-inference/en/index
To use it, you need to specify it in a request like so using OpenAI's chat completion API: https://huggingface.co/blog/llama3#how-to-prompt-llama-3
from openai import OpenAI
# initialize the client but point it to TGI
client = OpenAI(
base_url="https://api-inference.huggingface.co/models/CohereForAI/c4ai-command-r-plus" + "/v1/", # replace with your endpoint url
api_key="<HF_API_TOKEN>", # replace with your token
)
chat_completion = client.chat.completions.create(
model="tgi",
messages=[
{"role": "user", "content": "Why is open-source software important?"},
],
stream=True,
max_tokens=500
)
# iterate and print stream
for message in chat_completion:
print(message.choices[0].delta.content, end="")
This works.
Or, to use this exact example with postman.
Post Endpoint: https://api-inference.huggingface.co/models/CohereForAI/c4ai-command-r-plus/v1/chat/completions
JSON body:
{
"model": "tgi",
"messages": [
{"role": "user", "content": "Who is Elon Musk?"}
],
"stream": false,
"max_tokens": 500
}
Reponse:
{
"id": "",
"object": "text_completion",
"created": 1714021803,
"model": "text-generation-inference/commandrplus-medusa",
"system_fingerprint": "1.4.5-native",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Elon Musk is a business magnate, investor, and philanthropist. He is the founder, CEO, and Chief Engineer of SpaceX; angel investor, CEO and Product Architect of Tesla, Inc.; owner and CEO of Twitter, Inc.; founder of the Boring Company; co-founder of Neuralink; and president of the philanthropic Musk Foundation. With an estimated net worth of around $191 billion, Musk is the second-wealthiest person in the world, behind Bernard Arnault as of August 3, 2023."
},
"logprobs": null,
"finish_reason": "eos_token"
}
],
"usage": {
"prompt_tokens": 11,
"completion_tokens": 109,
"total_tokens": 120
}
}
Closing it as it seems resolved!