Instructions to use ibm-granite/granite-3.1-2b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-3.1-2b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-3.1-2b-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3.1-2b-instruct")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-3.1-2b-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-3.1-2b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-3.1-2b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.1-2b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-3.1-2b-instruct

SGLang

How to use ibm-granite/granite-3.1-2b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-3.1-2b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.1-2b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-3.1-2b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.1-2b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-3.1-2b-instruct with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-3.1-2b-instruct
```

RE-ADD float32 please.

by ctranslate2-4you - opened Dec 28, 2024

Discussion

ctranslate2-4you

Dec 28, 2024

A lot of use prefer to use float32 versions of these kinds of models. The reason is, if a model is originally in float32 we can convert at runtime (e.g. using bitsandbytes) into either float16 or bfloat16 depending on whether a user has cuda compute level 8.6 or above (as required for bfloat16). Also, having it in float32 allows users to run it on a CPU natively rather than having to convert at runtime a bfloat16 model to float32 (as vanilla CPU usage requires float32).

Anytime a model is converted from bfloat16/float16 to float32 there is accuracy loss...Likewise, if a model is converted to float16 from bfloat16 (and vice versa) there is accuracy loss since the precisions have a different floating point format...

In summary...there are still use cases for keeping the model in float32. What I'm asking for is:

(1) Keep float32 versions of things on the repository for this model and any other Granite models...this is IN ADDITION to float16, bfloat16 or whatever other precisions you want to have...

(2) This allows people to download specific versions that they want to use.

quantflex

Jan 2, 2025

@ctranslate2-4you it's still in the commit history.

quantflex

Jan 2, 2025

but yes, I agree.

mayank-mishra

Jan 8, 2025

@ctranslate2-4you the model was saved in bf16, so the fp32 model is just an upcast.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment