Instructions to use ibm-granite/granite-8b-code-base-4k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-8b-code-base-4k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-8b-code-base-4k")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-8b-code-base-4k")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-8b-code-base-4k")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ibm-granite/granite-8b-code-base-4k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-8b-code-base-4k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-8b-code-base-4k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-8b-code-base-4k

SGLang

How to use ibm-granite/granite-8b-code-base-4k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-8b-code-base-4k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-8b-code-base-4k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-8b-code-base-4k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-8b-code-base-4k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ibm-granite/granite-8b-code-base-4k with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-8b-code-base-4k
```

granite-8b-code-base-4k

Commit History

Deprecation notice

d21e874
verified

daviddcox commited on 25 days ago

update context length

8210a41
verified

rpand002 commited on Sep 2, 2024

Update README.md

08fdeca
verified

mayank-mishra commited on Jul 9, 2024

granite tag

f68b973
verified

mayank-mishra commited on May 10, 2024

Update README.md

145f47b
verified

mayank-mishra commited on May 8, 2024

Update README.md

fb59940
verified

mayank-mishra commited on May 7, 2024

Update README.md

6afdc02
verified

mayank-mishra commited on May 7, 2024

Update README.md

6c3c44f
verified

mayank-mishra commited on May 6, 2024

Update README.md

cd33711
verified

mayank-mishra commited on May 6, 2024

Update README.md

c7e28c0
verified

mayank-mishra commited on May 6, 2024

add warning

0aee173
verified

mayank-mishra commited on May 6, 2024

disable inference

401644c
verified

mayank-mishra commited on May 6, 2024

Update README.md

9e0eb38
verified

mayank-mishra commited on May 6, 2024

update example

78c2269
verified

mayank-mishra commited on May 6, 2024

removed HelpSteer dataset

cc940cd
verified

amezasor commited on May 5, 2024

code comments removed

3a6d5f0
verified

amezasor commited on May 5, 2024

metadata update

0016d33
verified

amezasor commited on May 4, 2024

model summary update

ea2cd2d
verified

amezasor commited on May 4, 2024

p3

9e5f74e

mayank-mishra commited on May 3, 2024

fixed model name in generation section

d7f06ab
verified

amezasor commited on May 3, 2024

fix model size

1a8823e
verified

amezasor commited on May 3, 2024

First commit granite-8b-code-base model card

fa42e7b
verified

amezasor commited on May 3, 2024

llama

c76e138

mayank-mishra commited on May 3, 2024

downcast to bf16

c33ce4f

mayank-mishra commited on May 2, 2024

upload model

0e6c38f

Mayank Mishra commited on Apr 24, 2024

initial commit

e82da12
verified

mayank-mishra commited on Apr 21, 2024

Commit History

Deprecation notice d21e874 verified

update context length 8210a41 verified

Update README.md 08fdeca verified

granite tag f68b973 verified

Update README.md 145f47b verified

Update README.md fb59940 verified

Update README.md 6afdc02 verified

Update README.md 6c3c44f verified

Update README.md cd33711 verified

Update README.md c7e28c0 verified

add warning 0aee173 verified

disable inference 401644c verified

Update README.md 9e0eb38 verified

update example 78c2269 verified

removed HelpSteer dataset cc940cd verified

code comments removed 3a6d5f0 verified

metadata update 0016d33 verified

model summary update ea2cd2d verified

p3 9e5f74e

fixed model name in generation section d7f06ab verified

fix model size 1a8823e verified

First commit granite-8b-code-base model card fa42e7b verified

llama c76e138

downcast to bf16 c33ce4f

upload model 0e6c38f

initial commit e82da12 verified

Deprecation notice

d21e874
verified

update context length

8210a41
verified

Update README.md

08fdeca
verified

granite tag

f68b973
verified

Update README.md

145f47b
verified

Update README.md

fb59940
verified

Update README.md

6afdc02
verified

Update README.md

6c3c44f
verified

Update README.md

cd33711
verified

Update README.md

c7e28c0
verified

add warning

0aee173
verified

disable inference

401644c
verified

Update README.md

9e0eb38
verified

update example

78c2269
verified

removed HelpSteer dataset

cc940cd
verified

code comments removed

3a6d5f0
verified

metadata update

0016d33
verified

model summary update

ea2cd2d
verified

p3

9e5f74e

fixed model name in generation section

d7f06ab
verified

fix model size

1a8823e
verified

First commit granite-8b-code-base model card

fa42e7b
verified

llama

c76e138

downcast to bf16

c33ce4f

upload model

0e6c38f

initial commit

e82da12
verified