Instructions to use ibm-granite/granite-20b-code-base-8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-20b-code-base-8k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-20b-code-base-8k")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-20b-code-base-8k")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-20b-code-base-8k")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ibm-granite/granite-20b-code-base-8k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-20b-code-base-8k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-20b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-20b-code-base-8k

SGLang

How to use ibm-granite/granite-20b-code-base-8k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-20b-code-base-8k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-20b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-20b-code-base-8k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-20b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ibm-granite/granite-20b-code-base-8k with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-20b-code-base-8k
```

granite-20b-code-base-8k

Commit History

Update README.md

8bbfb1d
verified

daviddcox commited on 29 days ago

update context length

c2a5bbb
verified

rpand002 commited on Sep 2, 2024

update to r1.1

0a81cba

mayank-mishra commited on Aug 30, 2024

granite tag

862809c
verified

mayank-mishra commited on May 10, 2024

update paper

f649927
verified

mayank-mishra commited on May 8, 2024

Update README.md

a1040eb
verified

mayank-mishra commited on May 7, 2024

Update README.md

dfb1b57
verified

mayank-mishra commited on May 6, 2024

Update README.md

caae741
verified

mayank-mishra commited on May 6, 2024

Update README.md

511b737
verified

mayank-mishra commited on May 6, 2024

Update README.md

13ab154
verified

mayank-mishra commited on May 6, 2024

update example

72d5ba9
verified

mayank-mishra commited on May 6, 2024

removed HelpSteer dataset

bcb550f
verified

amezasor commited on May 5, 2024

code comments removed

3234b38
verified

amezasor commited on May 5, 2024

metadata update

c57a393
verified

amezasor commited on May 4, 2024

model summary update

c6cddd9
verified

amezasor commited on May 4, 2024

fixed model name in generation section

8acb183
verified

amezasor commited on May 3, 2024

fixed model name in how to section

e87769a
verified

amezasor commited on May 3, 2024

First commit granite-20b-code-base model card

a37be0b
verified

amezasor commited on May 3, 2024

upload tokenizer

8de579d

Mayank Mishra commited on May 3, 2024

upload model

7267be4

mayank-mishra commited on May 3, 2024

initial commit

8747afc
verified

mayank-mishra commited on Apr 21, 2024

Commit History

Update README.md 8bbfb1d verified

update context length c2a5bbb verified

update to r1.1 0a81cba

granite tag 862809c verified

update paper f649927 verified

Update README.md a1040eb verified

Update README.md dfb1b57 verified

Update README.md caae741 verified

Update README.md 511b737 verified

Update README.md 13ab154 verified

update example 72d5ba9 verified

removed HelpSteer dataset bcb550f verified

code comments removed 3234b38 verified

metadata update c57a393 verified

model summary update c6cddd9 verified

fixed model name in generation section 8acb183 verified

fixed model name in how to section e87769a verified

First commit granite-20b-code-base model card a37be0b verified

upload tokenizer 8de579d

upload model 7267be4

initial commit 8747afc verified

Update README.md

8bbfb1d
verified

update context length

c2a5bbb
verified

update to r1.1

0a81cba

granite tag

862809c
verified

update paper

f649927
verified

Update README.md

a1040eb
verified

Update README.md

dfb1b57
verified

Update README.md

caae741
verified

Update README.md

511b737
verified

Update README.md

13ab154
verified

update example

72d5ba9
verified

removed HelpSteer dataset

bcb550f
verified

code comments removed

3234b38
verified

metadata update

c57a393
verified

model summary update

c6cddd9
verified

fixed model name in generation section

8acb183
verified

fixed model name in how to section

e87769a
verified

First commit granite-20b-code-base model card

a37be0b
verified

upload tokenizer

8de579d

upload model

7267be4

initial commit

8747afc
verified