Instructions to use ibm-granite/granite-34b-code-base-8k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-34b-code-base-8k with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-34b-code-base-8k")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-34b-code-base-8k")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-34b-code-base-8k")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ibm-granite/granite-34b-code-base-8k with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-34b-code-base-8k"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-34b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-34b-code-base-8k

SGLang

How to use ibm-granite/granite-34b-code-base-8k with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-34b-code-base-8k" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-34b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-34b-code-base-8k" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-34b-code-base-8k",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ibm-granite/granite-34b-code-base-8k with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-34b-code-base-8k
```

granite-34b-code-base-8k

Commit History

Update README.md

291a64f
verified

daviddcox commited on 28 days ago

update context length

1a00a60
verified

rpand002 commited on Sep 2, 2024

granite tag

89ac8e4
verified

mayank-mishra commited on May 10, 2024

update paper

32e171a
verified

mayank-mishra commited on May 8, 2024

Update README.md

b618164
verified

mayank-mishra commited on May 7, 2024

Update README.md

f9a1f24
verified

mayank-mishra commited on May 6, 2024

Update README.md

668e323
verified

mayank-mishra commited on May 6, 2024

Update README.md

261f160
verified

mayank-mishra commited on May 6, 2024

Update README.md

58e3fe3
verified

mayank-mishra commited on May 6, 2024

update example

d6dedc3
verified

mayank-mishra commited on May 6, 2024

removed HelpSteer dataset

6b5821c
verified

amezasor commited on May 5, 2024

metadata update

67a85e7
verified

amezasor commited on May 5, 2024

metadata update

fd89ca2
verified

amezasor commited on May 4, 2024

mode summary update

9c95ae0
verified

amezasor commited on May 4, 2024

mode summary update

6b11f07
verified

amezasor commited on May 4, 2024

First commit for granite-34b-code-base model card

f60e5ca
verified

amezasor commited on May 3, 2024

upload model

9c24586

Mayank Mishra commited on May 2, 2024

initial commit

dfd9dcf
verified

mayank-mishra commited on Apr 21, 2024

Commit History

Update README.md 291a64f verified

update context length 1a00a60 verified

granite tag 89ac8e4 verified

update paper 32e171a verified

Update README.md b618164 verified

Update README.md f9a1f24 verified

Update README.md 668e323 verified

Update README.md 261f160 verified

Update README.md 58e3fe3 verified

update example d6dedc3 verified

removed HelpSteer dataset 6b5821c verified

metadata update 67a85e7 verified

metadata update fd89ca2 verified

mode summary update 9c95ae0 verified

mode summary update 6b11f07 verified

First commit for granite-34b-code-base model card f60e5ca verified

upload model 9c24586

initial commit dfd9dcf verified

Update README.md

291a64f
verified

update context length

1a00a60
verified

granite tag

89ac8e4
verified

update paper

32e171a
verified

Update README.md

b618164
verified

Update README.md

f9a1f24
verified

Update README.md

668e323
verified

Update README.md

261f160
verified

Update README.md

58e3fe3
verified

update example

d6dedc3
verified

removed HelpSteer dataset

6b5821c
verified

metadata update

67a85e7
verified

metadata update

fd89ca2
verified

mode summary update

9c95ae0
verified

mode summary update

6b11f07
verified

First commit for granite-34b-code-base model card

f60e5ca
verified

upload model

9c24586

initial commit

dfd9dcf
verified