trufae/granite3-instruct-8b-gguf

Granite3 GGUF

Source: ibm-granite/granite-3.0-8b-instruct

Setup:

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
python -m venv venv
. venv/bin/activate
pip install .

Download the tensors:

from huggingface_hub import snapshot_download
model_id="ibm-granite/granite-3.0-8b-instruct"
snapshot_download(repo_id=model_id, local_dir="granite3",
                  local_dir_use_symlinks=False, revision="main")

Build the gguf:

python convert_hf_to_gguf.py --outfile granite3.gguf --outtype q8_0 granite3

Test with r2ai

r2ai -m "$PWD/granite3.gguf"