Instructions to use Mungert/granite-7b-base-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Mungert/granite-7b-base-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Mungert/granite-7b-base-GGUF",
	filename="granite-7b-base-bf16.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Mungert/granite-7b-base-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Mungert/granite-7b-base-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Mungert/granite-7b-base-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Mungert/granite-7b-base-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Mungert/granite-7b-base-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Mungert/granite-7b-base-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Mungert/granite-7b-base-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Mungert/granite-7b-base-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Mungert/granite-7b-base-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Mungert/granite-7b-base-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use Mungert/granite-7b-base-GGUF with Ollama:
```
ollama run hf.co/Mungert/granite-7b-base-GGUF:Q4_K_M
```

Unsloth Studio

How to use Mungert/granite-7b-base-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Mungert/granite-7b-base-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Mungert/granite-7b-base-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Mungert/granite-7b-base-GGUF to start chatting

Docker Model Runner
How to use Mungert/granite-7b-base-GGUF with Docker Model Runner:
```
docker model run hf.co/Mungert/granite-7b-base-GGUF:Q4_K_M
```

Lemonade

How to use Mungert/granite-7b-base-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Mungert/granite-7b-base-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.granite-7b-base-GGUF-Q4_K_M

List all available models

lemonade list

A newer version of this model is available: ibm-granite/granite-3.3-8b-base

granite-7b-base GGUF Models

Model Generation Details

This model was generated using llama.cpp at commit 5dd942de.

Quantization Beyond the IMatrix

I've been experimenting with a new quantization approach that selectively elevates the precision of key layers beyond what the default IMatrix configuration provides.

In my testing, standard IMatrix quantization underperforms at lower bit depths, especially with Mixture of Experts (MoE) models. To address this, I'm using the --tensor-type option in llama.cpp to manually "bump" important layers to higher precision. You can see the implementation here:
👉 Layer bumping with llama.cpp

While this does increase model file size, it significantly improves precision for a given quantization level.

I'd love your feedback—have you tried this? How does it perform for you?

Click here to get info on choosing the right GGUF model format

Model Name: Granite-7b-base

License: Apache-2.0

Languages: Primarily English

Architecture: The model architecture is a replica of Meta’s Llama2-7B base variant with MHA, trained with 1M batch size on 2T tokens.

Context Length: 4k tokens

Tokenizer: Llama2

Model Developers: IBM Research

Representing IBM’s commitment to open source innovation IBM has released granite-7b-base, a base pre-trained LLM from IBM’s Granite model series, under an apache-2.0 license for community and commercial use. Granite-7b-base was pre-trained from scratch on IBM-curated data as an open reference implementation of Meta’s Llama-2-7B. In a commitment to data transparency and fostering open innovation, the data sources, sampling proportions, and URLs for access are provided below.

For more information about training this model, please check out the blog: https://pytorch.org/blog/maximizing-training/

Pre-Training Data

The model was trained on 2T tokens, with sampling proportions designed to match the sampling distributions released in the Llama1 paper as closely as possible.

Dataset	Description	Sampling Proportion	URL
Common Crawl	Open repository of web crawl data with snapshots ranging from 2021 to 2023.	77%	https://data.commoncrawl.org/
Github_Clean	Code data from CodeParrot covering a variety of coding languages.	5.50%	https://huggingface.co/datasets/codeparrot/github-code-clean
Wikipedia and Wikimedia	Eight Wikimedia projects (enwiki, enwikibooks, enwikinews, enwikiquote, enwikisource, enwikiversity, enwikivoyage, enwiktionary). containing extracted plain text from pages and articles.	2%	https://dumps.wikimedia.org
USPTO	US patents granted from 1975 to May 2023, excluding design patents.	5%	https://bulkdata.uspto.gov/
PubMed Central	Biomedical and life sciences papers.	1.75%	https://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_package/
arXiv	Over 1.8 million scientific paper pre-prints posted to arXiv.	2.50%	https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T
StackExchange	Anonymized set of all user-contributed content on the Stack Exchange network, a popular collection of websites centered around user-contributed questions and answers.	1%	https://archive.org/details/stackexchange_20221206
PG19	A repository of free e-books with focus on older works for which U.S. copyright has expired.	0.25%	https://github.com/google-deepmind/pg19
Webhose	Unstructured web content converted into machine-readable data feeds purchased by IBM.	5%	N/A

Evaluation Results

LM-eval Harness Scores

Evaluation metric	Llama2-7B (baseline)	Granite-7b-base
MMLU (zero shot)	0.41	0.43
MMLU (5-shot weighted avg)	0.47	0.50
Arc challenge	0.46	0.44
Arc easy	0.74	0.71
Boolq	0.78	0.76
Copa	0.87	0.83
Hellaswag	0.76	0.74
Openbookqa	0.44	0.42
Piqa	0.79	0.79
Sciq	0.91	0.91
Winogrande	0.69	0.67
Truthfulqa	0.39	0.39
GSM8k (8-shot)	0.13	0.11

Bias, Risks, and Limitations

Granite-7b-base is a base model and has not undergone any safety alignment, there it may produce problematic outputs. In the absence of adequate safeguards and RLHF, there exists a risk of malicious utilization of these models for generating disinformation or harmful content. Caution is urged against complete reliance on a specific language model for crucial decisions or impactful information, as preventing these models from fabricating content is not straightforward. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in ungrounded generation scenarios due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.

🚀 If you find these models useful

Help me test my AI-Powered Quantum Network Monitor Assistant with quantum-ready security checks:

👉 Quantum Network Monitor

The full Open Source Code for the Quantum Network Monitor Service available at my github repos ( repos with NetworkMonitor in the name) : Source Code Quantum Network Monitor. You will also find the code I use to quantize the models if you want to do it yourself GGUFModelBuilder

💬 How to test:
Choose an AI assistant type:

TurboLLM (GPT-4.1-mini)
HugLLM (Hugginface Open-source models)
TestLLM (Experimental CPU-only)

What I’m Testing

I’m pushing the limits of small open-source models for AI network monitoring, specifically:

Function calling against live network services
How small can a model go while still handling:
- Automated Nmap security scans
- Quantum-readiness checks
- Network Monitoring tasks

🟡 TestLLM – Current experimental model (llama.cpp on 2 CPU threads on huggingface docker space):

✅ Zero-configuration setup
⏳ 30s load time (slow inference but no API costs) . No token limited as the cost is low.
🔧 Help wanted! If you’re into edge-device AI, let’s collaborate!

Other Assistants

🟢 TurboLLM – Uses gpt-4.1-mini :

**It performs very well but unfortunatly OpenAI charges per token. For this reason tokens usage is limited.
Create custom cmd processors to run .net code on Quantum Network Monitor Agents
Real-time network diagnostics and monitoring
Security Audits
Penetration testing (Nmap/Metasploit)

🔵 HugLLM – Latest Open-source models:

🌐 Runs on Hugging Face Inference API. Performs pretty well using the lastest models hosted on Novita.

💡 Example commands you could test:

"Give me info on my websites SSL certificate"
"Check if my server is using quantum safe encyption for communication"
"Run a comprehensive security audit on my server"
'"Create a cmd processor to .. (what ever you want)" Note you need to install a Quantum Network Monitor Agent to run the .net code on. This is a very flexible and powerful feature. Use with caution!

Final Word

I fund the servers used to create these model files, run the Quantum Network Monitor service, and pay for inference from Novita and OpenAI—all out of my own pocket. All the code behind the model creation and the Quantum Network Monitor project is open source. Feel free to use whatever you find helpful.

If you appreciate the work, please consider buying me a coffee ☕. Your support helps cover service costs and allows me to raise token limits for everyone.

I'm also open to job opportunities or sponsorship.

Thank you! 😊

Downloads last month: 12

GGUF

Model size

7B params

Architecture

llama

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including Mungert/granite-7b-base-GGUF

Granite Models

Collection

IBM Granite Models • 54 items • Updated Mar 2 • 2