How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf wredd/medgemma-4b-gguf:# Run inference directly in the terminal:
llama cli -hf wredd/medgemma-4b-gguf:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf wredd/medgemma-4b-gguf:# Run inference directly in the terminal:
./llama-cli -hf wredd/medgemma-4b-gguf:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf wredd/medgemma-4b-gguf:# Run inference directly in the terminal:
./build/bin/llama-cli -hf wredd/medgemma-4b-gguf:Use Docker
docker model run hf.co/wredd/medgemma-4b-gguf:Quick Links
MedGemma 4B GGUF - Quantized for African Healthcare
Quantized versions of google/medgemma-1.5-4b-it optimized for on-device medical AI in resource-constrained settings.
Available Models
| File | Quantization | Size | RAM | MedQA (vs Unquantized) | Use Case |
|---|---|---|---|---|---|
medgemma-4b-iq2_xs.gguf |
IQ2_XS (2-bit) + Medical imatrix | ~0.85GB | ~1.9GB | 43.8% (63% retained) | Ultra-budget phones |
medgemma-4b-q2_k.gguf |
Q2_K (2-bit) | ~1.3GB | ~2.3GB | 34.7% (50% retained) | Budget phones |
medgemma-4b-q4_k_m.gguf |
Q4_K_M (4-bit) | ~2.3GB | ~3GB | 56.0% (81% retained) | Budget phones |
Medical Importance Matrix
The IQ2_XS model was quantized using a custom importance matrix (imatrix) calibrated on:
- African primary care scenarios (malaria, typhoid, cholera, respiratory infections)
- Maternal and child health (pregnancy complications, childhood diarrhea, nutrition)
- Emergency triage (snake bites, severe dehydration, trauma)
- Multi-language symptoms (Twi, Hausa, Yoruba, English)
This preserves medical diagnostic accuracy while aggressively compressing general knowledge.
Usage with llama.cpp
./llama-cli -m medgemma-4b-iq2_xs.gguf -p "Patient has fever, chills, and headache for 3 days. What could this be?"
License
Subject to Gemma Terms of Use.
Part of the Nku Project
Built for the Google MedGemma Impact Challenge - bringing AI-powered healthcare to underserved African communities.
- Downloads last month
- 39
Hardware compatibility
Log In to add your hardware
1-bit
2-bit
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for wredd/medgemma-4b-gguf
Base model
google/medgemma-1.5-4b-it
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf wredd/medgemma-4b-gguf:# Run inference directly in the terminal: llama cli -hf wredd/medgemma-4b-gguf: