models/README.md

models/

Place the GGUF model file for the local llama.cpp server in this directory.

Which file

Download the main model from the public Hugging Face repo:

• Repo: unsloth/gemma-4-26B-A4B-it-qat-GGUF
• File: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf

The filename must match GEMMA_MODEL_FILENAME in your .env (see .env.example). If you use a different quantization, set GEMMA_MODEL_FILENAME to the file you downloaded so the server can find it.

How it is used

docker-compose.yaml mounts this directory into the llama.cpp container at /models, and the server is launched with -m /models/${GEMMA_MODEL_FILENAME}. Once the GGUF file sits here with the matching name, docker compose up will serve it.

This directory is otherwise git-ignored (only this README is tracked) — model files are large and are not committed.