models/README.md

# models/

Place the GGUF model file for the local llama.cpp server in this directory.

## Which file

Download the main model from the public Hugging Face repo:

- **Repo:** `unsloth/gemma-4-26B-A4B-it-qat-GGUF`
- **File:** `gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf`

The filename must match `GEMMA_MODEL_FILENAME` in your `.env` (see `.env.example`).
If you use a different quantization, set `GEMMA_MODEL_FILENAME` to the file you
downloaded so the server can find it.

## How it is used

`docker-compose.yaml` mounts this directory into the llama.cpp container at
`/models`, and the server is launched with `-m /models/${GEMMA_MODEL_FILENAME}`.
Once the GGUF file sits here with the matching name, `docker compose up` will
serve it.

This directory is otherwise git-ignored (only this README is tracked) — model
files are large and are not committed.