Presets

Preconfigured presets and Docker-based build/run scripts for llama.cpp and turboquant.

Hardware

Linux host has:

• RTX 2080 SUPER: CUDA device 0, compute capability sm_75, ~8 GB VRAM
• GTX 980: CUDA device 1, compute capability sm_52, ~4 GB VRAM
• Proprietary NVIDIA driver 580

llama.cpp CUDA builds target both GPUs with:

CMAKE_CUDA_ARCHITECTURES=52;75

The host keeps CUDA 13.4. The llama.cpp build uses CUDA 12.9 inside an Ubuntu 24.04 container because CUDA 12.9 is incompatible with host Ubuntu 26 headers.

Presets

Two preset files define model configurations with server parameters, quantization, and speculative decoding settings:

File	Purpose
preset.ini	Standard llama.cpp presets
turboquant.ini	TurboQuant presets with KV cache tuning (cache-type-k, cache-type-v)

GPU settings in preset.ini:

device = CUDA0,CUDA1
split-mode = layer
main-gpu = 0
fit = on

Automatic fitting handles model VRAM overflow. layer split minimizes PCIe traffic; the GTX 980 is connected at PCIe x1.

Docker setup

Host requirements:

• Docker
• NVIDIA Container Toolkit
• Proprietary NVIDIA driver 580

Build image once after changing Dockerfile:

sudo docker compose build

Files:

File	Purpose
Dockerfile	CUDA 12.9.1 build image with CMake, OpenSSL, GCC, and ccache
docker-compose.yml	GPU-enabled llama.cpp build service
build-llama.cpp.sh	Pull source and run containerized build

The image includes libssl-dev, so llama.cpp can download Hugging Face models over HTTPS.

Scripts

Build (Linux)

sudo docker compose build    # First time or after Dockerfile changes
./build-llama.cpp.sh         # Build standard llama.cpp
./build-llama.cpp-tq.sh      # Build turboquant fork

The standard build writes binaries to:

/mnt/MODELS/llama.cpp/build/bin/

No host CUDA toolkit or host compiler is required for the standard build.

Build (Windows)

build-llama.cpp.bat         # Build standard llama.cpp
build-llama.cpp-tq.bat      # Build turboquant fork

Server

./server.sh                 # Start llama.cpp server with preset.ini
./server-tq.sh              # Start turboquant server with turboquant.ini

server.sh uses the prebuilt CUDA 12.9 image, exposes port 8080, and serves both GPUs. From another computer:

http://10.0.0.74:8080

The server runs in router mode and loads models on demand. Set an API key before exposing it beyond a trusted LAN; current presets intentionally have no key configured.

Benchmark

Targeted Qwen3.6 dual-GPU benchmark:

./bench-qwen3.6-35b.sh

It tests:

• GPU layer offload (-ngl)
• CPU thread counts (-t)
• Generation lengths (-n)
• Load modes (auto, mmap, mlock, mmap+mlock, dio)
• Lazy loading (auto, on, off)
• Batch and microbatch sizes
• Layer and row split modes
• Manual tensor split ratios
• CPU MoE placement
• KV cache precision

The benchmark uses CUDA 12.9 in the prebuilt container and targets both GPUs. llama-bench device syntax is slash-separated:

-dev CUDA0/CUDA1

This differs from server/preset syntax, which is comma-separated:

device = CUDA0,CUDA1

Results save to bench-results/. Analyze completed dual-GPU results:

./analyze-bench.py

The analyzer ignores old single-GPU results, ranks prompt and generation throughput separately, and prints a recommended preset block. It does not modify preset.ini.

General benchmark script remains available:

./llama-bench.sh [model] [--llama|--tq] [--quant q1,q2] [--cache-type-k k1,k2] [--cache-type-v v1,v2] [--ncmoe m1,m2]

Quick Start

cd /mnt/MODELS/presets
sudo docker compose build
./build-llama.cpp.sh
./server.sh

Then use the OpenAI-compatible API on port 8080.