llama.cpp/docs/backend/VirtGPU/development.md

# Development and Testing

## Development

### Code Generation

The backend uses code generation from YAML configuration:

```bash

# Regenerate protocol code

cd ggml-virtgpu/

python regenerate_remoting.py

```

### Adding New Operations

1. Add function definition to `ggmlremoting_functions.yaml`
2. Regenerate code with `regenerate_remoting.py`
3. Implement guest-side forwarding in `virtgpu-forward-*.cpp`
4. Implement host-side handling in `backend-dispatched-*.cpp`

## Testing

This document provides instructions for building and testing the GGML-VirtGPU backend on macOS with containers.

### Prerequisites

The testing setup requires:

- macOS host system
- Container runtime with `libkrun` provider (podman machine)
- Access to development patchset for VirglRenderer

### Required Patchsets

The backend requires patches that are currently under review:

- **Virglrenderer APIR upstream PR**: https://gitlab.freedesktop.org/virgl/virglrenderer/-/merge_requests/1590 (for reference)

- **MacOS Virglrenderer (for krunkit)**: https://gitlab.freedesktop.org/kpouget/virglrenderer/-/tree/main-macos

- **Linux Virglrenderer (for krun)**: https://gitlab.freedesktop.org/kpouget/virglrenderer/-/tree/main-linux



### Build Instructions



#### 1. Build ggml-virtgpu-backend (Host-side, macOS)



```bash

# Build the backend that runs natively on macOS

mkdir llama.cpp

cd llama.cpp

git clone https://github.com/ggml-org/llama.cpp.git src

cd src



LLAMA_MAC_BUILD=$PWD/build/ggml-virtgpu-backend



cmake -S . -B $LLAMA_MAC_BUILD \

      -DGGML_NATIVE=OFF \
      -DLLAMA_CURL=ON \

      -DGGML_REMOTINGBACKEND=ONLY \

      -DGGML_METAL=ON


TARGETS="ggml-metal"
cmake --build $LLAMA_MAC_BUILD --parallel 8 --target $TARGETS

# Build additional tools for native benchmarking
EXTRA_TARGETS="llama-run llama-bench"

cmake --build $LLAMA_MAC_BUILD --parallel 8 --target $EXTRA_TARGETS
```



#### 2. Build virglrenderer (Host-side, macOS)



```bash

# Build virglrenderer with APIR support

mkdir virglrenderer

git clone https://gitlab.freedesktop.org/kpouget/virglrenderer -b main-macos src

cd src



VIRGL_BUILD_DIR=$PWD/build



# -Dvenus=true and VIRGL_ROUTE_VENUS_TO_APIR=1 route the APIR requests via the Venus backend, for easier testing without a patched hypervisor



meson setup $VIRGL_BUILD_DIR \

      -Dvenus=true \

      -Dapir=true



ninja -C $VIRGL_BUILD_DIR

```

#### 3. Build ggml-virtgpu (Guest-side, Linux)

Option A: Build from a script:

```bash

# Inside a Linux container

mkdir llama.cpp

git clone https://github.com/ggml-org/llama.cpp.git src

cd src



LLAMA_LINUX_BUILD=$PWD//build-virtgpu



cmake -S . -B $LLAMA_LINUX_BUILD \

      -DGGML_VIRTGPU=ON



ninja -C $LLAMA_LINUX_BUILD

```

Option B: Build container image with frontend:

```bash

cat << EOF > remoting.containerfile

FROM quay.io/fedora/fedora:43

USER 0



WORKDIR /app/remoting



ARG LLAMA_CPP_REPO="https://github.com/ggml-org/llama.cpp.git"

ARG LLAMA_CPP_VERSION="master"

ARG LLAMA_CPP_CMAKE_FLAGS="-DGGML_VIRTGPU=ON"

ARG LLAMA_CPP_CMAKE_BUILD_FLAGS="--parallel 4"



RUN dnf install -y git cmake gcc gcc-c++ libcurl-devel libdrm-devel



RUN git clone "\${LLAMA_CPP_REPO}" src \\

 && git -C src fetch origin \${LLAMA_CPP_VERSION} \\

 && git -C src reset --hard FETCH_HEAD



RUN mkdir -p build \\

 && cd src \\

 && set -o pipefail \\

 && cmake -S . -B ../build \${LLAMA_CPP_CMAKE_FLAGS} \\

 && cmake --build ../build/ \${LLAMA_CPP_CMAKE_BUILD_FLAGS}



ENTRYPOINT ["/app/remoting/src/build/bin/llama-server"]

EOF



mkdir -p empty_dir

podman build -f remoting.containerfile ./empty_dir -t localhost/llama-cpp.virtgpu

```

### Environment Setup

#### Set krunkit Environment Variables

```bash

# Define the base directories (adapt these paths to your system)

VIRGL_BUILD_DIR=$HOME/remoting/virglrenderer/build

LLAMA_MAC_BUILD=$HOME/remoting/llama.cpp/build-backend



# For krunkit to load the custom virglrenderer library

export DYLD_LIBRARY_PATH=$VIRGL_BUILD_DIR/src



# For Virglrenderer to load the ggml-remotingbackend library

export VIRGL_APIR_BACKEND_LIBRARY="$LLAMA_MAC_BUILD/bin/libggml-virtgpu-backend.dylib"



# For llama.cpp remotingbackend to load the ggml-metal backend

export APIR_LLAMA_CPP_GGML_LIBRARY_PATH="$LLAMA_MAC_BUILD/bin/libggml-metal.dylib"

export APIR_LLAMA_CPP_GGML_LIBRARY_REG=ggml_backend_metal_reg

```

#### Launch Container Environment

```bash

# Set container provider to libkrun

export CONTAINERS_MACHINE_PROVIDER=libkrun

podman machine start

```

#### Verify Environment

Confirm that krunkit is using the correct virglrenderer library:

```bash

lsof -c krunkit | grep virglrenderer

# Expected output:

# krunkit 50574 user  txt  REG  1,14  2273912  10849442 ($VIRGL_BUILD_DIR/src)/libvirglrenderer.1.dylib

```

### Running Tests

#### Launch Test Container

```bash

# Optional model caching

mkdir -p models

PODMAN_CACHE_ARGS="-v models:/models --user root:root --cgroupns host --security-opt label=disable -w /models"



podman run $PODMAN_CACHE_ARGS -it --rm --device /dev/dri localhost/llama-cpp.virtgpu

```

#### Test llama.cpp in Container

```bash



# Run performance benchmark

/app/remoting/build/bin/llama-bench -m ./llama3.2

```

Expected output (performance may vary):
```

| model                          |       size |     params | backend    | ngl |          test |                  t/s |

| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |

| llama 3B Q4_K - Medium         |   1.87 GiB |     3.21 B | ggml-virtgpu |  99 |         pp512 |        991.30 ± 0.66 |

| llama 3B Q4_K - Medium         |   1.87 GiB |     3.21 B | ggml-virtgpu |  99 |         tg128 |         85.71 ± 0.11 |

```

### Troubleshooting

#### SSH Environment Variable Issues

⚠️ **Warning**: Setting `DYLD_LIBRARY_PATH` from SSH doesn't work on macOS. Here is a workaround:

**Workaround 1: Replace system library**
```bash

VIRGL_BUILD_DIR=$HOME/remoting/virglrenderer/build  # ⚠️ adapt to your system

BREW_VIRGL_DIR=/opt/homebrew/Cellar/virglrenderer/0.10.4d/lib

VIRGL_LIB=libvirglrenderer.1.dylib



cd $BREW_VIRGL_DIR

mv $VIRGL_LIB ${VIRGL_LIB}.orig

ln -s $VIRGL_BUILD_DIR/src/$VIRGL_LIB

```