[API Error: Connection error. (cause: fetch failed)]
Environment
Hardware: AMD Strix Halo APU (gfx11, PCI ID 1002:1586)
Kernel: Linux 7.0.0-27-generic
Vulkan driver: Mesa RADV
Backend: Vulkan
Build: rocmfp4-llama (Strix FP4 branch)
Issue
After serving requests successfully for approximately 45–60 minutes, llama-server consistently terminates with:
terminate called after throwing an instance of 'vk::DeviceLostError'
what(): vk::Queue::submit: ErrorDeviceLost
Immediately beforehand, RADV reports:
radv/amdgpu: The CS has been cancelled because the context is lost. This context is innocent.
The stack trace ends in the Vulkan backend:
libggml-vulkan.so
ggml_backend_sched_graph_compute_async()
llama_context::decode()
common_speculative_state_draft_mtp::process()
which indicates the failure occurs while processing speculative decoding (MTP/draft model).
Kernel log
The kernel reports that the GPU compute queue hangs before the application aborts:
amdgpu: ring comp_1.2.0 timeout
Process llama-server pid XXXXX
Starting comp_1.2.0 ring reset
reset compute queue (1:2:0)
Ring comp_1.2.0 reset succeeded
device wedged, but recovered through reset
After the compute queue reset, Vulkan returns VK_ERROR_DEVICE_LOST, causing llama-server to abort.
Earlier in the session, the kernel repeatedly logs:
amdgpu: Fence fallback timer expired on ring comp_1.2.0
which may indicate the compute queue is degrading before eventually timing out.
Workload characteristics
Vulkan backend
Very long running server
Context grows to approximately 88k tokens
Context checkpoints reach ~480 MiB
Speculative decoding/MTP is enabled
The crash does not appear to be an application-side exception first—the GPU compute queue times out, the kernel resets it, and Vulkan subsequently reports VK_ERROR_DEVICE_LOST.
I have also seen separate Linux OOM kills under different runs, but this particular failure is not an OOM event. It is a GPU compute timeout followed by a successful queue reset.