I had a user ask if these would run on a CPU and the answer surprised me kind of a lot:

$ CUDA_VISIBLE_DEVICES="" ~/llama.cpp/build/bin/llama-cli --model kat-coder-v2.5-dev/KAT-Coder-V2.5-Dev-MTP-APEX-i-quality.gguf -c 512 -ngl 0 --cpu-moe --no-mmap -fa on --spec-type draft-mtp --spec-draft-n-max 4 --reasoning off --numa distribute
0.00.015.670 E ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected
warning: no usable GPU found, --gpu-layers option will be ignored
warning: one possible reason is that llama.cpp was compiled without GPU support
warning: consult docs/build.md for compilation instructions

Loading model...

▄▄ ▄▄
██ ██
██ ██ ▀▀█▄ ███▄███▄ ▀▀█▄ ▄████ ████▄ ████▄
██ ██ ▄█▀██ ██ ██ ██ ▄█▀██ ██ ██ ██ ██ ██
██ ██ ▀█▄██ ██ ██ ██ ▀█▄██ ██ ▀████ ████▀ ████▀
██ ██
▀▀ ▀▀

build : b10068-571d0d540
model : kat-coder-v2.5-dev/KAT-Coder-V2.5-Dev-MTP-APEX-i-quality.gguf
ftype : Q4_K - Medium
modalities : text

available commands:
/exit or Ctrl+C stop or exit
/regen regenerate the last response
/clear clear the chat history
/read add a text file
/glob

How would I check the free memory in python?

Checking Free Memory in Python

There are several approaches depending on what level of memory information you need:

1. Using psutil (Most Common)

import psutil

# Total, available, used memory, and free memory
mem = psutil.virtual_memory()

print(f"Total memory:     {mem.total / (1024**3):.2f} GB")
print(f"Available memory: {mem.available / (1024**3):.2f} GB")
print(f"Used memory:      {mem.used / (1024**3):.2f} GB")
print(f"Free memory:      {mem.free / (1024**3):.2f} GB")
print(f"Memory usage %:   {mem.percent}%")

Install with: pip install psutil

2. Using resource Module (Linux/macOS Only)

import resource

# In kilobytes
mem_info = resource.getrusage(resource.RUSAGE_SELF)
print(f"Max resident set size (KB): {mem_info.ru_maxrss}")
print(f"Page faults (soft): {mem_info.ru_minflt}")

3. Reading from /proc/meminfo (Linux Only)

def get_memory_info():
    with open('/proc/meminfo', 'r') as f:
        lines = f.readlines()
    
    mem_info = {}
    for line in lines:
        key, value = line.split(':')
        mem_info[key.strip()] = int(value.strip().split()[0])  # in kB
    
    return {
        'total_mb': mem_info['MemTotal'] / 1024,
        'free_mb': mem_info['MemFree'] / 1024,
        'available_mb': mem_info['MemAvailable'] / 1024,
    }

print(get_memory_info())

4. Checking Python-Specific Memory Usage

import sys
import tracemalloc

# Track current Python object memory allocations
tracemalloc.start()

# ... your code ...

current, peak = tracemalloc.get_tr

[ Prompt: 17.0 t/s | Generation: 11.4 t/s ]

>