GPU enabled

README.md

@@ -7,12 +7,12 @@ sdk: gradio
 sdk_version: 6.5.1
 app_file: app.py
 pinned: false
-suggested_hardware: cpu-basic
+suggested_hardware: t4-small
 license: artistic-2.0
 short_description: CPU-only local Nemotron GGUF chat
 ---
 
-Dreadzone is a CPU-only Hugging Face Space that runs a local GGUF model with
+Dreadzone is a Hugging Face Space that runs a local GGUF model with
 `llama-cpp-python` and Gradio ChatInterface.
 
 The app downloads
@@ -21,16 +21,18 @@ automatically on first launch and streams responses from
 `NVIDIA-Nemotron-3-Nano-4B-Q5_K_M.gguf`.
 
 No hosted inference API, OAuth token, secrets, or external inference services are
-used.
+used. The default dependency pin uses the CUDA 12.4 `llama-cpp-python` wheel for
+GPU Spaces.
 
-## CPU settings
+## Runtime settings
 
-The defaults are intentionally conservative for low-memory CPU inference:
+The defaults are intentionally conservative while enabling GPU offload:
 
 - `N_CTX=2048`
 - `N_BATCH=128`
 - `MAX_HISTORY_TURNS=6`
 - `N_THREADS` defaults to one fewer than the detected CPU count
+- `N_GPU_LAYERS=-1` offloads all possible layers to GPU
 
 You can override the model or runtime settings with Space variables:
 
@@ -40,4 +42,5 @@ You can override the model or runtime settings with Space variables:
 - `N_CTX`
 - `N_BATCH`
 - `N_THREADS`
+- `N_GPU_LAYERS`
 - `MAX_HISTORY_TURNS`

app.py

@@ -13,6 +13,7 @@ MODEL_FILE = os.getenv("MODEL_FILE", "NVIDIA-Nemotron-3-Nano-4B-Q5_K_M.gguf")
 N_CTX = int(os.getenv("N_CTX", "2048"))
 N_BATCH = int(os.getenv("N_BATCH", "128"))
 N_THREADS = int(os.getenv("N_THREADS", str(max(1, (os.cpu_count() or 2) - 1))))
+N_GPU_LAYERS = int(os.getenv("N_GPU_LAYERS", "-1"))
 MAX_HISTORY_TURNS = int(os.getenv("MAX_HISTORY_TURNS", "6"))
 
 DEFAULT_SYSTEM_PROMPT = (
@@ -23,7 +24,11 @@ DEFAULT_SYSTEM_PROMPT = (
 
 @lru_cache(maxsize=1)
 def get_llm() -> Llama:
-    print(f"Loading model {MODEL_REPO}/{MODEL_FILE}", flush=True)
+    print(
+        f"Loading model {MODEL_REPO}/{MODEL_FILE} "
+        f"with n_gpu_layers={N_GPU_LAYERS}",
+        flush=True,
+    )
     model_path = hf_hub_download(repo_id=MODEL_REPO, filename=MODEL_FILE)
     print(f"Model file ready: {model_path}", flush=True)
 
@@ -32,7 +37,7 @@ def get_llm() -> Llama:
         n_ctx=N_CTX,
         n_batch=N_BATCH,
         n_threads=N_THREADS,
-        n_gpu_layers=0,
+        n_gpu_layers=N_GPU_LAYERS,
         use_mmap=True,
         use_mlock=False,
         logits_all=False,

requirements.txt

@@ -1,4 +1,4 @@
---extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
+--extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124
 
 gradio==6.5.1
 huggingface_hub>=0.23.0