File size: 3,684 Bytes
0824501 f596de4 0824501 f596de4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 |
---
license: mit
---
# RTX 5000 Series–Ready `llama-cpp-python` Wheel (Python 3.12, Windows)
**Status:** ✅ CONFIRMED WORKING — No more “invalid resource handle” errors
**Wheel:** `llama_cpp_python-0.3.16-cp312-cp312-win_amd64.whl`
**License:** MIT (same as upstream `llama-cpp-python`)
**Platform:** Windows 10/11 x64
**Python:** 3.12
**CUDA:** 12.8 (optimized for Blackwell)
---
## 🚀 Performance (Verified on RTX 5090)
- ~64 tokens/sec on *Mistral Small 24B* (5-bit quant)
- Full GPU offload (`n_gpu_layers = -1`) working as expected
- ~1.83× faster than RTX 3090 in the same setup (35 tok/s → 64 tok/s)
- 32 GB VRAM fully utilized (no kernel crashes)
> Notes: numbers vary with quant, context, and params; these are representative.
---
## 🔧 Why This Works
The wheel forces **cuBLAS** instead of ggml’s custom CUDA kernels.
On RTX 5090 (Blackwell, `sm_120`), ggml’s custom kernels can trigger:
“CUDA error: invalid resource handle”.
cuBLAS is stable on 5090 and avoids those kernel issues.
**Key CMake flags used:**
-DGGML_CUDA=ON
-DGGML_CUDA_FORCE_CUBLAS=1 # Use cuBLAS instead of custom kernels
-DGGML_CUDA_NO_PINNED=1 # Avoid pinned memory issues with GDDR7
-DGGML_CUDA_F16=0 # Disable problematic FP16 code paths
-DCMAKE_CUDA_ARCHITECTURES=all-major # Ensure sm_120 is included
---
## 📋 Requirements
- NVIDIA RTX 5090 (or other Blackwell GPU)
- NVIDIA drivers 570.86.10+
- CUDA Toolkit 12.8
- Python 3.12
- Windows 10/11 x64
- Microsoft Visual C++ Redistributable 2015–2022
---
## 🛠️ Installation
1) Download the wheel:
`llama_cpp_python-0.3.16-cp312-cp312-win_amd64.whl`
2) Install:
pip install llama_cpp_python-0.3.16-cp312-cp312-win_amd64.whl
---
## ✅ Quick Verification
from llama_cpp import Llama
# Full GPU offload on 5090
llm = Llama(
model_path="your_model.gguf",
n_gpu_layers=-1, # full GPU
n_ctx=2048,
verbose=True
)
out = llm("Hello, how are you?", max_tokens=20)
print(out["choices"][0]["text"])
**What to look for in stdout:**
- CUDA device assignment lines (e.g., using CUDA:0)
- VRAM allocations *without* any “invalid resource handle” errors
---
## 🏗️ Build It Yourself (Advanced)
**Prereqs:** CUDA 12.8, Visual Studio Build Tools 2022 (with C++), Python 3.12
mkdir C:\wheels
cd C:\wheels
set FORCE_CMAKE=1
set CMAKE_BUILD_PARALLEL_LEVEL=15
set CMAKE_ARGS=-DGGML_CUDA=ON -DGGML_CUDA_FORCE_CUBLAS=1 -DGGML_CUDA_NO_PINNED=1 -DGGML_CUDA_F16=0 -DCMAKE_CUDA_ARCHITECTURES=all-major
pip wheel llama-cpp-python --no-cache-dir --wheel-dir C:\wheels --verbose
**Build time:** ~10 minutes on a modern CPU
**Wheel size:** ~231 MB (larger due to cuBLAS inclusion)
---
## 🐛 Troubleshooting
**“Invalid resource handle” errors**
- This wheel specifically fixes this. If you still see them, verify:
- CUDA 12.8 is installed
- Latest NVIDIA drivers are installed
- No other CUDA apps are interfering
**CPU fallback**
- If GPU isn’t detected, check `nvidia-smi` and ensure `CUDA_VISIBLE_DEVICES` isn’t set.
---
## 🙏 Credits
Built using the open-source `llama-cpp-python` project by **abetlen** and the `llama.cpp` project by **ggml-org**.
This wheel provides RTX 5090 compatibility by configuring cuBLAS fallback; it is not an official upstream release.
- For issues with this specific wheel: *open an issue here (this repo/thread).*
- For general `llama-cpp-python` issues: use the official repository.
---
Finally — RTX 5000 series owners can use their flagship GPU for local LLM inference without crashes! 🎉 |