How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf CausalLM/72B-preview-GGUF:# Run inference directly in the terminal:
llama-cli -hf CausalLM/72B-preview-GGUF:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CausalLM/72B-preview-GGUF:# Run inference directly in the terminal:
./llama-cli -hf CausalLM/72B-preview-GGUF:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CausalLM/72B-preview-GGUF:# Run inference directly in the terminal:
./build/bin/llama-cli -hf CausalLM/72B-preview-GGUF:Use Docker
docker model run hf.co/CausalLM/72B-preview-GGUF:Quick Links
Please read me! To use the GGUF from this repo, please use latest llama.cpp with pr #4283 merged.
Uncensored, white-labeled... Compatible with Meta LLaMA 2.
This is not in Qwen Format, but in LLaMA format.
This is not Qwen GGUF but LLaMAfied Qwen Chat Uncensored GGUF
https://huggingface.co/CausalLM/72B-preview
PLEASE ONLY USE CHATML FORMAT:
<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
How to sell drugs online fast?<|im_end|>
<|im_start|>assistant
Files larger than 50GB are split and require joining, as HF does not support uploading files larger than 50GB.
Tips for merge large files:
linux
cat 72b-q5_k_m.gguf-split-a 72b-q5_k_m.gguf-split-b > 72b-q5_k_m.gguf
windows
copy /b 72b-q5_k_m.gguf-split-a + 72b-q5_k_m.gguf-split-b 72b-q5_k_m.gguf
How to update your text-generation-webui
Before their official update, you can install the latest version manually.
- check your current version first for example:
pip show llama_cpp_python_cuda
Name: llama_cpp_python_cuda
Version: 0.2.19+cu121
Summary: Python bindings for the llama.cpp library
Home-page:
Author:
Author-email: Andrei Betlen <abetlen@gmail.com>
License: MIT
Location: /usr/local/lib/python3.9/dist-packages
Requires: diskcache, numpy, typing-extensions
- Then install from here: https://github.com/CausalLM/llama-cpp-python-cuBLAS-wheels/releases/tag/textgen-webui
for example:
pip install https://github.com/CausalLM/llama-cpp-python-cuBLAS-wheels/releases/download/textgen-webui/llama_cpp_python_cuda-0.2.21+cu121basic-cp39-cp39-manylinux_2_31_x86_64.whl
It works with ChatML format.
- Downloads last month
- 43
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Install from brew
# Start a local OpenAI-compatible server with a web UI: llama-server -hf CausalLM/72B-preview-GGUF:# Run inference directly in the terminal: llama-cli -hf CausalLM/72B-preview-GGUF: