How to use from
llama.cppInstall from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf QuantFactory/UltraLlama-3.1-8B-GGUF:# Run inference directly in the terminal:
llama cli -hf QuantFactory/UltraLlama-3.1-8B-GGUF:Use pre-built binary
# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf QuantFactory/UltraLlama-3.1-8B-GGUF:# Run inference directly in the terminal:
./llama-cli -hf QuantFactory/UltraLlama-3.1-8B-GGUF:Build from source code
git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf QuantFactory/UltraLlama-3.1-8B-GGUF:# Run inference directly in the terminal:
./build/bin/llama-cli -hf QuantFactory/UltraLlama-3.1-8B-GGUF:Use Docker
docker model run hf.co/QuantFactory/UltraLlama-3.1-8B-GGUF:Quick Links
QuantFactory/UltraLlama-3.1-8B-GGUF
This is quantized version of mlabonne/UltraLlama-3.1-8B created using llama.cpp
Original Model Card
UltraLlama-3.1-8B
Llama 3.1 8B model trained on a high-quality magpie dataset to measure its quality:
| Model | MMLU | Hellaswag | ARC-C | GSM8K | TruthfulQA | Winogrande | IFEval | MMLU-Pro | MATH Lvl 5 | GPQA | MuSR | BBH |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Meta-Llama-3.1-8B | 65.28 | 82.09 | 59.22 | 51.02 | 45.15 | 77.58 | 11.45 | 32.74 | 4.38 | 30.93 | 7.98 | 46.77 |
| Meta-Llama-3-8B-Instruct | 65.60 | 78.79 | 61.95 | 75.28 | 51.66 | 75.77 | 47.43 | 5.87 | 7.95 | 30.11 | 37.92 | 49.04 |
| FineLlama-3.1-8B | 62.22 | 80.30 | 55.55 | 51.02 | 49.51 | 75.30 | 1.68 | 30.90 | 4.12 | 27.45 | 35.77 | 44.22 |
| UltraLlama-3.1-8B | 54.36 | 74.98 | 55.46 | 51.10 | 49.93 | 72.05 | 10.19 | 26.63 | 3.08 | 25.47 | 40.93 | 42.44 |
Magpie underperforms FineTome-100k. The quality looks okay, but not as good as high-quality open-source datasets.
- Downloads last month
- 173
Hardware compatibility
Log In to add your hardware
2-bit
3-bit
4-bit
5-bit
6-bit
8-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for QuantFactory/UltraLlama-3.1-8B-GGUF
Base model
unsloth/Meta-Llama-3.1-8B
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf QuantFactory/UltraLlama-3.1-8B-GGUF:# Run inference directly in the terminal: llama cli -hf QuantFactory/UltraLlama-3.1-8B-GGUF: