---
license: apache-2.0
base_model:
- FirstPrinciplesLabs/quark
base_model_relation: quantized
pipeline_tag: text-generation
language:
- en
tags:
- gguf
- llama.cpp
- physics
- science
- reasoning
- math
---

# Quark int8

[Quark](https://huggingface.co/FirstPrinciplesLabs/quark), a 9B reasoning model for physics, science and
maths by [FirstPrinciples](https://firstprinciples.com/), quantized to int8 for
[llama.cpp](https://github.com/ggml-org/llama.cpp) and tools built on it (Ollama, LM Studio). It is also on
Ollama as [FirstPrinciples/quark-q8](https://ollama.com/FirstPrinciples/quark-q8).

| File | Format | Size | Input |
|---|---|---|---|
| `quark-Q8_0.gguf` | Q8_0 (int8) | 9.8 GB | text |

Q8_0 is llama.cpp's int8 format: weights are stored as 8-bit integers, with one scale shared by each
block of 32.

## Results

The int8 version matches the original bf16 model on GPQA Diamond, AIME 2025, MMLU-Pro Physics and
Humanity's Last Exam, and generates faster. See the Quark model card for the
[benchmark scores](https://huggingface.co/FirstPrinciplesLabs/quark#results) and
[throughput](https://huggingface.co/FirstPrinciplesLabs/quark#throughput) on an NVIDIA H200 and a
MacBook Pro (M4, 16 GB).

## Usage

### Ollama

The int8 model is on Ollama as [FirstPrinciples/quark-q8](https://ollama.com/FirstPrinciples/quark-q8),
with the recommended sampling settings and a 131,072-token context built in. Install
[Ollama](https://ollama.com/download), then run:

```bash
ollama run FirstPrinciples/quark-q8
```

A 131,072-token context needs about 15 GB of GPU memory. On a 16 GB Mac, lower it inside the chat with
`/set parameter num_ctx 32768` (longer answers are then cut off)

### llama.cpp

Install llama.cpp, which includes `llama-server`, following its
[installation guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md) (Homebrew,
winget, conda, Nix, and builds for NVIDIA GPUs).

Then start a server; it downloads the model on first run and serves an OpenAI-compatible API and a chat
page at `http://localhost:8080`:

```bash
llama-server -hf FirstPrinciplesLabs/quark_quantized_weight_only-Q8_0 --jinja -c 131072 \
  --temp 1.0 --top-k 20 --top-p 0.95 --presence-penalty 1.5
```

This sets the recommended sampling settings for thinking mode (from the
[Quark model card](https://huggingface.co/FirstPrinciplesLabs/quark#usage)). A 131,072-token context needs about
15 GB of GPU memory; on a 16 GB Mac, use `-c 32768` instead (longer answers are then cut off).

## License

Apache 2.0