Instructions to use FirstPrinciplesLabs/quark with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FirstPrinciplesLabs/quark with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="FirstPrinciplesLabs/quark")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("FirstPrinciplesLabs/quark")
model = AutoModelForMultimodalLM.from_pretrained("FirstPrinciplesLabs/quark", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FirstPrinciplesLabs/quark with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FirstPrinciplesLabs/quark"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FirstPrinciplesLabs/quark

SGLang

How to use FirstPrinciplesLabs/quark with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FirstPrinciplesLabs/quark" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FirstPrinciplesLabs/quark" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use FirstPrinciplesLabs/quark with Docker Model Runner:
```
docker model run hf.co/FirstPrinciplesLabs/quark
```

chengfp-eng commited on 4 days ago

Commit

10fa12c

verified ·

1 Parent(s): bd96215

README: call the 8-bit GGUF Q8_0 (model weights quantized to 8 bits) instead of int8

Browse files

Files changed (1) hide show

README.md +13 -13

README.md CHANGED Viewed

@@ -24,19 +24,19 @@ Quark is a 9B reasoning model for physics, science and maths, built by [FirstPri
 our curated physics, maths and coding mix.
 It outperforms the base model on a range of benchmarks, and reaches
-graduate-level science scores comparable to much larger open models. The weights are available in bf16 and as an [int8 version](#int8-version) for llama.cpp, Ollama and
-LM Studio, which matches bf16 on every benchmark below.
 Quark is the first in a series of small, powerful scientific reasoning models. You can run it locally (see [Usage](#usage)) or on [Theo Platform](https://firstprinciples.com/sign-up), a research platform for complex problems, where you can use it alongside other frontier models, check its results with Python, Mathematica and Sage, and keep each session as a research graph you can revise one step at a time.
 ## Results
-Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) and vLLM (the int8 version
 with llama.cpp): thinking mode, up to 131,072 output tokens, and following the Qwen3.5 model card's sampling
 settings. AIME 2025 is averaged over 16 samples per problem and GPQA Diamond over 8. HLE (text-only, 2,158
 questions) is graded by Qwen3.5-27B as judge.
-| Benchmark | Qwen3.5-9B | Quark (bf16) | Quark ([int8](#int8-version)) |
 | --- | --- | --- | --- |
 | GPQA Diamond | 81.6 | 87.4 | **87.8** |
 | AIME 2025 | 86.9 | 91.7 | **94.0** |
@@ -44,7 +44,7 @@ questions) is graded by Qwen3.5-27B as judge.
 | Humanity's Last Exam (HLE), text | 15.9 | **22.7** | 22.3 |
 | HLE, physics subset | 8.4 | 15.8 | **16.3** |
-All were measured by us with the same setup. The int8 version matches Quark: the differences are within
 run-to-run variation.
 **Compared with other open models** (scores as reported by each developer):
@@ -52,7 +52,7 @@ run-to-run variation.
 | Model | Size | GPQA Diamond | AIME 2025 | HLE |
 | --- | --- | --- | --- | --- |
 | Quark (fp16) | 9B | 87.4 | 91.7 | 22.7 |
-| Quark (int8) | 9B | 87.8 | 94.0 | 22.3 |
 | Qwen3-8B | 8B | 59.6 | 69.3 | 4.4 |
 | NVIDIA Nemotron Nano 9B v2 | 9B | 64.0 | 72.1 | 6.5 |
 | gpt-oss-20b | 21B | 71.5 | 91.7 | 10.9 |
@@ -62,19 +62,19 @@ run-to-run variation.
 | DeepSeek-R1-0528 | 671B | 81.0 | 87.5 | 17.7 |
 | Kimi K2 Thinking | 1T | 84.5 | **94.5** | 23.9 |
-— = not reported. Sources: each model's official model card; Quark (bf16) and its int8 version measured by us
-(see above), with the int8 version's differences from Quark within run-to-run variation.
-## int8 version
-An int8 version for llama.cpp, Ollama and LM Studio (in their GGUF format) is available at
 [FirstPrinciplesLabs/quark-GGUF](https://huggingface.co/FirstPrinciplesLabs/quark-GGUF). Its accuracy is
 in the [Results](#results) table: it matches Quark on all benchmarks.
 ## Throughput
 Tokens per second on an NVIDIA H200 and a MacBook Pro, for the original bf16 weights and the
-[int8 version](#int8-version).
 ### llama.cpp
@@ -82,7 +82,7 @@ Throughput for a single user with llama.cpp, by how much context is already fill
 Output generation, tokens per second:
-| Hardware | Context | int8 | bf16 |
 | --- | --- | --- | --- |
 | NVIDIA H200 | empty | **216** | 185 |
 | NVIDIA H200 | 32K | **191** | 167 |
@@ -126,7 +126,7 @@ Or run it locally with [Ollama](https://ollama.com/FirstPrinciples/quark) (BF16
 ollama run firstprinciples/quark
 ```
-For the [int8 version](#int8-version) with llama.cpp:
 ```bash
 llama-server -hf FirstPrinciplesLabs/quark-GGUF:Q8_0 --jinja -c 131072 \

 our curated physics, maths and coding mix.
 It outperforms the base model on a range of benchmarks, and reaches
+graduate-level science scores comparable to much larger open models. We also quantized the model weights to 8 bits (Q8_0) for llama.cpp, Ollama and LM Studio; this
+[Q8_0 version](#q8_0-version) matches bf16 on every benchmark below.
 Quark is the first in a series of small, powerful scientific reasoning models. You can run it locally (see [Usage](#usage)) or on [Theo Platform](https://firstprinciples.com/sign-up), a research platform for complex problems, where you can use it alongside other frontier models, check its results with Python, Mathematica and Sage, and keep each session as a research graph you can revise one step at a time.
 ## Results
+Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) and vLLM (the Q8_0 version
 with llama.cpp): thinking mode, up to 131,072 output tokens, and following the Qwen3.5 model card's sampling
 settings. AIME 2025 is averaged over 16 samples per problem and GPQA Diamond over 8. HLE (text-only, 2,158
 questions) is graded by Qwen3.5-27B as judge.
+| Benchmark | Qwen3.5-9B | Quark (bf16) | Quark ([Q8_0](#q8_0-version)) |
 | --- | --- | --- | --- |
 | GPQA Diamond | 81.6 | 87.4 | **87.8** |
 | AIME 2025 | 86.9 | 91.7 | **94.0** |
 | Humanity's Last Exam (HLE), text | 15.9 | **22.7** | 22.3 |
 | HLE, physics subset | 8.4 | 15.8 | **16.3** |
+All were measured by us with the same setup. The Q8_0 version matches Quark: the differences are within
 run-to-run variation.
 **Compared with other open models** (scores as reported by each developer):
 | Model | Size | GPQA Diamond | AIME 2025 | HLE |
 | --- | --- | --- | --- | --- |
 | Quark (fp16) | 9B | 87.4 | 91.7 | 22.7 |
+| Quark (Q8_0) | 9B | 87.8 | 94.0 | 22.3 |
 | Qwen3-8B | 8B | 59.6 | 69.3 | 4.4 |
 | NVIDIA Nemotron Nano 9B v2 | 9B | 64.0 | 72.1 | 6.5 |
 | gpt-oss-20b | 21B | 71.5 | 91.7 | 10.9 |
 | DeepSeek-R1-0528 | 671B | 81.0 | 87.5 | 17.7 |
 | Kimi K2 Thinking | 1T | 84.5 | **94.5** | 23.9 |
+— = not reported. Sources: each model's official model card; Quark (bf16) and its Q8_0 version measured by us
+(see above), with the Q8_0 version's differences from Quark within run-to-run variation.
+## Q8_0 version
+A Q8_0 version (model weights quantized to 8 bits) for llama.cpp, Ollama and LM Studio (in their GGUF format) is available at
 [FirstPrinciplesLabs/quark-GGUF](https://huggingface.co/FirstPrinciplesLabs/quark-GGUF). Its accuracy is
 in the [Results](#results) table: it matches Quark on all benchmarks.
 ## Throughput
 Tokens per second on an NVIDIA H200 and a MacBook Pro, for the original bf16 weights and the
+[Q8_0 version](#q8_0-version).
 ### llama.cpp
 Output generation, tokens per second:
+| Hardware | Context | Q8_0 | bf16 |
 | --- | --- | --- | --- |
 | NVIDIA H200 | empty | **216** | 185 |
 | NVIDIA H200 | 32K | **191** | 167 |
 ollama run firstprinciples/quark
 ```
+For the [Q8_0 version](#q8_0-version) with llama.cpp:
 ```bash
 llama-server -hf FirstPrinciplesLabs/quark-GGUF:Q8_0 --jinja -c 131072 \