Instructions to use FirstPrinciplesLabs/quark with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use FirstPrinciplesLabs/quark with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="FirstPrinciplesLabs/quark")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("FirstPrinciplesLabs/quark")
model = AutoModelForMultimodalLM.from_pretrained("FirstPrinciplesLabs/quark", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use FirstPrinciplesLabs/quark with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FirstPrinciplesLabs/quark"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/FirstPrinciplesLabs/quark

SGLang

How to use FirstPrinciplesLabs/quark with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "FirstPrinciplesLabs/quark" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "FirstPrinciplesLabs/quark" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FirstPrinciplesLabs/quark",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use FirstPrinciplesLabs/quark with Docker Model Runner:
```
docker model run hf.co/FirstPrinciplesLabs/quark
```

Matthew Tchouikine commited on 5 days ago

Commit

b373146

1 Parent(s): ee399ef

clean up model card

Browse files

Files changed (1) hide show

README.md +50 -75

README.md CHANGED Viewed

@@ -30,105 +30,79 @@ Quark is the first in a series of small, powerful scientific reasoning models. Y
 ## Results
-Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) and vLLM (the GGUF with
-llama.cpp): thinking mode, up to 131,072 output tokens, and following the Qwen3.5 model card's sampling
 settings. AIME 2025 is averaged over 16 samples per problem and GPQA Diamond over 8. HLE (text-only, 2,158
 questions) is graded by Qwen3.5-27B as judge.
-| Benchmark | Qwen3.5-9B | **Quark** | Quark Q8_0 ([GGUF](#quantized-version-gguf)) |
-|---|---|---|---|
-| GPQA Diamond | 81.6 | **87.4** | 87.8 |
-| AIME 2025 | 86.9 | **91.7** | 94.0 |
-| MMLU-Pro Physics | 90.7 | 91.0 | 91.6 |
 | Humanity's Last Exam (HLE), text | 15.9 | **22.7** | 22.3 |
-| HLE, physics subset | 8.4 | **15.8** | 16.3 |
-All were measured by us with the same setup. The 8-bit GGUF matches Quark: the differences are within
 run-to-run variation.
 **Compared with other open models** (scores as reported by each developer):
 | Model | Size | GPQA Diamond | AIME 2025 | HLE |
-|---|---|---|---|---|
-| **Quark** | 9B | **87.4** | **91.7** | **22.7** |
-| Quark Q8_0 (GGUF) | 9B, 8-bit | 87.8 | 94.0 | 22.3 |
 | Qwen3-8B | 8B | 59.6 | 69.3 | 4.4 |
 | NVIDIA Nemotron Nano 9B v2 | 9B | 64.0 | 72.1 | 6.5 |
 | gpt-oss-20b | 21B | 71.5 | 91.7 | 10.9 |
 | Qwen3.5-27B | 27B | 85.5 | — | 24.3 |
-| Qwen3.8-27B | 27B | 89.2 | — | 30.8 |
 | gpt-oss-120b | 117B | 80.1 | 92.5 | 14.9 |
 | DeepSeek-R1-0528 | 671B | 81.0 | 87.5 | 17.7 |
-| Kimi K2 Thinking | 1T | 84.5 | 94.5 | 23.9 |
-— = not reported. Sources: each model's official model card; Quark and its GGUF measured by us (see
-above), with the GGUF's differences from Quark within run-to-run variation.
-## Quantized version (GGUF)
-An 8-bit GGUF for llama.cpp, Ollama and LM Studio is available at
-[FirstPrinciplesLabs/quark-GGUF](https://huggingface.co/FirstPrinciplesLabs/quark-GGUF), with the vision
-encoder for image input. Its accuracy is in the [Results](#results) table: it matches Quark on all
-benchmarks.
-## Speed
-Tokens per second on an NVIDIA H200 and a MacBook Pro, for the original bf16 weights and the Q8_0
-[GGUF](#quantized-version-gguf).
 ### llama.cpp
-Speed for a single user with llama.cpp, by how much context is already filled.
-Generation (writing the answer), tokens per second:
-| Hardware | Context | Q8_0 | bf16 |
-|---|---|---|---|
-| NVIDIA H200 | empty | 216 | 185 |
-| NVIDIA H200 | 32K | 191 | 167 |
-| NVIDIA H200 | 100K | 156 | 139 |
-| MacBook Pro, M4, 16 GB | empty | 11.2 | does not fit (18.4 GB) |
-| MacBook Pro, M4, 16 GB | 32K | 8.7 | does not fit |
-Prompt reading (a 512-token prompt), tokens per second:
-| Hardware | Context | Q8_0 | bf16 |
-|---|---|---|---|
-| NVIDIA H200 | empty | 9,106 | 16,587 |
-| NVIDIA H200 | 32K | 7,677 | 12,200 |
-| NVIDIA H200 | 100K | 5,544 | 7,800 |
-| MacBook Pro, M4, 16 GB | empty | 208 | does not fit |
-| MacBook Pro, M4, 16 GB | 32K | 139 | does not fit |
-Generation is limited by memory speed, where the smaller Q8_0 file is faster. Prompt reading is limited
-by compute, where llama.cpp runs bf16 faster on NVIDIA GPUs. For thinking-mode answers, generation
-takes most of the time.
-On Apple Silicon, speed scales with memory bandwidth (about 90% of it is used), so Macs with faster
-memory run proportionally faster. The model needs about 10 GB of memory plus room for the context;
-16 GB Macs can run it with shorter contexts.
 ### vLLM
 vLLM targets NVIDIA and AMD GPUs; on Macs, use llama.cpp or tools built on it.
-With vLLM on one NVIDIA H200, 512-token prompts and 4,096-token answers in thinking mode, speed of each
-request, tokens per second:
-| Concurrent requests | bf16 | Q8_0 GGUF |
-|---|---|---|
-| 1 | 184 | 66 |
-| 8 | 162 | 32 |
-| 24 | 145 | 16 |
-| 64 | 104 | 6.4 |
-| 128 | 71 | 3.3 |
-With vLLM, use the bf16 weights: vLLM runs GGUF files only through the
-[vllm-gguf-plugin](https://github.com/vllm-project/vllm-gguf-plugin), which is much slower (measured with
-vLLM 0.29 and plugin version 0.0.5). The GGUF is meant for llama.cpp, where it is the fastest option for
-a single user (see [llama.cpp](#llamacpp)). For a single user, vLLM with bf16 and llama.cpp run at about
-the same speed (184 and 185 tokens per second); vLLM's advantage is serving many users at once, with each
-of 128 concurrent requests still getting 71 tokens per second. Measured with `evalscope perf`.
 ## Training data
@@ -154,7 +128,7 @@ ollama run firstprinciples/quark
 Recommended sampling (from the Qwen3.5 model card):
 | | Thinking (default) | Non-thinking |
-|---|---|---|
 | `chat_template_kwargs` | `{"enable_thinking": true}` | `{"enable_thinking": false}` |
 | temperature / top_p / top_k | 1.0 / 0.95 / 20 | 1.0 / 1.0 / 40 |
 | presence penalty | 1.5 | 2.0 |
@@ -168,7 +142,7 @@ A hard problem from AIME 2025 (II, problem 15), in thinking mode with the recomm
 > There exist exactly three positive real values of \\(k\\) such that \\(f\\) has a minimum at exactly two real values of \\(x\\). Find the sum of these three values of \\(k\\).
 | | Qwen3.5-9B | **Quark** |
-|---|---|---|
 | Correct, out of 16 attempts | 1 | **10** |
 | Most common answer | 188 ✗ (14 of 16 attempts) | **240 ✓** |
 | Approach | Finds no way in and guesses that k must be one of the existing roots: 18 + 72 + 98 = 188 | Turns "a minimum at two points" into an equation for k and solves it |
@@ -176,22 +150,25 @@ A hard problem from AIME 2025 (II, problem 15), in thinking mode with the recomm
 **How Quark solves it** (summarised from its answer):
 1. If the minimum value \\(m\\) is reached at two points \\(u\\) and \\(v\\), the numerator minus \\(mx\\) must be a perfect square:
 $$
 (x-18)(x-72)(x-98)(x-k)-mx=(x-u)^2(x-v)^2 .
 $$
-2. Matching coefficients and eliminating \\(u\\) and \\(v\\) leaves one equation in \\(k\\):
 $$
 (k^2-376k-5120)^2 = 8128512\,k .
 $$
-3. It factors as \\((k-8)(k-32)(k-200)(k-512)=0\\), and checking the sign before squaring rules out \\(k=512\\).
-4. So \\(k = 8, 32, 200\\), and the answer is **240**.
 Both models' full written answers (each follows several thousand tokens of reasoning, not shown):
 <details>
 <summary><b>Quark</b>: 240 (correct)</summary>
 We interpret “ \\(f\\) has a minimum at exactly two real values of \\(x\\)” as meaning that the minimum value is attained at exactly two distinct \\(x\\)-values (the intended interpretation; otherwise \\(f\\) has no global minimum on all real \\(x\neq0\\)).
 Let
@@ -341,13 +318,11 @@ $$
 \boxed{240}
 $$
 </details>
 <details>
 <summary><b>Qwen3.5-9B</b>: 188 (incorrect)</summary>
 The function is given by \\(f(x)=\frac{(x-18)(x-72)(x-98)(x-k)}{x}\\).
 We are looking for the number of local minima.

 ## Results
+Accuracy (%), measured with [EvalScope](https://github.com/modelscope/evalscope) and vLLM (the int8 version
+with llama.cpp): thinking mode, up to 131,072 output tokens, and following the Qwen3.5 model card's sampling
 settings. AIME 2025 is averaged over 16 samples per problem and GPQA Diamond over 8. HLE (text-only, 2,158
 questions) is graded by Qwen3.5-27B as judge.
+| Benchmark | Qwen3.5-9B | Quark (bf16) | Quark ([int8](#int8-version)) |
+| --- | --- | --- | --- |
+| GPQA Diamond | 81.6 | 87.4 | **87.8** |
+| AIME 2025 | 86.9 | 91.7 | **94.0** |
+| MMLU-Pro Physics | 90.7 | 91.0 | **91.6** |
 | Humanity's Last Exam (HLE), text | 15.9 | **22.7** | 22.3 |
+| HLE, physics subset | 8.4 | 15.8 | **16.3** |
+All were measured by us with the same setup. The int8 version matches Quark: the differences are within
 run-to-run variation.
 **Compared with other open models** (scores as reported by each developer):
 | Model | Size | GPQA Diamond | AIME 2025 | HLE |
+| --- | --- | --- | --- | --- |
+| Quark (fp16) | 9B | 87.4 | 91.7 | 22.7 |
+| Quark (int8) | 9B | 87.8 | 94.0 | 22.3 |
 | Qwen3-8B | 8B | 59.6 | 69.3 | 4.4 |
 | NVIDIA Nemotron Nano 9B v2 | 9B | 64.0 | 72.1 | 6.5 |
 | gpt-oss-20b | 21B | 71.5 | 91.7 | 10.9 |
 | Qwen3.5-27B | 27B | 85.5 | — | 24.3 |
+| Qwen3.8-27B | 27B | **89.2** | — | **30.8** |
 | gpt-oss-120b | 117B | 80.1 | 92.5 | 14.9 |
 | DeepSeek-R1-0528 | 671B | 81.0 | 87.5 | 17.7 |
+| Kimi K2 Thinking | 1T | 84.5 | **94.5** | 23.9 |
+— = not reported. Sources: each model's official model card; Quark (bf16) and its int8 version measured by us
+(see above), with the int8 version's differences from Quark within run-to-run variation.
+## int8 version
+An int8 version for llama.cpp, Ollama and LM Studio (in their GGUF format) is available at
+[FirstPrinciplesLabs/quark-GGUF](https://huggingface.co/FirstPrinciplesLabs/quark-GGUF). Its accuracy is
+in the [Results](#results) table: it matches Quark on all benchmarks.
+## Throughput
+Tokens per second on an NVIDIA H200 and a MacBook Pro, for the original bf16 weights and the
+[int8 version](#int8-version).
 ### llama.cpp
+Throughput for a single user with llama.cpp, by how much context is already filled.
+Output generation, tokens per second:
+| Hardware | Context | int8 | bf16 |
+| --- | --- | --- | --- |
+| NVIDIA H200 | empty | **216** | 185 |
+| NVIDIA H200 | 32K | **191** | 167 |
+| NVIDIA H200 | 100K | **156** | 139 |
+| MacBook Pro, M4, 16 GB | empty | **11.2** | does not fit (18.4 GB) |
+| MacBook Pro, M4, 16 GB | 32K | **8.7** | does not fit |
 ### vLLM
 vLLM targets NVIDIA and AMD GPUs; on Macs, use llama.cpp or tools built on it.
+With vLLM on one NVIDIA H200 (bf16 weights), 512-token prompts and 4,096-token answers in thinking mode,
+throughput of each request, tokens per second:
+| Concurrent requests | Throughput per request |
+| --- | --- |
+| 1 | 184 |
+| 8 | 162 |
+| 24 | 145 |
+| 64 | 104 |
+| 128 | 71 |
 ## Training data
 Recommended sampling (from the Qwen3.5 model card):
 | | Thinking (default) | Non-thinking |
+| --- | --- | --- |
 | `chat_template_kwargs` | `{"enable_thinking": true}` | `{"enable_thinking": false}` |
 | temperature / top_p / top_k | 1.0 / 0.95 / 20 | 1.0 / 1.0 / 40 |
 | presence penalty | 1.5 | 2.0 |
 > There exist exactly three positive real values of \\(k\\) such that \\(f\\) has a minimum at exactly two real values of \\(x\\). Find the sum of these three values of \\(k\\).
 | | Qwen3.5-9B | **Quark** |
+| --- | --- | --- |
 | Correct, out of 16 attempts | 1 | **10** |
 | Most common answer | 188 ✗ (14 of 16 attempts) | **240 ✓** |
 | Approach | Finds no way in and guesses that k must be one of the existing roots: 18 + 72 + 98 = 188 | Turns "a minimum at two points" into an equation for k and solves it |
 **How Quark solves it** (summarised from its answer):
 1. If the minimum value \\(m\\) is reached at two points \\(u\\) and \\(v\\), the numerator minus \\(mx\\) must be a perfect square:
 $$
 (x-18)(x-72)(x-98)(x-k)-mx=(x-u)^2(x-v)^2 .
 $$
+1. Matching coefficients and eliminating \\(u\\) and \\(v\\) leaves one equation in \\(k\\):
 $$
 (k^2-376k-5120)^2 = 8128512\,k .
 $$
+1. It factors as \\((k-8)(k-32)(k-200)(k-512)=0\\), and checking the sign before squaring rules out \\(k=512\\).
+2. So \\(k = 8, 32, 200\\), and the answer is **240**.
 Both models' full written answers (each follows several thousand tokens of reasoning, not shown):
 <details>
 <summary><b>Quark</b>: 240 (correct)</summary>
 We interpret “ \\(f\\) has a minimum at exactly two real values of \\(x\\)” as meaning that the minimum value is attained at exactly two distinct \\(x\\)-values (the intended interpretation; otherwise \\(f\\) has no global minimum on all real \\(x\neq0\\)).
 Let
 \boxed{240}
 $$
 </details>
 <details>
 <summary><b>Qwen3.5-9B</b>: 188 (incorrect)</summary>
 The function is given by \\(f(x)=\frac{(x-18)(x-72)(x-98)(x-k)}{x}\\).
 We are looking for the number of local minima.