chengfp-eng commited on
Commit
e06240b
·
verified ·
1 Parent(s): 7f5699e

Add Transformers Python example to Usage

Browse files
Files changed (1) hide show
  1. README.md +34 -0
README.md CHANGED
@@ -145,6 +145,40 @@ MMLU-Pro, SuperGPQA and HLE (text-only) sets.
145
 
146
  Quark runs anywhere Qwen3.5-9B does (Transformers, vLLM, SGLang). We recommend an output budget of 131,072 tokens for best performance.
147
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
148
  ```bash
149
  vllm serve FirstPrinciplesLabs/quark \
150
  --max-model-len 135168 --reasoning-parser qwen3
 
145
 
146
  Quark runs anywhere Qwen3.5-9B does (Transformers, vLLM, SGLang). We recommend an output budget of 131,072 tokens for best performance.
147
 
148
+ With [Transformers](https://github.com/huggingface/transformers) (`pip install -U transformers accelerate`; the bf16
149
+ weights need about 20 GB of GPU memory):
150
+
151
+ ```python
152
+ from transformers import AutoModelForCausalLM, AutoTokenizer
153
+
154
+ model_id = "FirstPrinciplesLabs/quark"
155
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
156
+ model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")
157
+
158
+ messages = [{"role": "user", "content": (
159
+ "A particle is in the ground state of a 1D infinite square well of width L. The well suddenly expands to "
160
+ "width 2L. What is the probability of finding the particle in the ground state of the new well?"
161
+ )}]
162
+ inputs = tokenizer.apply_chat_template(
163
+ messages, add_generation_prompt=True, enable_thinking=True, return_tensors="pt", return_dict=True
164
+ ).to(model.device)
165
+
166
+ output = model.generate(
167
+ **inputs,
168
+ max_new_tokens=32768, # hard problems can need up to 131,072
169
+ do_sample=True, temperature=1.0, top_p=0.95, top_k=20,
170
+ )
171
+ text = tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
172
+
173
+ thinking, finished, answer = text.partition("</think>")
174
+ if not finished:
175
+ print("Stopped while still thinking; raise max_new_tokens.")
176
+ print(answer.strip()) # expected: 32 / (9 pi^2), about 0.36
177
+ ```
178
+
179
+ Transformers' `generate` has no presence penalty, so for long answers and for serving, we recommend vLLM, which
180
+ supports the full recommended sampling settings:
181
+
182
  ```bash
183
  vllm serve FirstPrinciplesLabs/quark \
184
  --max-model-len 135168 --reasoning-parser qwen3