Diogenes-4B
Reasoning in ~12x fewer tokens. Diogenes-4B is a LoRA adapter for Qwen/Qwen3.5-4B that teaches the model to think in a compact keyword/symbol notation instead of long prose: one step per sentence, math written as symbols, no filler, ending with answer \boxed{...} (math) or answer code. plus a fenced program (code).
The notation is a propositional-logic-style language: short statements joined by connectives (if, then, so, and, or, not, because), derived facts marked with so, case splits closed with ok or contradiction, and math written in plain symbols. It is not a formal logic system, but a trace reads like a chain of propositions and inferences.
Named after Diogenes, who said what he had to say and no more.
Results
Qwen3.5-4B in thinking mode averages ~13.4k reasoning tokens on MATH-500. With Diogenes-4B it averages ~1.1k, while keeping most of the accuracy and all of the code ability on our test.
| Eval (seed 42) | Base Qwen3.5-4B | Diogenes-4B |
|---|---|---|
| MATH-500, n=100 | 84 (32k cap, still truncated on 20 problems) | 75 (regraded) / 64 (strict) |
| Reasoning tokens, MATH-500 | 13.4k | 1.1k (~12x fewer) |
| GSM8K, n=100 | 88 | 80 |
| Reasoning tokens, GSM8K | 3.1k | 114 (~27x fewer) |
| Code, 20 HumanEval-style tasks | 14/20 | 14/20 |
| Valid notation | n/a | 83% MATH-500, 99% GSM8K |
- About 89% of the base's MATH-500 accuracy and 91% of its GSM8K accuracy at 8% and 4% of the tokens. Code ties.
- "Regraded" accepts equivalent answer forms (e.g.
x=3vs3) and never changes the value; "strict" is the raw string match. - Base numbers on MATH-500 are a floor, since 20 base generations hit the 32k cap.
- Single seed, our own harness, small samples (n=100; 20 code tasks).
Before and after
A GSM8K test problem (both answers correct):
Martha's cat is 5 times faster than her turtle. If the cat can run 15 feet/second, how many feet can her turtle crawl in 40 seconds?
The raw reasoning text each model produced, untouched.
| Qwen3.5-4B, thinking mode (1022 tokens) | Diogenes-4B (58 tokens) |
|---|---|
Thinking Process: |
cat_speed = 15. cat_speed = 5*turtle_speed. so turtle_speed = cat_speed/5. so turtle_speed = 3. time = 40. distance = turtle_speed*time. so distance = 120. answer 120. |
Same answer, about 18x fewer tokens. (Line breaks in the Diogenes column were added for readability; the model writes it as one line.)
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = "Qwen/Qwen3.5-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "ademRguez/Diogenes-4B")
msgs = [{"role": "user", "content": "What is the sum of the first 10 positive odd integers?"}]
prompt = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=True)
out = model.generate(**tok(prompt, return_tensors="pt").to(model.device), max_new_tokens=2048,
do_sample=True, temperature=1.0, top_p=0.95, top_k=20)
print(tok.decode(out[0], skip_special_tokens=False))
Use thinking mode (enable_thinking=True); the compressed notation appears in the thinking. The notation is described in NOTATION_SPEC.md.
Training
Fresh LoRA (r=16, alpha=32, attention + MLP projections) on Qwen/Qwen3.5-4B, 6,666 rows: 4,062 OpenR1-Math traces, 1,000 GSM8K and 1,604 OpenThoughts code traces, all rewritten into the notation and validated locally. 3 epochs (627 steps), bf16, max sequence length 2048.
Support
If you find this useful, consider buying me a coffee to support continued development:
- Downloads last month
- 43