Instructions to use thu-pacman/Puro-2B-Base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use thu-pacman/Puro-2B-Base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="thu-pacman/Puro-2B-Base")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("thu-pacman/Puro-2B-Base")
model = AutoModelForCausalLM.from_pretrained("thu-pacman/Puro-2B-Base", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use thu-pacman/Puro-2B-Base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "thu-pacman/Puro-2B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thu-pacman/Puro-2B-Base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/thu-pacman/Puro-2B-Base

SGLang

How to use thu-pacman/Puro-2B-Base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "thu-pacman/Puro-2B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thu-pacman/Puro-2B-Base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "thu-pacman/Puro-2B-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "thu-pacman/Puro-2B-Base",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use thu-pacman/Puro-2B-Base with Docker Model Runner:
```
docker model run hf.co/thu-pacman/Puro-2B-Base
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Under our fixed 15-benchmark base-model evaluation, one checkpoint in the Puro-2B collection beats Qwen2-1.5B at about $4.4K; the canonical final model goes further, approaching Qwen2.5-1.5B at a measured rental-equivalent accelerator cost of $6,891.

How Far Can a Poor Lab Go with RTX 5090s?

Puro-2B (普罗-2B) is a 2B-parameter dense causal language model pretrained from scratch on 1.4T tokens. It uses a Qwen3-1.7B-compatible architecture with untied input and output embeddings, blockwise FP8 training, the MuonH optimizer, and a two-phase data recipe. Training ran entirely on consumer-grade NVIDIA RTX 5090 GPUs.

The architecture is based on the Qwen3-1.7B configuration, not on pretrained Qwen weights. Puro-2B starts from random initialization.

Puro-2B cost and quality comparison

Why Puro-2B?

Puro-2B is intended to make billion-parameter pretraining inspectable and affordable for smaller research groups. The release covers more than the final weights:

A canonical 2B base model and intermediate or controlled checkpoints: this repo.
Materialized pretraining data: https://huggingface.co/datasets/thu-pacman/Puro-2B.
Training implementation: https://github.com/thu-pacman/Puro-Megatron.
Data-processing implementation: https://github.com/thu-pacman/Kaiyuan-Spark.
Technical report: https://arxiv.org/abs/2608.27370.

The main recipe combines RTX 5090 infrastructure, blockwise FP8, MuonH with hyperball constraints, proxy-guided data selection, and a curriculum-aware late continuation followed by checkpoint averaging.

Puro-2B end-to-end training pipeline

The $5,090 Result, Explained

The collection contains multiple checkpoints with different Phase 2 budgets and recipes. The report's approximately $4.4K result is an observed uniform-recipe checkpoint that already exceeds Qwen2-1.5B on the report's 15-task aggregate. It is not the canonical final checkpoint.

The canonical Puro-2B-Base model is the strongest released endpoint. Its production run used 22,514 measured active-training GPU-hours, corresponding to $6,891 under the report's normalized RTX 5090 rental rate.

These figures are accelerator-only reproduction estimates. They exclude data acquisition and preprocessing, proxy and ablation experiments, failed runs, post-training, evaluation, storage, networking, and research labor. They should not be read as the total cost of developing the project.

Puro-2B estimated efficiency factors

Puro Cost Scaling Law across five Phase 2 budgets

The scaling-law panel labels points by cumulative reproduction cost. The model catalog below maps those costs to Phase 2 budget fractions. Each fraction applies only to Phase 2 data exposure, while the cost includes the shared Phase 1 run.

Model Details

Property	Value
Model type	Dense decoder-only causal language model
Parameters	Approximately 2B
Initialization	From scratch
Architecture	Qwen3-1.7B configuration with untied embeddings
Hidden size	2,048
Transformer layers	28
Attention heads / KV heads	16 / 8
Feed-forward size	6,144
Vocabulary size	151,936
Context length	4,096 tokens
Export class	`Qwen3ForCausalLM`
Weight format	Safetensors

This is a pretrained base model. It has not been instruction-tuned or preference-aligned and should not be expected to behave like a chat assistant.

Quickstart

Use a Transformers release that supports the Qwen3 configuration:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thu-pacman/Puro-2B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

prompt = "The central limit theorem states that"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Evaluation

All numbers below come from the same deterministic OpenCompass pipeline in the technical report. The comparison uses pretrained/base checkpoints throughout. Generation tasks use greedy decoding; multiple-choice tasks use fixed token-likelihood ranking. Scores are percentages.

Model	Math + Code (4)	Reasoning + Knowledge (11)	Overall (15)
Qwen2-1.5B	40.29	60.54	55.14
Puro-2B	43.50	63.02	57.81
Qwen2.5-1.5B	47.52	65.53	60.73

The four math and code tasks are GSM8K, MATH, sanitized-MBPP, and HumanEval. The eleven reasoning and knowledge tasks are MMLU, MMLU-Pro, ARC-Challenge, ARC-Easy, BoolQ, CommonsenseQA, HellaSwag, PIQA, SocialIQA, WinoGrande, and BBH. Each displayed average is an unweighted arithmetic mean.

The Puro Cost Scaling Law fits five single-run Phase 2 uniform-budget points. It is a recipe-specific empirical scale-down relationship, not a universal law. The fit has no uncertainty interval, and the available experiments do not isolate curriculum ordering, constant-LR continuation, and checkpoint averaging as independent causal gains.

Training

Setting	Phase 1	Phase 2
Tokens consumed	439B	960B
RTX 5090 GPUs	24	96
Parallelism (TP / PP / DP)	1 / 2 / 12	1 / 4 / 24
Base learning rate	`5.00e-3 -> 1.04e-3`	`1.04e-3 -> 1.00e-5`
Schedule	Power decay	Linear decay, then selected constant-LR continuation
Median TFLOP/s/GPU	238	192

Both phases use a sequence length of 4,096, a global batch size of 1,536 sequences, and a micro-batch size of 2. Main Transformer linear-layer GEMMs use blockwise E4M3 FP8; numerically sensitive operations, master weights, and optimizer states remain in BF16 or FP32 as appropriate.

Selected approximately scale-invariant matrix weights are updated by MuonH with hyperball projection and zero weight decay. The remaining parameters use AdamW with weight decay 0.1. The MuonH matrix group applies a 10x multiplier to the shared base learning-rate schedule.

The final model uses an equal-weight parameter average of six checkpoints from the constant-LR branch resumed at optimizer step 218,000:

222100, 222200, 222300, 222400, 222500, 222569

Only model parameters are averaged; optimizer states are not.

Model Catalog

Repository	Role
Puro-2B-Base	Canonical final model; equal-weight SMA of six checkpoints from the constant-LR curriculum branch resumed at step 218,000.
Puro-2B-Base-Phase1	Shared Phase 1 endpoint used to initialize the released uniform and curriculum Phase 2 branches.
Puro-2B-Curriculum-DecayFinal	Unaveraged endpoint of the curriculum-ordering Phase 2 linear-decay trajectory; the no-constant-LR comparison.
Puro-2B-Curriculum-SMA6-Inputs	Artifact collection containing the six late checkpoints averaged to produce Puro-2B-Base; not a separately averaged model.
Puro-2B-Uniform	Full-budget uniform-ordering Phase 2 endpoint; approximately $6.9K cumulative reproduction cost.
Puro-2B-Uniform-Phase2-1of2	Endpoint of the uniform-ordering branch using 1/2 of the Phase 2 token budget; approximately $4.4K cumulative reproduction cost.
Puro-2B-Uniform-Phase2-1of4	Endpoint of the uniform-ordering branch using 1/4 of the Phase 2 token budget; approximately $3.1K cumulative reproduction cost.
Puro-2B-Uniform-Phase2-1of8	Endpoint of the uniform-ordering branch using 1/8 of the Phase 2 token budget; approximately $2.5K cumulative reproduction cost.
Puro-2B-Uniform-Phase2-1of16	Endpoint of the uniform-ordering branch using 1/16 of the Phase 2 token budget; approximately $2.2K cumulative reproduction cost.

Author-controlled model weights, training code, processing code, and documentation are released under Apache License 2.0 where marked. The materialized dataset is distributed under other because its components retain different upstream terms; see the dataset card's license matrix and notices.

Intended Use and Limitations

Puro-2B is intended for research on pretraining, data recipes, optimization, model scaling, continued pretraining, and downstream adaptation. It can also be used as a compact base model for task-specific post-training.

The model may produce inaccurate, biased, unsafe, offensive, or copyrighted content. Its pretraining data includes web text, code, mathematics, Chinese and English material, synthetic data, and instruction-formatted examples. The release does not claim exhaustive removal of personal information, benchmark contamination, or undesirable content. Evaluate and post-train the model for your domain before deployment, and add application-specific safeguards where people could be affected by its outputs.

License

The Puro-2B model weights are released under the Apache License 2.0. The training data remains subject to the dataset repository's documented upstream licenses and terms.

Citation

Please cite our technical report if you find our work useful:

@misc{luo2026puro2b,
  title        = {Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within {\$}5090},
  author       = {Kairong Luo and Jiarui Cui and Yaorui Yin and Shengqi Chen and
                  Yiming Yang and Linxiang Gao and Yanmohan Wang and Mingzhe Zhang and
                  Kaiyue Wen and Kaifeng Lyu and Wenguang Chen},
  year         = {2026},
  eprint       = {2608.27370},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url          = {https://arxiv.org/abs/2608.27370}
}