Instructions to use Soulfate24/ReaderLM-v2-Paretrix with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Soulfate24/ReaderLM-v2-Paretrix with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Soulfate24/ReaderLM-v2-Paretrix")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Soulfate24/ReaderLM-v2-Paretrix", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Soulfate24/ReaderLM-v2-Paretrix with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Soulfate24/ReaderLM-v2-Paretrix
# Run inference directly in the terminal:
llama cli -hf Soulfate24/ReaderLM-v2-Paretrix

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Soulfate24/ReaderLM-v2-Paretrix
# Run inference directly in the terminal:
llama cli -hf Soulfate24/ReaderLM-v2-Paretrix

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Soulfate24/ReaderLM-v2-Paretrix
# Run inference directly in the terminal:
./llama-cli -hf Soulfate24/ReaderLM-v2-Paretrix

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Soulfate24/ReaderLM-v2-Paretrix
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Soulfate24/ReaderLM-v2-Paretrix

Use Docker

docker model run hf.co/Soulfate24/ReaderLM-v2-Paretrix

LM Studio
Jan

vLLM

How to use Soulfate24/ReaderLM-v2-Paretrix with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Soulfate24/ReaderLM-v2-Paretrix"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Soulfate24/ReaderLM-v2-Paretrix",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Soulfate24/ReaderLM-v2-Paretrix

SGLang

How to use Soulfate24/ReaderLM-v2-Paretrix with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Soulfate24/ReaderLM-v2-Paretrix" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Soulfate24/ReaderLM-v2-Paretrix",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Soulfate24/ReaderLM-v2-Paretrix" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Soulfate24/ReaderLM-v2-Paretrix",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use Soulfate24/ReaderLM-v2-Paretrix with Ollama:
```
ollama run hf.co/Soulfate24/ReaderLM-v2-Paretrix
```
Unsloth Desktop
Docker Model Runner
How to use Soulfate24/ReaderLM-v2-Paretrix with Docker Model Runner:
```
docker model run hf.co/Soulfate24/ReaderLM-v2-Paretrix
```

Lemonade

How to use Soulfate24/ReaderLM-v2-Paretrix with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Soulfate24/ReaderLM-v2-Paretrix

Run and chat with the model

lemonade run user.ReaderLM-v2-Paretrix-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

🧮 Paretrix GGUF quantizations of jinaai/ReaderLM-v2

Model	MiB	PPL	ΔPPL	KLD	RMS Δp	top-p	Pareto
Q8_0 (stock)	1570	12.4136	+0.0284	0.0016	0.99%	97.9%	★
Fidelity-48pc	1422	12.4161	+0.0309	0.0031	1.39%	96.9%	★
Precision-42pc	1214	12.3712†	−0.0140	0.0055	1.84%	95.9%	≡ Q6_K-imx
Q6_K-imx (stock)	1214	12.3712†	−0.0140	0.0055	1.84%	95.9%	≡ Precision-42pc
Quality-36pc	1073	12.4244	+0.0392	0.0165	3.25%	93.1%	≡ Q5_K_M-imx
Q5_K_M-imx (stock)	1073	12.4244	+0.0392	0.0165	3.25%	93.1%	≡ Quality-36pc
Compact-33pc ⭐	979	12.4617	+0.0765	0.0309	4.35%	90.9%	★
Mini-30pc	890	12.6359	+0.2507	0.0553	5.95%	87.7%	≈ IQ4_XS-imx (+0.0027 · −36 MiB)
IQ4_XS-imx (stock)	854	12.5393	+0.1541	0.0580	5.96%	87.6%	★
Nano-27pc	802	12.6418	+0.2566	0.0887	7.39%	84.9%	★
IQ3_M-imx (stock)	741	13.3774	+0.9922	0.1706	10.37%	79.6%	★

ℹ️ About Paretrix Quantization Suite

Paretrix is an empirical, activation-aware quantization suite for GGUF language models and speculative decoding modules.

Pareto + Matrix → Paretrix. Each tier is priced so that no better trade exists at that compression. The suite measures real activation sensitivity (ΔKLD/MiB) per tensor class through llama-imatrix, learns rate tables from cross-architecture campaigns (THE MATRIX), and allocates bitwidths under exact budget targets: flat recipes where uniformity wins, rate-calibrated knapsack where heterogeneity pays.

Standard uniform quantization applies one bitwidth family across the whole network. Paretrix reads the model's own activation field (which classes are expensive to cut, which are nearly free, and where depth matters), then spends the budget where measurements show the greatest return.

Downloads last month: 298

GGUF

Model size

2B params

Architecture

qwen2

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for Soulfate24/ReaderLM-v2-Paretrix

Base model

jinaai/ReaderLM-v2

Quantized

(39)

this model