Instructions to use jackcloudman/Leanstral-2603-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jackcloudman/Leanstral-2603-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jackcloudman/Leanstral-2603-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf jackcloudman/Leanstral-2603-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jackcloudman/Leanstral-2603-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf jackcloudman/Leanstral-2603-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jackcloudman/Leanstral-2603-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf jackcloudman/Leanstral-2603-GGUF:Q4_K_M
Use Docker
docker model run hf.co/jackcloudman/Leanstral-2603-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use jackcloudman/Leanstral-2603-GGUF with Ollama:
ollama run hf.co/jackcloudman/Leanstral-2603-GGUF:Q4_K_M
- Unsloth Desktop
- Docker Model Runner
How to use jackcloudman/Leanstral-2603-GGUF with Docker Model Runner:
docker model run hf.co/jackcloudman/Leanstral-2603-GGUF:Q4_K_M
- Lemonade
How to use jackcloudman/Leanstral-2603-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jackcloudman/Leanstral-2603-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Leanstral-2603-GGUF-Q4_K_M
List all available models
lemonade list
- Atomic Chat
Leanstral 119B A6B - GGUF
GGUF quantizations of mistralai/Leanstral-2603 for use with llama.cpp.
Leanstral is the first open-source code agent designed for Lean 4, a proof assistant for formal mathematics and software verification. Built as part of the Mistral Small 4 family, it combines multimodal capabilities with an efficient MoE + MLA architecture.
Available Quantizations
| File | Quant | Size | Description |
|---|---|---|---|
mistralai_Leanstral-128x3.9B-2603-IQ3_XXS.gguf |
IQ3_XXS | 45.8 GB | Smallest. imatrix calibrated on Lean 4. Fits fully in 2x RTX 4090 (48 GB VRAM) at 8k context |
mistralai_Leanstral-128x3.9B-2603-imatrix-Q4_K_M.gguf |
Q4_K_M | 72.2 GB | Q4_K_M with the Lean-calibrated imatrix. Same size and hardware needs as the static one |
mistralai_Leanstral-128x3.9B-2603-Q4_K_M.gguf |
Q4_K_M | 72.2 GB | Static (no imatrix). Best balance of quality and size. Runs on 2x RTX 4090 + RAM offload |
mistralai_Leanstral-128x3.9B-2603-Q8_0.gguf |
Q8_0 | 126 GB | Near-lossless. Good base for custom requantization |
The plain Q4_K_M and Q8_0 are static quants (no imatrix). IQ3_XXS and imatrix-Q4_K_M use the importance matrix below.
Lean-calibrated imatrix
Leanstral-2603-lean-v1.imatrix.gguf is an importance matrix computed with llama-imatrix over 337 chunks of 512 tokens (final PPL 4.28 on the calibration text). You can use it to make your own quants:
./llama-quantize --allow-requantize \
--imatrix Leanstral-2603-lean-v1.imatrix.gguf \
mistralai_Leanstral-128x3.9B-2603-Q8_0.gguf \
mistralai_Leanstral-128x3.9B-2603-Q6_K.gguf Q6_K
Calibration mix, by size: about 45% Mathlib sources, 25% excerpts from a recent large Lean 4 proof development, 5% Mathematics in Lean, and 20% general text (calibration_datav3). The calibration text itself is not included in this repo.
Notes:
- The matrix was computed on the Q4_K_M model, not on full-precision weights.
token_embdandoutputhave no imatrix data (--process-outputwas not used).- One routed expert in block 18 never fired on the calibration text (99.22% coverage for that block).
- No quality comparison against a generic imatrix has been run yet.
Architecture
- Type: Mixture of Experts (MoE) + Multi-head Latent Attention (MLA)
- GGUF arch:
deepseek2(Mistral 4 uses the same architecture as DeepSeek V3) - Total parameters: 119B (6.5B active per token)
- Experts: 128 routed + 1 shared, 4 active per token
- MLA: q_lora_rank=1024, kv_lora_rank=256, qk_rope_head_dim=64
- Context: Up to 1M tokens (256k recommended)
- RoPE: YaRN scaling (factor=128, original_ctx=8192)
- Vocab: 131,072 tokens (Tekken tokenizer)
How to Run
llama-server (recommended)
./llama-server \
-m mistralai_Leanstral-128x3.9B-2603-Q4_K_M.gguf \
-fit on -fa on \
--host 0.0.0.0 \
--ctx-size 128000 \
--jinja \
--chat-template-file chat_template.jinja
Note: You need a chat template that supports
[THINK]blocks for reasoning. Download the template from the original model repo.
Reasoning
The model supports reasoning_effort via the chat template:
"high"- Enables thinking (recommended for Lean 4 proofs and complex tasks)"none"- Direct answers without reasoning
Pass reasoning_effort in your API request body, or modify the chat template default.
Performance
On 2x RTX 4090 (48GB VRAM) + 192GB RAM with Q4_K_M:
- ~34 tokens/s generation speed
- Model splits between GPU and system RAM automatically with
-fit on
On 2x RTX 4090 with IQ3_XXS, fully on GPU (-ngl 999 -c 8192):
- ~143 tokens/s generation speed (short prompt, single request)
- About 1 GB of VRAM left on the first GPU, so larger contexts need a third GPU or RAM offload
Conversion Details
- Source: FP8 (e4m3) consolidated weights from mistralai/Leanstral-2603
- Pipeline: FP8 consolidated โ dequant to BF16 โ Q8_0 GGUF โ Q4_K_M GGUF (with
--allow-requantize) - IQ3_XXS and imatrix-Q4_K_M: requantized from Q8_0 with
--allow-requantize --imatrix Leanstral-2603-lean-v1.imatrix.gguf - Converter:
convert_hf_to_gguf.pywith--mistral-formatflag (llama.cpp) - Tokenizer: Tekken v15 (requires
mistral-common >= 1.10.0for conversion)
License
Apache 2.0 - same as the original model.
Credits
- Original model by Mistral AI
- Quantized by jackcloudman
- Downloads last month
- 98
Model tree for jackcloudman/Leanstral-2603-GGUF
Base model
mistralai/Leanstral-2603