Instructions to use biscet/Whittle-35B-A3B-DFlash2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use biscet/Whittle-35B-A3B-DFlash2 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16 # Run inference directly in the terminal: llama cli -hf biscet/Whittle-35B-A3B-DFlash2:F16
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16 # Run inference directly in the terminal: llama cli -hf biscet/Whittle-35B-A3B-DFlash2:F16
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16 # Run inference directly in the terminal: ./build/bin/llama cli -hf biscet/Whittle-35B-A3B-DFlash2:F16
Use Docker
docker model run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16
- Llama new
- LM Studio
- Jan
- vLLM
How to use biscet/Whittle-35B-A3B-DFlash2 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "biscet/Whittle-35B-A3B-DFlash2" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "biscet/Whittle-35B-A3B-DFlash2", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16
- Ollama
How to use biscet/Whittle-35B-A3B-DFlash2 with Ollama:
ollama run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16
- Unsloth Desktop
- Pi
How to use biscet/Whittle-35B-A3B-DFlash2 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "biscet/Whittle-35B-A3B-DFlash2:F16" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use biscet/Whittle-35B-A3B-DFlash2 with Docker Model Runner:
docker model run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16
- Lemonade
How to use biscet/Whittle-35B-A3B-DFlash2 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull biscet/Whittle-35B-A3B-DFlash2:F16
Run and chat with the model
lemonade run user.Whittle-35B-A3B-DFlash2-F16
List all available models
lemonade list
- Hermes Agent
How to use biscet/Whittle-35B-A3B-DFlash2 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default biscet/Whittle-35B-A3B-DFlash2:F16
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use biscet/Whittle-35B-A3B-DFlash2 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "biscet/Whittle-35B-A3B-DFlash2:F16" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Whittle 35B-A3B DFlash2
A focused draft for Whittle.
526M parameters · F16 GGUF · v1.2.2 · Apache-2.0
Experimental community release. This drafter is intended for research and evaluation of speculative decoding with the Whittle Q5_K_M target.
Latest training checkpoint — November 16, 2026. v1.2.1 packages the best saved checkpoint from the latest KL-distillation run: step 52, selected by validation correct-prefix length, then token accuracy and selector loss. Training was stopped at step 660; the selected best weights are preserved.
A DFlash2 drafting adapter for speculative decoding with Whittle-Qwen-3.8-35B-A3B, adapted to its Q5_K_M GGUF. Run it alongside the separate Whittle target in a compatible DFlash2 runtime. The target stays frozen; the latest stage updates the feature projection, normalization, last two draft layers and candidate selector.
At a glance
| Specification | |
|---|---|
| Release | v1.2.2, documentation update for the step 52 checkpoint |
| Draft | 526,251,520 parameters · 6 layers · 16-token architecture block |
| Initialization | v1.2.0 followed by selector adaptation; original ancestry: Ornith 1.5 DFlash2 |
| Target | Whittle Q5_K_M GGUF |
| Download | F16 GGUF, about 0.991 GiB · Safetensors |
| License | Apache-2.0, with inherited notices in NOTICE |
What's improved
Recorded offline metrics for v1.0.0 and the checkpoint first published in v1.2.1.
| Metric | v1.0.0 | v1.2.1, step 52 | v1.0.0 vs Last |
|---|---|---|---|
| Validation loss ↓ | 4.2199 | 2.9266 | −30.65% |
| Draft-token accuracy ↑ | 46.90% | 59.64% | +12.73 percentage points |
| Correct-prefix proxy ↑ | 2.3965 tokens | 1.6641 tokens | −30.56% |
| Gold-token recall in top 16 ↑ | 71.23% | 83.59% | +12.36 percentage points |
The latest run processed 2,640 training contexts across 42 completed shards, reaching step 660. The selected checkpoint at step 52 had processed 208 contexts. Selection prioritizes the self-conditioned correct prefix, then accuracy and lower selector loss; it does not simply choose the last step or lowest loss. Full numeric history: EVALUATION.json and training/validation-history.jsonl.
These are offline draft-training metrics.
Training data and recipe
This training stage uses DFlash DeepSeek TraceMix, with a fresh selection of RU/EN project-code, text, reasoning and agent/tool-use contexts. The planned pool contains 16,384 contexts, balanced equally between Russian and English; 2,640 reached the optimizer before the stop. Earlier adaptation used DFlash ChatGPT TraceMix Lite and prompts selected from Tulu 3 SFT Mixture.
Whittle Q5 supplied the training completions, hidden features and full-vocabulary probability targets. DeepSeek Flash acted as a training mentor, reviewing aggregate metrics and advising continuation and learning-rate changes.
The latest stage uses forward KL distillation over all 248,320 vocabulary entries, plus selector cross-entropy. 282,898,944 of 526,251,520 draft parameters are updated; the first four draft layers remain frozen. Training uses AdamW, batch 4, a 384-token feature context, eight anchors per context, FP32 master weights and FP16 autocast. The selected step used LR 2e-6 with a 32-step warmup. See TRAINING.md, DATA_SOURCES.md and PROVENANCE.json.
Next training stage
The latest training run is stopped, with the best export and full local resume state retained. Further training can continue from the saved state when requested; no additional stage has been started for this release.
Files and use
Whittle-35B-A3B-DFlash2-v1.2.1-F16.gguf— inference draft, with author, version, license and checkpoint step embedded in GGUF metadata.tensors/best-model.safetensors— the same 96 trained tensors with their original F16/F32 dtypes, in PyTorch module naming, plus tensor inventory. No optimizer state or target weights.training/v1.2.1/— portable KL trainer, AMP retry helper and recipe;training/validation-history.jsonlrecords the training metrics.
Use the compatible Linux/NVIDIA DFlash2 runtime and a separate Whittle Q5 target. USAGE.md lists the launch parameters and download verification. File and weight checksums are listed in SHA256SUMS.
Previous release measurements and limitations
The v1.2.0 first-stage release selected continuation step 9,074 with the following offline metrics: validation loss 3.8691, draft-token accuracy 50.39%, correct-prefix proxy 2.6211 and top-16 recall 73.83%. Stage one covered 36,296 usable traces across 142 of 176 first-pass shards; the full epoch was not completed. Archived evaluation.
Anonymization of publication text does not imply unlearning of earlier training data; see PRIVACY.md.
main contains v1.2.2. The previous release card is available on the v1.2.1 branch. The v1.2.0 branch, v1.1.0 branch and v1.0.0 branch remain available.
Adaptation by Egor Prokhorov · Published by biscet · License & notices
- Downloads last month
- 191
16-bit
Model tree for biscet/Whittle-35B-A3B-DFlash2
Base model
Qwen/Qwen3.6-35B-A3B