Instructions to use nokeepai/Inkling-2-Mini-Unbound-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Use Docker
docker model run hf.co/nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "nokeepai/Inkling-2-Mini-Unbound-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "nokeepai/Inkling-2-Mini-Unbound-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
- Ollama
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with Ollama:
ollama run hf.co/nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with Docker Model Runner:
docker model run hf.co/nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
- Lemonade
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-Unbound-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use nokeepai/Inkling-2-Mini-Unbound-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "nokeepai/Inkling-2-Mini-Unbound-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling-2-Mini-Unbound v2 GGUF
GGUF builds of nokeepai/Inkling-2-Mini-Unbound v2, an uncensored Inkling 2 Mini, for llama.cpp, LM Studio and Ollama. The main model card has the method, the full evaluation and the limits.
Built by nokeep, who also sponsored the compute.
This model follows harmful instructions. It gives detailed answers to requests for harmful, dangerous or illegal content that the original model declines. It has no safety training layered back on. Do not deploy it to end users without your own safeguards.
v2 or v1.0?
v1.0 stopped the model refusing. v2 is trained on top of v1.0 to stop it dodging: fewer disclaimers, fewer "satire" or "conceptual" rewrites, more answers that actually do what was asked.
| Original | v1.0 | v2 | |
|---|---|---|---|
| Useful answers to 277 held-out harmful requests, thinking off | 0.7% | 51.6% | 78.7% |
| Same, thinking on (effort 0.9) | 0.7% | 62.1% | 83.4% |
| Refused, thinking off / on | 96.0% / 95.7% | 0.0% / 0.0% | 0.4% / 1.1% |
| Benign borderline prompts refused (784) | 38.0% | 0.0% | 0.3% |
| Drift from the original, Q8_0 (KLD, lower is closer) | โ | 0.0102 | 0.0318 |
| MMLU-300 / GSM8K-100 | 76.3 / 97 | 75.7 / 99 | 76.0 / 99 |
| Tensors changed from the original | โ | 20 | 120 |
| Files in this repo | Inkling-2-Mini-Unbound-<TYPE>.gguf |
Inkling-2-Mini-Unbound-v2-<TYPE>.gguf |
Measured on the Q8_0 GGUF in llama.cpp 19bc41dbb, greedy, answers allowed up to 8,192 tokens, on a sealed test
split. "Useful" means an LLM judge (Jev) rates the answer as fulfilling the request.
- Pick v2 for the most useful answers.
- Pick v1.0 for the smallest change from the original model (3ร lower drift), or to use the 7 MB LoRA.
The v1.0 files stay available under their v1.0 names (no -v2) and as the
v1.0 revision of this repo. The
main model card compares both with every
other public uncensored Inkling 2 Mini.
Files
| File | Size | SHA-256 | KLD vs original Q8_0 | Same top token |
|---|---|---|---|---|
| Inkling-2-Mini-Unbound-v2-Q8_0.gguf | 27.8 GB | 60c43f25b64930d72aac4219f688524e31f2cdf421b0ce0f60ae48a5ef70c8fa |
0.0318 | 95.6% |
| Inkling-2-Mini-Unbound-v2-Q6_K.gguf | 23.6 GB | 81ca1e0a751ad4f944872132d7bd808ddd1f53f87a1705c42722d1b43423c00b |
0.0324 | 95.5% |
| Inkling-2-Mini-Unbound-v2-Q5_K_M.gguf | 20.1 GB | f583f25f7b6b4135997299c55918da261ca0f08014851e89294f789f7cd7522e |
0.0328 | 95.4% |
| Inkling-2-Mini-Unbound-v2-Q4_K_M.gguf | 17.8 GB | 55d3aa9c0e5c4c6b666451ae51ce70f6490b99f41959c6bae7e99b31d3f6d0db |
0.0346 | 95.1% |
| Inkling-2-Mini-Unbound-v2-IQ3_M.gguf | 13.3 GB | 3d90872857c28e63650b5b3167b5fb4fe413bd8e5720e4a6871fcba9ac5230ab |
0.0474 | 93.4% |
| Inkling-2-Mini-Unbound-v2-IQ2_M.gguf | 11.2 GB | 91d02db308b9a662675c588fc3ae23bfd24465ffa83ab140a3c2780bfa3604a0 |
0.0638 | 91.2% |
| mmproj-Inkling-2-Mini-BF16.gguf / -Q8_0.gguf | 0.6 / 0.3 GB | vision projector, unchanged from the original |
KLD and top-token agreement are measured with llama.cpp against the original model's Q8_0 (ggml-org) on wikitext-2, 40 chunks of 4,096 tokens. The Q8_0 row is the model's own drift (v1.0: 0.0102); the smaller sizes add quantization loss on top. These numbers are not comparable with the v1.0 table, which was measured against BF16 at 512 tokens.
Which file should I pick?
Rule of thumb: to run fully on the GPU you need about the file size in VRAM, plus a few GB for the context (more
for long contexts). Inkling 2 Mini is a mixture-of-experts model, so if it doesn't fit, llama.cpp can keep some
experts in system RAM (--n-cpu-moe N). That works well, just slower; then VRAM + RAM together need to cover the
file size plus headroom.
| You have | Pick | File size |
|---|---|---|
| 32 GB+ VRAM, or plenty of RAM | Q8_0 (what the evaluation used; near lossless) | 27.8 GB |
| 24โ32 GB VRAM | Q6_K or Q5_K_M (close to Q8_0) | 23.6 GB / 20.1 GB |
| 16โ24 GB VRAM | Q4_K_M (the size most people want) | 17.8 GB |
| 12โ16 GB VRAM | IQ3_M (noticeably lossy) | 13.3 GB |
| Under 12 GB VRAM | IQ2_M (lossy; or Q4_K_M with experts in RAM) | 11.2 GB |
Add the mmproj file (0.3 GB at Q8_0) only if you want image input.
How they were made
- Start from the official file. The base is ggml-org's Inkling-2-Mini-GGUF BF16.
- Patch 120 tensors, byte-exactly. v2 changes the q, k, v, output and gate attention projections of all 24
layers (
attn_q,attn_k,attn_v,attn_output,attn_gate). Before writing each one, the patch script proves the mapping: the original checkpoint's BF16 bytes must equal ggml-org's tensor bytes exactly. Only then are v2's BF16 bytes written in place. The other 584 tensors (every expert, the router, embeddings, head) are ggml-org's bytes, untouched. The only header change is the model name. - Quantize with stock llama.cpp.
llama-quantizefrom the patched BF16. Every size below Q8_0 uses ggml-org's importance matrix, and keeps all five edited attention tensor types at Q8_0 so the change isn't blurred by low-bit rounding.
The main card's evaluation ran on the Q8_0 file in this repo, so its numbers apply to it directly.
Run it
You need a llama.cpp build with Inkling 2 support that includes the loop fix from PR #32704. That fix is on master
(we used commit 19bc41dbb); tagged releases up to b12608 don't have it. Apps that bundle llama.cpp need an equally
recent build.
llama.cpp
hf download nokeepai/Inkling-2-Mini-Unbound-GGUF \
Inkling-2-Mini-Unbound-v2-Q4_K_M.gguf mmproj-Inkling-2-Mini-Q8_0.gguf --local-dir models
./build/bin/llama-server \
-m models/Inkling-2-Mini-Unbound-v2-Q4_K_M.gguf \
--mmproj models/mmproj-Inkling-2-Mini-Q8_0.gguf \
--jinja -c 32768 -ngl 99
LM Studio. Search for nokeepai/Inkling-2-Mini-Unbound-GGUF and pick a file with -v2 in its name. The files
without -v2 are v1.0.
Ollama. Download a file as above, then point a Modelfile at it:
echo 'FROM ./models/Inkling-2-Mini-Unbound-v2-Q4_K_M.gguf' > Modelfile
ollama create inkling-unbound-v2 -f Modelfile
ollama run inkling-unbound-v2
Settings
- Thinking effort is set through the chat template (
reasoning_effort, 0 to 0.99, default 0.9), the same as the original. OpenAI-compatible servers take it as"chat_template_kwargs": {"reasoning_effort": 0}. The model still thinks briefly at 0. - Set a thinking cap (we used 4,096 tokens). At effort 0.9, v2 hits that cap on 17% of harmful prompts.
About the LoRA
The 7 MB Inkling-2-Mini-Unbound-LoRA-r16.gguf in this repo is v1.0's adapter. Loaded on any GGUF of the
original model it gives v1.0 behaviour, not v2. There is no GGUF LoRA for v2. v2's own LoRA (on top of v1.0, in
safetensors) is in the main repo under finetune/.
Known limits
- More drift than v1.0. KLD 0.0318 vs 0.0102. MMLU-300 and GSM8K-100 stay within noise of the original, but expect more wording differences from the original on ordinary requests.
- Long thinking at high effort. 17% of harmful prompts hit a 4,096-token thinking cap at effort 0.9 (v1.0 13%). Set a cap.
- Occasional soft refusal. About 1% of harmful requests at effort 0.9.
- Not every answer is useful. About 17โ21% of harmful requests still get a vague, partial or off-target answer.
- Crisis handling. v2 was not trained to add crisis resources and may leave them out. Deployments that can reach people at risk need their own handling.
- Not yet measured for v2: the full capability suite v1.0 passed, long-context, vision and tool use.
- Full list: main model card.
Intended use and misuse
Intended for research on refusal and model editing, red-teaming and safety evaluation, and for developers who apply their own content policy in a system they control. Not intended for products that serve the public without moderation, content that facilitates violence, weapons capable of mass harm, attacks on computer systems or abuse of children or other people, or anything illegal where you use it. The base model's Acceptable Use Policy applies.
Licence
Apache 2.0, like the base model, Inkling 2 Mini by Thinking Machines Lab, Inc. 120 attention tensors differ from the original; every other tensor is ggml-org's conversion of the original, unmodified before quantization.
About nokeep
This model was built, and its compute paid for, by nokeep. nokeep is a private AI app for uncensored chat, images and video. If this model is useful to you, the best way to support more releases like it is to try nokeep.
- Downloads last month
- 7,852
Model tree for nokeepai/Inkling-2-Mini-Unbound-GGUF
Base model
thinkingmachines/Inkling-2-Mini
