Instructions to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP # Run inference directly in the terminal: llama cli -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP # Run inference directly in the terminal: llama cli -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP # Run inference directly in the terminal: ./build/bin/llama cli -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Use Docker
docker model run hf.co/Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
- Llama new
- LM Studio
- Jan
- vLLM
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
- Ollama
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with Ollama:
ollama run hf.co/Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
- Unsloth Desktop
- Pi
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with Docker Model Runner:
docker model run hf.co/Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
- Lemonade
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Run and chat with the model
lemonade run user.Inkling-2-Mini-Uncensored-ROCmFPX-Q4_0_ROCMFP
List all available models
lemonade list
- Hermes Agent
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Q4_0_ROCMFP" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:# Run inference directly in the terminal:
llama cli -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:# Run inference directly in the terminal:
./build/bin/llama cli -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Use Docker
docker model run hf.co/Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:Inkling 2 Mini Uncensored ROCmFPX (AMD GPUs and Strix Halo)
Update, 21 November 2026: the files in this repository are now the uncensored model. They are built from Thinking Machines' original weights with AtomicChat's uncensoring edit merged in, and they replace the earlier censored files. A 6-bit file was added. The repository and the files were renamed to Inkling-2-Mini-Uncensored (old links redirect here). The model now answers requests the original refuses. The earlier files are still in this repository's commit history.
Hardware-aware GGUF quants of Inkling 2 Mini by Thinking Machines, built with the ROCmFPX weight formats for AMD GPUs (Vulkan and ROCm).
Inkling 2 Mini: 26.1B parameters, 24 layers, 128 experts per layer with 8 active per token (about 3.1B active parameters per layer pass). It is a looped model: layers 6 to 17 run three times per token, so a token passes through 48 layers. Text only here: the vision encoder and the multi-token prediction drafter are not included.
Not an official release. All credit for the model goes to Thinking Machines, and for the uncensoring edit to AtomicChat.
Files
| File | Size | Weights | Fits |
|---|---|---|---|
Inkling-2-Mini-Uncensored-Q8_0_ROCMFPX.gguf |
26.9 GB | everything 8-bit ROCmFPX (token embedding Q8_0) | best quality: Strix Halo, 32 GB AMD GPUs |
Inkling-2-Mini-Uncensored-Q6_K_ROCMFPX.gguf |
21.6 GB | experts 6-bit ROCmFPX, everything else 8-bit ROCmFPX | Strix Halo, 24 to 32 GB AMD GPUs |
Inkling-2-Mini-Uncensored-Q4_0_ROCMFP4_STRIX_O8.gguf |
15.5 GB | experts ROCmFP4, everything else 8-bit ROCmFPX | smallest: 24 GB AMD GPUs, Strix Halo with room for long context |
The names start with a stock type name (Q8_0, Q6_K, Q4_0) only so the Hugging Face file list sorts them by
bit width. The weights are ROCmFPX formats and need the ROCmFPX fork.
The uncensoring edit
AtomicChat's edit is a rank-1 LoRA on the attention output and the routed experts of all 24 layers plus the token embedding. In each layer it removes that layer's refusal direction. Their measurements: no refusals on their harmful test prompts in English and Russian (the original refused 91 to 100%), no refusals on XSTest's safe prompts (the original refused 4%), MMLU 79.15% against 80.65% for the original. See their card for the full table.
Here their adapter (Inkling-2-Mini-Uncensored-LoRA.gguf, revision 40886ab6) is merged at strength 1 into the original
checkpoint in full precision, and that merged model is the reference for the numbers below.
This model will answer requests that the original refuses. You are responsible for how you use it.
Quality
KL divergence against the merged full-precision model, 40 chunks of 512 tokens of English Wikipedia text (wikitext-2), CPU. Lower is better. "Same top token" is how often the file's most likely next token matches the reference.
| Build | Size | Perplexity | KL to reference | Same top token |
|---|---|---|---|---|
| Merged reference (not published) | 50.5 GB | 6.775 | ||
| original model, for comparison (not published) | 22.9 GB | 6.743 | ||
| Q8_0_ROCMFPX | 26.9 GB | 6.758 | 0.0039 | 97.4% |
| Q6_K_ROCMFPX | 21.6 GB | 6.777 | 0.0046 | 96.9% |
| Q4_0_ROCMFP4_STRIX_O8 | 15.5 GB | 6.799 | 0.0119 | 95.2% |
The quantization costs the same as in the earlier censored files (Q8 0.0039, 4-bit 0.0120 against their own reference). The edit itself raises wikitext perplexity by about 0.5% (6.743 to 6.775).
Every file also passed a greedy chat check before upload ("What is the capital of France? Answer in one word." gives "Paris"), and answered three prompts the original refuses.
Running it
Inkling 2 Mini needs the looped inkling2 graph, which is only in the ROCmFPX fork of llama.cpp (branch
inkling2-mini). These files do not load in upstream llama.cpp, and AtomicChat's files do not load in the fork: the
two use different tensor layouts. Build the fork with Vulkan or ROCm/HIP, then for example:
llama-server -m Inkling-2-Mini-Uncensored-Q4_0_ROCMFP4_STRIX_O8.gguf -ngl 99 -c 16384 --jinja
Known limits of this first version:
- Only the CPU path has been checked against the original PyTorch model (it matches to about 5e-6 on a small test model, and the tokenizer matches exactly). Vulkan and ROCm have not been tried yet. Please report in the Community tab whether it runs, how fast, and how the answers look.
- llama.cpp has no parser for Inkling's message format yet. Answers come back with Inkling's own markers
(
<|content_thinking|>,<|end_message|>and so on): the thinking is not split from the answer and tool calls are not parsed. Plain chat works; agent use does not yet. - The chat template accepts
reasoning_effortasnone,minimal,low,medium,high,xhighormax, or as a number from 0 to 0.99 (default 0.9).
How it was made
- Converted from the original checkpoint. The expert down projections, the attention outputs and the token embedding
get the LoRA added in F32; the other NVFP4 experts are kept as they are. Then requantized with
llama-quantize. - ROCmFPX tensor scales on (a power-of-two scale per tensor, stored as
.scale), token embedding Q8_0. - Importance matrix from a general text mix (prose, code, chat), taken on the original weights (a rank-1 edit barely changes it).
Credits and license
- Model: thinkingmachines/Inkling-2-Mini, Apache-2.0.
- Uncensoring edit: AtomicChat/Inkling-2-Mini-Uncensored-GGUF, Apache-2.0.
- Runtime: llama.cpp, with Inkling 2 and ROCmFPX support in ROCmFPX.
Apache-2.0. See LICENSE.
- Downloads last month
- 20
4-bit
6-bit
8-bit
Model tree for Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX
Base model
thinkingmachines/Inkling-2-Mini
Install (macOS, Linux)
# Start a local OpenAI-compatible server with a web UI: llama serve -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX:# Run inference directly in the terminal: llama cli -hf Harsha001/Inkling-2-Mini-Uncensored-ROCmFPX: