Instructions to use nativ-community/Inkling-2-Mini-nvfp4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use nativ-community/Inkling-2-Mini-nvfp4 with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("nativ-community/Inkling-2-Mini-nvfp4") config = load_config("nativ-community/Inkling-2-Mini-nvfp4") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use nativ-community/Inkling-2-Mini-nvfp4 with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "nativ-community/Inkling-2-Mini-nvfp4"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "nativ-community/Inkling-2-Mini-nvfp4" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use nativ-community/Inkling-2-Mini-nvfp4 with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "nativ-community/Inkling-2-Mini-nvfp4"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default nativ-community/Inkling-2-Mini-nvfp4
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use nativ-community/Inkling-2-Mini-nvfp4 with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "nativ-community/Inkling-2-Mini-nvfp4"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "nativ-community/Inkling-2-Mini-nvfp4" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling 2 Mini — MLX
MLX conversion of thinkingmachines/Inkling-2-Mini, preserving its native mixed precision. The checkpoint accepts text and images and generates text.
The first 20 MoE layers retain NVFP4 expert weights; the last four retain MXFP8. Other parameters retain their original BF16/FP32 dtypes. The 24 physical decoder layers execute 48 layer visits through the model's three-pass middle loop. The original bundled DSpark draft is included as mtp.safetensors.
Target weights: 19.53 GB across four shards. Bundled draft: 0.64 GB. The upstream model recommends Apple silicon with 32 GB or more memory; longer contexts need additional memory.
Required runtime
This release includes a patched MLX-VLM build with Inkling 2 Mini loop support. Use this build for the examples below; the upstream base revision did not yet support inkling2_loop.
python -m pip install --upgrade "https://huggingface.co/nativ-community/Inkling-2-Mini-nvfp4/resolve/main/runtime/mlx_vlm-0.8.0+inkling2mini-py3-none-any.whl"
The wheel is based on MLX-VLM commit 392052b4439c. Its source changes are available in runtime/inkling2-mini.patch. The wheel contains the Python runtime source, and its code retains the MLX-VLM license and upstream notices.
Text generation
from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
model_id = "nativ-community/Inkling-2-Mini-nvfp4"
model, processor = load(model_id)
prompt = apply_chat_template(
processor, model.config,
"Explain in one sentence why ice floats on water.",
reasoning_effort="none",
)
result = generate(
model, processor, prompt,
max_tokens=256, temperature=0, prefill_step_size=128,
)
print(result.text)
The model may still emit reasoning when reasoning_effort="none". Its message tokens distinguish reasoning from the final answer; stripping all special tokens concatenates these sections.
Image input
prompt = apply_chat_template(
processor, model.config, "Describe this image.",
num_images=1, reasoning_effort="none",
)
result = generate(
model, processor, prompt, image=["image.png"],
max_tokens=256, temperature=0, prefill_step_size=128,
)
print(result.text)
Bundled speculative draft
from mlx_vlm.speculative.drafters import load_drafter
draft, kind = load_drafter(model_id)
result = generate(
model, processor, prompt, image=["image.png"],
draft_model=draft, draft_kind=kind,
max_tokens=256, temperature=0, prefill_step_size=128,
)
print(result.text)
Validation and provenance
- Strict checkpoint reload succeeded; all 1,271 target tensors are exactly equal to the sanitized source tensors, including packed weights, scales, and floating-point parameters.
- 67 targeted tests passed, covering native and converted loading, mixed precision, loop caches, image inputs, and speculative generation/reset.
- The loop message gates and message attention were compared with the upstream Transformers implementation on small FP32 tensors; maximum absolute errors were below 4e-7.
- Real-weight smoke checks on an Apple M3 Ultra correctly answered arithmetic, explained ice buoyancy, and recognized a red circle. Standard and bundled-draft arithmetic generation produced the same 45 greedy tokens.
These checks establish conversion integrity and basic inference behavior. They do not establish full-model numerical parity with the original runtime or benchmark accuracy. MLX-VLM uses floating-point activations rather than the reference runtime's activation quantization. The advertised 1M context window was not exercised in these checks.
Source revision: 6e7c8248f75265bdd4fbab5273304243b5d8c2cb. See conversion.json, validation.json, and the upstream model card for architecture, training, evaluations, limitations, and licensing.
Reproduce the conversion
With the included runtime installed:
python -m mlx_vlm convert \
--hf-path thinkingmachines/Inkling-2-Mini \
--revision 6e7c8248f75265bdd4fbab5273304243b5d8c2cb \
--mlx-path Inkling-2-Mini-nvfp4
Copy the original mtp.safetensors into the output folder to retain the bundled draft. This conversion does not requantize the source experts.
- Downloads last month
- -
4-bit
Model tree for nativ-community/Inkling-2-Mini-nvfp4
Base model
thinkingmachines/Inkling-2-Mini