Instructions to use araysuter/Inkling-2-Mini-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use araysuter/Inkling-2-Mini-MLX-4bit with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("araysuter/Inkling-2-Mini-MLX-4bit") config = load_config("araysuter/Inkling-2-Mini-MLX-4bit") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use araysuter/Inkling-2-Mini-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "araysuter/Inkling-2-Mini-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "araysuter/Inkling-2-Mini-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use araysuter/Inkling-2-Mini-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "araysuter/Inkling-2-Mini-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default araysuter/Inkling-2-Mini-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use araysuter/Inkling-2-Mini-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "araysuter/Inkling-2-Mini-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "araysuter/Inkling-2-Mini-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Inkling 2 Mini — MLX 4bit
Converted from the official Thinking Machines checkpoint, revision 6e7c8248f75265bdd4fbab5273304243b5d8c2cb.
The source is the authors' trained NVFP4/MXFP8 release, not an original BF16 checkpoint. Conversion to affine MLX adds reconstruction error; 5-bit conversion does not recover higher-precision training weights. This is an independent conversion, not an official Thinking Machines release.
Quantization
- 4-bit affine matrices, group size 32.
- Token embeddings and output head: 8-bit.
- Vision encoder, expert routing and loop-control gates: retained at their source floating-point precision. Vision is included.
- All 24 stored layers and 48 layer applications per token are retained. The twelve middle layers still run three times.
- Weight-only per-group clipping search: factors
1, .995, .985, .97, .94, .9, .85, .8, selected by reconstruction MSE. This is not activation-calibrated GPTQ or AWQ. - Sum of per-group MSE decreased 7.99% versus min/max round-to-nearest for the matrices included in the clipping search. This is a weight statistic, not a benchmark score.
- Target plus vision weights: 17.49 GB / 16.29 GiB. Scale/bias metadata and protected tensors mean the actual average exceeds the nominal bit label.
The architecture has about 26.4B logical parameters, per the upstream model card. A Hub counter that counts packed integer storage may show a different number. The original auxiliary mtp.safetensors is not included or qualified by this conversion; no speculative-decoding speedup is claimed.
Required runtime
This model needs the pinned Inkling MLX-LM port. Stock released MLX-LM did not support this architecture when this artifact was prepared.
python3 -m venv inkling-env
source inkling-env/bin/activate
pip install 'mlx-lm @ git+https://github.com/ml-explore/mlx-lm.git@0c4c99192230f1943dd468d297f0b8a64ae7b7c2'
MLX_ENABLE_TF32=0 mlx_lm generate --model araysuter/Inkling-2-Mini-MLX-4bit --prompt 'Explain a binary search in two sentences.' -m 256
Validated with MLX 0.32.4, Python 3.14, and an Apple M5 Max with 64 GiB unified memory. The port's eight model tests pass with MLX_ENABLE_TF32=0. The default M5 reduced-precision float32 path fails three of those tests; MLX documents the precision switch. The recommended flag retains full float32 execution for those operations. This does not require converting all weights to float32.
For images, the pinned port includes the Inkling image processor and embed_inputs; the included validate_inkling.py shows the tested path with image embeddings. The full upstream context setting is retained in config; 128K and 1M contexts have not been qualified here.
Small validation sample
All artifacts were reloaded from disk before these probes. Same tokenizer and text were used for the source and conversion; the native trained source ran with the same MLX port and precision setting.
| Check | Observed converted output |
|---|---|
| arithmetic | 391 |
| coding | python def larger(a, b): return max(a, b) |
| vision | The image shows a red square on the left and a blue circle on the right. |
| Synthetic text probe | Native source | This conversion |
|---|---|---|
| Mean negative log likelihood | 2.26608 | 2.24673 |
| Perplexity | 9.64151 | 9.45676 |
Delta NLL: -0.01935. This is one short, synthetic text passage plus three smoke prompts, not a standard benchmark or evidence of general benchmark accuracy. Full prompt text, responses, timing and checks are in VALIDATION.json. Reasoning tokens count toward the recorded generation lengths and speeds.
Reproduce
convert_inkling.py converts one profile at a time with bounded tensor tiles; it does not materialize a full BF16 checkpoint. Put the pinned original checkpoint in source/, then run MLX_ENABLE_TF32=0 python convert_inkling.py 4bit. Use INKLING_WORKDIR to select the working directory. The included scripts use the runtime pinned above. QUANTIZATION_MANIFEST.json records every quantized matrix and its selected precision.
Attribution
Original model: Thinking Machines Lab, Inc. Original license: Apache 2.0; see LICENSE, the upstream model card, and its linked Model Acceptable Use Policy. MLX runtime: the MLX contributors. Published quantized weights retain the upstream model license.
- Downloads last month
- -
4-bit
Model tree for araysuter/Inkling-2-Mini-MLX-4bit
Base model
thinkingmachines/Inkling-2-Mini