Instructions to use Hygaard/Inkling-2-Small-oQ2e with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use Hygaard/Inkling-2-Small-oQ2e with MLX:
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("Hygaard/Inkling-2-Small-oQ2e") config = load_config("Hygaard/Inkling-2-Small-oQ2e") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use Hygaard/Inkling-2-Small-oQ2e with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Hygaard/Inkling-2-Small-oQ2e"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Hygaard/Inkling-2-Small-oQ2e" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Hermes Agent
How to use Hygaard/Inkling-2-Small-oQ2e with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Hygaard/Inkling-2-Small-oQ2e"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default Hygaard/Inkling-2-Small-oQ2e
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use Hygaard/Inkling-2-Small-oQ2e with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "Hygaard/Inkling-2-Small-oQ2e"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "Hygaard/Inkling-2-Small-oQ2e" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from Hygaard/Inkling-2-Small-oQ2e: direct link, hf CLI and curl.
- Browser
- Download file 7.7 kB
-
https://huggingface.co/Hygaard/Inkling-2-Small-oQ2e/resolve/main/README.md
- Command line
-
hf download hf://Hygaard/Inkling-2-Small-oQ2e/README.md
-
curl -L -o README.md https://huggingface.co/Hygaard/Inkling-2-Small-oQ2e/resolve/main/README.md
7.7 kB
| license: apache-2.0 | |
| base_model: thinkingmachines/Inkling-2-Small-MXFP8 | |
| base_model_relation: quantized | |
| library_name: mlx | |
| pipeline_tag: image-text-to-text | |
| tags: | |
| - mlx | |
| - omlx | |
| - inkling2 | |
| - oq2e | |
| - quantized | |
| - image-text-to-text | |
| # Inkling-2-Small oQ2e | |
| **oQ2e mixed-precision MLX conversion of [Thinking Machinesβ Inkling-2-Small-MXFP8](https://huggingface.co/thinkingmachines/Inkling-2-Small-MXFP8) for Apple Silicon.** | |
| **Effective weight storage: 2.61 BPW Β· 194.72 GB Β· group size 64.** | |
| The full expert set, vision encoder/projector, original tokenizer and chat template are retained, along with the original BF16 multi-token-prediction (MTP) sidecar. All 48 text layers retain 384 routed experts and top-8 routing. | |
| Other precision: [mixed 2/3-bit oQe (3.07 BPW)](https://huggingface.co/Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX), **preferred for 256 GB Apple Silicon hardware**. | |
| ## Conversion | |
| - **Source:** published MXFP8 checkpoint, `thinkingmachines/Inkling-2-Small-MXFP8@086a047e8b7ff2ac902dd52355328ddb66b16914`. | |
| - **Format:** MLX safetensors; oQ2e mixed precision, affine packing, group size 64. | |
| - **Tooling:** oMLX native Inkling 2/oQe converter, [PR #5229](https://github.com/jundot/omlx/pull/5229). | |
| - **Calibration:** 384 samples of 512 tokens with text/image inputs; full expert coverage. | |
| - **Weights:** 33 main shards plus the MTP sidecar, **194.72 GB / 181.34 GiB**. | |
| Conversion records are in [`CONVERSION.json`](./CONVERSION.json). | |
| ## Mixed precision | |
| The packed quantized matrices average **2.066 bits per weight**. The **2.61 effective BPW** includes quantization metadata, unquantized tensors and safetensors headers. | |
| | Component | Weight precision | Original scalar parameters | Stored GB | | |
| |---|---:|---:|---:| | |
| | Attention-residual controls | FP32 | 516,096 | 0.002064 | | |
| | MTP / draft | BF16 | 1,711,710,384 | 3.423421 | | |
| | Output head | 8-bit | 926,318,592 | 0.984214 | | |
| | Text attention | 8-bit, BF16 | 4,713,811,968 | 4.862202 | | |
| | Text convolutions | BF16 | 2,162,688 | 0.004325 | | |
| | Text normalization | BF16 | 525,312 | 0.001051 | | |
| | Text routed experts | 2-bit | 587,068,342,272 | 183.458857 | | |
| | Text routers | FP32 | 84,953,088 | 0.339812 | | |
| | Token embeddings | 8-bit | 926,318,592 | 0.984214 | | |
| | Vision encoder | BF16 | 292,041,728 | 0.584083 | | |
| | Vision merger / projector | BF16 | 35,651,584 | 0.071303 | | |
| | Safetensors headers | β | β | 0.000201 | | |
| | **Total** | **2.61 effective BPW** | **595,762,352,304** | **194.715747** | | |
| Stored GB uses decimal units. MTP is included; no tensors are counted twice. | |
| **Calculation:** `8 Γ 194,715,747,050 stored bytes Γ· 595,762,352,304 original scalar parameters = 2.614677 BPW`. Tokenizer/config files, KV cache and runtime workspace are excluded. | |
| <details> | |
| <summary>Per-block precision map</summary> | |
| | Block | Routed expert matrices | Other quantized matrices | Unquantized tensors | Stored GB | | |
| |---|---:|---:|---|---:| | |
| | Text global | β | 8-bit | BF16, FP32 | 1.968464 | | |
| | Text 0 | 2-bit | 8-bit | BF16, FP32 | 3.930511 | | |
| | Text 1 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 2 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 3 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 4 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 5 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 6 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 7 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 8 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 9 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 10 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 11 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 12 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 13 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 14 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 15 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 16 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 17 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 18 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 19 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 20 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 21 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 22 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 23 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 24 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 25 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 26 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 27 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 28 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 29 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 30 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 31 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 32 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 33 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 34 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 35 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 36 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 37 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 38 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 39 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 40 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 41 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Text 42 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 43 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 44 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 45 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 46 | 2-bit | 8-bit | BF16, FP32 | 3.930530 | | |
| | Text 47 | 2-bit | 8-bit | BF16, FP32 | 3.930889 | | |
| | Vision global | β | β | BF16 | 0.075866 | | |
| | Vision 0 | β | β | BF16 | 0.025197 | | |
| | Vision 1 | β | β | BF16 | 0.025197 | | |
| | Vision 2 | β | β | BF16 | 0.025197 | | |
| | Vision 3 | β | β | BF16 | 0.025197 | | |
| | Vision 4 | β | β | BF16 | 0.025197 | | |
| | Vision 5 | β | β | BF16 | 0.025197 | | |
| | Vision 6 | β | β | BF16 | 0.025197 | | |
| | Vision 7 | β | β | BF16 | 0.025197 | | |
| | Vision 8 | β | β | BF16 | 0.025197 | | |
| | Vision 9 | β | β | BF16 | 0.025197 | | |
| | Vision 10 | β | β | BF16 | 0.025197 | | |
| | Vision 11 | β | β | BF16 | 0.025197 | | |
| | Vision 12 | β | β | BF16 | 0.025197 | | |
| | Vision 13 | β | β | BF16 | 0.025197 | | |
| | Vision 14 | β | β | BF16 | 0.025197 | | |
| | Vision 15 | β | β | BF16 | 0.025197 | | |
| | Vision 16 | β | β | BF16 | 0.025197 | | |
| | Vision 17 | β | β | BF16 | 0.025197 | | |
| | Vision 18 | β | β | BF16 | 0.025197 | | |
| | Vision 19 | β | β | BF16 | 0.025197 | | |
| | Vision 20 | β | β | BF16 | 0.025197 | | |
| | Vision 21 | β | β | BF16 | 0.025197 | | |
| | Vision 22 | β | β | BF16 | 0.025197 | | |
| | MTP global | β | β | BF16 | 0.120264 | | |
| | MTP 0 | β | β | BF16 | 0.660631 | | |
| | MTP 1 | β | β | BF16 | 0.660631 | | |
| | MTP 2 | β | β | BF16 | 0.660631 | | |
| | MTP 3 | β | β | BF16 | 0.660631 | | |
| | MTP 4 | β | β | BF16 | 0.660631 | | |
| </details> | |
| Complete per-module/tensor accounting: [CSV](./WEIGHT_PRECISION.csv) and [JSON](./WEIGHT_PRECISION.json). | |
| ## Running | |
| Requires oMLXβs native Inkling 2 implementation ([PR #5229](https://github.com/jundot/omlx/pull/5229), tested at `2a18263b91e14ee92ed984f8c5a56d0053178cea`). Native loader: `omlx.patches.inkling2.vlm.load`. Keep the included processor, tokenizer, chat template and complete `tiktoken/` directory; generic `mlx-lm` loading is not supported. | |
| **Cold loading, text and image generation tested on a 256 GB M5 Ultra.** Main-model MLX allocator peak: **179.32 GiB**. Runtime and context state need additional memory. MTP decoding, broader quality and full-context capacity remain untested. | |
| ## Credits and license | |
| Model: **Thinking Machines Lab**. Conversion tooling: **oMLX / oQe**. Community conversion: **Hygaard**. | |
| Licensed under [Apache-2.0](./LICENSE). See [`NOTICE`](./NOTICE) for attribution and modification notes. | |