Inkling-2-Mini GGUF

Inkling-2-Mini GGUFs from Nace.AI, starting at 15.734 GB and 4.8 bits per weight.

Results

The 15.734 GB file reaches 0.007144 held-out KL.

KL measures the difference between token distributions; lower is closer. These full-vocabulary raw-logit scores cover 4,088 scored tokens against the original checkpoint with BF16 activations. Calibration, development and evaluation use disjoint text.

File Size, GB Bits per weight
IQ4_XS ยท 4.8 bits per weight 15.734 4.83
Native NVFP4 19.103 5.86

Sizes use decimal GB and include the complete text-backbone GGUF file, excluding vision and MTP. Bits per weight include file overhead.

4.8-bit evaluation ยท Native evaluation

Nace Weight Space Quantization

The 4.8-bit file retains native NVFP4 expert weights and compresses the remaining expert weights. Calibration-fitted error feedback improves ten attention matrices.

Run it

Download the 4.8-bit file and required runtime patch:

hf download nace-ai/Inkling-2-Mini-GGUF \
  Inkling-2-Mini-IQ4_XS-4.8bpw.gguf \
  runtime/inkling2-physical-scale.patch --local-dir .

Build the CUDA runtime used for evaluation on an NVIDIA B200:

git clone --filter=blob:none https://github.com/ggml-org/llama.cpp
git -C llama.cpp fetch origin refs/pull/32704/head
git -C llama.cpp checkout 2a28df0f9bd55c4975b6bffb35229a6020720f13
git -C llama.cpp apply ../runtime/inkling2-physical-scale.patch
cmake -S llama.cpp -B llama.cpp/build \
  -DGGML_CUDA=ON -DGGML_METAL=OFF -DCMAKE_CUDA_ARCHITECTURES=100
cmake --build llama.cpp/build --target llama-cli -j8
./llama.cpp/build/bin/llama-cli \
  -m Inkling-2-Mini-IQ4_XS-4.8bpw.gguf -ngl 999 \
  -p 'Explain merge sort.'

To use the native file, download Inkling-2-Mini-NVFP4.gguf from this repository and use its path with -m.

License

Apache 2.0. See NOTICE, modifications and the source model card's linked use policy.

Downloads last month
820
GGUF
Model size
26B params
Architecture
inkling2_loop
Hardware compatibility
Log In to add your hardware

4-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for nace-ai/Inkling-2-Mini-GGUF

Quantized
(180)
this model