Instructions to use RegalInvader/Marco-Nano-Instruct-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use RegalInvader/Marco-Nano-Instruct-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0
# Run inference directly in the terminal:
llama cli -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0
# Run inference directly in the terminal:
llama cli -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0
# Run inference directly in the terminal:
./build/bin/llama cli -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Use Docker

docker model run hf.co/RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Llama new
LM Studio
Jan

vLLM

How to use RegalInvader/Marco-Nano-Instruct-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "RegalInvader/Marco-Nano-Instruct-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "RegalInvader/Marco-Nano-Instruct-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Ollama
How to use RegalInvader/Marco-Nano-Instruct-GGUF with Ollama:
```
ollama run hf.co/RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0
```
Unsloth Desktop

How to use RegalInvader/Marco-Nano-Instruct-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use RegalInvader/Marco-Nano-Instruct-GGUF with Docker Model Runner:
```
docker model run hf.co/RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0
```

Lemonade

How to use RegalInvader/Marco-Nano-Instruct-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Run and chat with the model

lemonade run user.Marco-Nano-Instruct-GGUF-TQ1_0

List all available models

lemonade list

Hermes Agent

How to use RegalInvader/Marco-Nano-Instruct-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use RegalInvader/Marco-Nano-Instruct-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "RegalInvader/Marco-Nano-Instruct-GGUF:TQ1_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Marco-Nano-Instruct-GGUF / code /README.md

RegalInvader

Release: Marco-Nano-Instruct 1-, 2- and 3-bit GGUF for phones

e5235fb verified 3 days ago

preview code

Raw History Blame Contribute Delete

3.44 kB

Low-bit Marco-Nano-Instruct: quantisation and evaluation code

The codebook and quantiser code and the benchmark and evaluation scripts behind the 1-, 2- and 3-bit GGUF builds of Marco-Nano-Instruct (8B parameters, 0.6B active, 232 experts). The training code (the trainer, losses, data pipeline and run configs) is not published.

This is research code, published as it was used. Machine-specific paths are replaced by placeholders (/path/to/data, /path/to/repo, /path/to/llama.cpp), so set them for your machine before running anything.

What is here

Path	Contents
`src/lowbit/quant/`	The quantisers and codebook formats: ternary and N-bit grids, the IQ codebooks (IQ2_XXS, IQ2_S, IQ3_S, IQ4_NL), importance weighting, GPTQ-style rounding
`src/lowbit/export/`	GGUF reading and writing for those formats, bit-exact with llama.cpp
`src/lowbit/model/`	The quantised student model the export and KL tools load
`src/lowbit/eval/`, `src/lowbit/gen/`	Evaluation helpers (GSM8K scoring, retention, and the RS suites, which are fixed question subsets of the benchmarks for comparing a quantised model with the original) and the loop detector
`scripts/`	Quantisation and GGUF tools and the benchmarks. Each file's docstring says what it does and how to run it
`configs/eval/`, `configs/precision/`	The evaluation suites and the per-part formats
`tests/`	Unit tests for the formats, GGUF round trips and the evaluation code. Some need a GPU or tens of GB of memory

Quantisation and GGUF tools

Some file names carry the internal task or decision they came from: t27 is the GGUF export task, d58 the codebook-training decision and cb stands for codebook.

scripts/ptq_baseline.py: llama.cpp's own quantisations of a model (imatrix) and their KL divergence to the bf16 model (llama-perplexity --kl-divergence, 40 × 4,096 held-out tokens).
scripts/cb_import.py / scripts/cb_gate.py: read a llama.cpp GGUF's codebook codes and scales into the quantised model, and check that it exports back bit-exactly.
scripts/t27_gguf.py build (GGUF export): write a quantised model as a GGUF in its format.
scripts/tq1_repack.py: repack 2-bit gate and up projections as llama.cpp's ternary TQ1_0.
scripts/rtn_gguf.py, scripts/imatrix_to_importance.py: round-to-nearest quantisation with importance weights.

Benchmarks and evaluation

scripts/compare_quality.py: multiple choice (MMLU-Redux, MMLU-Pro, Global-MMLU, Belebele), GSM8K, MATH-500 and IFEval through each model's own chat template, one llama-server per model.
scripts/code_eval.py: HumanEval+ and MBPP+ with EvalPlus's tests in a no-network Docker sandbox.
scripts/ship_rule.py: paired comparison of two files on the same items (2-standard-error rule) plus KL.
scripts/sampling_check.py: looping replies under different sampling settings.
scripts/pod_kl.py: quick KL of a quantised model against the original with PyTorch.
scripts/card_tables.py (card tables), scripts/d58_scaling_chart.py (scaling chart): the model card's tables and chart from the result files.

Requirements

Python 3.12, PyTorch, and the packages in pyproject.toml.
llama.cpp for quantisation, KL and serving. Q2_0 and TQ1_0 are standard llama.cpp types.
Docker for the code benchmark.

Licence

Apache-2.0 (see LICENSE).