README.md

---
base_model: kd13/Type-o1-mini-instruct
library_name: transformers
pipeline_tag: text-generation
license: mit
language:
- en
- hi
tags:
- gguf
- llama.cpp
- quantized
---

# Type-o1-mini-instruct - GGUF

GGUF quantizations of [kd13/Type-o1-mini-instruct](https://huggingface.co/kd13/Type-o1-mini-instruct), a compact general-purpose instruct model (~1B parameters) for everyday assistant use.

Converted with [llama.cpp](https://github.com/ggml-org/llama.cpp).
The IQ quant was produced with an importance matrix; the rest are static quants.

> **Read the Usage section before running these files.** This model uses a custom chat
> template, so llama.cpp requires the `--jinja` flag. Without it you will get
> `this custom template is not supported`.

## Provided quants

Sorted by size, which is not the same as sorted by quality. IQ-quants are often preferable to non-IQ quants of a similar size.

| Link | Type | Size/GB | Notes |
|:-----|:-----|--------:|:------|
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q2_K.gguf) | Q2_K | 0.6 |  |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q3_K_S.gguf) | Q3_K_S | 0.6 |  |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q3_K_M.gguf) | Q3_K_M | 0.7 | lower quality |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q3_K_L.gguf) | Q3_K_L | 0.7 |  |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.IQ4_XS.gguf) | IQ4_XS | 0.7 |  |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q4_K_S.gguf) | Q4_K_S | 0.8 | fast, recommended |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q4_K_M.gguf) | Q4_K_M | 0.8 | fast, recommended |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q5_K_S.gguf) | Q5_K_S | 0.9 |  |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q5_K_M.gguf) | Q5_K_M | 0.9 |  |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q6_K.gguf) | Q6_K | 1.0 | very good quality |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.Q8_0.gguf) | Q8_0 | 1.3 | fast, best quality |
| [GGUF](https://huggingface.co/kd13/Type-o1-mini-instruct-GGUF/resolve/main/Type-o1-mini-instruct.f16.gguf) | f16 | 2.5 | 16 bpw, overkill |

## Which one should I pick?

For a model this small the practical range is **Q4_K_M through Q8_0**. A 1B model has little redundancy to give up, so the very low-bit quants lose more than they would on a 7B. `Q2_K` and `Q3_K_S` are included for completeness rather than as recommendations.

## Chat template

This model was fine-tuned on a custom template, not the standard Llama 3 header format.
Each message is wrapped as:

'''
<|begin_of_text|>{role}
{content}<|end_of_text|>
'''

and generation is prompted with a trailing `<|begin_of_text|>assistant\n`. The full Jinja template is embedded in every GGUF file in this repo, so any runtime with Jinja support applies it automatically.

Because this format is not one of llama.cpp's built-in recognised templates, its C++ template matcher will reject it. Passing `--jinja` tells llama.cpp to use the embedded Jinja template instead, which is what you want.

## Usage

### llama.cpp

```bash
llama-completion -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja \
  -sys "You are a helpful assistant." \
  -p "Explain photosynthesis in two sentences."
```

Recent llama.cpp builds renamed `llama-cli` to `llama-completion`; on older builds use `llama-cli` with the same flags. For raw text completion with no template applied at all, add `-no-cnv` and drop `--jinja`.

Server:

```bash
llama-server -m Type-o1-mini-instruct.Q4_K_M.gguf --jinja -c 4096
```

Omitting `--jinja` produces `this custom template is not supported` — that is a template-matching error, not a corrupt file.

### Ollama

```bash
ollama run hf.co/kd13/Type-o1-mini-instruct-GGUF:Q4_K_M
```