---
license: apache-2.0
base_model: CohereLabs/North-Mini-Code-1.0
tags:
  - gguf
  - llama.cpp
  - cohere
  - code
  - moe
  - quantized
  - north-mini-code
pipeline_tag: text-generation
language:
  - en
---

# North-Mini-Code-1.0-GGUF

GGUF conversions and quantizations of **CohereLabs/North-Mini-Code-1.0** for use with:

- llama.cpp
- LM Studio
- Ollama
- Jan
- KoboldCpp
- Text Generation WebUI
- Open WebUI
- Other GGUF-compatible runtimes

---

# About the Model

North-Mini-Code-1.0 is a code-focused Mixture-of-Experts (MoE) model released by CohereLabs.

This repository provides ready-to-use GGUF conversions for local inference across a range of hardware configurations.

---

# Available Files

### Full Precision

- `North-Mini-Code-1.0-F16.gguf`

### Quantized Versions

- `North-Mini-Code-1.0-Q4_K_M.gguf`
- `North-Mini-Code-1.0-Q5_K_M.gguf`
- `North-Mini-Code-1.0-Q6_K.gguf`
- `North-Mini-Code-1.0-Q8_0.gguf`

---

# Recommended Quantization

For most users:

```text
North-Mini-Code-1.0-Q4_K_M.gguf
```

It offers the best balance of:

- Quality
- Memory usage
- Inference speed

If you have more available RAM/VRAM, consider:

```text
North-Mini-Code-1.0-Q5_K_M.gguf
```

or

```text
North-Mini-Code-1.0-Q6_K.gguf
```

for slightly higher output quality.

---

# Approximate File Sizes

```text
F16      ~60+ GB
Q4_K_M   ~20 GB
Q5_K_M   ~23 GB
Q6_K     ~27 GB
Q8_0     ~34 GB
```

Actual sizes may vary slightly depending on conversion tooling versions.

---

# Usage

## llama.cpp

Prompt mode:

```bash
./llama-cli \
  -m North-Mini-Code-1.0-Q4_K_M.gguf \
  -p "Write a Python function that reverses a linked list."
```

Chat mode:

```bash
./llama-cli \
  -m North-Mini-Code-1.0-Q4_K_M.gguf \
  -cnv
```

---

## LM Studio

1. Download your preferred GGUF file.
2. Open LM Studio.
3. Import the model.
4. Start chatting.

---

## Ollama

Create a `Modelfile`:

```text
FROM North-Mini-Code-1.0-Q4_K_M.gguf
```

Create the model:

```bash
ollama create north-mini-code -f Modelfile
```

Run it:

```bash
ollama run north-mini-code
```

---

# Hardware Recommendations

### Q4_K_M

Recommended minimum:

```text
24 GB RAM
```

### Q5_K_M

Recommended minimum:

```text
32 GB RAM
```

### Q6_K

Recommended minimum:

```text
32-40 GB RAM
```

### Q8_0

Recommended minimum:

```text
48+ GB RAM
```

### F16

Recommended minimum:

```text
80+ GB RAM
```

---

# Prompting Tips

This model is optimized for programming-related tasks.

Example prompts:

```text
Implement a fast Rust HTTP server.
```

```text
Explain this C++ compiler error.
```

```text
Write comprehensive unit tests for the following Python code.
```

```text
Convert this JavaScript function to TypeScript.
```

```text
Optimize this SQL query.
```

---

# Base Model

Base model:

```text
CohereLabs/North-Mini-Code-1.0
```

All training, architecture, benchmarks, licensing terms, and usage restrictions belong to the original model authors.

Please refer to the original repository for official documentation and licensing information.

---

# Conversion Details

Converted using:

```text
llama.cpp
```

Generated quantizations:

```text
F16
Q4_K_M
Q5_K_M
Q6_K
Q8_0
```

A tokenizer compatibility workaround was applied during conversion to support current GGUF conversion tooling.

---

# Credits

- Base Model: CohereLabs
- GGUF Conversion & Quantization: NANI-Nithin
- Tooling: llama.cpp

---

# Repository

👉 https://huggingface.co/NANI-Nithin/north-mini-code-gguf