README.md

---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-3B-Instruct
tags:
  - gguf
  - llama-cpp
  - tool-calling
  - agent
  - local
  - qwen2
library_name: gguf
pipeline_tag: text-generation
---

# Meltdown Q8

**Meltdown_Q8.gguf** is a 3B local agent model, exported as Q8_0 GGUF. Runs fully offline — no API key required at inference time.

| | |
|---|---|
| **Parameters** | 3B |
| **Quantization** | Q8_0 (GGUF) |
| **Context length** | 264,768 tokens |
| **Base model** | [Qwen2.5-Coder-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct) |
| **Format** | GGUF (iRun, llama.cpp, LM Studio, KoboldCPP, etc.) |
| **VRAM (recommended)** | 4–6 GB+ |

## Quick Start

```bash
llama-server -m Meltdown_Q8.gguf -c 32000 -ngl 99 --host 127.0.0.1 --port 8080
```

Use **temperature 0.2** and load `system_prompt.txt` as the system message. OpenAI-compatible API: `http://127.0.0.1:8080/v1/chat/completions`.

Inference defaults are in `config.recommended.json`.

## Evaluation Results

Measured locally on **NVIDIA RTX 3060 12GB** with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) v0.4.12 and llama-server (**20 examples per task**). Frontier scores are vendor-published references (see `frontier_reference.json`).

**Evaluated:** 2026-07-15

### Overall Comparison

![Benchmark comparison](benchmark_comparison.png)

| Benchmark | What it tests | Meltdown Q8 (3B) | GPT-4.1 | Claude Sonnet 4 | Gemini 2.5 Pro |
|---|---|---:|---:|---:|---:|
| GSM8K (8-shot, n=20) | Grade-school math word problems | 65.0% | 95.2% | 94.0% | 93.5% |
| MMLU HS Mathematics (n=20) | High school math (generative MC) | 0.0% | 90.2% | 88.5% | 89.0% |
| MMLU HS Computer Science (n=20) | HS CS knowledge (generative MC) | 0.0% | 90.2% | 88.5% | 89.0% |
| IFEval (strict, n=20) | Instruction following | 60.0% | 87.5% | 86.0% | 85.5% |
| Agent Eval (n=20) | Structured tool-use on held-out prompts | 60.0% | — | — | — |

Raw JSON: `eval_standard.json`, `eval_agent.json`.

While the benchmark scores themselves are low, it is due to image-related benchmarks. For chat, the APE format overrides the base chat template, which is why it scores low in chat.
In day to day agentic workflows, with abbreviated MCP tools and unified instructions, Meltdown performs exceptionally. Escpecially while utilizing local Agentic harnesses like iRun's ReAct and APE.

### Agent Eval Breakdown

![Agent eval by category](agent_category_chart.png)

| Split | Pass Rate | Tests |
|---|---:|---:|
| Tool & file tasks | 80.0% | 12/15 |
| Conversational tasks | 0.0% | 0/5 |
| **Overall** | **60.0%** | **12/20** |

### Local vs Cloud

![Local vs cloud](local_vs_cloud.png)

| | Meltdown Q8 | GPT-4.1 | Claude Sonnet 4 | Gemini 2.5 Pro |
|---|:---:|:---:|:---:|:---:|
| Runs offline | Yes | No | No | No |
| API key required | No | Yes | Yes | Yes |
| Data leaves your machine | No | Yes | Yes | Yes |
| Parameters | 3B | — | — | — |

Meltdown is optimized for **local agent work** (privacy, zero API cost, offline use), not for beating frontier models on broad knowledge benchmarks.

## Files in This Repo

| File | Description |
|---|---|
| `Meltdown_Q8.gguf` | Model weights (~3.1 GB) — upload via Git LFS |
| `system_prompt.txt` | Recommended Meltdown system prompt |
| `config.recommended.json` | Inference + harness parameters |
| `benchmark_comparison.png` | Eval chart |
| `agent_category_chart.png` | Agent eval by category |
| `local_vs_cloud.png` | Operational comparison |
| `eval_standard.json` | lm-eval harness results |
| `eval_agent.json` | Agent eval results |
| `frontier_reference.json` | Frontier comparison score sources |

## License

Derived from Qwen2.5-Coder-3B-Instruct. See the [base model license](https://huggingface.co/Qwen/Qwen2.5-Coder-3B-Instruct).