---
title: Konjo AI
emoji: 🗜
colorFrom: gray
colorTo: blue
sdk: static
pinned: false
---

# Konjo AI

Local AI infrastructure for Apple Silicon. We make models that already exist
run faster on the hardware you already own.

🌐 [squish.run](https://squish.run) · 💻 [github.com/konjoai](https://github.com/konjoai)

---

## squish — Local LLM inference for Apple Silicon

[squish](https://github.com/konjoai/squish) is an MLX-based local inference
server with a block-level paged KV cache and INT3 quantization support for the
Qwen3 family. On a 16 GB M3 MacBook against Ollama:

- **5.4× faster** end-to-end response at 4000-token prompts (12.78s vs 69.6s)
- **1.5× faster** end-to-end on 75-token prompts (5.50s vs 8.09s)
- **33% less RAM** during inference (3.36 GB vs ~5 GB)
- **INT3 support** for Qwen3 with no measurable accuracy loss (Ollama doesn't ship INT3)

The honest tradeoff: Ollama still wins first-token latency on short prompts.
squish wins when you care about total response time on real workloads.

**Install:**
```bash
brew tap konjoai/squish && brew install squish
# or
pip install squish-ai
```

**Use:**
```bash
squish pull konjoai/Qwen3-8B-squished
squish run Qwen3-8B-squished
```

[Full benchmarks](https://github.com/konjoai/squish/blob/main/docs/RESULTS.md) ·
[Repo](https://github.com/konjoai/squish) ·
[Issues](https://github.com/konjoai/squish/issues)

---

## Pre-Compressed Models

This org hosts models pre-compressed by squish. Pull once, load instantly every
time after.

| Model | Squish ID | Quantization | Disk size | Context |
|---|---|---|---|---|
| _Available after first publish batch_ |

The format is `mlx_lm`-compatible — you can also use these models directly:

```python
from mlx_lm import load, generate

model, tokenizer = load("konjoai/Qwen2.5-7B-Instruct-squished")
response = generate(model, tokenizer, prompt="Hello", max_tokens=100)
print(response)
```

---

## How models are compressed

squish uses a three-tier pipeline:

1. **INT4/INT3 quantization** via a Rust extension (`squish_quant_rs`) with ARM
   NEON acceleration
2. **Block-level paged KV cache** — KV state is chunked into fixed-size blocks
   for prefix reuse across sessions
3. **Quantization safeguards** — squish hard-blocks INT3 on model families where
   it collapses (e.g. Gemma-3 loses ~15pp on common benchmarks); INT3 ships only
   for families that hold accuracy (Qwen3 specifically)

---

## Other projects

We also build [squash](https://github.com/konjoai/squash), a security and EU AI
Act compliance scanner for HuggingFace models. Independent codebase, related
mission.

---

## License

squish is BUSL-1.1. Compressed models inherit their base model's license — Qwen3
is Apache-2.0, Llama is the Llama Community License, etc. Check each model's card
for specifics.

---

## Requirements

- macOS 13.0 or later
- Apple Silicon (M1 / M2 / M3 / M4 / M5)
- Enough unified memory for the model (table above)

Intel Macs and Linux are not supported.