squishai/gemma-3-4b-it-bf16-squished

Gemma-3-4B-Instruct — Squished for Apple Silicon

This is Gemma-3-4B-Instruct (4B parameters) compressed with Squish — a local inference engine for Apple Silicon.

Weights are INT4-quantized using Squish's ARM NEON-accelerated pipeline and load in under a second on M-series hardware.

Quick start

brew tap konjoai/squish
brew install squish
squish pull gemma3:4b
squish run gemma3:4b

Model details

Property	Value
Parameters	4B
Family	Gemma 3
Developer	Google DeepMind
Raw size	8.1 GB
Squished size	5.4 GB
Context window	131,072 tokens
Minimum RAM	8 GB unified memory
Quantization	INT4 (Squish pipeline)
Format	MLX-compatible safetensors

Use case

Balanced Gemma model with long context. Strong multilingual support and instruction following.

Requirements

• macOS 13.0 or later
• Apple Silicon (M1, M2, M3, M4, M5)
• 8 GB unified memory minimum

Intel Macs, Linux, and Windows are not supported.

How to use with Squish

# Pull and run
squish pull gemma3:4b
squish run gemma3:4b

# OpenAI-compatible API on port 11435
curl http://localhost:11435/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma3:4b","messages":[{"role":"user","content":"Hello"}]}'

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11435/v1", api_key="squish")
response = client.chat.completions.create(
    model="gemma3:4b",
    messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)

Load with mlx_lm directly

from mlx_lm import load, generate

model, tokenizer = load("squishai/gemma-3-4b-it-bf16-squished")
response = generate(model, tokenizer, prompt="Hello", max_tokens=100)
print(response)

Compression details

This model was compressed using Squish's three-tier pipeline:

• INT4 quantization via squish_quant_rs Rust extension with ARM NEON acceleration
• Compressed weight loader — weights decompress directly into Metal-mapped memory at load time
• KV cache quantization — attention cache stored at reduced precision during generation

Source weights: mlx-community/gemma-3-4b-it-bf16

License

The original model weights are subject to the license of the source model (Google DeepMind). The compression and tooling are MIT licensed. See Squish license for details.

---

Pre-compressed by Konjo AI · squish.run