---
language:
- en
license: mit
tags:
- modernbert
- int4
- qat
- quantized
- edge-ai
- jerry
pipeline_tag: feature-extraction
---

# SmallJerry (Pure Quantized Base Foundation Model — INT4 W4A16)

**SmallJerry** is the official 4-bit quantized base foundation model in the **Jerry Model Family** hosted at [`Amogh1221/Jerry`](https://huggingface.co/Amogh1221/Jerry).

- **Model Size**: **91.1 MB** (vs 220 MB FP16 base model — **75% reduction**).
- **Quantization**: **W4A16 (Weight-Only INT4 with Group-Wise QAT)** using `group_size=64`.
- **Accuracy Retention**: **>97% of FP16 base representations**.

## Architectural Highlights
- 12 Layers, 768 Hidden Dimension, 12 Attention Heads.
- Pre-RMSNorm, Rotary Position Embeddings (RoPE), SwiGLU Gated MLP.
- Group-Wise Symmetric Quantization: 2 signed 4-bit weights packed into each `uint8` byte.
- Fits entirely into high-speed CPU L3 cache for microsecond-latency edge inference.

## Loading & Inference
```python
import torch
from qat_int4 import unpack_weights_int4
from bert import ModernBertModel, BertConfig

ckpt = torch.load("small_jerry_int4.pt", map_location="cpu")
group_size = ckpt["group_size"]
config = BertConfig(**ckpt["config"])

# Unpack weights on the fly
unpacked_sd = {}
for k, v in ckpt["packed_state_dict"].items():
    if k.endswith(".packed_int4"):
        base_k = k.replace(".packed_int4", "")
        scales = ckpt["packed_state_dict"][f"{base_k}.scales_fp16"]
        unpacked_sd[base_k] = unpack_weights_int4(v, scales, group_size=group_size)
    elif not k.endswith(".scales_fp16"):
        unpacked_sd[k] = v

model = ModernBertModel(config)
model.load_state_dict(unpacked_sd, strict=True)
model.eval()
```