SmallJerry/README.md

metadata

language:
  - en
license: mit
tags:
  - modernbert
  - int4
  - qat
  - quantized
  - edge-ai
  - jerry
pipeline_tag: feature-extraction

SmallJerry (Pure Quantized Base Foundation Model — INT4 W4A16)

SmallJerry is the official 4-bit quantized base foundation model in the Jerry Model Family hosted at Amogh1221/Jerry.

• Model Size: 91.1 MB (vs 220 MB FP16 base model — 75% reduction).
• Quantization: W4A16 (Weight-Only INT4 with Group-Wise QAT) using group_size=64.
• Accuracy Retention: >97% of FP16 base representations.

Architectural Highlights

• 12 Layers, 768 Hidden Dimension, 12 Attention Heads.
• Pre-RMSNorm, Rotary Position Embeddings (RoPE), SwiGLU Gated MLP.
• Group-Wise Symmetric Quantization: 2 signed 4-bit weights packed into each uint8 byte.
• Fits entirely into high-speed CPU L3 cache for microsecond-latency edge inference.

Loading & Inference

import torch
from qat_int4 import unpack_weights_int4
from bert import ModernBertModel, BertConfig

ckpt = torch.load("small_jerry_int4.pt", map_location="cpu")
group_size = ckpt["group_size"]
config = BertConfig(**ckpt["config"])

# Unpack weights on the fly
unpacked_sd = {}
for k, v in ckpt["packed_state_dict"].items():
    if k.endswith(".packed_int4"):
        base_k = k.replace(".packed_int4", "")
        scales = ckpt["packed_state_dict"][f"{base_k}.scales_fp16"]
        unpacked_sd[base_k] = unpack_weights_int4(v, scales, group_size=group_size)
    elif not k.endswith(".scales_fp16"):
        unpacked_sd[k] = v

model = ModernBertModel(config)
model.load_state_dict(unpacked_sd, strict=True)
model.eval()