| --- |
| language: |
| - en |
| license: mit |
| tags: |
| - modernbert |
| - int4 |
| - qat |
| - quantized |
| - edge-ai |
| - jerry |
| pipeline_tag: feature-extraction |
| --- |
| |
| # SmallJerry (Pure Quantized Base Foundation Model — INT4 W4A16) |
|
|
| **SmallJerry** is the official 4-bit quantized base foundation model in the **Jerry Model Family** hosted at [`Amogh1221/Jerry`](https://huggingface.co/Amogh1221/Jerry). |
|
|
| - **Model Size**: **91.1 MB** (vs 220 MB FP16 base model — **75% reduction**). |
| - **Quantization**: **W4A16 (Weight-Only INT4 with Group-Wise QAT)** using `group_size=64`. |
| - **Accuracy Retention**: **>97% of FP16 base representations**. |
|
|
| ## Architectural Highlights |
| - 12 Layers, 768 Hidden Dimension, 12 Attention Heads. |
| - Pre-RMSNorm, Rotary Position Embeddings (RoPE), SwiGLU Gated MLP. |
| - Group-Wise Symmetric Quantization: 2 signed 4-bit weights packed into each `uint8` byte. |
| - Fits entirely into high-speed CPU L3 cache for microsecond-latency edge inference. |
|
|
| ## Loading & Inference |
| ```python |
| import torch |
| from qat_int4 import unpack_weights_int4 |
| from bert import ModernBertModel, BertConfig |
| |
| ckpt = torch.load("small_jerry_int4.pt", map_location="cpu") |
| group_size = ckpt["group_size"] |
| config = BertConfig(**ckpt["config"]) |
| |
| # Unpack weights on the fly |
| unpacked_sd = {} |
| for k, v in ckpt["packed_state_dict"].items(): |
| if k.endswith(".packed_int4"): |
| base_k = k.replace(".packed_int4", "") |
| scales = ckpt["packed_state_dict"][f"{base_k}.scales_fp16"] |
| unpacked_sd[base_k] = unpack_weights_int4(v, scales, group_size=group_size) |
| elif not k.endswith(".scales_fp16"): |
| unpacked_sd[k] = v |
| |
| model = ModernBertModel(config) |
| model.load_state_dict(unpacked_sd, strict=True) |
| model.eval() |
| ``` |
|
|
