Ternary1B BITNET (SmolLM2 1.7B)

HuggingFaceTB/SmolLM2-1.7B stores each weight in 16 bits. This model stores the weights of every attention and feed forward layer in 1.58 (ternary) bits, so the whole model is several times smaller and can run on phones and cheap CPUs. It is one of six models in the Ternary1B collection, each using a different way to survive extreme compression, all scored on the same tests.

Method

BitNet b1.58 conversion. Every linear layer becomes a BitLinear layer: weights are scaled by their average absolute value and rounded to minus 1, 0 or plus 1, and activations are rounded to 8 bits. A warmup blends full precision into ternary step by step so the converted model does not collapse, with distillation from the original model.

Embeddings and normalization layers stay in 16 bit, as in BitNet. Training data: AmareshHebbar/Ternary1B_Data_SmolLM2_1.7B.

Why train instead of just rounding

Retrieval and prompting cannot make a model smaller. Plain rounding to 2 bits or ternary destroys it, which the baseline rows show. Every gain over those rows comes from changing the weights.

Results

Perplexity on WikiText2 test (lower is better). Benchmarks are zero shot accuracy from lm evaluation harness, limited to 1000 examples per task. Size is the packed file size in packed/: real 2 bit or ternary codes plus their scales, with embeddings in 16 bit.

ModelBits per weightSize (MB)Smaller byWikiText2 perplexityhellaswagarc_easypiqammluBenchmark mean
SmolLM2 1.7B (original bf16)16.0032641.0x8.1260.873.377.848.865.2
SmolLM2 1.7B (plain 2 bit rounding)2.506724.9x942140.0127.727.549.324.332.2
SmolLM2 1.7B (plain ternary rounding)1.604996.5x17663863.9126.426.450.324.031.8
Ternary1B GPTQ2.506724.9x2538.1930.726.952.924.333.7
Ternary1B GPTQ_LORA2.508104.0x21.7246.353.364.425.847.4
Ternary1B BITNET1.604996.5x35.0540.748.163.823.143.9

size_vs_quality

benchmarks

Speed on a CPU

llama.cpp b6500, 128 threads on AMD EPYC 7702 64-Core Processor.

Model fileSize (MB)Prompt tokens/sGenerated tokens/s
SmolLM2 1.7B f16326667.34.0
SmolLM2 1.7B Q4_K_M (4 bit reference)100758.09.0
Ternary1B BITNET as TQ2_059085.69.5
Ternary1B BITNET as TQ1_051863.26.2

Only the ternary model maps exactly onto llama.cpp's ternary formats, so 2 bit models are not timed on CPU.

Sample completions (greedy)

  • The capital of France is the only country in the world that has a population of 1.5 million.

The French government has been working hard to improve the quality of

  • Photosynthesis is the process by which the process is carried out.

The process of the production of a product is the process of the production of a product. The process of the production

  • def fibonacci(n):

Files

  • Repository root: the same weights expanded to bf16 safetensors, so it loads in transformers with no extra code. For faithful output, also quantize activations to 8 bits as below.
  • packed/: the real compressed form. quant.safetensors holds packed codes (4 weights per byte at 2 bits, 5 weights per byte for ternary) and scales, rest.safetensors holds embeddings and norms, and quant_config.json describes the layout.
  • tq_runtime.py: unpacks packed/ into a working model.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET")
model = AutoModelForCausalLM.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", torch_dtype="bfloat16")
from huggingface_hub import hf_hub_download
import importlib.util
spec = importlib.util.spec_from_file_location("tq_runtime", hf_hub_download("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "tq_runtime.py"))
rt = importlib.util.module_from_spec(spec)
spec.loader.exec_module(rt)
rt.patch_act_quant(model)
ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, do_sample=False)[0]))

To build the model from the packed files only: download the repo, then tq_runtime.load_model(path).

Training hyperparameters

group_size64
bitnet_lr8e-05
bitnet_steps1800
bitnet_warmup_steps50
bitnet_lambda_warmup_frac0.3
train_seq_len1024
train_micro_batch2
train_grad_accum16
train_logits_chunk256
train_kd_weight1.0
train_kd_temp1.0
train_weight_decay0.0
size_packed_bytes523689912
size_packed_mb499.4296188354492
size_bf16_bytes3422777848
size_compression6.5358865419580585
size_bits_per_quantized_weight1.6000033378601082
size_quantized_params1610612736
size_formatternary 1.58 bit
Downloads last month
197
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET

Quantized
(63)
this model

Dataset used to train AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET

Collection including AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET