Instructions to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET") model = AutoModelForCausalLM.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET
- SGLang
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET with Docker Model Runner:
docker model run hf.co/AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET
Ternary1B BITNET (SmolLM2 1.7B)
HuggingFaceTB/SmolLM2-1.7B stores each weight in 16 bits. This model stores the weights of every attention and feed forward layer in 1.58 (ternary) bits, so the whole model is several times smaller and can run on phones and cheap CPUs. It is one of six models in the Ternary1B collection, each using a different way to survive extreme compression, all scored on the same tests.
Method
BitNet b1.58 conversion. Every linear layer becomes a BitLinear layer: weights are scaled by their average absolute value and rounded to minus 1, 0 or plus 1, and activations are rounded to 8 bits. A warmup blends full precision into ternary step by step so the converted model does not collapse, with distillation from the original model.
Embeddings and normalization layers stay in 16 bit, as in BitNet. Training data: AmareshHebbar/Ternary1B_Data_SmolLM2_1.7B.
Why train instead of just rounding
Retrieval and prompting cannot make a model smaller. Plain rounding to 2 bits or ternary destroys it, which the baseline rows show. Every gain over those rows comes from changing the weights.
Results
Perplexity on WikiText2 test (lower is better). Benchmarks are zero shot accuracy from lm evaluation harness, limited to 1000 examples per task. Size is the packed file size in packed/: real 2 bit or ternary codes plus their scales, with embeddings in 16 bit.
| Model | Bits per weight | Size (MB) | Smaller by | WikiText2 perplexity | hellaswag | arc_easy | piqa | mmlu | Benchmark mean |
|---|---|---|---|---|---|---|---|---|---|
| SmolLM2 1.7B (original bf16) | 16.00 | 3264 | 1.0x | 8.12 | 60.8 | 73.3 | 77.8 | 48.8 | 65.2 |
| SmolLM2 1.7B (plain 2 bit rounding) | 2.50 | 672 | 4.9x | 942140.01 | 27.7 | 27.5 | 49.3 | 24.3 | 32.2 |
| SmolLM2 1.7B (plain ternary rounding) | 1.60 | 499 | 6.5x | 17663863.91 | 26.4 | 26.4 | 50.3 | 24.0 | 31.8 |
| Ternary1B GPTQ | 2.50 | 672 | 4.9x | 2538.19 | 30.7 | 26.9 | 52.9 | 24.3 | 33.7 |
| Ternary1B GPTQ_LORA | 2.50 | 810 | 4.0x | 21.72 | 46.3 | 53.3 | 64.4 | 25.8 | 47.4 |
| Ternary1B BITNET | 1.60 | 499 | 6.5x | 35.05 | 40.7 | 48.1 | 63.8 | 23.1 | 43.9 |
Speed on a CPU
llama.cpp b6500, 128 threads on AMD EPYC 7702 64-Core Processor.
| Model file | Size (MB) | Prompt tokens/s | Generated tokens/s |
|---|---|---|---|
| SmolLM2 1.7B f16 | 3266 | 67.3 | 4.0 |
| SmolLM2 1.7B Q4_K_M (4 bit reference) | 1007 | 58.0 | 9.0 |
| Ternary1B BITNET as TQ2_0 | 590 | 85.6 | 9.5 |
| Ternary1B BITNET as TQ1_0 | 518 | 63.2 | 6.2 |
Only the ternary model maps exactly onto llama.cpp's ternary formats, so 2 bit models are not timed on CPU.
Sample completions (greedy)
- The capital of France is the only country in the world that has a population of 1.5 million.
The French government has been working hard to improve the quality of
- Photosynthesis is the process by which the process is carried out.
The process of the production of a product is the process of the production of a product. The process of the production
- def fibonacci(n):
Files
- Repository root: the same weights expanded to bf16 safetensors, so it loads in transformers with no extra code. For faithful output, also quantize activations to 8 bits as below.
packed/: the real compressed form.quant.safetensorsholds packed codes (4 weights per byte at 2 bits, 5 weights per byte for ternary) and scales,rest.safetensorsholds embeddings and norms, andquant_config.jsondescribes the layout.tq_runtime.py: unpackspacked/into a working model.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET")
model = AutoModelForCausalLM.from_pretrained("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", torch_dtype="bfloat16")
from huggingface_hub import hf_hub_download
import importlib.util
spec = importlib.util.spec_from_file_location("tq_runtime", hf_hub_download("AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET", "tq_runtime.py"))
rt = importlib.util.module_from_spec(spec)
spec.loader.exec_module(rt)
rt.patch_act_quant(model)
ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=30, do_sample=False)[0]))
To build the model from the packed files only: download the repo, then tq_runtime.load_model(path).
Training hyperparameters
| group_size | 64 |
| bitnet_lr | 8e-05 |
| bitnet_steps | 1800 |
| bitnet_warmup_steps | 50 |
| bitnet_lambda_warmup_frac | 0.3 |
| train_seq_len | 1024 |
| train_micro_batch | 2 |
| train_grad_accum | 16 |
| train_logits_chunk | 256 |
| train_kd_weight | 1.0 |
| train_kd_temp | 1.0 |
| train_weight_decay | 0.0 |
| size_packed_bytes | 523689912 |
| size_packed_mb | 499.4296188354492 |
| size_bf16_bytes | 3422777848 |
| size_compression | 6.5358865419580585 |
| size_bits_per_quantized_weight | 1.6000033378601082 |
| size_quantized_params | 1610612736 |
| size_format | ternary 1.58 bit |
- Downloads last month
- 197
Model tree for AmareshHebbar/Ternary1B_SmolLM2_1.7B_BITNET
Base model
HuggingFaceTB/SmolLM2-1.7B
