|
|
|
|
|
--- |
|
|
language: en |
|
|
license: apache-2.0 |
|
|
tags: |
|
|
- efficient-llm |
|
|
- quantization |
|
|
- ternary |
|
|
- bitnet |
|
|
- pytorch |
|
|
- tinystories |
|
|
- language-modeling |
|
|
datasets: |
|
|
- roneneldan/TinyStories |
|
|
arxiv: 2602.07374 |
|
|
--- |
|
|
|
|
|
# TernaryLM-132M |
|
|
|
|
|
TernaryLM-132M is a 132M parameter Transformer trained natively using ternary weights {-1, 0, +1}. |
|
|
|
|
|
Unlike post-training quantization methods, this model learns quantized representations during training. |
|
|
|
|
|
## Architecture |
|
|
|
|
|
- Parameters: 132M |
|
|
- Layers: 12 |
|
|
- Hidden Size: 768 |
|
|
- Attention Heads: 12 |
|
|
- Context Length: 512 |
|
|
- Quantization: Native Ternary Training |
|
|
|
|
|
## Training |
|
|
|
|
|
- Dataset: TinyStories (~60k stories) |
|
|
- Optimizer: AdamW (betas=(0.9, 0.98)) |
|
|
- LR: 3e-4 |
|
|
- Scheduler: OneCycleLR |
|
|
- Epochs: 15 |
|
|
- Hardware: Multi-GPU T4 setup (Kaggle) |
|
|
|
|
|
## Intended Use |
|
|
|
|
|
Research on: |
|
|
- Efficient Transformers |
|
|
- Quantization-aware training |
|
|
- Edge deployment |
|
|
|
|
|
## Limitations |
|
|
|
|
|
- Not instruction-tuned |
|
|
- Limited dataset scale |
|
|
- Research prototype |
|
|
|
|
|
## Citation |
|
|
|
|
|
```bibtex |
|
|
@misc{nargund2026ternarylmmemoryefficientlanguagemodeling, |
|
|
title={TernaryLM: Memory-Efficient Language Modeling via Native 1-Bit Quantization with Adaptive Layer-wise Scaling}, |
|
|
author={Nisharg Nargund and Priyesh Shukla}, |
|
|
year={2026}, |
|
|
eprint={2602.07374}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL}, |
|
|
url={https://arxiv.org/abs/2602.07374}, |
|
|
} |
|
|
``` |
