---
language: en
license: apache-2.0
tags:
- efficient-llm
- quantization
- ternary
- bitnet
- pytorch
- tinystories
- language-modeling
datasets:
- roneneldan/TinyStories
arxiv: 2602.07374
---

# TernaryLM-132M

TernaryLM-132M is a 132M parameter Transformer trained natively using ternary weights {-1, 0, +1}.

Unlike post-training quantization methods, this model learns quantized representations during training.

## Architecture

- Parameters: 132M
- Layers: 12
- Hidden Size: 768
- Attention Heads: 12
- Context Length: 512
- Quantization: Native Ternary Training

## Training

- Dataset: TinyStories (~60k stories)
- Optimizer: AdamW (betas=(0.9, 0.98))
- LR: 3e-4
- Scheduler: OneCycleLR
- Epochs: 15
- Hardware: Multi-GPU T4 setup (Kaggle)

## Intended Use

Research on:
- Efficient Transformers
- Quantization-aware training
- Edge deployment

## Limitations

- Not instruction-tuned
- Limited dataset scale
- Research prototype

## Citation

```bibtex
@misc{nargund2026ternarylmmemoryefficientlanguagemodeling,
      title={TernaryLM: Memory-Efficient Language Modeling via Native 1-Bit Quantization with Adaptive Layer-wise Scaling}, 
      author={Nisharg Nargund and Priyesh Shukla},
      year={2026},
      eprint={2602.07374},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.07374}, 
}
```