README.md

---
license: mit
tags:
- ternary-quantization
- 1.58-bit
- experimental
- language-model
- tiny-stories
- adafactor
- bitlinear
license_name: mit
license_link: https://opensource.org/licenses/MIT
---

# ARTN - Adaptive Ternary Recurrent Network

## 1.58-bit Ternary Quantization for Language Models

**Experimental implementation of ternary (1.58-bit) quantization for training small language models on extremely limited hardware.**

**Status: EXPERIMENTAL - Not Production Ready**

---

## Project Overview

This project explores whether **ternary quantization** (weights constrained to -1, 0, +1) can be used to train small language models on hardware with:

| Constraint | Value |
|------------|-------|
| RAM | 4GB |
| CPU | 2-core (no GPU) |
| Storage | ~20GB |

---

## Model Architecture

| Parameter | Value |
|-----------|-------|
| Model type | BitLinearGPT |
| Parameters | 6,856,608 (~6.8M) |
| Hidden size (d_model) | 256 |
| Layers | 6 |
| Attention heads | 8 |
| Feed-forward size | 1024 |
| Max sequence length | 256 |
| Vocabulary size | 4,000 |
| Quantization | Ternary (1.58-bit) |

---

## Checkpoints Included

### 1. checkpoint-step5000-experiment.pt

| Attribute | Value |
|-----------|-------|
| Steps | 5,000 |
| Final Loss | 8.45 |
| Entropy | Not monitored |
| Status | SUCCESS (pipeline verified) |
| Description | First test training, proved architecture works |
| Duration | ~4 hours |

**Purpose:** Verified that training pipeline works correctly.

---

### 2. checkpoint-step50000-failed-collapsed.pt

| Attribute | Value |
|-----------|-------|
| Steps | 50,000 |
| Final Loss | 3.16 |
| Final Entropy | 8.2% (COLLAPSED) |
| Top Token | "time" at 94% |
| Status | FAILED (collapsed) |
| Description | Beta1=None caused model collapse |
| Duration | 79 hours |
| Root Cause | Missing momentum (beta1=None) |

**Warning:** This checkpoint is COLLAPSED. Model outputs repetitive text like "time time time...". Use only as a learning example of what NOT to do.

---

### 3. checkpoint-step50000-experiment-gibberish.pt

| Attribute | Value |
|-----------|-------|
| Steps | 50,000 |
| Final Loss | 7.56 |
| Final Entropy | 96.2% (HEALTHY) |
| Top Token | "the" at 0.56% |
| Status | HEALTHY but NOT USEFUL |
| Description | Beta1=0.9 prevented collapse but learning too slow |
| Duration | 83 hours |
| Issue | Beta1=0.9 too conservative |

**Note:** This checkpoint is HEALTHY (entropy normal) but produces gibberish. Model learned slowly due to overly conservative optimizer settings.

---

## Training Configuration

| Setting | Value |
|---------|-------|
| Dataset | TinyStories (roneneldan/TinyStories) |
| Samples | 50,000 |
| Batch size | 4 |
| Gradient accumulation | 8 |
| Effective batch size | 32 |
| Optimizer | Adafactor |
| Max Learning Rate | 0.0003 |
| Min Learning Rate | 0.00003 |
| Warmup steps | 1,000 |
| Weight decay | 0.01 |
| Gradient clipping | 1.0 |

---

## Key Findings

### Beta1 Trade-off

| Beta1 Value | Learning Speed | Stability | Result |
|-------------|---------------|-----------|--------|
| None (0.0) | FAST | UNSTABLE | COLLAPSED at 50k |
| 0.9 | SLOW | STABLE | HEALTHY but not useful |
| 0.5 - 0.7 | UNKNOWN | UNKNOWN | Need to test |

**Critical Lesson:** Beta1=None causes collapse. Beta1=0.9 too conservative. Need to find optimal value.

---

### Training Metrics Comparison

| Training | Loss @ 50k | Entropy | Top Token | Status |
|----------|-----------|---------|-----------|--------|
| Attempt 2 (beta1=None) | 3.16 | 8.2% | 94% "time" | COLLAPSED |
| Attempt 4 (beta1=0.9) | 7.56 | 96.2% | 0.56% "the" | HEALTHY (gibberish) |

**Paradox:** Lower loss does NOT mean better model. Attempt 2 had lower loss but was collapsed.

---

## Total Time Invested

| Attempt | Duration | Outcome |
|---------|----------|---------|
| 1 (5k test) | 4 hours | SUCCESS (pipeline verified) |
| 2 (50k collapsed) | 79 hours | FAILED (beta1=None) |
| 3 (stopped) | 1 hour | STOPPED (wrong accumulation) |
| 4 (50k gibberish) | 83 hours | COMPLETED (not useful) |
| **Total** | **167 hours** | |

---

## Lessons Learned

1. **Entropy monitoring is CRITICAL** - Loss alone doesn't indicate model health
2. **Beta1=None causes collapse** - Model falls into repetitive patterns
3. **Beta1=0.9 is too conservative** - Learning becomes extremely slow
4. **Lower loss ≠ better model** - Collapsed model had lower loss (3.16 vs 7.56)
5. **Need optimal beta1** - Testing 0.5 and 0.7 as candidates
6. **Ternary quantization is challenging** - Requires careful optimizer tuning

---

## How to Use These Checkpoints

### For Research/Learning

```python
import torch
from src.model.bitlinear import BitLinearGPT
from src.tokenizer.bpe_tokenizer import BPETokenizer

# Load model architecture
model = BitLinearGPT(
    vocab_size=4000,
    d_model=256,
    n_heads=8,
    n_layers=6,
    d_ff=1024,
    max_seq_len=256
)

# Load checkpoint (choose one)
checkpoint = torch.load('checkpoint-step50000-experiment-gibberish.pt', map_location='cpu')
model.load_state_dict(checkpoint)
model.eval()

# Load tokenizer
tokenizer = BPETokenizer.load('tokenizer.json')

# Generate text
input_ids = torch.tensor([tokenizer.encode("Once upon a time")])
# ... generation code ...
```

### Not For Production

These checkpoints are experimental. They will NOT produce coherent text. Use them to:
- Learn about ternary quantization challenges
- Understand model collapse patterns
- Compare optimizer settings
- Train your own models with different configs

---

## Tokenizers

| File | Description |
|------|-------------|
| tokenizer.json | Main tokenizer (BPE, vocab size 4000) |
| tokenizer-backup-50k.json | Backup from collapsed training |

---

## Source Code

Full source code available at GitHub:

**https://github.com/kishxrx/ARTN**

Includes:
- Complete training scripts
- Model architecture
- Training history documentation
- Monitoring tools

---

## Training History

Complete documentation of all experiments:

**https://github.com/kishxrx/ARTN/blob/main/TRAINING_HISTORY.md**

---

## Next Steps

| Phase | Plan |
|-------|------|
| 1 | Test beta1=0.5 (10k steps) |
| 2 | Test beta1=0.7 (10k steps) |
| 3 | Compare entropy, loss, samples |
| 4 | Select optimal beta1 |
| 5 | Full 50k training with best config |
| 6 | Release v1.0 if successful |

---

## Citation

If you use this work for learning or research:

```bibtex
@misc{artn-ternary-2026,
  author = {kishxrx},
  title = {ARTN: Adaptive Ternary Recurrent Network - 1.58-bit Quantization Experiments},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/kishxrx/artn-ternary-experimental}
}
```

---

## License

MIT License

---

## Disclaimer

**THIS IS EXPERIMENTAL SOFTWARE.**

- Models do NOT produce coherent text
- Checkpoints are for learning purposes only
- Not suitable for any production use
- No guarantees of any kind

---

## Contact

- GitHub: https://github.com/kishxrx/ARTN
- HuggingFace: https://huggingface.co/kishxrx

---

**Last Updated:** August 2026

**Version:** v0.1-experimental