Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,39 @@
+# PureBit Transformer
+
+**A transformer that operates on raw binary bits instead of tokens.**
+
+## Architecture
+- **Vocab size**: 2 (just 0 and 1!)
+- **d_model**: 256
+- **Layers**: 6
+- **Heads**: 8
+- **Parameters**: ~18M
+
+## Training
+- Trained on raw UTF-8 bytes converted to bits
+- Best loss achieved: **0.6863** (random = 0.693)
+- Training data: ~70MB of text = 560M bits
+
+## Key Insight
+This explores whether transformers can learn at the bit level. Results show minimal learning beyond random - predicting individual bits is extremely hard without byte-level structure.
+
+## Usage
+```python
+import torch
+
+# Load checkpoint
+ckpt = torch.load('purebit_best_70mb.pt')
+print(f"Loss: {ckpt['loss']:.4f}")
+print(f"Bits seen: {ckpt['bits']:,}")
+
+# Model architecture in model.py
+```
+
+## Files
+- `purebit_best_70mb.pt` - Best checkpoint (loss 0.6863)
+- `model.py` - Model architecture
+- `train.py` - Training script
+- `infer.py` - Inference script
+
+## Author
+OpenTransformers - Experimental architecture research