File size: 2,577 Bytes
d20ca56 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 | ---
license: mit
language:
- en
- fr
- de
- es
- it
- pt
- nl
tags:
- braille
- accessibility
- language-model
- emergent-patterns
- grade-infinity
datasets:
- project-gutenberg
pipeline_tag: text-generation
---
# Braille256-v2: Scaled Braille Language Model
A **25.6M parameter** language model trained natively on Braille Unicode characters (U+2800-U+28FF).
## Model Description
Braille256-v2 is a scaled version trained on **64.8 million Braille tokens** from 79 Project Gutenberg books. The model learns to predict Braille patterns directly, potentially discovering emergent contraction-like patterns similar to Grade 2/3 Braille.
### Key Features
- **Native Braille**: Works directly with 256 Braille Unicode characters
- **Dot-Pattern Embeddings**: Custom initialization based on physical dot patterns
- **Emergent Contractions**: May learn compression patterns similar to human-designed Braille contractions
- **Scaled Architecture**: 512 hidden size, 8 layers, 8 attention heads
### Training Details
| Metric | Value |
|--------|-------|
| Parameters | 25.6M |
| Training Tokens | 64.8M |
| Training Steps | 10,000 |
| Final Loss | 1.19 |
| Training Time | 4h 27m (MPS) |
### Architecture
```
Hidden Size: 512
Layers: 8
Attention Heads: 8
Vocabulary: 256 (Braille) + 5 (special tokens)
Max Sequence Length: 512
```
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("ryanscottbarrett/braille256-v2")
tokenizer = AutoTokenizer.from_pretrained("ryanscottbarrett/braille256-v2")
# Generate Braille text
prompt = "⠠⠞⠓⠑⠀" # "The " in Braille
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))
```
## Research Goals
This model is part of the **Grade Infinity Braille** research project exploring:
1. Can neural networks discover efficient Braille contractions from scratch?
2. Do emergent patterns match human-designed Grade 2/3 contractions?
3. Can cross-linguistic training reveal universal compression patterns?
## Next Steps
- **braille256-v3**: Multilingual training on 100M+ tokens (French, German, Spanish, Italian, Portuguese, Dutch)
- **Grade Infinity**: Universal contracted Braille that works across all languages
## Citation
```bibtex
@misc{braille256v2,
author = {Ryan Barrett},
title = {Braille256-v2: Scaled Braille Language Model},
year = {2024},
publisher = {HuggingFace},
url = {https://huggingface.co/ryanscottbarrett/braille256-v2}
}
```
## License
MIT
|