Braille256-v2: Scaled Braille Language Model

A 25.6M parameter language model trained natively on Braille Unicode characters (U+2800-U+28FF).

Model Description

Braille256-v2 is a scaled version trained on 64.8 million Braille tokens from 79 Project Gutenberg books. The model learns to predict Braille patterns directly, potentially discovering emergent contraction-like patterns similar to Grade 2/3 Braille.

Key Features

• Native Braille: Works directly with 256 Braille Unicode characters
• Dot-Pattern Embeddings: Custom initialization based on physical dot patterns
• Emergent Contractions: May learn compression patterns similar to human-designed Braille contractions
• Scaled Architecture: 512 hidden size, 8 layers, 8 attention heads

Training Details

Metric	Value
Parameters	25.6M
Training Tokens	64.8M
Training Steps	10,000
Final Loss	1.19
Training Time	4h 27m (MPS)

Architecture

Hidden Size: 512
Layers: 8
Attention Heads: 8
Vocabulary: 256 (Braille) + 5 (special tokens)
Max Sequence Length: 512

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ryanscottbarrett/braille256-v2")
tokenizer = AutoTokenizer.from_pretrained("ryanscottbarrett/braille256-v2")

# Generate Braille text
prompt = "⠠⠞⠓⠑⠀"  # "The " in Braille
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=100)
print(tokenizer.decode(outputs[0]))

Research Goals

This model is part of the Grade Infinity Braille research project exploring:

1. Can neural networks discover efficient Braille contractions from scratch?
2. Do emergent patterns match human-designed Grade 2/3 contractions?
3. Can cross-linguistic training reveal universal compression patterns?

Next Steps

• braille256-v3: Multilingual training on 100M+ tokens (French, German, Spanish, Italian, Portuguese, Dutch)
• Grade Infinity: Universal contracted Braille that works across all languages

Citation

@misc{braille256v2,
  author = {Ryan Barrett},
  title = {Braille256-v2: Scaled Braille Language Model},
  year = {2024},
  publisher = {HuggingFace},
  url = {https://huggingface.co/ryanscottbarrett/braille256-v2}
}

License

MIT