license: apache-2.0
language:
- en
pipeline_tag: text-generation
library_name: transformers
tags:
- i3-arhitecture
i3-tiny
i3-tiny is a compact, efficient character-level language model designed for experimentation and exploration in text generation. Despite its small size, it packs a surprising punch for creative and research-oriented tasks, generating sequences that are quirky, unpredictable, and full of “human-like” character-level errors.
Model Overview
i3-tiny is trained to predict the next character in a sequence, making it ideal for character-level language modeling, creative text generation, and research on lightweight, efficient models. Its small footprint allows rapid experimentation, even on modest hardware, and it provides a playground for studying how models learn patterns in sequences of characters.
The model is intentionally experimental — it’s not aligned, fact-checked, or polished. Instead, it showcases how a compact architecture can capture patterns in text, learn from repetition, and generate outputs that are sometimes surprisingly coherent, sometimes hilariously garbled.
Training Details
- Dataset: ~45,830 characters (a curated text corpus repeated to improve exposure)
- Vocabulary: 34 characters (all lowercased)
- Sequence length: 128
- Training iterations: 2,000
- Batch size: 2
- Optimizer: AdamW, learning rate 3e-4
- Model parameters: 711,106
- Performance notes: Each iteration takes roughly 400–500 ms; 100 iterations take ~45 s on average. Loss steadily decreased from 3.53 to 2.15 over training.
Example generation (iteration 1200):
Prompt: "The quick"
Generated: the quick efehn. dethe cans the fice the fpeens antary of eathetint, an thadat hitimes the and cow thig, and
These outputs capture the chaotic creativity of a character-level model: a mixture of readable words, invented forms, and surprising sequences.
Intended Uses
- Character-level text generation experiments
- Research and education: studying lightweight language models, sequence learning, and text modeling
- Creative exploration: generating quirky text or procedural content for games, demos, or artistic projects
⚠️ i3-tiny is experimental and not intended for production or high-stakes applications. Text may be repetitive, nonsensical, or inconsistent.
Limitations
- Small vocabulary and character-level modeling limit natural language fluency
- Outputs are highly experimental and not fact-checked
- Generated sequences can be repetitive or unexpectedly garbled
- Not aligned or safety-checked
Model Weights
- Stored in
model.bin - Compatible with PyTorch