i3-lab/i3-12m

i3 Model - Ultra-Efficient Pretraining Language Model

Model Description

The i3 Model is designed to optimize pretraining efficiency while retaining core language modeling capabilities.
Its architecture allows training on memory-constrained hardware, including CPU-only setups, without sacrificing sequence modeling performance.

The i3 architecture is present within the model for highly efficient pretraining. It is designed to reduce memory usage, speed up training, and allow pretraining from scratch on tiny hardware. Internal details are abstracted for simplicity.

---

Use

from transformers import pipeline

pipe = pipeline("text-generation", model="FlameF0X/i3-12m")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

---

Model Statistics

• Vocabulary Size: 4,466 (variable-length chunks)

• Hidden Dimension: 512

• Number of Layers: 12

• Max Sequence Length: 256

• Total Parameters: 12,691,186

• Tokenization: Memory-efficient variable-length chunking (2–3 characters)

  • Total tokens: 334,524,736

---

Key Features

1. Memory-Optimized: Designed to train on tiny hardware with minimal RAM usage
2. Pretraining-Focused Architecture: i3 layers provide efficient sequence modeling, low-rank linear updates, and factorized attention
3. Variable-Length Tokenization: 2–3 character chunks for compact embeddings
4. Conversational Readiness: Optimized for dialogue and text generation

---

i3 Architecture (Abstract Overview)

Design Philosophy

The i3 model targets CPU-friendly, memory-constrained pretraining, emphasizing:

• Long-range sequence modeling
• Low-rank weight updates for memory savings
• Efficient factorized attention
• 4-bit weights and microbatching for minimal memory footprint

Technologies used in the i3 Architecture that are open-sourced by me:

• Low-Rank Pre-training - LoRa for pre-training.

Conceptual Layout

Input Tokens
    │
+-----------------+
| Embedding Layer |
+-----------------+
    │
+-----------------+
| i3 Architecture |
+-----------------+
    │
+------------------------+
| KQV Low-Rank Attention |
+------------------------+
    │
+-----------------------+
| LayerNorm + Residuals |
+-----------------------+
    │
+-------------------+
| Output Projection |
+-------------------+
    │
Predicted Tokens

Key idea: Every component is optimized for memory efficiency and pretraining speed on small hardware, while preserving essential transformer dynamics.

---

Training Details

• Sequence length: 128–512 tokens
• Model size: ~12M parameters (CPU-friendly)
• Optimizer: AdamW or Lion (4-bit / mixed precision)
• Dataset: TinyChat (~50–200 MB)
• Training loop: gradient checkpointing + recomputation
• Objective: token prediction / text generation

Citation

@software{lorpt2025,
  title={LoRPt: Low-Rank Pretraining for Resource-Efficient Language Models},
  author={[FlameF0X]},
  year={2025},
  url={https://github.com/FlameF0X/Low-Rank-Pretraining}
}