i3-lab
/

i3-tiny

@@ -8,83 +8,62 @@ tags:
 - i3-arhitecture
 ---
-# Model Card: i3Model (Hybrid Efficient LLM)
-## Overview
-**i3Model** is a research-focused, ultra-efficient large language model prototype designed for exploring advanced hybrid architectures that balance **performance, scalability, and memory efficiency**. It integrates several experimental mechanisms for sequence modeling, low-rank parameterization, and quantization-aware training to achieve strong performance under resource constraints.
-This model was developed for experimentation in lightweight large language modeling, particularly for tasks such as:
-* Character- or token-level language modeling
-* Text generation and continuation
-* Research into efficient training and deployment techniques
-> **Note:** Architectural details are proprietary and are intentionally omitted.
 ---
-## Intended Use
-The model is intended for:
-* Academic research on hybrid recurrent-transformer architectures
-* Prototyping efficient LLMs for low-resource environments
-* Studying low-rank adaptation and quantization for model compression
-It is **not** optimized or tested for production deployment, safety-critical applications, or real-world text generation beyond controlled research settings.
 ---
-## Key Features
-* **Hybrid Recurrent–Sequence Modeling:** Combines sequence-mixing and dynamic state-space mechanisms for temporal reasoning.
-* **Low-Rank Parameterization:** Reduces parameter footprint while maintaining expressivity.
-* **Quantization-Aware Design:** Uses a 4-bit quantization scheme with FP32 master weights for training stability.
-* **Causal Autoregressive Training:** Enables next-token prediction and controlled text generation.
-* **Modular and Extensible:** Supports layer-wise experimentation and scalable configuration.
----
-## Training Details
-* **Training Objective:** Next-token prediction (causal language modeling)
-* **Dataset:** Custom small-scale character-level corpus derived from public domain text passages
-* **Sequence Length:** 128 tokens (for prototype training)
-* **Optimization:** AdamW optimizer with weight decay and gradient-based updates
-* **Learning Rate:** 3e-4
-* **Training Duration:** ~2000 iterations on a small dataset
-* **Batch Size:** 2
-The model was trained primarily for demonstration and performance measurement purposes rather than benchmark-level convergence.
 ---
-## Evaluation
-Evaluation focused on:
-* **Training stability**
-* **Generation coherence at small scale**
-* **Speed and memory performance metrics**
-While not benchmarked on large-scale NLP datasets, the model demonstrates promising early results in lightweight text generation with efficient runtime characteristics.
 ---
 ## Limitations
-* The model is trained on a very limited dataset and may produce incoherent or repetitive outputs.
-* It lacks fine-tuning for alignment, safety, or factual consistency.
-* It is unsuitable for deployment in sensitive or user-facing contexts.
-* Generation quality is constrained by vocabulary and training corpus diversity.
 ---
-## Ethical Considerations
-This model is intended solely for **research and educational use**. Users should:
-* Avoid using it to generate misleading or harmful content.
-* Not deploy it in systems interacting with the public without additional alignment and safety layers.
-* Attribute the model appropriately if adapted or redistributed.

 - i3-arhitecture
 ---
+# i3-tiny
+**i3-tiny** is a compact, efficient character-level language model designed for experimentation and exploration in text generation. Despite its small size, it packs a surprising punch for creative and research-oriented tasks, generating sequences that are quirky, unpredictable, and full of “human-like” character-level errors.
 ---
+## Model Overview
+i3-tiny is trained to predict the next character in a sequence, making it ideal for **character-level language modeling**, **creative text generation**, and **research on lightweight, efficient models**. Its small footprint allows rapid experimentation, even on modest hardware, and it provides a playground for studying how models learn patterns in sequences of characters.
+The model is **intentionally experimental** — it’s not aligned, fact-checked, or polished. Instead, it showcases how a compact architecture can capture patterns in text, learn from repetition, and generate outputs that are sometimes surprisingly coherent, sometimes hilariously garbled.
 ---
+## Training Details
+* **Dataset:** ~45,830 characters (a curated text corpus repeated to improve exposure)
+* **Vocabulary:** 34 characters (all lowercased)
+* **Sequence length:** 128
+* **Training iterations:** 2,000
+* **Batch size:** 2
+* **Optimizer:** AdamW, learning rate 3e-4
+* **Model parameters:** 711,106
+* **Performance notes:** Each iteration takes roughly 400–500 ms; 100 iterations take ~45 s on average. Loss steadily decreased from 3.53 to 2.15 over training.
+**Example generation (iteration 1200):**
+```
+Prompt: "The quick"
+Generated: the quick efehn. dethe cans the fice the fpeens antary of eathetint, an thadat hitimes the and cow thig, and
+```
+These outputs capture the **chaotic creativity** of a character-level model: a mixture of readable words, invented forms, and surprising sequences.
 ---
+## Intended Uses
+* **Character-level text generation experiments**
+* **Research and education**: studying lightweight language models, sequence learning, and text modeling
+* **Creative exploration**: generating quirky text or procedural content for games, demos, or artistic projects
+> ⚠️ i3-tiny is experimental and **not intended for production or high-stakes applications**. Text may be repetitive, nonsensical, or inconsistent.
 ---
 ## Limitations
+* Small vocabulary and character-level modeling limit natural language fluency
+* Outputs are **highly experimental** and not fact-checked
+* Generated sequences can be repetitive or unexpectedly garbled
+* Not aligned or safety-checked
 ---
+## Model Weights
+* Stored in `model.bin`
+* Compatible with PyTorch