Upload README.md

README.md

@@ -1,5 +1,5 @@
 ---
-license: apache-2.0
+license: gpl-3.0
 datasets:
 - HuggingFaceFW/fineweb-edu
 - mattwesney/General_Inquiry_Thinking-Chain-Of-Thought
@@ -11,63 +11,60 @@ language:
 - en
 tags:
 - small
-- haiku
-new_version: CompactAI-O/TMLM-Haiku-2
+- glint
+new_version: CompactAI-O/Glint-0.3
 ---
-# TinyMemoryLM (Haiku)
 
-> **⚠️ IMPORTANT NOTICE**
-> 1. **The model is really dumb.** This is a sub-1M parameter research model designed for experimentation, not production use.
-> 2. **Do not expect it to answer any questions.** It is prone to repetition, hallucination, and format collapse.
+# Glint-0.2
 
-## Overview
+> The pipe character incident. We do not talk about the pipe character incident.
 
-TinyMemoryLM is an ultra-lightweight language model optimized for edge cases and architectural experimentation. Despite its small footprint, it incorporates several novel training innovations aimed at stabilizing tiny model convergence, including hybrid tokenization, loss boosting strategies, and context-aware relevance modeling.
+Glint-0.2 was supposed to be the smart one. It has weight-tied layers, grouped-query attention, sliding windows, multi-token prediction heads. Fancy stuff. And sometimes it still outputs `|fdish||||!@|`.
 
-This release includes both **Pretrained Weights** (base language modeling, completion) and **Instruction Weights** (fine-tuned for chat).
+Progress is not a straight line.
 
-## Files Provided
+## What you get
 
-| File | Description |
+| File | What it is |
 | :--- | :--- |
-| `tokenizer.json` | Hybrid word/character tokenizer vocabulary (2,133 tokens). |
-| `pretrain.pt` | Base pretrained checkpoint (language modeling). |
-| `model.pt` | Instruction-tuned checkpoint (SFT/Chat). |
-| `samples.jsonl` | Sample generations with NLL/PPL metrics at checkpoints. |
-| `loss_curve.png` | Training loss progression across all phases. |
+| `tokenizer.json` | Hybrid word/char tokenizer (~2,133 tokens) |
+| `pretrain.pt` | Base pretrained checkpoint |
+| `model.pt` | Instruction-tuned checkpoint (SFT) |
+| `samples.jsonl` | Sample generations with metrics at checkpoints |
+| `loss_curve.png` | Training loss across all phases |
 
-## Model Specifications
+## Specs
 
-| Parameter | Value |
+| Thing | Value |
 | :--- | :--- |
 | **Architecture** | Transformer Decoder (GQA) |
 | **Parameters** | ~700K |
-| **Context Length** | 2,048 tokens |
+| **Context** | 2,048 tokens |
 | **Sliding Window** | 512 tokens |
-| **Dimensions** | `d_model=128`, `unique_layers=8`, `logical_layers=16`, `heads=4`, `kv_heads=2`, `ffn=224` |
-| **Vocabulary** | ~2,133 tokens (Hybrid Char + Word) |
-| **Normalization** | RMSNorm |
-| **Embeddings** | Rotary Embeddings (RoPE, 25% fraction) |
+| **d_model** | 128 |
+| **Unique Layers** | 8 (tied to make 16 logical) |
+| **Heads** | 4 |
+| **KV Heads** | 2 |
+| **FFN** | 224 |
+| **Vocab** | ~2,133 (Hybrid Char + Word) |
+| **Norm** | RMSNorm |
+| **Position** | RoPE (25% fraction) |
 | **Activation** | SwiGLU |
-| **Multi-Token Prediction** | Horizons at 2, 3, 4 |
+| **Multi-Token Prediction** | Horizons 2, 3, 4 |
 
-## Architecture Highlights
+## Fancy tricks
 
-TinyMemoryLM implements several research-focused modifications to standard transformer architectures:
+- **Weight-tied layers:** 8 unique transformer blocks repeated to make 16 layers. Every 3rd layer gets global attention instead of sliding window. Cheap and surprisingly effective.
+- **GQA:** 4 attention heads sharing 2 KV heads. Less cache, less compute.
+- **Sliding window:** 512 tokens local, with periodic global layers for long-range context.
+- **MTP:** Extra prediction heads at offsets 2, 3, and 4. Weighted at 0.3 during training.
+- **Hybrid tokenizer:** Word-level where possible, char fallback for the weird stuff.
+- **Word token loss boost:** 3x loss on multi-character tokens so the model actually learns words.
+- **Response-start weighting:** First 20 tokens of assistant responses get 3x weight.
 
-*   **Weight-Tied Logical Layers:** 8 unique transformer blocks are repeated to create 16 logical layers (every 3rd layer uses global attention vs. sliding window), drastically reducing parameter count.
-*   **Grouped-Query Attention (GQA):** 4 attention heads share 2 KV heads, reducing KV cache and compute.
-*   **Sliding Window Attention:** Local attention within 512-token windows, with periodic global layers for long-range context.
-*   **Multi-Token Prediction (MTP):** Auxiliary prediction heads at horizons 2, 3, and 4 with dedicated adapters and norms, weighted at 0.3 during training.
-*   **Hybrid Tokenizer:** Combines character-level fallback with frequent word tokens to balance compression and vocabulary size.
-*   **Word Token Loss Boosting:** Upweights loss signals for multi-character tokens (3x) to prevent the model from ignoring them in favor of character-level spelling.
-*   **Response-Start Weighting:** Prioritizes the first 20 tokens of assistant responses (3x weight) to improve prompt conditioning.
-*   **Embedding Scale:** Learned scaling factor applied to token embeddings for improved training dynamics.
+## Training
 
-
-### Training Hyperparameters
-
-| Parameter | Value |
+| Thing | Value |
 | :--- | :--- |
 | **Batch Size** | 48 |
 | **Pretrain LR** | 8e-4 (min 1e-5) |
@@ -75,26 +72,20 @@ TinyMemoryLM implements several research-focused modifications to standard trans
 | **Warmup** | 300 steps |
 | **Weight Decay** | 0.02 |
 | **Max Grad Norm** | 1.0 |
-| **MTP Weight** | 0.3 |
-| **Word Token Loss Boost** | 3.0x |
-| **Response-Start Boost** | 3.0x (first 20 tokens) |
-| **Checkpointing** | Every 1,000 steps |
+| **Checkpoint** | Every 1,000 steps |
 | **Sampling** | Every 5,000 steps |
 
-## Training Loss Curve
-
-Training loss progression across pretrain and SFT phases:
-
-![Training Loss Curve](model/loss_curve.png)
+## Loss curve
 
-## Limitations & Expectations
+![loss](model/loss_curve.png)
 
-Please manage your expectations when using TinyMemoryLM:
+## Limitations
 
-*   **Repetition:** Tiny models are prone to collapsing into repetitive token loops.
-*   **Knowledge:** The model has limited world knowledge due to parameter constraints.
-*   **Usage:** This model is intended for **research, educational purposes, and architectural benchmarking**. It is not suitable for assistant tasks or reliable information retrieval.
+- Repeats itself.
+- Knows almost nothing.
+- Research only. Not an assistant.
+- Sometimes hallucinates pipes.
 
 ---
 
-*Generated for research purposes. Use responsibly.*
+*Built by [CompactAI](https://huggingface.co/CompactAI-O). We learn by failing.*