Spaces:

CompactAI-O
/

Homepage

Running

App Files Files Community

CompactAI commited on 10 days ago

Commit

906d75d

verified ·

1 Parent(s): dccefd3

Create Weird Training Idea: Embeddings First, Everything Else Later.html

Browse files

Files changed (1) hide show

Weird Training Idea: Embeddings First, Everything Else Later.html +184 -0

Weird Training Idea: Embeddings First, Everything Else Later.html ADDED Viewed

	@@ -0,0 +1,184 @@

+<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>A Weird Training Idea: Embeddings First, Everything Else Later | FMN-GPT - CompactAI</title>
+    <link rel="stylesheet" href="bluesheet.css">
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
+    <link href="https://fonts.googleapis.com/css2?family=Geist:wght@400:500:600:700&family=Geist+Mono&display=swap" rel="stylesheet">
+<style>
+        :root {
+            --blue-900: #000000;
+            --blue-800: #0a0a0a;
+            --blue-700: #111111;
+            --blue-600: #1a1a1a;
+            --blue-500: #333333;
+            --blue-400: #555555;
+            --blue-300: #777777;
+            --blue-200: #888888;
+            --blue-100: #aaaaaa;
+            --white: #ffffff;
+            --white-soft: #f5f5f5;
+            --white-muted: #e0e0e0;
+            --grid-line: rgba(255, 255, 255, 0.03);
+            --grid-line-major: rgba(255, 255, 255, 0.06);
+            --accent: #ededed;
+            --accent-muted: #888888;
+            --font-sans: 'Geist', -apple-system, BlinkMacSystemFont, sans-serif;
+            --font-mono: 'Geist Mono', 'SF Mono', 'Fira Code', monospace;
+            --container-max: 1100px;
+        }
+        * { box-sizing: border-box; margin: 0; padding: 0; }
+        html { font-size: 16px; scroll-behavior: smooth; }
+        body { font-family: var(--font-sans); background: var(--blue-900); color: var(--white-muted); line-height: 1.7; -webkit-font-smoothing: antialiased; }
+        a { color: var(--white); text-decoration: none; transition: color 0.15s ease; }
+        a:hover { color: var(--accent); }
+        .container { max-width: var(--container-max); margin: 0 auto; padding: 0 24px; }
+        nav { position: fixed; top: 0; left: 0; right: 0; z-index: 100; background: rgba(0, 0, 0, 0.85); backdrop-filter: blur(12px); border-bottom: 1px solid var(--blue-600); padding: 16px 0; }
+        nav .container { display: flex; justify-content: space-between; align-items: center; }
+        .nav-brand { font-size: 18px; font-weight: 600; color: var(--white); display: flex; align-items: center; gap: 8px; }
+        .nav-brand span { color: var(--accent); }
+        .nav-links { display: flex; gap: 32px; }
+        .nav-links a { font-size: 14px; font-weight: 500; color: var(--blue-200); }
+        .nav-links a:hover { color: var(--white); }
+        .post { padding: 140px 0 80px; }
+        .post-back { display: inline-block; color: var(--blue-200); font-size: 14px; margin-bottom: 32px; }
+        .post-back:hover { color: var(--accent); }
+        .post-back::before { content: '← '; }
+        .post-meta { display: flex; gap: 12px; margin-bottom: 20px; }
+        .post-date { font-size: 13px; color: var(--blue-200); font-family: var(--font-mono); }
+        .post-tag { font-size: 11px; font-weight: 600; text-transform: uppercase; letter-spacing: 0.05em; color: var(--white); background: rgba(255, 255, 255, 0.08); padding: 4px 10px; border-radius: 4px; }
+        .post h1 { font-size: 36px; font-weight: 700; color: var(--white); margin-bottom: 32px; line-height: 1.2; letter-spacing: -0.02em; }
+        .post-body p { font-size: 17px; line-height: 1.8; margin-bottom: 24px; color: var(--blue-200); }
+        .post-body p:first-of-type { font-size: 20px; color: var(--white-muted); }
+        .post-body h2 { font-size: 24px; font-weight: 600; color: var(--white); margin: 48px 0 20px; }
+        .post-body blockquote { border-left: 3px solid var(--accent); padding: 20px 24px; margin: 32px 0; background: var(--blue-800); border-radius: 0 8px 8px 0; }
+        .post-body blockquote p { font-size: 16px; font-style: italic; color: var(--blue-200); margin: 0; }
+        .post-body hr { border: none; height: 1px; background: var(--blue-600); margin: 48px 0; }
+        .code-block { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; margin: 24px 0; font-family: var(--font-mono); font-size: 13px; overflow-x: auto; }
+        .code-block .comment { color: var(--blue-200); font-style: italic; display: block; margin-top: 4px; }
+        .stats-grid { display: grid; grid-template-columns: 1fr 1fr; gap: 16px; margin: 24px 0; }
+        .stat-card { background: var(--blue-800); border: 1px solid var(--blue-600); border-radius: 8px; padding: 20px; text-align: center; }
+        .stat-card .number { font-size: 32px; font-weight: 700; color: var(--accent); font-family: var(--font-mono); }
+        .stat-card .label { font-size: 13px; color: var(--blue-200); margin-top: 8px; }
+        .post-footer { margin-top: 48px; padding-top: 32px; border-top: 1px solid var(--blue-600); }
+        .post-footer p { font-size: 14px; color: var(--blue-200); font-style: italic; margin: 0; }
+        footer { padding: 40px 0; background: var(--blue-800); border-top: 1px solid var(--blue-600); text-align: center; }
+        footer p { color: var(--blue-200); font-size: 14px; margin-bottom: 8px; }
+        footer a { color: var(--blue-200); }
+        footer a:hover { color: var(--accent); }
+        @media (max-width: 768px) { .post h1 { font-size: 28px; } .nav-links { display: none; } .stats-grid { grid-template-columns: 1fr; } }
+</style>
+</head>
+<body>
+    <nav>
+        <div class="container">
+            <a href="index.html" class="nav-brand"><span>/</span>FMN-GPT</a>
+            <div class="nav-links">
+                <a href="blog.html">Blog</a>
+                <a href="status.html">Model Status</a>
+                <a href="https://huggingface.co/CompactAI-O" target="_blank">HuggingFace Org</a>
+            </div>
+        </div>
+    </nav>
+    <main>
+        <article class="post">
+            <div class="container">
+                <a href="blog.html" class="post-back">Back to Blog</a>
+                <header>
+                    <div class="post-meta">
+                        <span class="post-date">2026-05-15</span>
+                        <span class="post-tag">Architecture Ideas</span>
+                    </div>
+                    <h1>A Weird Training Idea: Embeddings First, Everything Else Later</h1>
+                </header>
+                <div class="post-body">
+                    <p>I have a weird idea. It might be stupid. It might be brilliant. I do not know yet. The idea is simple. Instead of training the entire model at once, train the embedding layers first. Then train the rest of the model to utilize those embeddings. Two stages. Two objectives. One hope.</p>
+                    <blockquote>
+                        <p>Weird ideas are the only ideas worth having. The standard approaches are well documented. The weird approaches are where progress hides.</p>
+                    </blockquote>
+                    <h2>The Core Concept</h2>
+                    <p>Here is how it would work. Stage one takes 100 billion tokens and shoves them into the embedding layers. The embeddings learn to represent tokens in a rich, structured space. The rest of the model sits idle. Stage two freezes those embeddings. The rest of the model trains on another 100 billion tokens, learning to use the fixed embeddings as input.</p>
+                    <div class="code-block">
+                        <span class="comment"># Two-stage training in pseudocode</span><br>
+                        # Stage 1: Train embeddings only<br>
+                        embeddings = EmbeddingLayer(vocab_size, hidden_dim)<br>
+                        for token_batch in dataset_100B:<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;loss = embedding_objective(token_batch, embeddings)<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;embeddings.update(loss)<br>
+                        <br>
+                        # Stage 2: Freeze embeddings, train the rest<br>
+                        embeddings.freeze()<br>
+                        model = Transformer(rest_of_architecture)<br>
+                        for token_batch in dataset_100B_round2:<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;embedded = embeddings(token_batch)<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;loss = language_modeling_objective(embedded, model)<br>
+                        &nbsp;&nbsp;&nbsp;&nbsp;model.update(loss)<br>
+                        <span class="comment"># Simple. Weird. Possibly broken.</span>
+                    </div>
+                    <p>The intuition is that embeddings carry most of the semantic weight. If embeddings are well trained, the rest of the model has an easier job. It does not need to learn what tokens mean. It only needs to learn how to combine those meanings. That division of labor might accelerate learning. It might also create a bottleneck. Both outcomes are informative.</p>
+                    <h2>Why This Might Help</h2>
+                    <div class="stats-grid">
+                        <div class="stat-card">
+                            <div class="number">2</div>
+                            <div class="label">Training Stages</div>
+                        </div>
+                        <div class="stat-card">
+                            <div class="number">200B</div>
+                            <div class="label">Total Tokens</div>
+                        </div>
+                        <div class="stat-card">
+                            <div class="number">???</div>
+                            <div class="label">Actual Benefit</div>
+                        </div>
+                        <div class="stat-card">
+                            <div class="number">∞</div>
+                            <div class="label">My Curiosity</div>
+                        </div>
+                    </div>
+                    <p>Training embeddings separately could reduce interference. When all parameters update simultaneously, gradients compete. Embedding gradients might conflict with attention gradients. Separating the phases isolates the objectives. Each phase optimizes a clear target. That clarity might improve convergence.</p>
+                    <p>Fixed embeddings could also stabilize training. The rest of the model sees consistent input representations. It does not need to adapt to shifting embeddings. That stability might allow deeper architectures or higher learning rates. It might also limit expressivity. The trade-off is the experiment.</p>
+                    <h2>Potential Problems</h2>
+                    <p>The embeddings might overfit to stage one objectives. They might learn representations that are optimal for embedding loss but suboptimal for language modeling. The rest of the model then inherits those limitations. It cannot request better embeddings. It must work with what it is given.</p>
+                    <p>There is also the risk of capacity mismatch. If embeddings are too expressive, the rest of the model might underutilize them. If embeddings are too constrained, the rest of the model might struggle to compensate. Finding the right balance requires tuning. Tuning requires time. Time is the resource I lack.</p>
+                    <blockquote>
+                        <p>Every architectural choice is a bet. This bet is that staged training reduces interference. The house always wins. I am placing the bet anyway.</p>
+                    </blockquote>
+                    <h2>How I Might Test It</h2>
+                    <p>I would start small. A 1M parameter model. A 500 token vocabulary. Two training stages of 10B tokens each. Compare against a baseline trained end-to-end on the same 20B tokens. Measure loss, perplexity, and downstream task performance. The difference, if any, will guide the next iteration.</p>
+                    <div class="code-block">
+                        <span class="comment"># Minimal experiment design</span><br>
+                        Baseline: Train full model on 20B tokens<br>
+                        Experiment: Stage 1 embeddings on 10B, Stage 2 rest on 10B<br>
+                        Metrics: Wikitext-2 loss, BLiMP accuracy, ARC-Easy score<br>
+                        Decision: Keep the approach if metrics improve by 5%+<br>
+                        <span class="comment"># Measure twice. Cut once. Iterate slowly.</span>
+                    </div>
+                    <p>If the staged approach helps at small scale, I would scale up. Larger vocabularies. More tokens. Deeper architectures. The principle remains the same. The implementation grows. The risk grows too. That is the nature of research.</p>
+                    <h2>Why I Am Thinking About This</h2>
+                    <p>I train tiny models. I care about efficiency. I care about making every parameter count. Embeddings are a large fraction of tiny model parameters. If embeddings can be trained more effectively, the whole model benefits. That potential motivates the idea.</p>
+                    <p>I also like weird ideas. I like asking questions that might have simple answers. I like exploring concepts that might just work. This is how I learn. This is how I have fun. This is how I end up with blogs about staged embedding training.</p>
+                    <h2>Final Thoughts</h2>
+                    <p>Train embeddings first. Freeze them. Train the rest to use them. The idea is simple. The implementation is uncertain. The outcome is unknown. That uncertainty is the point. That uncertainty is the opportunity.</p>
+                    <p>I might test this idea. I might fail. I might learn something. That is the cycle. That is the process. That is how progress happens in my tiny corner of AI research.</p>
+                    <p>If you have thoughts on this idea, let me know. If you have tried something similar, share your results. If you think this is a terrible idea, you are probably right. But terrible ideas sometimes lead to good ones. That is how innovation works. That is how I work.</p>
+                    <hr>
+                </div>
+                <footer class="post-footer">
+                    <p>Current status: Weird idea documented. Experiment design drafted. Risks acknowledged. Optimism maintained. Progress is weird. Curiosity is higher.</p>
+                </footer>
+            </div>
+        </article>
+    </main>
+    <footer>
+        <div class="container">
+            <p>Built with curiosity over compute</p>
+            <p>FMN-GPT by <a href="https://huggingface.co/CompactAI-O" target="_blank">CompactAI-O</a> | 2026</p>
+        </div>
+    </footer>
+</body>
+</html>