Spaces:

CompactAI
/

Built-with-curiosity-not-compute

Running

App Files Files Community

CompactAI commited on 6 days ago

Commit

fcf16d3

verified ·

1 Parent(s): 23610a1

Delete index.html

Browse files

Files changed (1) hide show

index.html +0 -531

index.html DELETED Viewed

@@ -1,531 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-<head>
-    <meta charset="UTF-8">
-    <meta name="viewport" content="width=device-width, initial-scale=1.0">
-    <title>FMN-GPT | CompactAI</title>
-    <link rel="stylesheet" href="styles.css">
-    <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet">
-</head>
-<body>
-    <nav class="main-nav">
-        <div class="container">
-            <a href="index.html" class="nav-brand">FMN-GPT</a>
-            <div class="nav-links">
-                <a href="blog.html">Blog</a>
-                <a href="status.html">Model Status</a>
-                <a href="https://huggingface.co/CompactAI" target="_blank">HuggingFace</a>
-            </div>
-        </div>
-    </nav>
-    <main>
-        <!-- Hero Section -->
-        <section class="hero">
-        <div class="hero-content">
-            <div class="apology-badge">A New Approach to Small Models</div>
-            <h1 class="hero-title">Why I Stopped<br><span class="highlight">Compressing Models</span></h1>
-            <p class="hero-subtitle">And started building something that doesn't need 8 H100s to think</p>
-            <div class="scroll-indicator">
-                <span>Scroll for the truth</span>
-                <div class="scroll-arrow"></div>
-            </div>
-        </div>
-        <div class="hero-visual">
-            <canvas id="neuron-canvas"></canvas>
-        </div>
-    </section>
-    <!-- A Note Before We Begin -->
-    <section class="preface" id="preface">
-        <div class="container">
-            <h2 class="section-title">A Note Before We Begin</h2>
-            <div class="preface-content">
-                <p class="drop-cap">You might have noticed that everything on my profile got deleted. That was intentional. Let me explain why.</p>
-                <p>I used to fill my profile with compressed models. Dozens of them. But I realized that quantity was masking the real problem. I wasn't building anything new. Just cloning and shrinking other people's work.</p>
-                <p>So I wiped the slate clean. A fresh start. This time, I'm here to <strong>build from scratch</strong>.</p>
-            </div>
-        </div>
-    </section>
-    <!-- The Confession -->
-    <section class="confession" id="confession">
-        <div class="container">
-            <h2 class="section-title">The Confession</h2>
-            <div class="confession-content">
-                <div class="confession-text">
-                    <p class="drop-cap">Let me be honest with you. For months, I was that person. You know the one. Leaving my computer on overnight, running distillation scripts, trying to squeeze GPT-4 into something that could run on a potato.</p>
-                    <p>And you know what? <strong>It was boring.</strong></p>
-                    <p>Who actually enjoys watching loss curves descend at 3 AM? Who gets excited about shaving off 2% of parameters while the model forgets how to count?</p>
-                    <blockquote>
-                        "I was cloning someone else's work and compressing it. The process lacked real creation. Just digital photocopying with extra steps."
-                    </blockquote>
-                    <p>So I stopped. And I started asking a different question:</p>
-                    <div class="big-question">
-                        <span class="question-mark">?</span>
-                        <p>What if a model could be small <em>by design</em>, avoiding compression entirely?</p>
-                    </div>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- What I'm Building -->
-    <section class="what-building" id="what">
-        <div class="container">
-            <h2 class="section-title">What I'm Building Instead</h2>
-            <div class="feature-grid">
-                <div class="feature-card main-feature">
-                    <div class="feature-icon">
-                        <svg viewBox="0 0 100 100" class="neuron-icon">
-                            <circle cx="50" cy="30" r="15" class="neuron-node"/>
-                            <circle cx="25" cy="70" r="12" class="neuron-node"/>
-                            <circle cx="75" cy="70" r="12" class="neuron-node"/>
-                            <path d="M50 45 L30 58" class="neuron-connection"/>
-                            <path d="M50 45 L70 58" class="neuron-connection"/>
-                            <path d="M37 70 L63 70" class="neuron-connection" stroke-dasharray="5,3"/>
-                        </svg>
-                    </div>
-                    <h3>FMN-GPT</h3>
-                    <p class="feature-subtitle">Factored Multiplicative Neuron Transformer</p>
-                    <p>A transformer architecture where each neuron can call backward into the network. A fundamentally different way to think, designed from scratch.</p>
-                    <div class="feature-stats">
-                        <div class="stat">
-                            <span class="stat-value">491</span>
-                            <span class="stat-label">Vocab Size</span>
-                        </div>
-                        <div class="stat">
-                            <span class="stat-value">65K</span>
-                            <span class="stat-label">Context Length</span>
-                        </div>
-                        <div class="stat">
-                            <span class="stat-value">120</span>
-                            <span class="stat-label">Max Loops/Neuron</span>
-                        </div>
-                    </div>
-                    <p class="disclaimer-text">Everything is subject to change.</p>
-                </div>
-            </div>
-            <!-- Architecture Diagram -->
-            <div class="architecture-section">
-                <h3>The Architecture</h3>
-                <div class="architecture-diagram" id="arch-diagram">
-                    <div class="arch-layer" data-layer="input">
-                        <div class="layer-box input-layer">
-                            <span>Input Embeddings</span>
-                        </div>
-                    </div>
-                    <div class="arch-arrow">v</div>
-                    <div class="arch-layer" data-layer="recursive">
-                        <div class="layer-box recursive-layer">
-                            <span>Shared Transformer Block x6</span>
-                            <div class="layer-details">
-                                <div class="detail-item">
-                                    <span class="detail-label">Multi-Query Attention</span>
-                                    <span class="detail-value">4 heads</span>
-                                </div>
-                                <div class="detail-item">
-                                    <span class="detail-label">FMN Feedforward</span>
-                                    <span class="detail-value">Rank 40</span>
-                                </div>
-                                <div class="detail-item">
-                                    <span class="detail-label">Dynamic Routing</span>
-                                    <span class="detail-value">REINFORCE based</span>
-                                </div>
-                            </div>
-                        </div>
-                    </div>
-                    <div class="arch-arrow loop-arrow">
-                        <span>Recurrent State</span>
-                    </div>
-                    <div class="arch-layer" data-layer="output">
-                        <div class="layer-box output-layer">
-                            <span>Output (Weight-Tied)</span>
-                        </div>
-                    </div>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- How It Works -->
-    <section class="how-it-works" id="how">
-        <div class="container">
-            <h2 class="section-title">How It Actually Works</h2>
-            <div class="mechanism-tabs">
-                <button class="tab-btn active" data-tab="fmn">FMN Neurons</button>
-                <button class="tab-btn" data-tab="routing">Dynamic Routing</button>
-                <button class="tab-btn" data-tab="recurrent">Recurrent Mixer</button>
-                <button class="tab-btn" data-tab="loops">Loop Counter</button>
-            </div>
-            <div class="tab-content">
-                <!-- FMN Tab -->
-                <div class="tab-pane active" id="fmn-pane">
-                    <div class="pane-content">
-                        <div class="pane-text">
-                            <h3>Factored Multiplicative Neurons</h3>
-                            <p>Traditional neurons compute <code>y = σ(Wx + b)</code>. Simple, but limited.</p>
-                            <p>FMN neurons compute:</p>
-                            <div class="equation">
-                                <code>gate = tanh(W1(x)) * sigmoid(W2(x))</code>
-                            </div>
-                            <div class="equation">
-                                <code>output = V(gate)</code>
-                            </div>
-                            <p>Each FMN uses a multiplicative gating mechanism with two weight matrices W1 and W2 that project to a rank 40 latent space, then combines via tanh and sigmoid before projecting back through V.</p>
-                            <ul class="feature-list">
-                                <li>Rank 40 latent dimension for efficient computation</li>
-                                <li>Multiplicative gating enables complex interactions</li>
-                                <li>Optional SwiGLU variant available</li>
-                                <li>Initialized with Xavier uniform for stability</li>
-                            </ul>
-                        </div>
-                        <div class="pane-visual">
-                            <canvas id="fmn-canvas" width="400" height="300"></canvas>
-                        </div>
-                    </div>
-                </div>
-                <!-- Routing Tab -->
-                <div class="tab-pane" id="routing-pane">
-                    <div class="pane-content">
-                        <div class="pane-text">
-                            <h3>Dynamic Neuron Routing</h3>
-                            <p>Each neuron can route its output to any layer and any neuron in the network using <strong>REINFORCE policy gradients</strong> for true gradient flow through hard routing decisions.</p>
-                            <div class="equation">
-                                <code>should_route ~ Bernoulli(sigmoid(logits))</code>
-                            </div>
-                            <div class="equation">
-                                <code>target_layer ~ Categorical(softmax(layer_logits))</code>
-                            </div>
-                            <p>The router learns:</p>
-                            <ul class="feature-list">
-                                <li>Whether to route via learned sigmoid gate</li>
-                                <li>Which layer to target (0 to n_layers)</li>
-                                <li>Which neuron to target (0 to d_model)</li>
-                                <li>Routing strength via learned parameter</li>
-                            </ul>
-                        </div>
-                        <div class="pane-visual">
-                            <canvas id="routing-canvas" width="400" height="300"></canvas>
-                        </div>
-                    </div>
-                </div>
-                <!-- Recurrent Tab -->
-                <div class="tab-pane" id="recurrent-pane">
-                    <div class="pane-content">
-                        <div class="pane-text">
-                            <h3>Recurrent Mixer</h3>
-                            <p>A per-channel gated recurrent state that persists across layer iterations. Think of it as a tiny LSTM for each dimension.</p>
-                            <div class="equation">
-                                <code>g = sigmoid(x * w_x + s * w_s + b)</code>
-                            </div>
-                            <div class="equation">
-                                <code>s_new = s * (1 - g) + x * g</code>
-                            </div>
-                            <p>This allows the model to accumulate information across the 6 layer passes, creating a form of <strong>internal memory</strong>. The state scale parameter controls how much the recurrent state influences each layer.</p>
-                        </div>
-                        <div class="pane-visual">
-                            <canvas id="recurrent-canvas" width="400" height="300"></canvas>
-                        </div>
-                    </div>
-                </div>
-                <!-- Loops Tab -->
-                <div class="tab-pane" id="loops-pane">
-                    <div class="pane-content">
-                        <div class="pane-text">
-                            <h3>Loop Counter</h3>
-                            <p>Each neuron has a hard limit on how many times it can participate in routing loops. This prevents infinite cycles and forces the model to be efficient.</p>
-                            <div class="equation">
-                                <code>loop_exhausted = loop_counts >= max_loops</code>
-                            </div>
-                            <p>With <code>max_loops = 120</code>, each neuron can participate in at most 120 routing events before being silenced. This creates a form of <strong>computational budget</strong>.</p>
-                            <div class="loop-demo">
-                                <label>Max Loops: <span id="loop-value">120</span></label>
-                                <input type="range" id="loop-slider" min="1" max="200" value="120">
-                                <div class="loop-indicator" id="loop-indicator"></div>
-                            </div>
-                        </div>
-                        <div class="pane-visual">
-                            <canvas id="loops-canvas" width="400" height="300"></canvas>
-                        </div>
-                    </div>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- Explain Like I'm Five -->
-    <section class="elief-section" id="elief">
-        <div class="container">
-            <h2 class="section-title">Explain Like I'm Five</h2>
-            <p class="section-subtitle">How does this tiny model think?</p>
-            <div class="elief-content">
-                <div class="elief-card">
-                    <h3>What is FMN-GPT?</h3>
-                    <p>Imagine a really tiny brain. Most AI brains today are huge, like a library with millions of books. FMN-GPT is more like a small notebook. But here's the trick. It can read that notebook over and over, each time understanding a little more.</p>
-                </div>
-                <div class="elief-card">
-                    <h3>Why Character-Level?</h3>
-                    <p>Most AI models learn whole words at a time. We taught this one to read letter by letter, like a child learning to read. This keeps it small. It only needs to know 491 characters instead of thousands of words. Every letter matters.</p>
-                </div>
-                <div class="elief-card">
-                    <h3>How Does It Think?</h3>
-                    <p>When you ask a question, the model passes it through the same brain circuit 6 times. Each pass, it thinks a little deeper. Like asking yourself "what's 2+2?" and then checking, "Wait, let me think again. 2+2... that's adding two groups of two... so that makes 4!"</p>
-                </div>
-                <div class="elief-card">
-                    <h3>The Magic Number: 491 Characters</h3>
-                    <p>The model uses a character-level vocabulary of exactly 491 tokens. This includes ASCII characters, special symbols, and custom thinking tokens (the thinking emoji and light bulb emoji). Every character matters.</p>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- Why I Stopped -->
-    <section class="why-stopped" id="why">
-        <div class="container">
-            <h2 class="section-title">Why I Really Stopped</h2>
-            <div class="reasons-grid">
-                <div class="reason-card">
-                    <div class="reason-number">01</div>
-                    <h3>It Was Boring</h3>
-                    <p>There's no creativity in distillation. You're just making a smaller copy of someone else's breakthrough. Where's the fun in that?</p>
-                </div>
-                <div class="reason-card">
-                    <div class="reason-number">02</div>
-                    <h3>Diminishing Returns</h3>
-                    <p>Every 1% of parameter reduction came with a measurable drop in capability. The tradeoff wasn't worth it.</p>
-                </div>
-                <div class="reason-card">
-                    <div class="reason-number">03</div>
-                    <h3>Overnight Runs</h3>
-                    <p>Who leaves their computer on overnight to clone someone else's work and compress it? The process lacks real creation. Mere photocopying in disguise.</p>
-                </div>
-                <div class="reason-card">
-                    <div class="reason-number">04</div>
-                    <h3>A Better Question</h3>
-                    <p>Instead of asking "how do I make this smaller?", I started asking "what if it was designed to be small from the start?"</p>
-                </div>
-            </div>
-            <div class="comparison-section">
-                <h3>The Old Way vs. The New Way</h3>
-                <div class="comparison-grid">
-                    <div class="comparison-item old">
-                        <h4>Model Compression</h4>
-                        <ul>
-                            <li>Start with 7B parameters</li>
-                            <li>Distill, quantize, prune</li>
-                            <li>End with 1B parameters</li>
-                            <li>Capability: ???</li>
-                            <li>Time: Weeks of compute</li>
-                        </ul>
-                    </div>
-                    <div class="comparison-arrow">-></div>
-                    <div class="comparison-item new">
-                        <h4>Small by Design</h4>
-                        <ul>
-                            <li>Start with ~100K parameters</li>
-                            <li>Novel architecture</li>
-                            <li>End with ~100K parameters</li>
-                            <li>Capability: Emergent</li>
-                            <li>Time: One GPU, one night</li>
-                        </ul>
-                    </div>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- Roadmap -->
-    <section class="roadmap-section" id="roadmap">
-        <div class="container">
-            <h2 class="section-title">Roadmap</h2>
-            <p class="section-subtitle">Where we're headed (everything is subject to change)</p>
-            <div class="roadmap-timeline">
-<div class="roadmap-item completed">
-                <div class="roadmap-marker"></div>
-                <div class="roadmap-content">
-                    <h4>Phase 1: Core Architecture</h4>
-                    <p>FMN neurons with rank 40, REINFORCE based dynamic routing, recurrent mixer, QK normalization, gated residuals. The foundation is complete.</p>
-                    <span class="roadmap-status">Completed</span>
-                </div>
-            </div>
-            <div class="roadmap-item in-progress">
-                <div class="roadmap-marker"></div>
-                <div class="roadmap-content">
-                    <h4>Phase 2: Training Pipeline</h4>
-                    <p>Character-level tokenization (491 vocab), 7 instruction datasets, pretraining on English-Pretraining-Dataset, AdamW optimizer, bfloat16 precision.</p>
-                    <span class="roadmap-status">In Progress</span>
-                </div>
-            </div>
-                <div class="roadmap-item">
-                    <div class="roadmap-marker"></div>
-                    <div class="roadmap-content">
-                        <h4>Phase 3: CoT Reasoning</h4>
-                        <p>Teaching the model to think step-by-step with explicit reasoning traces.</p>
-                        <span class="roadmap-status">Planned</span>
-                    </div>
-                </div>
-                <div class="roadmap-item">
-                    <div class="roadmap-marker"></div>
-                    <div class="roadmap-content">
-                        <h4>Phase 4: Evaluation Suite</h4>
-                        <p>Comprehensive benchmarks to measure what 100K parameters can actually do.</p>
-                        <span class="roadmap-status">Planned</span>
-                    </div>
-                </div>
-                <div class="roadmap-item">
-                    <div class="roadmap-marker"></div>
-                    <div class="roadmap-content">
-                        <h4>Phase 5: Model Release</h4>
-                        <p>Open weights on HuggingFace for the community to experiment with.</p>
-                        <span class="roadmap-status">Planned</span>
-                    </div>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- Dataset Credits -->
-    <section class="credits-section" id="credits">
-        <div class="container">
-            <h2 class="section-title">Dataset Credits</h2>
-            <p class="section-subtitle">The data that trains our model (subject to change)</p>
-            <div class="credits-grid">
-                <div class="credit-card">
-                    <h4>Pretraining Dataset</h4>
-                    <p class="credit-name">shuyuej/English-Pretraining-Dataset</p>
-                    <p>Large-scale English text for initial language understanding and general knowledge.</p>
-                    <a href="https://huggingface.co/datasets/shuyuej/English-Pretraining-Dataset" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 1</h4>
-                    <p class="credit-name">TeichAI/Pony-Alpha-15k</p>
-                    <p>Conversational instruction data for learning dialogue patterns and responses.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/Pony-Alpha-15k" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 2</h4>
-                    <p class="credit-name">TeichAI/convo-v1</p>
-                    <p>Multi-turn conversation data for context handling and coherent dialogue.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/convo-v1" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 3</h4>
-                    <p class="credit-name">TeichAI/Step-3.5-Flash-2600x</p>
-                    <p>High-quality instruction-response pairs for fine-tuning reasoning capabilities.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/Step-3.5-Flash-2600x" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 4</h4>
-                    <p class="credit-name">TeichAI/sherlock-thinking-alpha-11000x</p>
-                    <p>Thinking and reasoning data for chain of thought training.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/sherlock-thinking-alpha-11000x" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 5</h4>
-                    <p class="credit-name">TeichAI/glm-4.7-2000x</p>
-                    <p>GLM model outputs for diverse response patterns.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/glm-4.7-2000x" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 6</h4>
-                    <p class="credit-name">TeichAI/claude-haiku-4.5-high-reasoning-1700x</p>
-                    <p>Claude reasoning outputs for advanced thinking patterns.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/claude-haiku-4.5-high-reasoning-1700x" target="_blank">View on HuggingFace</a>
-                </div>
-                <div class="credit-card">
-                    <h4>Instruction Dataset 7</h4>
-                    <p class="credit-name">TeichAI/gemini-3-flash-preview</p>
-                    <p>Gemini model outputs for additional diversity.</p>
-                    <a href="https://huggingface.co/datasets/TeichAI/gemini-3-flash-preview" target="_blank">View on HuggingFace</a>
-                </div>
-            </div>
-        </div>
-    </section>
-    <!-- Closing -->
-    <section class="closing" id="closing">
-        <div class="container">
-            <div class="closing-content">
-                <h2>Beyond Competing with GPT-4</h2>
-                <p>The real goal is understanding what's actually necessary for intelligence to emerge. Maybe 100K parameters is enough. Maybe it isn't. But we won't know until we try building from first principles instead of just compressing what already exists.</p>
-                <div class="closing-quote">
-                    <blockquote>
-                        "Smaller models serve as a means toward a larger goal. Understanding what makes models work in the first place is the real objective."
-                    </blockquote>
-                </div>
-                <div class="poem-section">
-                    <p class="poem-intro">It's coming.</p>
-                    <div class="poem">
-                        <p>Small enough to hide in plain sight,</p>
-                        <p>Big enough to twist the stars of night,</p>
-                        <p>Quiet as a shadow, sharp as a spark,</p>
-                        <p>A tiny flame that will light the dark.</p>
-                    </div>
-                </div>
-                <div class="cta-section">
-                    <p>This is an ongoing experiment. Everything described here is subject to change. Follow along if you're curious about what this architecture can actually do.</p>
-                </div>
-            </div>
-        </div>
-    </section>
-    </main>
-    <!-- Footer -->
-    <footer class="footer">
-        <div class="container">
-            <div class="footer-content">
-                <p class="footer-text">Built with curiosity over compute.</p>
-                <p class="footer-subtext">FMN-GPT by <a href="https://huggingface.co/CompactAI" target="_blank">CompactAI</a> - 2026</p>
-            </div>
-        </div>
-    </footer>
-    <script src="main.js"></script>
-</body>
-</html>