Upload 2 files

index.html

@@ -612,22 +612,22 @@
                     <div class="feature-card">
                         <div class="feature-icon">M</div>
                         <h3>External Memory Module</h3>
-                        <p>384 memory slots that the model can read from and write to. Think of it as the model's diary. It writes stuff down so it doesn't have to remember. Revolutionary concept.</p>
+                        <p>A recurrent memory module with 32-dimensional memory vectors is baked into the architecture, but currently disabled during SFT training due to AOT autograd compatibility. The architecture supports it — the training pipeline just isn't cooperating yet.</p>
                     </div>
                     <div class="feature-card">
                         <div class="feature-icon">C</div>
                         <h3>Precision Codebook</h3>
-                        <p>A 32-dimensional codebook at the output head. Instead of predicting directly into a ~2.1K token vocabulary, the model projects down to learnable codes that get mapped to the full vocabulary. Think of it as a compression layer that helps with efficiency while still producing readable output.</p>
+                        <p>Tied weight embeddings with a learnable per-token output bias. Instead of a separate codebook projection, the model ties input embeddings to output weights and learns a bias vector to compensate for word-token suppression. Simple, parameter-efficient, and surprisingly effective.</p>
                     </div>
                     <div class="feature-card">
                         <div class="feature-icon">T</div>
                         <h3>Makeshift MTP</h3>
-                        <p>Multi-token prediction with 2 horizons. The model predicts further ahead during training, then branches during inference. It's like looking both ways before crossing, then deciding to jaywalk anyway.</p>
+                        <p>Multi-token prediction adapters with horizon 2 are wired into the architecture but currently run with weight 0.0. They're there for future experiments. Think of them as emergency exits that nobody's allowed to use yet.</p>
                     </div>
                     <div class="feature-card">
                         <div class="feature-icon">R</div>
                         <h3>RTX 5090 Optimized</h3>
-                        <p>Fully optimized for the 5090 with torch.compile, gradient checkpointing, and chunked attention. We use every trick in the book. And some from the sequel.</p>
+                        <p>Tuned for RTX 5090 with chunked sliding-window attention (1024 window, 256 chunk), bf16 mixed precision, and batch size 64. torch.compile and gradient checkpointing are available but disabled for the Haiku tier — stability over speed.</p>
                     </div>
                 </div>
             </div>
@@ -649,22 +649,22 @@
                     <div class="arch-arrow">↓</div>
                     <div class="arch-layer">
                         <div class="arch-box main">
-                            <span>Transformer Block x2</span>
-                            <small>With External Memory</small>
+                            <span>Transformer Block ×6</span>
+                            <small>Memory Module (Disabled)</small>
                         </div>
                     </div>
                     <div class="arch-arrow">↓</div>
                     <div class="arch-layer">
                         <div class="arch-box">
-                            <span>Codebook Head</span>
-                            <small>16-dim Quantized</small>
+                            <span>Tied Output Head</span>
+                            <small>Learnable Bias</small>
                         </div>
                     </div>
                     <div class="arch-arrow">↓</div>
                     <div class="arch-layer">
                         <div class="arch-box">
                             <span>Output</span>
-                            <small>~2.1K vocab</small>
+                            <small>~2.1K Hybrid Vocab</small>
                         </div>
                     </div>
                     <div class="arch-details">
@@ -681,8 +681,8 @@
                             <span class="arch-detail-value">256</span>
                         </div>
                         <div class="arch-detail">
-                            <span class="arch-detail-label">memory_slots</span>
-                            <span class="arch-detail-value">384</span>
+                            <span class="arch-detail-label">memory_dim</span>
+                            <span class="arch-detail-value">32</span>
                         </div>
                         <div class="arch-detail">
                             <span class="arch-detail-label">code_dim</span>
@@ -697,6 +697,62 @@
             </div>
         </section>
 
+        <section class="model-series" style="padding: 100px 0; background: var(--black); border-top: 1px solid var(--gray-2);">
+            <div class="container">
+                <div class="section-header">
+                    <h2>Model Series</h2>
+                    <p>Three tiers following Chinchilla scaling. Yes, we borrowed the naming scheme. No, we're not sorry.</p>
+                </div>
+                <div style="display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 24px; margin-top: 48px;">
+                    <div style="background: var(--gray-1); border: 1px solid var(--gray-2); border-radius: 12px; padding: 32px;">
+                        <div style="display: flex; align-items: center; gap: 12px; margin-bottom: 16px;">
+                            <span style="font-size: 32px; font-weight: 700; color: var(--accent);">Haiku</span>
+                            <span style="font-size: 13px; color: var(--gray-5); background: var(--gray-2); padding: 4px 10px; border-radius: 6px;">~1M params</span>
+                        </div>
+                        <p style="color: var(--gray-6); font-size: 15px; margin-bottom: 20px;">Lightweight and experimental. Updated frequently. The scrappy underdog.</p>
+                        <div style="display: grid; grid-template-columns: 1fr 1fr; gap: 8px; font-size: 13px; font-family: var(--font-mono);">
+                            <span style="color: var(--gray-5);">dim</span><span style="color: var(--gray-7);">160</span>
+                            <span style="color: var(--gray-5);">layers</span><span style="color: var(--gray-7);">6</span>
+                            <span style="color: var(--gray-5);">heads</span><span style="color: var(--gray-7);">4</span>
+                            <span style="color: var(--gray-5);">ffn_dim</span><span style="color: var(--gray-7);">256</span>
+                            <span style="color: var(--gray-5);">context</span><span style="color: var(--gray-7);">2,048</span>
+                            <span style="color: var(--gray-5);">pretrain tokens</span><span style="color: var(--gray-7);">~1B</span>
+                        </div>
+                    </div>
+                    <div style="background: var(--gray-1); border: 1px solid var(--gray-2); border-radius: 12px; padding: 32px;">
+                        <div style="display: flex; align-items: center; gap: 12px; margin-bottom: 16px;">
+                            <span style="font-size: 32px; font-weight: 700; color: var(--accent);">Sonnet</span>
+                            <span style="font-size: 13px; color: var(--gray-5); background: var(--gray-2); padding: 4px 10px; border-radius: 6px;">~300M params</span>
+                        </div>
+                        <p style="color: var(--gray-6); font-size: 15px; margin-bottom: 20px;">Balanced and stable. Updated less often. The responsible middle child.</p>
+                        <div style="display: grid; grid-template-columns: 1fr 1fr; gap: 8px; font-size: 13px; font-family: var(--font-mono);">
+                            <span style="color: var(--gray-5);">dim</span><span style="color: var(--gray-7);">768</span>
+                            <span style="color: var(--gray-5);">layers</span><span style="color: var(--gray-7);">36</span>
+                            <span style="color: var(--gray-5);">heads</span><span style="color: var(--gray-7);">12</span>
+                            <span style="color: var(--gray-5);">ffn_dim</span><span style="color: var(--gray-7);">2,560</span>
+                            <span style="color: var(--gray-5);">context</span><span style="color: var(--gray-7);">2,048</span>
+                            <span style="color: var(--gray-5);">pretrain tokens</span><span style="color: var(--gray-7);">~300B</span>
+                        </div>
+                    </div>
+                    <div style="background: var(--gray-1); border: 1px solid var(--gray-2); border-radius: 12px; padding: 32px;">
+                        <div style="display: flex; align-items: center; gap: 12px; margin-bottom: 16px;">
+                            <span style="font-size: 32px; font-weight: 700; color: var(--accent);">Opus</span>
+                            <span style="font-size: 13px; color: var(--gray-5); background: var(--gray-2); padding: 4px 10px; border-radius: 6px;">~600M params</span>
+                        </div>
+                        <p style="color: var(--gray-6); font-size: 15px; margin-bottom: 20px;">Maximum quality. Heavy and most stable. The overachiever who never sleeps.</p>
+                        <div style="display: grid; grid-template-columns: 1fr 1fr; gap: 8px; font-size: 13px; font-family: var(--font-mono);">
+                            <span style="color: var(--gray-5);">dim</span><span style="color: var(--gray-7);">1,024</span>
+                            <span style="color: var(--gray-5);">layers</span><span style="color: var(--gray-7);">39</span>
+                            <span style="color: var(--gray-5);">heads</span><span style="color: var(--gray-7);">16</span>
+                            <span style="color: var(--gray-5);">ffn_dim</span><span style="color: var(--gray-7);">3,584</span>
+                            <span style="color: var(--gray-5);">context</span><span style="color: var(--gray-7);">2,048</span>
+                            <span style="color: var(--gray-5);">pretrain tokens</span><span style="color: var(--gray-7);">~600B</span>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </section>
+
         <section class="demo">
             <div class="container">
                 <div class="section-header">

status.html

@@ -506,15 +506,15 @@
                         <div class="features-grid">
                             <div class="feature-item">
                                 <span class="feature-name">External Memory</span>
-                                <span class="feature-status enabled">Enabled</span>
+                                <span class="feature-status disabled">Disabled</span>
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Precision Codebook</span>
-                                <span class="feature-status enabled">Enabled</span>
+                                <span class="feature-status disabled">Disabled</span>
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Makeshift MTP</span>
-                                <span class="feature-status enabled">Enabled</span>
+                                <span class="feature-status disabled">Disabled (weight=0.0)</span>
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Gradient Checkpointing</span>
@@ -522,7 +522,7 @@
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Torch Compile</span>
-                                <span class="feature-status enabled">Enabled</span>
+                                <span class="feature-status disabled">Disabled</span>
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Chunked Attention</span>
@@ -530,27 +530,47 @@
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Flash Attention</span>
-                                <span class="feature-status enabled">Enabled</span>
+                                <span class="feature-status disabled">Not Used</span>
                             </div>
                             <div class="feature-item">
                                 <span class="feature-name">Repetition Penalty</span>
                                 <span class="feature-status disabled">Disabled (1.0)</span>
                             </div>
+                            <div class="feature-item">
+                                <span class="feature-name">Tied Embeddings</span>
+                                <span class="feature-status enabled">Enabled</span>
+                            </div>
+                            <div class="feature-item">
+                                <span class="feature-name">Output Logit Bias</span>
+                                <span class="feature-status enabled">Enabled</span>
+                            </div>
+                            <div class="feature-item">
+                                <span class="feature-name">Word Token Loss Boost</span>
+                                <span class="feature-status enabled">Enabled (3x)</span>
+                            </div>
+                            <div class="feature-item">
+                                <span class="feature-name">Response-Start Boost</span>
+                                <span class="feature-status enabled">Enabled (3x, 20 tokens)</span>
+                            </div>
+                            <div class="feature-item">
+                                <span class="feature-name">Entropy Regularization</span>
+                                <span class="feature-status disabled">Disabled</span>
+                            </div>
                         </div>
                     </div>
 
                     <!-- Memory Configuration -->
                     <div class="status-card">
                         <div class="status-header">
-                            <h3>Memory Module Config</h3>
+                            <h3>Memory Module Config (Disabled)</h3>
                         </div>
                         <div class="specs-grid">
                             <div class="spec-item">
-                                <div class="spec-value">384</div>
+                                <div class="spec-value">&mdash;</div>
                                 <div class="spec-label">Memory Slots</div>
                             </div>
                             <div class="spec-item">
-                                <div class="spec-value">64</div>
+                                <div class="spec-value">32</div>
                                 <div class="spec-label">Memory Dim</div>
                             </div>
                             <div class="spec-item">
@@ -571,11 +591,19 @@
                         </div>
                         <div class="dataset-list">
                             <div class="dataset-item">
-                                <span class="dataset-name">shuyuej/English-Pretraining-Dataset</span>
+                                <span class="dataset-name">HuggingFaceFW/fineweb_100BT</span>
                                 <span class="dataset-info">Pretraining</span>
                             </div>
                             <div class="dataset-item">
-                                <span class="dataset-name">imdatta0/openthink_chat</span>
+                                <span class="dataset-name">mattwesney/General_Inquiry_Thinking-Chain-Of-Thought</span>
+                                <span class="dataset-info">Instruction Tuning</span>
+                            </div>
+                            <div class="dataset-item">
+                                <span class="dataset-name">tatsu-lab/alpaca</span>
+                                <span class="dataset-info">Instruction Tuning</span>
+                            </div>
+                            <div class="dataset-item">
+                                <span class="dataset-name">databricks/databricks-dolly-15k</span>
                                 <span class="dataset-info">Instruction Tuning</span>
                             </div>
                             <div class="dataset-item">
@@ -598,7 +626,7 @@
                             <div class="log-entry">
                                 <span class="log-date">2026-03-01</span>
                                 <span class="log-status active">Running</span>
-                                <span class="log-message">Training on RTX 5090 with torch.compile enabled</span>
+                                <span class="log-message">Training on RTX 5090 with torch.compile disabled, chunked attention active</span>
                             </div>
                             <div class="log-entry">
                                 <span class="log-date">2026-02-28</span>