| window.annotations = { |
| "switch-c-scale": { |
| "page": 1, |
| "type": "Scale note", |
| "title": "Big model smell", |
| "rects": [ |
| { |
| "left": 59.03, |
| "top": 27.4, |
| "width": 25.73, |
| "height": 1.66 |
| } |
| ], |
| "html": "<p>Finally, we have a model comparable in scale to Google's <a href=\"https://huggingface.co/google/switch-c-2048\">legendary Switch-C</a>! It's been <a href=\"https://arxiv.org/abs/2101.03961\">5 years</a> since Noam Shazeer & Co. scaled MoEs to an unprecedented size, though that model only had about 1.8B active parameters.</p>\n<p>To put things in perspective, Google spent roughly <span class=\"math-inline\" data-tex=\"3.3 \\times 10^{21}\\ \\text{FLOPs}\">\\(3.3 \\times 10^{21}\\ \\text{FLOPs}\\)</span> (floating-point operations) on that pretraining run. If we do some back-of-the-envelope math for DeepSeek-V4 Pro, we get:</p>\n<div class=\"math-display\" data-tex=\"\\begin{aligned}\n\\mathrm{FLOPs}_{\\mathrm{V4\\ Pro}}\n&\\approx 6 \\times N_{\\mathrm{active}} \\times D_{\\mathrm{tokens}} \\\\\n&\\approx 6 \\times 49\\mathrm{B} \\times 33\\mathrm{T} \\\\\n&\\approx 9.7 \\times 10^{24}\\ \\text{FLOPs} \\\\\n&\\approx 1 \\times 10^{25}\\ \\text{FLOPs}\n\\end{aligned}\">\\[\\begin{aligned}\n\\mathrm{FLOPs}_{\\mathrm{V4\\ Pro}}\n&\\approx 6 \\times N_{\\mathrm{active}} \\times D_{\\mathrm{tokens}} \\\\\n&\\approx 6 \\times 49\\mathrm{B} \\times 33\\mathrm{T} \\\\\n&\\approx 9.7 \\times 10^{24}\\ \\text{FLOPs} \\\\\n&\\approx 1 \\times 10^{25}\\ \\text{FLOPs}\n\\end{aligned}\\]</div>\n<p>That's a 3,000x difference in compute!</p>" |
| }, |
| "v32-baseline": { |
| "page": 1, |
| "type": "Baseline note", |
| "title": "Comparison with DeepSeek-V3.2", |
| "rects": [ |
| { |
| "left": 79.55, |
| "top": 47.85, |
| "width": 8.74, |
| "height": 1.66 |
| }, |
| { |
| "left": 11.63, |
| "top": 49.55, |
| "width": 17.4, |
| "height": 1.66 |
| } |
| ], |
| "html": "<p>...which is already a strong baseline. In fact, it actually pioneered some of the attention tweaks we’re now seeing in DeepSeek-V4, and it was incredibly efficient to boot.</p>\n<p><a href=\"https://arxiv.org/pdf/2512.02556\">Below is a comparison</a> of the inference costs between V3.1 and V3.2 across various context lengths. (Quick refresher: prefilling is when the model computes the KV cache for the prompt, while decoding is the actual one-by-one token generation.)</p>\n<figure>\n<img src=\"content/notes/media/001-v32-baseline-inference-costs.png\" alt=\"Inference costs comparison\">\n</figure>" |
| }, |
| "modeling-capability": { |
| "page": 4, |
| "type": "Architecture note", |
| "title": "What counts as modeling capability", |
| "rects": [ |
| { |
| "left": 26.24, |
| "top": 58.27, |
| "width": 29.54, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>This might sound a bit abstract, so let's clarify. \"Modeling capability\" generally refers to a model's ability to represent complex functions and capture intricate patterns. You can also think of it as topological expressivity—how flexibly the model can route, store, and combine information across different layers.</p>\n<p>As numerous mechanistic interpretability papers have shown (<a href=\"https://transformer-circuits.pub/\">1</a>, <a href=\"https://arxiv.org/abs/2312.12141v1\">2</a>), information in a Transformer flows sequentially through a single residual stream. This is the main highway that layers read from and write to. Every single layer (Attention, MLP, MoE) has to read from this stream, process the data, and write it back.</p>\n<figure>\n<img src=\"content/notes/media/002-modeling-capability-residual-stream.png\" alt=\"Transformer residual stream illustration\">\n<figcaption>From <a href=\"https://transformer-circuits.pub/2021/framework/index.html\">Anthropic’s blogpost</a>.</figcaption>\n</figure>\n\n<p>(This follows directly from how residual connections work: <code>x' = x + layer(x)</code>, omitting normalizations for clarity. We just chain multiple additions to an ever-changing <code>x'</code>).</p>\n<p>The surefire way to boost modeling capability and expressivity is to widen this \"highway\" by increasing the hidden or embedding dimension. Intuitively, you're just creating more slots to carry information between layers. But there's a catch: the computational cost (FLOPs) of Attention/FFN/MoE layers scales quadratically with the hidden dimension. Widening the stream makes the model drastically slower and much heavier to compute.</p>\n<p>This is exactly where HyperConnections (HC) and its upgraded version, mHC, come in. They allow us to massively increase the residual stream's bandwidth without a huge compute overhead. They effectively introduce a \"virtual\" width to keep representations distinct and rich. By expanding the residual stream by a factor of <span class=\"math-inline\" data-tex=\"n\">\\(n\\)</span> (say, 4x wider), it can carry four times as much information forward without overwriting the previous data.</p>" |
| }, |
| "future-hardware-efficiency": { |
| "page": 5, |
| "type": "Hardware note", |
| "title": "How FP4-FP8 can be more efficient", |
| "rects": [ |
| { |
| "left": 20.54, |
| "top": 29.35, |
| "width": 67.98, |
| "height": 1.82 |
| } |
| ], |
| "html": "<p>So why is it 1/3 more efficient? Tensor core computations are bottlenecked by physical resources. Specifically, it's about how many operand bits can be staged, decoded, and fed into the compute pipeline per cycle.</p>\n<p>When the size of matrix elements shrinks, we can push more elements through the datapath in the same amount of time. But there's a catch: the compute blocks themselves must natively support this format.</p>\n<p>Here are Nvidia's official specs. You can clearly see how cutting the operand size in half doubles the potential compute:</p>\n<figure>\n<img src=\"content/notes/media/003-future-hardware-efficiency-nvidia-compute-specs.png\" alt=\"Nvidia compute specs\">\n</figure>\n\n<p>Under the hood, this speedup happens because GPU instructions can process larger matrices in a single pass, taking a fixed number of clock cycles.</p>\n<p>Check out this screenshot from a <a href=\"https://www.scribd.com/document/768738241/Benchmarking-and-Dissecting-the-Nvidia-Hopper-GPU-Architecture\">paper</a> benchmarking different instructions on the H800:</p>\n<figure>\n<img src=\"content/notes/media/003-future-hardware-efficiency-h800-instructions.png\" alt=\"H800 instructions benchmark\">\n</figure>\n\n<p>Notice how the instruction names include the matrix chunk sizes <code>m-n-k</code>. For instance, <code>m64n256k16</code> (the first row for 16-bit formats) means multiplying a <code>64x16</code> matrix by a <code>16x256</code> matrix.</p>\n<p>Now look at the 8-bit FP8 format in the fourth row. It uses <code>m64n256k32</code>, which means multiplying <code>64x32</code> by <code>32x256</code>. That's the maximum possible size for this GPU type; larger matrices just get split into chunks.</p>\n<p>The inner dimension K has doubled, but the instruction still takes exactly 128 clock cycles to execute. So, by dropping down to 8 bits, the GPU manages to multiply twice as many elements in the same number of cycles.</p>\n<p>This brings us to an interesting conclusion. If a GPU natively supports FP4 x FP8, it can process more elements per instruction than FP8 x FP8. This gives us a theoretical speedup of <span class=\"math-inline\" data-tex=\"(8+8)/(4+8) = 16/12 = 1.33\\times\">\\((8+8)/(4+8) = 16/12 = 1.33\\times\\)</span>.</p>\n<p>But without this hardware support, developers are stuck calling instructions that perform exactly like standard FP8 x FP8.</p>\n<p>The current generation of Blackwell GPUs doesn't actually support this FP4 x FP8 efficiency boost. Sure, they allow multiplying any combination of <code>{f4,f6,f8} × {f4,f6,f8}</code>, but the throughput stays exactly the same. Doing <code>f4 x f8</code> is exactly as fast as <code>f8 x f8</code>. Yes, it's twice as fast as Hopper (see the blue box below), but that's it:</p>\n<figure>\n<img src=\"content/notes/media/003-future-hardware-efficiency-blackwell-specs.png\" alt=\"Blackwell GPU specifications\">\n</figure>\n\n<p>That 2x boost is just a baseline architectural improvement in the GPU, which you can also see for FP16 and TF32 (in the red box). A true 2x throughput jump only happens if you go full <code>f4 x f4</code> (in the green box).</p>\n<p>Because of this, getting that 1/3 extra efficiency out of the box just isn't happening right now. DeepSeek will either have to transition to <code>f4 x f4</code> compute, or we'll have to wait for the next generation of hardware—like Nvidia's Vera Rubin or, more likely, upcoming Huawei chips.</p>" |
| }, |
| "transformer-of-theseus": { |
| "page": 6, |
| "type": "Architecture note", |
| "title": "Transformer of Theseus", |
| "rects": [ |
| { |
| "left": 34.55, |
| "top": 83.32, |
| "width": 45.1, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>It's pretty funny that we still call these models \"Transformers\" in reference to the original 2017 architecture, considering we've changed almost everything since then.</p>\n<p>The attention mechanism is completely different now (the paper we're looking at actually uses two types!). We ditched standard FFNs for MoEs—which have evolved several times themselves—swapped out positional embeddings, and moved from LayerNorm to RMSNorm.</p>\n<p>Even the loss function has evolved to mix in multi-token prediction task. And all of this doesn't even mention the massive paradigm shift from the original encoder-decoder setup to decoder-only LLMs.</p>\n<p>Throw in the fact that the legendary Adam optimizer was replaced by AdamW, and now we're moving toward Muon.</p>\n<p>It’s the Transformer of Theseus. If you replace every original component over time, is it still a Transformer? I'll leave that as an exercise for the reader.</p>" |
| }, |
| "mtp-objective": { |
| "page": 7, |
| "type": "Training note", |
| "title": "Multi-Token Prediction", |
| "rects": [ |
| { |
| "left": 72.83, |
| "top": 53.13, |
| "width": 15.47, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 54.74, |
| "width": 44.18, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Multi-Token Prediction (MTP) does exactly what it sounds like: it predicts multiple future tokens at every position. In V4, we’re only predicting one extra token, but why bother?</p>\n<p>First, an MTP objective provides a denser training signal, which improves data efficiency. It also forces the model to \"think ahead\" and optimize its internal representations for future tokens.</p>\n<p>As shown in the diagram from the <a href=\"https://arxiv.org/abs/2412.19437\">original DeepSeek-V3 paper</a>, MTP predicts these extra tokens sequentially while maintaining the full causal chain at every step.</p>\n<figure>\n<img src=\"content/notes/media/005-mtp-objective-sequential-prediction.png\" alt=\"MTP sequential prediction illustration\">\n</figure>\n\n<p>Architecturally, each MTP module shares its input and output embeddings with the main model, but maintains its own full transformer block. This block includes an attention layer and—while the paper doesn't explicitly confirm it—likely an MoE layer (though it could just be a standard FFN).</p>\n<p>With MTP, we compute a standard cross-entropy loss. This acts as an auxiliary loss with some weight (likely somewhere between 0.1 and 0.3, if we go by the V3 paper).</p>\n<p>For training, MTP uses a standard cross-entropy loss. This functions as an auxiliary loss, with a weighting factor likely somewhere between 0.1 and 0.3, if we go by the V3 paper.</p>" |
| }, |
| "aux-loss-free-routing": { |
| "page": 7, |
| "type": "MoE note", |
| "title": "Auxiliary-loss-free expert routing", |
| "rects": [ |
| { |
| "left": 56.2, |
| "top": 36.05, |
| "width": 32.1, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 37.66, |
| "width": 6.89, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>When we train MoE models, token routing often suffers from \"routing collapse\". This happens when the network stubbornly sends most tokens to just a few experts. This completely destroys efficiency and creates massive GPU bottlenecks, leaving most of your hardware sitting idle. Historically, the standard fix has been an auxiliary loss—a penalty that forces the model to distribute tokens evenly.</p>\n<p>But this penalty acts as a regularization term that alters the gradients, which conflicts with the primary language modeling objective. As <a href=\"https://arxiv.org/abs/2408.15664\">shown here</a>, if you make the penalty strong enough to successfully balance the load, it degrades the model's overall performance.</p>\n<p>To escape this trade-off, the auxiliary-loss-free strategy shifts the balancing act away from the loss function and directly into the routing mechanism. It does this by applying a dynamic, expert-wise bias (just one scalar value per expert, per MoE layer).</p>\n<p>If an expert receives more than its fair share of tokens (check out the outer columns in the image below), its bias is <strong>decreased</strong> to suppress selection. If an expert is underutilized, its bias gets <strong>bumped up</strong>. By iteratively adjusting these biases based on batch statistics, we achieve a perfectly balanced expert load without compromising the training objective.</p>\n<figure>\n<img src=\"content/notes/media/006-aux-loss-free-routing-dynamic-bias.png\" alt=\"Dynamic bias and load balancing in MoE layers\">\n</figure>\n\n<p>The cool part is that this dynamic bias is only used to make the routing decisions. Once the top experts are chosen, the bias is stripped away. We then use the original router scores to weight the experts' actual outputs.</p>" |
| }, |
| "hash-routed-initial-moe": { |
| "page": 7, |
| "type": "MoE note", |
| "title": "Hash routing in early layers", |
| "rects": [ |
| { |
| "left": 79.27, |
| "top": 42.49, |
| "width": 9.03, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 44.1, |
| "width": 76.67, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 45.71, |
| "width": 11.38, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>In almost all recent MoE models, the first one to three layers are standard, dense FFNs. We do this out of pure necessity. If you place learned MoE routing in the earliest layers, you often run into \"routing collapse\"—the model just dumps almost all tokens onto one or two experts, which causes severe training instability.</p>\n<p>Why does this happen? The earliest layers of a Transformer act as feature extractors for universal, low-level traits like morphology and spelling. This happens before a token representation even absorbs its surrounding context. Since these hidden states are completely uncontextualized, standard MoE routers struggle to learn any meaningful, dynamic mappings for expert specialization.</p>\n<p>Dense FFNs solve this by forcing all tokens through the exact same weights. This builds a shared representational foundation. It's much more efficient than increasing the hidden size, which would quadratically increase compute across all the later layers. It's also better than expanding the vocabulary size. Massive vocabularies already suffer from a long tail of rare, undertrained tokens (if you haven't read the famous <a href=\"https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldmagikarp-plus-prompt-generation\">SolidGoldMagikarp</a> story, it highlights exactly this issue).</p>\n<p>Furthermore, even if training stabilizes, the absolute best an early-layer learned router can do is memorize a static mapping. It essentially learns a hardcoded rule like, \"Always send Token ID 405 to Expert 3.\" HashMoE simply leans into this by stripping out the learned router entirely and replacing it with a deterministic, uniform hash function.</p>" |
| }, |
| "sqrt-softplus-routing": { |
| "page": 7, |
| "type": "MoE note", |
| "title": "Sqrt-Softplus for router scores", |
| "rects": [ |
| { |
| "left": 28.21, |
| "top": 34.44, |
| "width": 60.08, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 35.88, |
| "width": 27.44, |
| "height": 1.82 |
| } |
| ], |
| "html": "<p>After selecting and computing the outputs for K experts in an MoE layer, we calculate their weighted sum. It's helpful to break this down into three distinct concepts:</p>\n<ol>\n<li>Router logits — the raw outputs of the router linear layer.</li>\n<li>Affinity scores — the positive scores you get after applying an activation function like <span class=\"math-inline\" data-tex=\"Softmax\">\\(Softmax\\)</span>.</li>\n<li>Routing weights — the selected top-K scores after normalization, which are actually used to mix the expert outputs.</li>\n</ol>\n<p>Classic MoE routers typically relied on <span class=\"math-inline\" data-tex=\"Softmax\">\\(Softmax\\)</span>. This ensures the selected expert weights are non-negative and normalized into a probability-like distribution. But <span class=\"math-inline\" data-tex=\"Softmax\">\\(Softmax\\)</span> can get way too sharp because of the exponential. For example, if the router logits for 4 experts are <code>[4, 4, 4, 8]</code>, the last expert grabs about 95% of the probability mass. The first three experts are left fighting over the remaining 5%. This leads to a harsh winner-take-most dynamic.</p>\n<p>DeepSeek-V3 tackled this by moving to <span class=\"math-inline\" data-tex=\"Sigmoid\">\\(Sigmoid\\)</span>-based affinity scores. Unlike <span class=\"math-inline\" data-tex=\"Softmax\">\\(Softmax\\)</span>, <span class=\"math-inline\" data-tex=\"Sigmoid\">\\(Sigmoid\\)</span> scores each expert independently. If one expert gets a massive score, it doesn't automatically suppress the others.</p>\n<p>But <span class=\"math-inline\" data-tex=\"Sigmoid\">\\(Sigmoid\\)</span> has its own fatal flaw: it saturates. Large positive logits all squash to roughly 1, and large negative logits drop to roughly 0. Gradients vanish on both ends. So, if two experts have logits of 4 and 8, <span class=\"math-inline\" data-tex=\"Sigmoid\">\\(Sigmoid\\)</span> just says, \"Yep, they're both basically 1.\" You lose critical information about how strongly that token actually matches the expert.</p>\n<p>This <span class=\"math-inline\" data-tex=\"Sigmoid\">\\(Sigmoid\\)</span>-style routing quickly became the standard for DeepSeek-like MoE models, including Kimi K2.x, GLM-5.x, and MiniMax-M2.x. However, with DeepSeek-V4, the team shifted gears again. They dropped <span class=\"math-inline\" data-tex=\"Sigmoid\">\\(Sigmoid\\)</span> for <span class=\"math-inline\" data-tex=\"Sqrt(Softplus)\">\\(Sqrt(Softplus)\\)</span>.</p>\n<p><span class=\"math-inline\" data-tex=\"Softplus\">\\(Softplus\\)</span> behaves a lot like a smooth <span class=\"math-inline\" data-tex=\"ReLU\">\\(ReLU\\)</span>: <span class=\"math-inline\" data-tex=\"Softplus(z)=\\log(1+e^z)\">\\(Softplus(z)=\\log(1+e^z)\\)</span>:</p>\n<figure>\n<img src=\"content/notes/media/008-sqrt-softplus-routing-softplus-graph.png\" alt=\"Softplus activation function graph\">\n</figure>\n\n<p>For highly negative values, it hugs zero. But for large positive values, it keeps growing roughly linearly instead of flattening out. This means positive router logits finally preserve their magnitude. A logit of 8 is now meaningfully stronger than a logit of 4.</p>\n<p>Still, plain <span class=\"math-inline\" data-tex=\"Softplus\">\\(Softplus\\)</span> can be a bit too aggressive. If one selected expert has a massive positive logit, it can easily dominate the normalized mixture. Taking the square root beautifully solves this by compressing the dynamic range. <span class=\"math-inline\" data-tex=\"Sqrt(Softplus)\">\\(Sqrt(Softplus)\\)</span> hits the perfect sweet spot.</p>\n<p>Let's check out the MoE <a href=\"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/inference/model.py#L546\">code</a>:</p>\n<pre><code class=\"language-python\"># Compute router logits\nscores = router(x)\n# Convert logits into positive affinity scores\nscores = F.softplus(scores).sqrt()\n# Select top-K experts by affinity\nindices = scores.topk(self.topk, dim=-1)[1]\nweights = scores.gather(1, indices)\n# Normalize relative weights among selected experts\nweights /= weights.sum(dim=-1, keepdim=True)\n# What's this?\nweights *= self.route_scale\n</code></pre>\n<p>We can see that the <code>weights</code> are normalized at the end so they sum to one. But why multiply them by <code>self.route_scale</code> right after?</p>\n<p>Imagine the selected top-6 experts output vectors with similar magnitudes, but they point in different directions. Not literally opposite, just capturing different features. When we average those vectors together, their coordinates partially cancel out. As a result, the final vector can end up much smaller than a single expert's output.</p>\n<p>RMS (root mean square) is just a handy way to measure this. It basically means the \"typical per-coordinate size\" of a vector. In the toy simulation below, averaging 6 somewhat independent expert outputs gives us a vector with an RMS of only <span class=\"math-inline\" data-tex=\"1/\\sqrt{6} \\approx 0.41\">\\(1/\\sqrt{6} \\approx 0.41\\)</span> compared to that of a single expert. Without any correction, the routed MoE branch would just become too weak.</p>\n<p>DeepSeek-V4 Pro uses top-6 routing and sets <code>route_scale = 2.5</code>. Multiplying by 2.5 scales that uniform top-6 average back up to roughly the scale of a single expert:</p>\n<pre><code class=\"language-python\">import torch\ntorch.manual_seed(0)\n\nexpert_out = torch.randn(4096, 6, 1024) # [tokens, top6 experts, hidden_dim]\nweights = torch.ones(6) / 6 # uniform top-6 routing\n\noriginal_rms = expert_out.pow(2).mean(-1).sqrt().mean()\n\nfor route_scale in [1.0, 2.5]:\n agg_out = route_scale * (expert_out * weights[None, :, None]).sum(dim=1)\n agg_rms = agg_out.pow(2).mean(-1).sqrt().mean()\n print(f"route_scale={route_scale}: aggregated/original RMS = {agg_rms / original_rms:.4f}")\n# route_scale=1.0: aggregated/original RMS = 0.4083\n# route_scale=2.5: aggregated/original RMS = 1.0209\n</code></pre>\n<p>For instance, Kimi K2.5 <a href=\"https://huggingface.co/moonshotai/Kimi-K2.6/blob/main/config.json#L145\">uses</a> top-8 routing and a scale of 2.827, which is almost exactly <span class=\"math-inline\" data-tex=\"\\sqrt{8}\">\\(\\sqrt{8}\\)</span>. It's not a strict rule, though.</p>" |
| }, |
| "hyperconnection-stream-expansion": { |
| "page": 7, |
| "type": "Architecture note", |
| "title": "How Hyper-Connections expand", |
| "rects": [ |
| { |
| "left": 38.97, |
| "top": 73.52, |
| "width": 49.33, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 75.13, |
| "width": 9.04, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>You might assume this means adding <span class=\"math-inline\" data-tex=\"n_{hc}\">\\(n_{hc}\\)</span> embedding layers at the start to pass multiple distinct embeddings down the residual stream. Surprisingly, that’s not the case. For each token, the embedding is simply repeated multiple times:</p>\n<pre><code class=\"language-python\">def forward(self, input_ids: torch.Tensor, start_pos: int = 0):\n h = self.embed(input_ids)\n # Expand to hc_mult copies for Hyper-Connections using repetition\n h = h.unsqueeze(2).repeat(1, 1, self.hc_mult, 1)\n # Traverse transformer blocks\n for layer in self.layers:\n h = layer(h, start_pos, input_ids)\n # Output LMHead\n logits = self.head(h, self.hc_head_fn, self.hc_head_scale, self.hc_head_base, self.norm)\n return logits\n</code></pre>\n<p>Under the hood, each <code>layer</code> contains the projection matrices we’ll discuss later. We need these to mix the <span class=\"math-inline\" data-tex=\"n_{hc}\">\\(n_{hc}\\)</span> channels into one before running the core operation (like Attention or MoE), and to separate them back out afterward. </p>\n<p>In practice, this means that while identical embeddings enter the first transformer layer across all <span class=\"math-inline\" data-tex=\"n_{hc}\">\\(n_{hc}\\)</span> streams, they will look different by the time they exit.</p>" |
| }, |
| "hyperconnection-equation-1": { |
| "page": 7, |
| "type": "Architecture note", |
| "title": "Hyper-Connections and Equations 1, 3, 5 explained", |
| "rects": [ |
| { |
| "left": 39.8, |
| "top": 83.65, |
| "width": 49.2, |
| "height": 2.95 |
| } |
| ], |
| "html": "<p>This equation, along with equations (3)-(5) below, might look incredibly dense and unintuitive. The key to understanding them is Figure 2 from the paper:</p>\n<figure>\n<img src=\"content/notes/media/010-hyperconnection-equation-1-figure-2.png\" alt=\"Figure 2 from the paper explaining HyperConnections\">\n</figure>\n\n<p>The transformer block now takes in 4 embeddings per token instead of just 1 (since <span class=\"math-inline\" data-tex=\"n_{hc} = 4\">\\(n_{hc} = 4\\)</span> for the DeepSeek-V4 Pro architecture). However, the core layers (DeepSeekMoE / CSA / HCA) still only process a single embedding. To bridge this gap, they added two components:</p>\n<ul>\n<li><strong>Pre-Block Mixing</strong> (bottom center). This is the input mapping <span class=\"math-inline\" data-tex=\"A_l\">\\(A_l\\)</span> in the formula. We flatten the 4 embeddings into a single vector and pass it through a linear layer with an output dimension of 4. These 4 numbers act as weights for a weighted sum of the input embeddings. The result is a single embedding that captures all 4 \"meanings\" in the right proportions. This single embedding is what actually goes into the core layer.</li>\n<li><strong>Post-Block Mixing</strong>. This corresponds to the output mapping <span class=\"math-inline\" data-tex=\"C_l\">\\(C_l\\)</span> in the formula. It does the reverse, distributing that single embedding back into the 4 residual streams. Importantly, we do not split the layer output into four different vectors. Instead, we broadcast the same layer update into the four hyper-connection lanes, just with different learned amplitudes. For instance, we might write <code>1.1 * u</code> to one residual stream and <code>0.3 * u</code> to another, where <span class=\"math-inline\" data-tex=\"u\">\\(u\\)</span> is the core layer's output.<ul>\n<li>It might sound counterintuitive, but the weights for this step are derived by concatenating the exact same 4 embeddings used in Pre-Block Mixing. The linear layer here has identical dimensions: <span class=\"math-inline\" data-tex=\"4d\">\\(4d\\)</span> inputs to <span class=\"math-inline\" data-tex=\"4\">\\(4\\)</span> outputs.</li>\n</ul>\n</li>\n</ul>\n<p>TL;DR: Four embeddings are compressed into a single embedding of the same size. This vector passes through the MoE/Attention layer with zero compute overhead and gets broadcast back into the 4 streams. In the equation, this is represented as <span class=\"math-inline\" data-tex=\"C_lF_l(A_lX_l)\">\\(C_lF_l(A_lX_l)\\)</span> instead of the standard <span class=\"math-inline\" data-tex=\"F_l(X_l)\">\\(F_l(X_l)\\)</span>. After that, the result is added back to the residual streams. Because the Post-Block Mixing weights are predicted <em>before</em> the core layer, they are completely independent of its output. The model decides in advance exactly what information it wants to write and to which streams.</p>\n<p>This diagram should help you get a better feel for pre- and post-mixing.</p>\n<figure>\n<img src=\"content/notes/media/010-hyperconnection-equation-1-pre-post-mixing.png\" alt=\"Pre- and post-mixing diagram\">\n<figcaption>Normalizations, biases, and scales are omitted for clarity.</figcaption>\n</figure>\n\n<p>The bottom arrow labeled \"Residual connection\" simply computes the standard transformer update: <span class=\"math-inline\" data-tex=\"x' = x + f(x)\">\\(x' = x + f(x)\\)</span>. But in HyperConnections (and mHC), there's more going on. Besides the pre- and post-mixing shown above, there's also <strong>Residual mixing</strong>. The concept is similar: it blends the 4 residual streams together.</p>\n<p>To do this, the model predicts 16 weights instead of 4. These are reshaped into a 4x4 matrix. Each row essentially acts like pre-mixing, defining how each stream influences all the others, like this:</p>\n<figure>\n<img src=\"content/notes/media/010-hyperconnection-equation-1-residual-matrix.png\" alt=\"4x4 residual mapping matrix visualization\">\n</figure>\n\n<p>ChatGPT visualizes the flow of these residual streams like this:</p>\n<figure>\n<img src=\"content/notes/media/010-hyperconnection-equation-1-stream-flow.png\" alt=\"Residual stream flow visualization\">\n<figcaption>Every residual stream reads from and writes to all streams.</figcaption>\n</figure>" |
| }, |
| "birkhoff-polytope": { |
| "page": 8, |
| "type": "Modeling note", |
| "title": "TF is Birkhoff polytope?", |
| "rects": [ |
| { |
| "left": 33.88, |
| "top": 23.99, |
| "width": 54.6, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 25.43, |
| "width": 2.95, |
| "height": 1.82 |
| } |
| ], |
| "html": "<p>\"What the hell is that?\" you might ask. I hate it when simple concepts get slapped with overly complex names. As you'll see, it's actually pretty straightforward.</p>\n<p>In this context, the residual mapping matrix <span class=\"math-inline\" data-tex=\"B_l\">\\(B_l\\)</span> is exactly that 4x4 matrix I mentioned earlier. It simply defines the weights used to combine the embeddings across the different residual streams.</p>\n<p>Researchers use all this fancy jargon to say one basic thing: let's just normalize the matrix so that every row and column sums to exactly one, and all elements stay non-negative. Matrices that follow this rule are called \"doubly stochastic.\" The mathematical set of all such matrices is known as the \"Birkhoff Polytope.\"</p>\n<p>This is the core difference between mHC and standard HC. Even before V4, researchers noticed that the 4x4 matrices controlling the residual stream mixing tended to break down. The deeper you go into the network, the crazier the absolute values get:</p>\n<figure>\n<img src=\"content/notes/media/011-birkhoff-polytope-layer-matrices.png\" alt=\"Matrices in layers 1, 30, and 60 respectively.\">\n<figcaption>Matrices in layers 1, 30, and 60 respectively.</figcaption>\n</figure>\n\n<p>In the top half of the image, you can see the <span class=\"math-inline\" data-tex=\"B_l\">\\(B_l\\)</span> matrices in a trained network without normalization. The row and column sums are printed along the left and bottom edges of each matrix. As you can see, those sums just blow up.</p>\n<p>If we combine the streams using these raw, unnormalized weights, the vector magnitudes become completely unbounded. Instead of staying stable, they can jump by factors of tens or hundreds. That's a total nightmare for optimization and causes serious training instability.</p>\n<p>But look at the bottom half of the image. mHC normalizes these matrices so these sums stay close to one. It's not always perfect, but hey, it works!</p>" |
| }, |
| "non-expansive-residuals": { |
| "page": 8, |
| "type": "Stability note", |
| "title": "Why constrain residual mixing in mHC?", |
| "rects": [ |
| { |
| "left": 11.65, |
| "top": 29.9, |
| "width": 76.65, |
| "height": 2.0 |
| }, |
| { |
| "left": 11.7, |
| "top": 31.76, |
| "width": 36.02, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Recall that whenever you multiply an embedding <span class=\"math-inline\" data-tex=\"x\">\\(x\\)</span> by a weight matrix <span class=\"math-inline\" data-tex=\"W\">\\(W\\)</span>, you're applying a linear transformation. Geometrically, this transformation takes the input space and rotates, squishes, and stretches it.</p>\n<p>The spectral norm of a matrix is simply its maximum stretch factor. Since the paper bounds this norm to 1, the matrix's maximum stretch factor is at most 1.0. As a result, no matter what vector you multiply by this matrix, the output will never be longer than the input.</p>\n<p>Why does this matter? In mHC, we repeatedly apply mixing transformations in the residual stream. If any of these matrices have a spectral norm greater than 1, certain directions in the activation space will keep getting amplified. Even a 3% expansion at each layer yields an amplification of <span class=\"math-inline\" data-tex=\"1.03^{60} \\approx 5.9\">\\(1.03^{60} \\approx 5.9\\)</span> (assuming a 60-layer network).</p>\n<p>By constraining <span class=\"math-inline\" data-tex=\"B\">\\(B\\)</span> to be doubly stochastic, mHC makes the residual mapping behave like a mass-preserving mixer. It can blend information across the expanded residual lanes, but it can't arbitrarily amplify the signal.</p>\n<div class=\"annotation-media-pair\">\n<p><a href=\"https://arxiv.org/abs/2512.24880\">The mHC paper</a> visualizes this directly, showing the products of residual mixing matrices across 60 layers for both the unconstrained HC setup and the constrained mHC setup. In the HC case, the composite mapping develops extreme positive and negative values. This means some paths through the residual stream strongly amplify the signal or gradients—exactly the kind of behavior that destabilizes optimization.</p>\n<figure>\n<img src=\"content/notes/media/012-non-expansive-residuals-products.png\" alt=\"Products of residual mixing matrices across 60 layers\">\n</figure>\n</div>\n\n<p>In the mHC case, the composite matrix remains well-behaved. Its entries are non-negative, and its row and column sums stay close to 1. They aren't exactly 1 because the underlying normalization algorithm is iterative, meaning the doubly stochastic constraint is only approximate. Crucially, however, the gains remain bounded instead of exploding.</p>" |
| }, |
| "sinkhorn-projection": { |
| "page": 8, |
| "type": "Optimization note", |
| "title": "The Birkhoff polytope", |
| "rects": [ |
| { |
| "left": 39.87, |
| "top": 78.42, |
| "width": 48.74, |
| "height": 1.82 |
| } |
| ], |
| "html": "<p>As mentioned earlier, this complicated phrasing just means that we normalize the matrix so that each row and column sums to 1, and all its elements are non-negative.</p>" |
| }, |
| "dual-kv-entry-series": { |
| "page": 9, |
| "type": "Attention note", |
| "title": "Two KV-entry series", |
| "rects": [ |
| { |
| "left": 68.39, |
| "top": 82.11, |
| "width": 19.9, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>One series, <span class=\"math-inline\" data-tex=\"C^b\">\\(C^b\\)</span>, contains features representing the \"current chunk,\" while the other, <span class=\"math-inline\" data-tex=\"C^a\">\\(C^a\\)</span>, represents the \"previous chunk.\" Effectively, we create two representations for each token. The first is used when the token is part of the current compression chunk and likely encodes its core meaning. The second is used when it acts as context, providing only supplementary detail.</p>" |
| }, |
| "positional-biases": { |
| "page": 10, |
| "type": "Attention note", |
| "title": "Learnable positional biases", |
| "rects": [ |
| { |
| "left": 80.28, |
| "top": 19.77, |
| "width": 8.01, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.65, |
| "top": 21.63, |
| "width": 13.74, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>These are just per-position biases. They allow the model to learn patterns like a decay effect within a chunk, where early tokens contribute less to the sum than later ones. Since we have <span class=\"math-inline\" data-tex=\"m\">\\(m\\)</span> tokens per chunk, we need <span class=\"math-inline\" data-tex=\"m\">\\(m\\)</span> of these bias vectors.</p>" |
| }, |
| "csa-formulas-11-12": { |
| "page": 10, |
| "type": "Attention note", |
| "title": "Equations 11-12 for CSA", |
| "rects": [ |
| { |
| "left": 19.75, |
| "top": 25.62, |
| "width": 68.9, |
| "height": 7.18 |
| } |
| ], |
| "html": "<p>Let's break down what's happening in these formulas in a bit more detail. We have three pairs of terms:</p>\n<ul>\n<li><span class=\"math-inline\" data-tex=\"C^a\">\\(C^a\\)</span> and <span class=\"math-inline\" data-tex=\"C^b\">\\(C^b\\)</span> — the token vectors we use to build the compressed chunk representation. As noted above, <span class=\"math-inline\" data-tex=\"a\">\\(a\\)</span> and <span class=\"math-inline\" data-tex=\"b\">\\(b\\)</span> denote the previous and current chunks. In CSA, each chunk has <span class=\"math-inline\" data-tex=\"m=4\">\\(m=4\\)</span> tokens (in HSA, it's <span class=\"math-inline\" data-tex=\"m=128\">\\(m=128\\)</span>, but the idea is exactly the same).<ul>\n<li>Intuitively, these vectors capture the actual meaning of the embedding.</li>\n</ul>\n</li>\n<li><span class=\"math-inline\" data-tex=\"Z^a\">\\(Z^a\\)</span> and <span class=\"math-inline\" data-tex=\"Z^b\">\\(Z^b\\)</span> — the local attention logits we use to weight different tokens during compression. Unlike standard Attention, where we compute one logit per token pair, here the logits are per-channel.<ul>\n<li>Intuitively, these vectors track the importance of the information inside each token.</li>\n</ul>\n</li>\n<li><span class=\"math-inline\" data-tex=\"B^a\">\\(B^a\\)</span> and <span class=\"math-inline\" data-tex=\"B^b\">\\(B^b\\)</span> — as I mentioned earlier, these are just per-position biases so the model can learn things like intra-chunk decay.<ul>\n<li>Intuitively, they downweight the importance of <span class=\"math-inline\" data-tex=\"Z^a\">\\(Z^a\\)</span> and <span class=\"math-inline\" data-tex=\"Z^b\">\\(Z^b\\)</span> depending on their position.</li>\n</ul>\n</li>\n</ul>\n<p>So, the argument passed to <span class=\"math-inline\" data-tex=\"Softmax_{row}\">\\(Softmax_{row}\\)</span> in the formula represents the importance of all <span class=\"math-inline\" data-tex=\"2m\">\\(2m\\)</span> tokens across both chunks (current and previous, as if they were concatenated), adjusted for their positions. The <span class=\"math-inline\" data-tex=\"Softmax_{row}\">\\(Softmax_{row}\\)</span> is then applied over the tokens separately for each channel. Again, this differs from a standard <span class=\"math-inline\" data-tex=\"Softmax\">\\(Softmax\\)</span>, which typically operates on scalars. Instead, we apply this operation for each hidden channel independently, across token positions in two consecutive chunks.</p>\n<p>As a result, <span class=\"math-inline\" data-tex=\"Softmax_{row}\">\\(Softmax_{row}\\)</span> outputs weighting scores <span class=\"math-inline\" data-tex=\"S^a\">\\(S^a\\)</span> and <span class=\"math-inline\" data-tex=\"S^b\">\\(S^b\\)</span>. We use these to weight <span class=\"math-inline\" data-tex=\"C^a\">\\(C^a\\)</span> and <span class=\"math-inline\" data-tex=\"C^b\">\\(C^b\\)</span> channel-by-channel, as shown in equation (12). It's essentially a smarter alternative to average pooling, where the tokens dictate their own weights.</p>\n<p>We decouple <span class=\"math-inline\" data-tex=\"C\">\\(C\\)</span> and <span class=\"math-inline\" data-tex=\"Z\">\\(Z\\)</span> to handle specific edge cases. Sometimes a token embedding channel is disproportionately large and would normally dominate the compressed representation in a standard average. Separating them ensures it won't contribute much if it's not actually relevant in that specific context.</p>\n<p>Ultimately, these formulas just describe a per-channel pooling operator over a local temporal window.</p>" |
| }, |
| "hadamard-product": { |
| "page": 10, |
| "type": "Math note", |
| "title": "The Hadamard product", |
| "rects": [ |
| { |
| "left": 25.98, |
| "top": 33.93, |
| "width": 20.02, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>The Hadamard product <span class=\"math-inline\" data-tex=\"\\odot\">\\(\\odot\\)</span> is simply channel-wise multiplication:</p>\n<figure>\n<img src=\"content/notes/media/017-hadamard-product-illustration.png\" alt=\"Hadamard product illustration\">\n</figure>\n\n<p>As I mentioned earlier, the weighting is applied per channel, rather than as a single scalar weight per token. That's exactly why we need the Hadamard product here.</p>" |
| }, |
| "grouped-output-projection": { |
| "page": 11, |
| "type": "Attention note", |
| "title": "The grouped projection trick", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 65.81, |
| "width": 50.03, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>From what I've gathered, this is the first time an approach like this has been used. You won't find it in any previous LLMs of this scale. You do occasionally see conceptually similar ideas though, such as in <a href=\"https://arxiv.org/abs/2603.08343\">Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers</a>.</p>" |
| }, |
| "large-cnh": { |
| "page": 11, |
| "type": "Scale note", |
| "title": "The \"quite large\" attention output", |
| "rects": [ |
| { |
| "left": 70.61, |
| "top": 64.2, |
| "width": 9.26, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>When the paper says \"<span class=\"math-inline\" data-tex=\"cn_h\">\\(cn_h\\)</span> is quite large\", they mean <span class=\"math-inline\" data-tex=\"c \\times n_h = 512 \\times 128 = 65536\">\\(c \\times n_h = 512 \\times 128 = 65536\\)</span>. So before the attention output projection, each token has a 65,536-dimensional concatenated attention output. Since the model's hidden size is <span class=\"math-inline\" data-tex=\"d = 7168\">\\(d = 7168\\)</span>, the output projection would require a Linear layer with nearly 470 million parameters—for every single layer.</p>\n<p>For reference, BERT-Large had 340M parameters in total. Oh, the brave new world of LLMs!</p>\n<p>That's why DeepSeek factorizes this massive matrix into several smaller ones. This trick cuts the parameter count per layer down to <span class=\"math-inline\" data-tex=\"16 \\times (4096 \\times 1024) + 16384 \\times 7168 \\approx 185\">\\(16 \\times (4096 \\times 1024) + 16384 \\times 7168 \\approx 185\\)</span>M parameters, which is a 2.55x reduction. Across 61 layers, you're looking at roughly 28.66B parameters for the naive approach versus 11.26B parameters with GOP. That saves 17.4B parameters just in the attention output projections alone.</p>\n<p>You might wonder why other models haven't used this trick. Well, most mainstream models have significantly smaller (compressed) per-head embedding sizes. For instance:</p>\n<ul>\n<li>In DeepSeek-V3.2, <span class=\"math-inline\" data-tex=\"c \\times n_h = 128 \\times 128 = 16384\">\\(c \\times n_h = 128 \\times 128 = 16384\\)</span>, and the <code>Linear(16384, 7168)</code> layer only has 117M parameters.</li>\n<li>For Kimi-2.5/2.6, <span class=\"math-inline\" data-tex=\"c \\times n_h = 128 \\times 64 = 8192\">\\(c \\times n_h = 128 \\times 64 = 8192\\)</span>, and the <code>Linear(8192, 7168)</code> layer has 59M parameters.</li>\n</ul>\n<p>So, this bottleneck only emerged because of the massive increase in embedding size. This is mainly because the sequence is compressed, with each chunk representation carrying the semantic information of multiple tokens at once.</p>" |
| }, |
| "partial-rope": { |
| "page": 13, |
| "type": "Attention note", |
| "title": "RoPE on 64 dimensions", |
| "rects": [ |
| { |
| "left": 17.04, |
| "top": 15.03, |
| "width": 34.26, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>The idea of applying RoPE to only a part of the embedding isn't exactly new. It surfaced recently in <a href=\"https://arc.net/l/quote/wvnvlywj\">Gemma 4</a>, for example. Let's dive into how and why this actually works.</p>\n<p>RoPE takes the Query and Key vectors and slices them up into pairs of values. A 64-dimensional vector has 32 of these pairs. We can look at each pair as a 2D vector pointing in a specific direction. RoPE applies a slight rotation to each pair, and angle of rotation decreases as we move through the embedding dimensions. As Maarten Grootendorst explains in his <a href=\"https://newsletter.maartengrootendorst.com/i/193064129/p-rope\">Gemma 4 breakdown</a>:</p>\n<figure>\n<img src=\"content/notes/media/020-partial-rope-frequency-ladder.png\" alt=\"image.png\">\n</figure>\n\n<p>This is known as the frequency. High-frequency pairs are extremely sensitive to small position changes because they undergo a large rotation. The low-frequency pairs, on the other hand, get a very slight rotation. They barely move at all from word to word.</p>\n<p>You could say the first pairs are high-frequency and handle short-range dependencies. The last pairs are low-frequency and manage long-range influence. Since they rotate so little over standard context lengths, relative position doesn't drastically change their dot-product contribution.</p>\n<p>But why use rotation in the first place? RoPE changes the relative angle between the Query and Key while preserving their norms. This directly affects their dot product, which is what we use to compute attention logits. Rotation lowers the dot product, which in turn lowers the attention scores. Here's a cool GIF from Lorenzo Cesconetto’s <a href=\"https://towardsdatascience.com/rope-clearly-explained/\">blog</a>:</p>\n<figure>\n<img src=\"content/notes/media/020-partial-rope-rotation-demo.gif\" alt=\"DeepSeek uses RMSNorm before core attention on both Qs and KVs, so calling it “cosine” is somehow justified.\">\n<figcaption>DeepSeek uses RMSNorm before core attention on both Qs and KVs, so calling it “cosine” is somehow justified.</figcaption>\n</figure>\n\n<p>The tempting intuition here is simple: the larger the relative distance, the more the rotation misaligns the Query and Key, so the attention score should decay. This holds up in a toy setup where Q and K are already aligned or nearly constant. But generally speaking, it's just not true! To quote the paper <a href=\"https://arxiv.org/abs/2410.06205\">Round and Round We Go! What makes Rotary Positional Encodings useful</a>:</p>\n<blockquote>\n<p>A common belief is that RoPE is useful because it helps to decay token dependency as relative distance increases. In this work, we argue that this is unlikely to be the core reason. We study the internals of a trained Gemma 7B model to understand how RoPE is being used at a mechanical level. We find that Gemma learns to use RoPE to construct robust \"positional\" attention patterns by exploiting the highest frequencies. We also find that, in general, Gemma greatly prefers to use the lowest frequencies of RoPE, which we suspect are used to carry semantic information.\n</p>\n<p><…> we carry out an ablation in which they truncate the very lowest frequencies of RoPE. The intuition is that <…> truncating the lowest frequencies should not harm performance. In fact, this allows RoPE to provide robust semantic channels that are distance agnostic. We call this modification p-RoPE, with <span class=\"math-inline\" data-tex=\"p\">\\(p\\)</span> being the fraction of RoPE “kept”.\n</p>\n</blockquote>\n<p>DeepSeek-V4 uses <strong>partial RoPE</strong>, not p-RoPE (and yes, they are actually different). The model reserves the last 64 dimensions of each 512-d attention head as a RoPE slice and computes a standard 64-d RoPE ladder directly within that slice. This differs from p-RoPE, which keeps the original full-head frequency scale but discards the lowest-frequency dimensions. To put it simply: DeepSeek reduces the rotary dimension, while p-RoPE truncates the frequency ladder.</p>\n<p>And, of course, you could just as easily use the first 64 dimensions instead of the last—it makes absolutely no difference.</p>\n<figure>\n<img src=\"content/notes/media/020-partial-rope-partial-vs-prope.png\" alt=\"image.png\">\n</figure>" |
| }, |
| "absolute-position-leak": { |
| "page": 13, |
| "type": "Attention note", |
| "title": "RoPE absolute-position leakage", |
| "rects": [ |
| { |
| "left": 55.58, |
| "top": 16.64, |
| "width": 32.95, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>To see why this happens, we need to dig into the math behind how RoPE actually works. In the previous note, we talked about slicing the Query and Key vectors into pairs of values, then rotating them by different angles.</p>\n<p>These angles are a deterministic function of the token's <strong>absolute</strong> position and the specific RoPE channel pair index. But wait, don't we want relative positional encoding? Yes! And that's exactly what we get when we multiply Q and K to compute the attention logits.</p>\n<p>To see why, let's recall two handy properties of rotation matrices:</p>\n<ol>\n<li>The transpose of a rotation matrix gives you the inverse rotation: <span class=\"math-inline\" data-tex=\"R_a^T = R_{-a}\">\\(R_a^T = R_{-a}\\)</span></li>\n<li>Multiplying two rotation matrices adds their angles together: <span class=\"math-inline\" data-tex=\"R_a R_b = R_{a+b}\">\\(R_a R_b = R_{a+b}\\)</span></li>\n</ol>\n<p>Now, imagine we're computing the attention logit between a query at position <span class=\"math-inline\" data-tex=\"t\">\\(t\\)</span> and a key at position <span class=\"math-inline\" data-tex=\"j\">\\(j\\)</span>. Without RoPE, that's just <span class=\"math-inline\" data-tex=\"attention_{logit}(t, j) = q_t^T k_j\">\\(attention_{logit}(t, j) = q_t^T k_j\\)</span>. Let's apply the RoPE rotation matrices to Q and K: <span class=\"math-inline\" data-tex=\"q_t\\rarr R_t q_t,\\space\\space k_j \\rarr R_j k_j\">\\(q_t\\rarr R_t q_t,\\space\\space k_j \\rarr R_j k_j\\)</span>. As mentioned earlier, these matrices depend <em>only</em> on their absolute positions in the sequence, <span class=\"math-inline\" data-tex=\"t\">\\(t\\)</span> and <span class=\"math-inline\" data-tex=\"j\">\\(j\\)</span>.</p>\n<p>So our new score becomes:</p>\n<div class=\"math-display\" data-tex=\"\\begin{aligned}\nscore(t, j)\n&= (R_t q_t)^T (R_j k_j) \\\\\n&= q_t^T R_t^T R_j k_j \\\\\n&= q_t^T R_{j-t} k_j\n\\end{aligned}\">\\[\\begin{aligned}\nscore(t, j)\n&= (R_t q_t)^T (R_j k_j) \\\\\n&= q_t^T R_t^T R_j k_j \\\\\n&= q_t^T R_{j-t} k_j\n\\end{aligned}\\]</div>\n<p>Boom! The extra term in our dot product no longer depends on <span class=\"math-inline\" data-tex=\"t\">\\(t\\)</span> and <span class=\"math-inline\" data-tex=\"j\">\\(j\\)</span> individually. It depends purely on their difference! <span class=\"math-inline\" data-tex=\"R_{j-t}\">\\(R_{j-t}\\)</span> will be exactly the same whether <span class=\"math-inline\" data-tex=\"j=995, \\space t=1000\">\\(j=995, \\space t=1000\\)</span> or <span class=\"math-inline\" data-tex=\"j=95,\\space t=100\">\\(j=95,\\space t=100\\)</span>. In both cases, the tokens are exactly 5 positions apart.</p>\n<p>Now, let's see how RoPE actually gets applied in the Attention layer. <a href=\"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/inference/model.py#L502\">Here</a> is the relevant part of the code:</p>\n<pre><code class=\"language-python\">kv = self.wkv(x)\nkv = self.kv_norm(kv)\napply_rotary_emb(kv[..., -rd:], freqs_cis)\n...\no = sparse_attn(q, kv, self.attn_sink, topk_idxs, self.softmax_scale)\n</code></pre>\n<p>After we call <code>apply_rotary_emb</code>, the last 64 dimensions of <code>kv</code> carry absolute positional info. Inside the attention block, we compute attention weights, which are then used to calculate a weighted sum of the values. But these values have also been RoPE-rotated based on their absolute positions. This happens because the KV vector serves as both the key and the value (which cuts our KV cache memory footprint in half).</p>\n<p>This side effect is exactly what the authors mean when they talk about absolute position embeddings leaking into the attention output (<code>o</code>).</p>" |
| }, |
| "supplementary-sliding-attention": { |
| "page": 13, |
| "type": "Attention note", |
| "title": "SWA + CSA/HCA = ?", |
| "rects": [ |
| { |
| "left": 23.08, |
| "top": 33.71, |
| "width": 65.22, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Here's a fun thought experiment: does this mean the exact same token can show up in the attention calculation twice? Like, once in the sliding window and again inside a compressed CSA/HCA chunk?</p>\n<p>Well, yes and no. The chunk doesn't contain the raw token itself; instead, its semantics are included into the chunk's overall representation.</p>\n<p>Let's walk through an example. Imagine our query token sits at position 200. We'll use the DeepSeek-V4 Pro hyperparameters: <code>window_size = 128</code> and <code>compress_ratio = 4</code> for CSA. The sliding window will cover tokens <code>73...200</code>. Meanwhile, compressed chunk #49 spans tokens <code>196...199</code>. Since the compression overlaps, it also pulls in information from the previous block, <code>192...195</code>.</p>\n<p>Now, suppose the indexer selects the two most recent chunks, #48 and #49. When that happens, the attention mechanism will simultaneously see:</p>\n<pre><code>...\nraw token 191\nraw token 192\n...\nraw token 198\nraw token 199\n...\ncompressed summary of tokens in chunk 192...199\ncompressed summary of tokens in chunk 188...195\n...\n</code></pre>\n<p>So, yes, the model can actually see the same piece of information twice or even three times (in our example, tokens <code>192...195</code>). It's just packaged in different formats.</p>" |
| }, |
| "attention-sink-formula": { |
| "page": 13, |
| "type": "Attention note", |
| "title": "The Attention-sink", |
| "rects": [ |
| { |
| "left": 11.55, |
| "top": 44.25, |
| "width": 76.95, |
| "height": 9.7 |
| } |
| ], |
| "html": "<p>The issue with these emergent attention sinks is that the attention mechanism can't just choose to look at nothing. It has to look somewhere. If all the Q-K pairs have the exact same score, the probability mass (which must sum to 1) gets smeared evenly across every single token.</p>\n<p>Ideally, an attention head wants to lock onto a specific interaction pattern between embeddings. But some of those patterns are rare and simply aren't present in a given sequence. So, LLMs learn a neat trick: they dump their \"excess\" attention mass onto the very first tokens. You can clearly see this happening from layer 2 onwards in the image below:</p>\n<figure>\n<img src=\"content/notes/media/023-attention-sink-formula-attention-maps.png\" alt=\"Attention maps showing sink tokens\">\n<figcaption>Image from the <a href=\"https://arxiv.org/abs/2309.17453\">paper</a></figcaption>\n</figure>\n\n<p>But here's where standard Sliding Window Attention (SWA) runs into trouble. As the window shifts forward, those initial tokens—the ones the model was using as an implicit sink—fall right out of the KV cache. You could fix this by constantly reprocessing the active window and recomputing its KVs, but that defeats the whole purpose of reusing the cache in the first place.</p>\n<figure>\n<img src=\"content/notes/media/023-attention-sink-formula-sliding-window.png\" alt=\"image.png\">\n</figure>\n\n<p>To solve this, we can simply add a dedicated sink term to the softmax denominator (though this is just one of several potential workarounds). This gives the attention head a clear \"none-of-the-above\" option. This way, any leftover attention mass safely drains into the sink instead of spilling over onto the active KV entries after the initial tokens drop out.</p>\n<p>Effectively, this formula creates a dummy token. It only participates in the softmax normalization and affects the final attention output as if its value vector consisted entirely of zeros.</p>" |
| }, |
| "fp4-indexer-attention": { |
| "page": 13, |
| "type": "Precision note", |
| "title": "FP4 inside the CSA indexer / attention", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 75.02, |
| "width": 76.82, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>I did some digging with GPT-5.5 Pro, and it looks like DeepSeek-V4 is probably the first public 100B+ LLM to use FP4 inside attention during training. Even in <a href=\"https://arxiv.org/abs/2604.12374\">Nvidia's most recent paper</a> on Nemotron 3 Super, where they advertise their GPU-accelerated NVFP4 format and its pre-training stability, they explicitly keep QKV and attention projections out of NVFP4 to \"maintain fidelity of few attention layers\":</p>\n<figure>\n<img src=\"content/notes/media/024-fp4-indexer-attention-nemotron-fp4.png\" alt=\"image.png\">\n</figure>\n\n<p>(Side note: keeping the final 15% of the network in higher precision is a really interesting trick. Duly noted 📝)</p>\n<p>It's worth noting that DeepSeek isn't doing full FP4 attention. They only use reduced precision in the indexer, which is responsible for picking candidate chunks. Why is this approach less insane than full FP4 attention? Full attention has several numerically nasty parts that can easily blow up:</p>\n<ul>\n<li>QK dot products can have heavy-tailed logits.</li>\n<li>Softmax is exponential and severely amplifies these logit errors.</li>\n<li>Small logit differences can wildly shift probability mass.</li>\n</ul>\n<p>DeepSeek-V4 sidesteps the worst of this. The indexer only needs high recall, not perfectly calibrated probabilities or exact rankings. The top-k is large enough (1024) that a little bit of score noise doesn't really matter. As long as the right block makes it into the candidate set, we're good, because it will get processed by higher-precision attention later anyway.</p>\n<p>That said, this is still a big deal.</p>" |
| }, |
| "kv-cache-two-percent": { |
| "page": 13, |
| "type": "Memory note", |
| "title": "Where ~2% KV cache claim comes from", |
| "rects": [ |
| { |
| "left": 14.08, |
| "top": 88.7, |
| "width": 53.23, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Let’s break down exactly which architectural tweaks drive this massive 50x reduction. Under the hood, the memory footprint actually shrinks by 200x, but each KV entry is now 4 times wider (512 dimensions instead of 128).</p>\n<p>Baseline KV per token per layer:</p>\n<div class=\"math-display\" data-tex=\"\\begin{aligned}\n2_{K,V} \\times 8_{KV\\space heads}\n\\times 128_{dim} \\times 2_{BF16\\space bytes}\n= 4096 \\space bytes\n\\end{aligned}\">\\[\\begin{aligned}\n2_{K,V} \\times 8_{KV\\space heads}\n\\times 128_{dim} \\times 2_{BF16\\space bytes}\n= 4096 \\space bytes\n\\end{aligned}\\]</div>\n<p>Here's a table summarizing how each change contributes to the final number:</p>\n<table>\n<thead>\n<tr>\n<th>Step</th>\n<th>Size change</th>\n<th>B/token/layer</th>\n<th>Cumulative reduction</th>\n<th>Intuition / notes</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Baseline: BF16 GQA8, K+V, 128-d</strong></td>\n<td>—</td>\n<td><strong>4096</strong></td>\n<td><strong>1×</strong></td>\n<td>starting point</td>\n</tr>\n<tr>\n<td><strong>1 KV head instead of 8</strong></td>\n<td>÷8</td>\n<td><strong>512</strong></td>\n<td><strong>8×</strong></td>\n<td><a href=\"https://arxiv.org/abs/2305.13245\">MQA-style cache</a>: one KV stream for all query heads</td>\n</tr>\n<tr>\n<td><strong>Shared KV instead of separate K and V</strong></td>\n<td>÷2</td>\n<td><strong>256</strong></td>\n<td><strong>16×</strong></td>\n<td>one vector acts as both key and value</td>\n</tr>\n<tr>\n<td><strong>Token-level compression: CSA/HCA</strong></td>\n<td>÷7.76</td>\n<td><strong>33</strong></td>\n<td><strong>124×</strong></td>\n<td>CSA stores 1 per 4 tokens; HCA stores 1 per 128 tokens</td>\n</tr>\n<tr>\n<td><strong>But V4 KV entry is 512-d, not 128-d</strong></td>\n<td>×4</td>\n<td><strong>132</strong></td>\n<td><strong>31×</strong></td>\n<td>each compressed entry is wider to summarize a token chunk</td>\n</tr>\n<tr>\n<td><strong>Mixed precision: 64 BF16 + 448 FP8 dims</strong></td>\n<td>÷~1.78</td>\n<td><strong>74.25</strong></td>\n<td><strong>55×</strong></td>\n<td></td>\n</tr>\n<tr>\n<td>(would be 220× without the 512-d expansion)</td>\n<td>512 BF16 dims would be 1024 B; V4 stores 576 B</td>\n<td></td>\n<td></td>\n<td></td>\n</tr>\n<tr>\n<td><strong>Small overheads (CSA indexer, etc)</strong></td>\n<td>+~8 B</td>\n<td><strong>~82 B</strong></td>\n<td><strong>~50×</strong></td>\n<td>aligns with the paper’s ≈2%</td>\n</tr>\n</tbody>\n</table>\n<p>The 7.76× effective token-compression factor comes from compression ratios for chunks of size 4 (CSA) and 128 (HCA):</p>\n<div class=\"math-display\" data-tex=\"\\frac{1}{\\frac{1}{2}\\left(\\frac{1}{4} + \\frac{1}{128}\\right)}\n= \\frac{1}{0.1289}\n\\approx 7.76\">\\[\\frac{1}{\\frac{1}{2}\\left(\\frac{1}{4} + \\frac{1}{128}\\right)}\n= \\frac{1}{0.1289}\n\\approx 7.76\\]</div>\n<p>assuming a 50/50 mix.</p>\n<p>Looking at the table, the two biggest contributors are MQA and token-level chunk compression. While MQA is standard practice these days, this chunk compression is the real architectural gem of DeepSeek-V4.</p>" |
| }, |
| "muon-optimizer": { |
| "page": 14, |
| "type": "Optimizer note", |
| "title": "The Muon optimizer", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 38.26, |
| "width": 18.27, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>New, efficient optimizers—and the field of optimization as a whole—are massive topics. I definitely can't cover them fully in a single note. Hell, they deserve their own hour-long lecture!</p>\n<p>If you want to dive deeper and build some intuition on your own, I highly recommend checking out these links:</p>\n<ul>\n<li>Original blog post: <a href=\"https://kellerjordan.github.io/posts/muon/\">Muon: An optimizer for hidden layers in neural networks</a></li>\n<li><a href=\"https://jasonjiaxiangli.github.io/blog/monotone-ns/\">Monotone Newton-Schulz: Near-Monotone Polynomial Iterations for Matrix Sign</a></li>\n<li><a href=\"https://www.notion.so/233ffa7f40c4800eafa5cc843e039327?pvs=21\">Understanding Muon: A Revolutionary Neural Network Optimizer</a></li>\n</ul>\n<p>For now, I'll just share a few important quotes from those posts:</p>\n<ul>\n<li>Traditional optimizers treat all parameters the same, but Muon is smarter. It understands that 2D weight matrices <in Linear layers> have unique geometric structures.</li>\n<li>When we examine the gradient matrices that traditional optimizers produce, we find they often have very high condition numbers. Think of this as how \"sensitive\" a matrix is to small changes. High condition numbers mean the matrix is nearly low-rank - most of its information is concentrated in just a few directions. Some parameters get huge updates while others get tiny ones, creating imbalanced learning.</li>\n<li>Muon solves this problem through a elegant mathematical insight: instead of applying raw momentum updates, it orthogonalizes them to create more balanced parameter changes.</li>\n<li>Think of orthogonalization like this: if traditional optimizers create updates that are like a bunch of arrows pointing in similar directions (wasteful and imbalanced), Muon transforms them into arrows that point in perpendicular directions - each carrying unique, non-redundant information.</li>\n<li>We speculate that orthogonalization effectively increases the scale of other “rare directions” which have small magnitude in the update but are nevertheless important for learning.</li>\n</ul>" |
| }, |
| "linear-stages-overlap": { |
| "page": 15, |
| "type": "Systems note", |
| "title": "2 (or 3) Linear layers inside an expert", |
| "rects": [ |
| { |
| "left": 59.29, |
| "top": 33.08, |
| "width": 29.24, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 34.69, |
| "width": 18.55, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Reading the paper, you might think a single expert has two linear layers. But that's not the case—there are actually three. <a href=\"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/inference/model.py#L587\">Here's</a> the code:</p>\n<pre><code class=\"language-python\">class Expert(nn.Module):\n """Single MoE expert: SwiGLU FFN (w1, w2, w3). Computation in float32 for stability."""\n def __init__(self, dim: int, inter_dim: int, dtype=None, swiglu_limit=0):\n super().__init__()\n self.w1 = Linear(dim, inter_dim, dtype=dtype)\n self.w2 = Linear(inter_dim, dim, dtype=dtype)\n self.w3 = Linear(dim, inter_dim, dtype=dtype)\n self.swiglu_limit = swiglu_limit\n\n def forward(self, x: torch.Tensor, weights: Optional[torch.Tensor] = None) -> torch.Tensor:\n dtype = x.dtype\n gate = self.w1(x).float()\n up = self.w3(x).float()\n if self.swiglu_limit > 0:\n up = torch.clamp(up, min=-self.swiglu_limit, max=self.swiglu_limit)\n gate = torch.clamp(gate, max=self.swiglu_limit)\n x = F.silu(gate) * up\n if weights is not None:\n x = weights * x\n return self.w2(x.to(dtype))\n</code></pre>\n<p>As you can see, <code>w1</code> (gate) and <code>w3</code> (up) have the exact same dimensions. Under the hood, they are actually <a href=\"https://github.com/deepseek-ai/DeepGEMM/blob/211d2678d9d178de7a77fbfefe7310e0f4f8821c/deep_gemm/include/deep_gemm/impls/sm100_fp8_fp4_mega_moe.cuh#L37\">fused</a> into a single linear layer (Linear-1) and computed in parallel. You can even spot this in Figure 5, where L1 takes about twice as long to compute as L2:</p>\n<figure>\n<img src=\"content/notes/media/027-linear-stages-overlap-timeline.png\" alt=\"Timeline comparison of L1 and L2 computation\">\n</figure>" |
| }, |
| "expert-parallelism-bottleneck": { |
| "page": 15, |
| "type": "Systems note", |
| "title": "Expert parallelism", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 15.99, |
| "width": 76.9, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.65, |
| "top": 17.6, |
| "width": 76.65, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 19.21, |
| "width": 19.88, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Borrowing from the excellent <a href=\"https://huggingface.co/spaces/nanotron/ultrascale-playbook?section=expert_parallelism\">Ultra-Scale Playbook</a>:</p>\n<p>The design of MoE layers makes it easy to implement parallelism across the experts dimension, for what we call expert parallelism (EP). Since the feedforward layers are fully independent, we can simply put each expert's feedforward layer on a different worker (GPU). Compared to TP (TensorParallelism), this approach is much more lightweight, since we don't need to split the matrix multiplication; we just need to route the hidden states of a token to the right expert.</p>\n<figure>\n<img src=\"content/notes/media/028-expert-parallelism-bottleneck-routing.png\" alt=\"Expert Parallelism routing illustration\">\n</figure>\n\n<p>There are a few tricks to make EP work efficiently, and they are closely tied to model design. For instance, DeepSeek-V3 enforces a constraint in the router, ensuring that each token is sent to at most <span class=\"math-inline\" data-tex=\"M\">\\(M\\)</span> nodes (in their case, 4) to keep the tokens on a single node and reduce communication overhead.</p>" |
| }, |
| "aten-operators": { |
| "page": 16, |
| "type": "Kernel note", |
| "title": "Torch ATen operators", |
| "rects": [ |
| { |
| "left": 11.65, |
| "top": 84.62, |
| "width": 18.43, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>ATen is PyTorch's low-level tensor operator library. When you write standard PyTorch code like this:</p>\n<pre><code class=\"language-python\">y = torch.softmax(x, dim=-1)\nz = torch.matmul(y, w)\nu = z + bias\n</code></pre>\n<p>PyTorch translates it into lower-level ATen operators:</p>\n<pre><code class=\"language-cpp\">aten::softmax\naten::matmul\naten::add\naten::view\naten::sum\n</code></pre>\n<p>While these ATen ops are great building blocks, chaining them together creates two major bottlenecks. First is the kernel launch overhead. Every ATen call requires the host CPU to schedule a separate GPU kernel. While individual launch times are microscopic, doing this for hundreds of tiny ops per layer adds up quickly.</p>\n<p>Second is the memory bandwidth bottleneck. Executing ops sequentially forces intermediate tensors to be written out to slow global GPU memory (VRAM), only to be immediately read back into fast on-chip memory (SRAM/registers) by the next operator.</p>\n<p>You can drastically speed up your code by replacing multiple ATen ops with a fused kernel. This minimizes CPU scheduling overhead and lets you keep intermediate results in fast memory, preventing expensive read/write cycles.</p>\n<p><img alt=\"Fused kernels illustration from the Ultra-Scale Playbook.\" src=\"content/notes/media/029-aten-operators-fused-kernel-playbook.png\"></p>\n<p>From <a href=\"https://huggingface.co/spaces/nanotron/ultrascale-playbook?section=fused_kernels\">the Ultra-Scale Playbook</a>.</p>" |
| }, |
| "integer-analysis": { |
| "page": 17, |
| "type": "Compiler note", |
| "title": "Why integer analysis shows up", |
| "rects": [ |
| { |
| "left": 75.1, |
| "top": 51.65, |
| "width": 13.24, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 53.26, |
| "width": 49.85, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>When translating a high-level tensor operation into low-level GPU code, the compiler has to map exact memory slices to specific CUDA threads (compute units). You might write something innocent in Python like <code>x[:, ::2].T + bias</code>, but behind the scenes, that turns into an absolute of integer arithmetic over shapes, strides, offsets, block sizes, thread IDs, masks, and alignment constraints.</p>\n<p>This is where a solver comes in, acting as a mini theorem prover. The compiler hits it with questions like: \"Can this index ever go out of bounds?\", \"Will two threads write to the exact same address?\", or \"Are these elements guaranteed to be contiguous?\" If the solver can mathematically prove these properties, the compiler gets the green light to take a highly optimized fast path.</p>\n<p>Vectorized memory loading is a classic example. If several values are contiguous and properly aligned in GPU memory, the kernel can fetch them in a single instruction instead of using multiple scalar loads. But the moment you introduce slicing, transposing, padding, or custom layouts, logical \"neighbors\" in your tensor are rarely physical neighbors in memory anymore.</p>\n<p>Because the underlying address formula might involve strides, modulos, floor division, and symbolic dimensions, the compiler has to prove that <code>address(k + 1) == address(k) + 1</code> (and that <code>k</code> remains strictly within bounds). If it can't prove this, It falls back to a slow, conservative approach using scalar loads and runtime bounds checks.</p>\n<p>So, where does this solver actually live in everyday PyTorch? If you're running in standard eager mode, it doesn't. Eager mode relies entirely on pre-compiled kernels where human engineers have already done all the heavy mathematical lifting.</p>\n<p>But the second you wrap your model in torch.compile(), everything changes. PyTorch starts generating custom GPU kernels on the fly. Behind the scenes, it fires up SymPy—a symbolic math library—to act as this exact kind of solver. By symbolically analyzing dynamic shapes and strides, SymPy proves to TorchInductor (the compiler) exactly when it's safe to vectorize memory reads or fuse operations without unnecessary overhead.</p>" |
| }, |
| "no-split-kv": { |
| "page": 18, |
| "type": "Kernel note", |
| "title": "Why split-KV is unavailable", |
| "rects": [ |
| { |
| "left": 49.42, |
| "top": 52.4, |
| "width": 29.67, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>First, let's break down what the Split-KV method is and why we need it. Then we’ll look at why we can't actually use it here.</p>\n<p>In NVIDIA GPU architecture, the Streaming Multiprocessor (SM) is the fundamental computational building block. You can think of it loosely as a highly parallel GPU \"core\". A modern GPU like an H100 has over a hundred SMs, and we usually parallelize compute workloads across them.</p>\n<p>The famous FlashAttention algorithm works exceptionally well during training because it parallelizes across the batch and query dimensions. During pre-training or the prefill phase of inference, we process many query tokens in parallel, which naturally provides enough work to keep all SMs busy.</p>\n<p>But during decoding—for both standard inference and, more importantly, RL rollouts—the query length is tiny. It's often just a single token. Combine that with long contexts that force small batch sizes due to memory limits, and the GPU suddenly lacks parallel work. Most of the SMs sit completely idle, leading to terrible GPU utilization.</p>\n<p>To fix this, we use Split-KV. This decoding-focused optimization splits the long sequence dimension (the KV cache) into smaller chunks to be processed in parallel across multiple SMs. This allows all SMs to work on the exact same query simultaneously. Afterward, a separate step gathers the partial results from all SMs and reduces (sums) them to produce the final attention output:</p>\n<figure>\n<img src=\"content/notes/media/031-no-split-kv-method.gif\" alt=\"Split-KV method illustration\">\n</figure>\n\n<p>However, Split-KV's reduction step is problematic. DeepSeek requires bitwise batch invariance, meaning the same token must produce exactly the same bits regardless of where it appears in a batch.</p>\n<p>In standard floating-point arithmetic, the order of summation matters. Due to rounding errors, <span class=\"math-inline\" data-tex=\"(A+B)+C\\neq A+(B+C)\">\\((A+B)+C\\neq A+(B+C)\\)</span>. Since the final bitwise result strictly depends on the exact grouping and order of these additions, Split-KV can't guarantee the strict reproducibility that DeepSeek needs.</p>" |
| }, |
| "wave-quantization": { |
| "page": 18, |
| "type": "Kernel note", |
| "title": "When split-KV is removed", |
| "rects": [ |
| { |
| "left": 74.35, |
| "top": 55.62, |
| "width": 13.95, |
| "height": 1.65 |
| }, |
| { |
| "left": 16.28, |
| "top": 57.22, |
| "width": 72.01, |
| "height": 1.65 |
| }, |
| { |
| "left": 16.28, |
| "top": 58.84, |
| "width": 13.86, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>In the previous note, we saw how Split-KV slices the long sequence dimension (the KV cache) into smaller chunks to process them in parallel across multiple SMs. Without it, our GPU parallelization options shrink to just splitting over the batch and query length.</p>\n<p>Since the query length during decoding is exactly 1, there's nothing to parallelize there. We're left with just the batch.</p>\n<p>In the standard approach without Split-KV, a single batch element occupies all the resources of an entire SM. An H800 GPU physically has 132 SMs. I don't know the exact batch size DeepSeek uses for long-context RL rollouts, but let's assume it's 200 to keep things simple.</p>\n<p>Each SM picks up one batch element, multiplying the current token's Query by the KV cache of the entire context. The first 132 elements start processing immediately, fully saturating the GPU. This is the first wave. The remaining 68 requests (<span class=\"math-inline\" data-tex=\"200 - 132 = 68\">\\(200 - 132 = 68\\)</span>) have to wait for the second wave, since there are no free SMs left.</p>\n<p>Once the first wave finishes, the remaining 68 elements are assigned to the newly freed SMs. But now the GPU is half-idle. We're only using 68 out of 132 SMs, while the other 64 just sit there doing nothing.</p>\n<p>This hardware underutilization is exactly what Split-KV solved. It allowed us to spread the tail of the batch across all SMs. But the DeepSeek-V4 team decided to ditch it to preserve batch invariance.</p>\n<div class=\"callout\">\n<p>Interesting insight: this means bumping the batch size from 200 to 264 is essentially free in terms of latency (assuming you have enough VRAM for the KV cache). The compute time for batches of 200 and 264 will be exactly the same. In both cases, the GPU needs to execute exactly two waves.</p>\n<p>The <a href=\"https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#wave-quant\">Nvidia docs</a> have a great chart illustrating this effect for matrix multiplication of sizes <span class=\"math-inline\" data-tex=\"M \\times K\">\\(M \\times K\\)</span> and <span class=\"math-inline\" data-tex=\"K \\times N\">\\(K \\times N\\)</span>. If <span class=\"math-inline\" data-tex=\"K\">\\(K\\)</span> and <span class=\"math-inline\" data-tex=\"M\">\\(M\\)</span> are fixed while <span class=\"math-inline\" data-tex=\"N\">\\(N\\)</span> grows, the number of tiles (chunks) and execution time increase in discrete steps:</p>\n<figure>\n<img src=\"content/notes/media/032-wave-quantization-chart.png\" alt=\"Nvidia wave quantization chart\">\n</figure>\n<p>A matrix with <span class=\"math-inline\" data-tex=\"N=3100\">\\(N=3100\\)</span> will take the exact same amount of time to compute as a matrix with <span class=\"math-inline\" data-tex=\"N=4500\">\\(N=4500\\)</span>.</p>\n</div>\n\n<p>This exact scenario leads to severe wave-quantization inefficiencies. Without the ability to parallelize the workload at a finer granularity, we simply can't fully utilize the GPU.</p>" |
| }, |
| "split-k-gemm": { |
| "page": 18, |
| "type": "Kernel note", |
| "title": "Split-K for small batches", |
| "rects": [ |
| { |
| "left": 26.17, |
| "top": 78.15, |
| "width": 62.19, |
| "height": 1.65 |
| }, |
| { |
| "left": 16.28, |
| "top": 79.76, |
| "width": 13.12, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Conceptually, split-K is based on the exact same idea as the split-KV trick we discussed earlier, but this time, we’re applying it to GEMM (general matrix multiplication).</p>\n<p>We use GEMM all the time in linear layers to multiply inputs by a weight matrix. The operation <code>C = A @ B</code> has the following shapes:</p>\n<ul>\n<li>A: [M, K]</li>\n<li>B: [K, N]</li>\n<li>C: [M, K] x [K, N] = [M, N]</li>\n</ul>\n<p>We calculate each element <span class=\"math-inline\" data-tex=\"C[m, n]\">\\(C[m, n]\\)</span> as a sum over <span class=\"math-inline\" data-tex=\"K\">\\(K\\)</span>: <span class=\"math-inline\" data-tex=\"C[m, n] = \\sum_k A[m, k] * B[k, n]\">\\(C[m, n] = \\sum_k A[m, k] * B[k, n]\\)</span>.</p>\n<p>Usually, GEMM parallelizes the computation by slicing the <span class=\"math-inline\" data-tex=\"[m, n]\">\\([m, n]\\)</span> grid into tiles. A single CUDA block handles a specific tile <span class=\"math-inline\" data-tex=\"C[m0:m1, n0:n1]\">\\(C[m0:m1, n0:n1]\\)</span>. It loops through all values of <span class=\"math-inline\" data-tex=\"K\">\\(K\\)</span>, accumulates the sum, and writes out the finished tile.</p>\n<p>Split-K takes this a step further by splitting the computation along the <span class=\"math-inline\" data-tex=\"K\">\\(K\\)</span> dimension. Now, our tile looks like this:</p>\n<div class=\"math-display\" data-tex=\"C[m0:m1, n0:n1]\n= partial_0 + partial_1 + partial_2 + partial_3 + ...\">\\[C[m0:m1, n0:n1]\n= partial_0 + partial_1 + partial_2 + partial_3 + ...\\]</div>\n<p>We use this splitting technique in roughly the same scenarios as split-KV. When parallelizing across just two axes (especially when one is very small, like the batch size) doesn't generate enough parallel work, some SMs on the GPU end up idle. This reduces hardware utilization.</p>\n<p>Just like with split-KV, the order in which we accumulate the partial sums <span class=\"math-inline\" data-tex=\"partial_0 + partial_1 + partial_2 + partial_3\">\\(partial_0 + partial_1 + partial_2 + partial_3\\)</span> actually matters. If we don't preserve it, we break batch invariance. Floating-point addition simply isn't associative: <span class=\"math-inline\" data-tex=\"(A+B)+C\\neq A+(B+C)\">\\((A+B)+C\\neq A+(B+C)\\)</span>.</p>\n<div class=\"callout\">\n<p>Note that even though \"split-KV\" and \"split-K\" sound very similar, the \"K\" refers to completely different things.</p>\n<p>In split-KV, it stands for Keys and Values in attention calculations. Split-KV slices the sequence dimension, and later we have to merge the partial softmax statistics and weighted values.</p>\n<p>In split-K, <code>K</code> refers to one of the GEMM dimensions. It's the reduction dimension, which usually corresponds to the input feature dimension in a <code>[batch_size, features_dim]</code> matrix.</p>\n</div>" |
| }, |
| "mhc-output-24": { |
| "page": 18, |
| "type": "Implementation note", |
| "title": "24 in mHC dimensions", |
| "rects": [ |
| { |
| "left": 49.9, |
| "top": 76.54, |
| "width": 32.96, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Let's quickly recap how mHC works. It takes <span class=\"math-inline\" data-tex=\"m\">\\(m\\)</span> embeddings of size <span class=\"math-inline\" data-tex=\"d\">\\(d\\)</span> from the residual stream, concatenates them, normalizes them, and passes the resulting <span class=\"math-inline\" data-tex=\"m*d\">\\(m*d\\)</span> vector through three linear layers:</p>\n<ul>\n<li>Pre-Block Mixing (4 output features)</li>\n<li>Post-Block Mixing (4 output features)</li>\n<li>Residual Mixing (<span class=\"math-inline\" data-tex=\"4\\times 4\">\\(4\\times 4\\)</span> output features)</li>\n</ul>\n<figure>\n<img src=\"content/notes/media/034-mhc-output-24-mixing-layers.png\" alt=\"mHC mixing layers diagram\">\n</figure>\n\n<p>We can actually fuse these three linear layers into a single layer with an output dimension of <span class=\"math-inline\" data-tex=\"4 + 4 + (4\\times4)=24\">\\(4 + 4 + (4\\times4)=24\\)</span>. This 24 is exactly the number of scalars mHC needs to generate for the three small mixing matrices when we set <code>n_hc = 4</code>. You can see this reflected in the <a href=\"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/inference/model.py#L666\">inference code</a>:</p>\n<pre><code class=\"language-python\"># assume hc_mult = 4\nmix_hc = (2 + hc_mult) * hc_mult\nhc_dim = hc_mult * args.dim\n\n# both of these are 24 x (4 * dim)\nself.hc_attn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))\nself.hc_ffn_fn = nn.Parameter(torch.empty(mix_hc, hc_dim))\n</code></pre>\n<p>Interestingly, this fusion slightly complicates the Muon optimizer logic we discussed earlier. We now have three completely different projections bundled together into a single 2D weight matrix (<code>hc_attn_fn</code> and <code>hc_ffn_fn</code>). The DeepSeek team likely anticipated this, and presumably only fuses these layers during the forward pass and at inference time.</p>" |
| }, |
| "post-training-fp4": { |
| "page": 19, |
| "type": "Precision note", |
| "title": "FP4 during post-training", |
| "rects": [ |
| { |
| "left": 46.07, |
| "top": 62.06, |
| "width": 25.69, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Back in DeepSeek-V3, the core training framework relied on FP8 mixed precision for both pretraining and RL. Just a quick refresher: this means we mix and match different data types—the big win here is that MoE weights are stored and multiplied in FP8, which saves a massive amount of memory. Meanwhile, attention and other sensitive components are kept in higher precision.</p>\n<figure>\n<img src=\"content/notes/media/035-post-training-fp4-mixed-precision.png\" alt=\"It really was quite something!\">\n</figure>\n\n<p>It really was quite something!</p>\n<p>Now, DeepSeek-V4 introduces a much more granular breakdown across the different training stages. I’ve put together a quick table summarizing all the precision types mentioned throughout the paper. It's super handy to keep this in mind as we go:</p>\n<table>\n<thead>\n<tr>\n<th>Stage</th>\n<th>Type</th>\n<th>Notes</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>LM Pretrain</td>\n<td><strong>FP8 mixed</strong></td>\n<td>Inherits the DS-V3 training framework. Core Linear GEMMs are in FP8. No FP4 whatsoever.</td>\n</tr>\n<tr>\n<td>Specialist SFT</td>\n<td><strong>simulated FP4→FP8</strong></td>\n<td>QAT in post-training. FP4 expert weights are de-quantized to FP8 for the forward and backward passes.</td>\n</tr>\n<tr>\n<td>Specialist RL / rollouts</td>\n<td><strong>native FP4</strong></td>\n<td>No backward pass. Uses FP4 weights directly for sampling.</td>\n</tr>\n<tr>\n<td>Specialist RL / train</td>\n<td><strong>simulated FP4→FP8</strong></td>\n<td>Backward pass over FP8 weights. Gradients flow into FP32 master weights via STE.</td>\n</tr>\n<tr>\n<td>OPD / student rollouts</td>\n<td><strong>native FP4</strong></td>\n<td>The student generates on-policy trajectories.</td>\n</tr>\n<tr>\n<td>OPD / teacher logits inference</td>\n<td><strong>native FP4</strong></td>\n<td>Inference-only forward passes, meaning no backward pass is needed.</td>\n</tr>\n<tr>\n<td>OPD / student train</td>\n<td><strong>simulated FP4→FP8</strong></td>\n<td>Reverse-KL / full-vocab distillation.</td>\n</tr>\n<tr>\n<td>Model serving (API)</td>\n<td><strong>native FP4</strong></td>\n<td>FP4 is used for MoE expert weights and the CSA indexer QK path.</td>\n</tr>\n</tbody>\n</table>\n<p>I want to reiterate: we aren't talking about compressing <em>all</em> the model weights here. The paper clarifies this later on (specifically, right in the same paragraph as this note):</p>\n<blockquote>\n<p>We apply FP4 (MXFP4) quantization to two components: (1) MoE expert weights, which are a major source of GPU memory occupancy and (2) the Query-Key (QK) path in the indexer of CSA, where QK activations are cached, loaded, and multiplied entirely in FP4.\n</p>\n</blockquote>\n<p>So, none of this affects how things like token embeddings are stored.</p>" |
| }, |
| "fp4-scale-absorption": { |
| "page": 19, |
| "type": "Precision note", |
| "title": "FP4-to-FP8 scale absorption", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 82.17, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 83.78, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 85.39, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 87.0, |
| "width": 18.9, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>At first glance, losslessly casting FP4 values to FP8 seems straightforward. But it's not totally clear if the quantization scales will actually fit. Let's peek under the hood to see how these quantizations work.</p>\n<p>DeepSeek-V3 was trained in FP8. Unlike our good old FP16 and FP32, this format comes with a catch. 16 and 32 bits give you enough dynamic range to accurately capture the spread of values in a neural network. But 8 bits just isn't enough—numbers quickly collapse into zeros or blow up to infinity due to overflow and underflow.</p>\n<p>To fix this, FP8 relies on a separate quantization scale that lives outside those 8 bits. We calculate this scale so that multiplying it back restores the original weight with minimal loss: <span class=\"math-inline\" data-tex=\"w \\approx w_{fp8} * scale\">\\(w \\approx w_{fp8} * scale\\)</span>.</p>\n<p>The memory savings come from sharing a single quantization scale across a whole block of FP8 values. Block sizes differ for activations and weights, but for this note, we'll stick to weights. In DeepSeek-V3, they group and scale weight elements in 128x128 blocks (i.e., per 128 input channels and 128 output channels).</p>\n<p>If a matrix is larger than that, we simply slice it into 128x128 tiles. For example, the FP8 weights for a <code>Linear(7168, 2048)</code> layer get chopped into <span class=\"math-inline\" data-tex=\"(7168/128) \\times (2048/128) = 896\">\\((7168/128) \\times (2048/128) = 896\\)</span> blocks. So, alongside the raw FP8 values, we also have to store 896 scaling factors. The matrix multiplications natively factor in these scales. Here's a great visualization from <a href=\"https://developer.nvidia.com/blog/per-tensor-and-per-block-scaling-strategies-for-effective-fp8-training/\">Nvidia's blog</a>:</p>\n<figure>\n<img src=\"content/notes/media/036-fp4-scale-absorption-fp8-scaling.png\" alt=\"FP8 block scaling visualization\">\n</figure>\n\n<p>DeepSeek-V4 applies the exact same principle to FP4. However, because FP4 quantization is much more aggressive, rounding errors accumulate faster. To counter this, they shrank the group size down to 1x32. This means a single 128x128 block of FP8 values now contains exactly <span class=\"math-inline\" data-tex=\"(128 / 1) \\times (128 / 32) = 512\">\\((128 / 1) \\times (128 / 32) = 512\\)</span> of these tiny FP4 chunks.</p>\n<p>Spot the problem yet? Standard FP8 expects exactly one scale per 128x128 block, but our new FP4 setup has 512 scales for that same block. DeepSeek claims the FP8 weights can just \"absorb\" these extra scales. Let's write some code to prove it:</p>\n<pre><code class=\"language-python\">import torch\n\n# FP8 block: 128x128; FP4 scale tile: 1x32\nR, C, G = 128, 128, 32\n\n# Values exactly representable by FP4\nfp4 = torch.tensor([0, .5, 1, 1.5, 2, 3, 4, 6,\n -.5, -1, -1.5, -2, -3, -4, -6],\n dtype=torch.float32)\n\n# Fake already-quantized FP4 payload, unpacked into fp32\nq4 = fp4[torch.randint(len(fp4), (R, C))].view(R, C // G, G)\n\n# One local scale per 1x32 tile\ns4 = (2.0 ** torch.randint(0, 15, (R, C // G)))\n\n# One shared FP8 scale for the whole 128x128 block\ns8 = torch.tensor(2.0 ** 8)\n\n# Absorb local FP4 scales into the FP8 payload\noriginal_data = q4 * s4[..., None]\nq8 = (original_data / s8).reshape(R, C)\nq8 = q8.to(torch.float8_e4m3fn)\n\n# Same weights: many FP4 local scales vs one FP8 block scale\nassert torch.equal(\n (q4 * s4[..., None]).reshape(R, C),\n q8.float() * s8,\n)\n</code></pre>\n<p>In this snippet, <code>fp4</code> lists the actual values that an FP4 number can represent. It can only express a small, fixed grid (values taken from <a href=\"https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/#:~:text=What%20is%20NVFP4?,same%20for%20the%20negative%20range\">here</a>), not arbitrary floats. We use <code>torch.randint(len(fp4), (R, C))</code> to sample random entries from this grid, simulating an already-quantized FP4 payload (MoE weights).</p>\n<p>Next, in <code>s4</code> we randomly sample one local power-of-two scale per 1x32 tile. Finally, <code>s8 = 2^8</code> is our shared 128x128 FP8 block scale. I picked this specific value for the toy example so the rescaled payload fits cleanly into FP8, though in reality, it might differ.</p>\n<p>Now for the magic. The line <code>original_data = q4 * s4[..., None]</code> reconstructs the original weights in higher precision. Then, <code>q8 = (original_data / s8).reshape(R, C)</code> folds that fine-grained FP4 scale information directly into the FP8 payload itself. If the final <code>assert</code> passes, it means the FP4→FP8 conversion didn't introduce any additional rounding error. <code>q4 * s4</code> and <code>q8 * s8</code> reconstruct the exact same weights.</p>\n<p>Keep in mind that information is inherently lost when you initially compress the weights down to the FP4 grid. FP4 simply can't store a value like 1.7—it has to snap to 1.5, as you can see in our <code>fp4</code> tensor.</p>" |
| }, |
| "cp-all-gather": { |
| "page": 21, |
| "type": "Systems note", |
| "title": "Context-parallel all-gather", |
| "rects": [ |
| { |
| "left": 86.19, |
| "top": 56.96, |
| "width": 2.1, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 58.57, |
| "width": 76.66, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 60.18, |
| "width": 6.3, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Yes, each rank receives the full KV cache for the entire sequence.</p>\n<p>You might expect this to eat up a ton of space and completely blow up your VRAM. But thanks to the incredible efficiency of CSA and HCA, the actual memory footprint is tiny, even for super long context windows.</p>\n<p>Let's do some quick math. For simplicity, we'll assume all elements are stored in BF16 (2 bytes per element), even though DeepSeek mentioned storing some dimensions in FP8.</p>\n<p>For a 64k token context, a single CSA layer stores <span class=\"math-inline\" data-tex=\"64\\text{k}/4 = 16,384\">\\(64\\text{k}/4 = 16,384\\)</span> KV entries (thanks to the compression factor <span class=\"math-inline\" data-tex=\"m=4\">\\(m=4\\)</span>). That comes out to just:</p>\n<div class=\"math-display\" data-tex=\"512 \\times 2 \\times 16,384\n= 16.7\\text{M} \\text{ bytes}\n\\approx 16.7\\text{ MB}\">\\[512 \\times 2 \\times 16,384\n= 16.7\\text{M} \\text{ bytes}\n\\approx 16.7\\text{ MB}\\]</div>\n<p>per layer, per sequence.</p>\n<p>For HCA, that number drops by a factor of <span class=\"math-inline\" data-tex=\"128/4 = 32\">\\(128/4 = 32\\)</span>. We're talking less than a single megabyte!</p>\n<p>Even scaling all the way up to a 1M token context, we're still only looking at roughly 256 MB and 8 MB, respectively. These numbers are so small they barely make a dent in either your network bandwidth or VRAM.</p>" |
| }, |
| "layer-kv-cache-sizes": { |
| "page": 22, |
| "type": "Memory note", |
| "title": "Layer-varying KV caches", |
| "rects": [ |
| { |
| "left": 70.46, |
| "top": 60.29, |
| "width": 17.84, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 61.9, |
| "width": 27.61, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>This is because CSA and HCA operate on different chunk sizes (for example, <span class=\"math-inline\" data-tex=\"m=4\">\\(m=4\\)</span> versus <span class=\"math-inline\" data-tex=\"m'=128\">\\(m'=128\\)</span>). To handle this, we have to keep token buffers of different sizes so we can compress them into a single chunk later on. Consequently, one layer will have vastly more token slots than another.</p>\n<p>We also have to account for the fact that CSA and HCA inherently produce different numbers of chunks due to their distinct compression factors. On top of that, CSA requires us to store the lightning indexer keys in the KV cache. </p>\n<p>Long story short, the layers differ a lot.</p>" |
| }, |
| "on-disk-kv-cache": { |
| "page": 24, |
| "type": "Serving note", |
| "title": "Is on-disk KV cache fast enough?", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 23.88, |
| "width": 28.57, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>You might think reading from disk is extremely slow and would massively spike response latency—especially in long-context scenarios with a massive KV cache.</p>\n<p>But it turns out, that's not the case. In their recent paper, <a href=\"https://arxiv.org/abs/2602.21548\">DualPath: Breaking the Storage Bandwidth Bottleneck in Agentic LLM Inference</a>, DeepSeek wrote: “Our cluster-wide 3FS has no internal DRAM cache and can saturate the 400Gbps bandwidth of the storage NIC.”</p>\n<p>3FS is the name of their custom file system. The interesting part is the lack of a DRAM cache. This means the system isn't faking its performance by keeping hot KV blocks in a massive DRAM data cache; everything is read directly from SSDs.</p>\n<p>Despite this, the storage architecture is designed to completely saturate the Network Interface Controller (NIC). In other words, you physically can't pump more data into a given GPU server without adding more NICs.</p>\n<p>For context, 400 Gbps is roughly 50 GB/s. That's about the bandwidth of a single fast DDR5 memory channel. Sure, it falls way short of the aggregate server DRAM bandwidth. But it's actually right on par with the relevant ingress path into a GPU. This path is typically the PCIe Gen5 I/O fabric, which pushes about 64 GB/s in one direction.</p>\n<p>So, a single fully saturated 400 Gbps storage NIC gets you pretty close to the per-GPU host ingress limit. Of course, this is still a drop in the bucket compared to the massive HBM bandwidth inside the GPU itself.</p>\n<p>But a standard host doesn't just have one GPU—it has eight. Since eight GPUs have to share that 400 Gbps connection, you're still looking at a pretty hefty bottleneck.</p>\n<p>This is exactly the problem DeepSeek tackles in that DualPath paper I mentioned earlier. They actually managed to nearly double the throughput by using the NIC of a neighboring host and routing the data over a blazing-fast GPU-to-GPU link (via a separate Storage NIC, or SNIC).</p>\n<p>Now, how do you saturate that 50 GB/s read speed using SSDs? That's obviously way beyond what a single SSD can handle. But it's totally doable for a small cluster of datacenter NVMe drives. On paper, roughly four PCIe 5.0 SSDs or eight PCIe 4.0 SSDs can easily max it out.</p>\n<p>This perfectly explains the hardware design described on <a href=\"https://github.com/deepseek-ai/3fs\">DeepSeek’s GitHub</a>. They use a 3FS storage node packed with 16 PCIe 4.0 NVMe drives and two 400 Gbps NICs. This SSD array can deliver up to 112 GB/s of sequential read bandwidth.</p>\n<figure>\n<img src=\"content/notes/media/039-on-disk-kv-cache-3fs-architecture.png\" alt=\"DeepSeek 3FS storage node architecture\">\n</figure>\n\n<p>This doesn't magically turn an SSD into RAM for cache reads. But it does explain why this setup isn't orders of magnitude slower than reading the cache straight out of DRAM.</p>" |
| }, |
| "adamw-exceptions": { |
| "page": 26, |
| "type": "Optimizer note", |
| "title": "How to combine AdamW + Muon", |
| "rects": [ |
| { |
| "left": 11.7, |
| "top": 41.77, |
| "width": 76.59, |
| "height": 1.67 |
| }, |
| { |
| "left": 11.7, |
| "top": 43.4, |
| "width": 76.9, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 45.01, |
| "width": 76.82, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 46.62, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 48.23, |
| "width": 9.37, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>As mentioned earlier in the paper, AdamW is also applied to the static biases and gating factors of the mHC modules.</p>\n<figure>\n<img src=\"content/notes/media/040-adamw-exceptions-equations-3-5.png\" alt=\"These six parameters from Equations 3-5.\">\n<figcaption>These six parameters from Equations 3-5.</figcaption>\n</figure>\n\n<p>But why do we make this specific split? Why train some layers with Muon and others with AdamW?</p>\n<p>Let's recall what we discussed about Muon in an earlier note. Traditional optimizers treat all parameters exactly the same. Muon is smarter. It understands that 2D weight matrices (like those in Linear layers) have unique geometric structures.</p>\n<p>During the backward pass, it doesn't just apply raw momentum updates. Instead, it orthogonalizes them to create much more balanced parameter changes.</p>\n<p>This means Muon’s operation only really makes sense when the parameter is a proper matrix transform. Let’s look at the exceptions:</p>\n<ul>\n<li><strong>Embedding module:</strong> Yes, technically it's a 2D matrix of shape <span class=\"math-inline\" data-tex=\"vocab\\_size \\times hidden\\_dim\">\\(vocab\\_size \\times hidden\\_dim\\)</span>. But semantically, it's not a normal dense linear map—it's a lookup table. The rows correspond to discrete tokens, not continuous input features. AdamW is a better fit here because it maintains per-parameter adaptive second moments. This allows rows for rare and frequent tokens to get different effective learning rates.</li>\n<li><strong>Prediction head</strong> (often called the LM head): it's It's a classifier over a categorical vocabulary. Its rows and columns are tied to token identities, not just hidden features.</li>\n<li><strong>RMSNorm:</strong> The weights here are usually just a 1D scale vector. There is no meaningful 2D matrix geometry to orthogonalize. You could try reshaping it to <span class=\"math-inline\" data-tex=\"1\\times d\">\\(1\\times d\\)</span>, but then Muon would just degenerate into normalizing or coupling the entire vector update. That's definitely not what you want.</li>\n<li><strong>mHC components:</strong> The same logic applies to the mHC static biases and gates.</li>\n</ul>\n<p>You can easily carry this logic over to your own experiments. Muon is fantastic for internal learned matrix transforms. For everything else, just stick to AdamW.</p>" |
| }, |
| "swiglu-clamp": { |
| "page": 27, |
| "type": "Training note", |
| "title": "SwiGLU clamping", |
| "rects": [ |
| { |
| "left": 42.14, |
| "top": 45.01, |
| "width": 46.16, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 46.45, |
| "width": 71.68, |
| "height": 1.82 |
| } |
| ], |
| "html": "<p>Even though they explicitly say \"throughout the <strong>training</strong>,\" you absolutely have to do this during inference too. Otherwise, generation <a href=\"https://github.com/sgl-project/sglang/issues/23752\">breaks</a>:</p>\n<figure>\n<img src=\"content/notes/media/041-swiglu-clamp-broken-generation.png\" alt=\"Broken generation issue\">\n</figure>\n\n<p>In DeepSeek's initial code release, the SwiGLU clipping for the shared expert was missing. There was even a comment right next to it, so leaving it out was a deliberate choice. (Did an LLM write that? 🤔)</p>\n<p>Fortunately, the open-source community quickly tracked down the issue: \"the missing clamp causes shared-expert SiLU output to grow to <strong>±2000+</strong> during inference. This pollutes the residual stream via <code>final_hidden_states += shared_output</code>, propagates through <code>mhc_post</code>, and degrades <code>lm_head</code> logits at sentence-boundary positions.\"</p>\n<p>DeepSeek quickly <a href=\"https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/commit/a1fd202632e91bc0efe4eedb63ce554649c53997\">rolled out a fix</a>:</p>\n<figure>\n<img src=\"content/notes/media/041-swiglu-clamp-bug-fix.png\" alt=\"DeepSeek bug fix commit\">\n</figure>\n\n<p>Long story short: don't forget to clip your activations!</p>" |
| }, |
| "grm-policy-trajectory": { |
| "page": 29, |
| "type": "Post-training note", |
| "title": "Generative Reward Models", |
| "rects": [ |
| { |
| "left": 66.62, |
| "top": 78.47, |
| "width": 21.68, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.64, |
| "top": 80.08, |
| "width": 30.43, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>A Generalist Reward Model (GRM) is a model trained to evaluate non-verifiable or weakly verifiable tasks. These evaluation scores allow us to run RLVR (Reinforcement Learning from Verifiable Rewards) algorithms like GRPO.</p>\n<p>Right now, DeepSeek has just <a href=\"https://arxiv.org/abs/2504.02495\">one paper</a> detailing their GRM. Personally, I suspect this specific approach is a bit outdated and has probably evolved (I'll touch on one of those changes in the next note). But I still want to give you a rough idea of how a GRM works.</p>\n<p>The GRM is trained using RL. Given a prompt and a policy trajectory, it learns to write an evaluation plan—basically a set of principles and their relative weights. From there, it generates a critique, a partial score for each principle, and a final weighted score.</p>\n<figure>\n<img src=\"content/notes/media/042-grm-evaluation-process.png\" alt=\"GRM evaluation process diagram\">\n<figcaption>From <a href=\"https://arxiv.org/abs/2504.02495\">this paper</a></figcaption>\n</figure>\n\n<p>This approach easily generalizes to tasks where a simple binary \"right/wrong\" score doesn't work. Think creative writing, report compilation, summarization, and many others.</p>" |
| }, |
| "grm-rl-optimization": { |
| "page": 29, |
| "type": "Post-training note", |
| "title": "Same GRM ⇔ Policy model", |
| "rects": [ |
| { |
| "left": 42.24, |
| "top": 80.08, |
| "width": 46.05, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 81.68, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 83.29, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 84.9, |
| "width": 10.1, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>As far as I know, DeepSeek hasn't published a standalone paper showing exactly how they train a general GRM alongside their base model. They do have a paper on <a href=\"https://arxiv.org/abs/2504.02495\">Inference-Time Scaling for Generalist Reward Modeling</a> that covers the core principles of GRM training. Interestingly, though, they don't even use it as a Reward Model for the LLM in GRPO in that work.</p>\n<p>That said, we can find a very similar approach in <a href=\"https://arxiv.org/abs/2511.22570\">DeepSeekMath-v2</a>. There, the authors train the exact same model in three distinct modes simultaneously:</p>\n<ol>\n<li><strong>Proof Generator</strong>: actually solves the math problems.</li>\n<li><strong>Verifier</strong>: identifies issues and scores proofs from the Proof Generator.</li>\n<li><strong>Meta-Verifier</strong>: reviews the proof analyses produced by the Verifier.</li>\n</ol>\n<p>As a result, the model masters all three skills. More importantly, this setup helps \"incentivize the Proof Generator to identify and resolve as many issues as possible in their own proofs before finalizing them.\" Since the model already knows how to find errors and critique itself, it can self-verify on the fly while generating a solution. This is exactly what drives up the quality!</p>\n<p>What's cool is that the authors didn't just combine all three modes right away. They arrived at this setup after running some error analysis:</p>\n<blockquote>\n<p><…> we observed a critical limitation: when prompted to both generate and analyze its own proof in one shot, the generator tends to claim correctness even when the external verifier easily identify flaws. In other words, while the generator can refine proofs based on external feedback, it fails to evaluate its own work with the same rigor as the dedicated verifier.</p>\n<p>This observation motivated us to endow the proof generator with genuine verification capabilities.\n</p>\n</blockquote>\n<p>The proof verifier and generator create a tight feedback loop. The verifier improves the generator, and as the generator gets better, it produces new proofs that push the verifier’s current limits.</p>\n<p>For the general domain, these verification capabilities might look a lot like the DeepSeek GRM. The model first comes up with \"Principles\" that determine the quality of a potential response for a given query. Then, it evaluates the answer based on these principles and calculates a final score using a weighted sum. Everything is learned end-to-end via RL, and the model writes its own reasoning to justify the score. Humans don't have to hardcode these principles at all.</p>\n<figure>\n<img src=\"content/notes/media/042-grm-evaluation-process.png\" alt=\"GRM evaluation process diagram\">\n<figcaption>From <a href=\"https://arxiv.org/abs/2504.02495\">this paper</a></figcaption>\n</figure>" |
| }, |
| "persistent-reasoning-history": { |
| "page": 31, |
| "type": "Agent note", |
| "title": "Reasoning across agent turns", |
| "rects": [ |
| { |
| "left": 25.59, |
| "top": 73.65, |
| "width": 49.81, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>When I was using Codex and ChatGPT last year, I kept wondering: can these models actually see their own reasoning from the previous steps? Half the time, it felt like they couldn't. I’d point out a specific step in their logic and ask, \"Why did you do that?\"—and they'd completely lose the plot. They genuinely had no clue what I was talking about.</p>\n<figure>\n<img src=\"content/notes/media/044-persistent-reasoning-history-hidden-cot.png\" alt=\"Hidden CoT example\">\n</figure>\n\n<p>Now the meta has shifted: models are explicitly fed their previous reasoning steps. This keeps them from getting stuck in endless logic loops. Not only is it a massive UX upgrade, but it also slashes token usage, especially for tool calling. A model can formulate a plan and execute several tool calls sequentially, without needing to constantly reinvent the plan for future calls because it was discarded.</p>\n<p>Even Anthropic, who are notoriously secretive about their closed-source Claude Code—talked about this in a recent <a href=\"https://www.anthropic.com/engineering/april-23-postmortem\">blog post</a>:</p>\n<figure>\n<img src=\"content/notes/media/044-persistent-reasoning-history-anthropic-reasoning-chains.png\" alt=\"Anthropic blog post snippet about reasoning chains\">\n</figure>" |
| }, |
| "reverse-kl-opd": { |
| "page": 33, |
| "type": "Distillation note", |
| "title": "Why reverse KL is used for on-policy distillation", |
| "rects": [ |
| { |
| "left": 53.14, |
| "top": 53.9, |
| "width": 16.15, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>Okay, what is reverse KL, and why don't we use forward KL? Let's break it down.</p>\n<p>Approaches to post-training a student model generally fall into two categories. To quote a great <a href=\"https://thinkingmachines.ai/blog/on-policy-distillation/#loss-function-reverse-kl\">blog post</a> by Thinking Machines:</p>\n<ul>\n<li><strong>On-policy</strong> training samples rollouts from the student model itself and assigns them some reward.</li>\n<li><strong>Off-policy</strong> training relies on target outputs from an external source that the student learns to imitate.</li>\n</ul>\n<p>The downside of off-policy training is that the student learns in contexts the teacher frequents, not the ones the student itself will actually encounter. This leads to compounding errors. If the student makes an early mistake that the teacher would never make, it drifts further and further away from the states it saw during training.</p>\n<p>So why not use on-policy methods all the time? It helps to compare on-policy distillation to RL. The student generates trajectories, and the teacher grades each of them to provide feedback. But standard RL has a massive downside: this feedback is very sparse. The student only learns that its final answer was wrong and avoids that path in the future. It doesn't learn exactly where it made the mistake.</p>\n<p>Intuitively, the core idea of on-policy distillation (OPD) is to sample trajectories from the student model and use a high-performing teacher to grade <strong>each token</strong> along the way.</p>\n<p>Let's recall that Kullback-Leibler (KL) divergence is a type of statistical distance. It measures how much an approximating probability distribution <span class=\"math-inline\" data-tex=\"Q\">\\(Q\\)</span> (the student) differs from a target or reference distribution <span class=\"math-inline\" data-tex=\"P\">\\(P\\)</span> (the teacher).</p>\n<p>Crucially, divergences are generally asymmetric, meaning <span class=\"math-inline\" data-tex=\"D_{KL}(P || Q) \\neq D_{KL}(Q || P)\">\\(D_{KL}(P || Q) \\neq D_{KL}(Q || P)\\)</span>. Because of this asymmetry, we have to choose the order of the arguments, which dictates whether we use Forward KL or Reverse KL.</p>\n<p>The formula for Forward KL, straight from <a href=\"https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence\">Wikipedia</a>, looks like this:</p>\n<div class=\"math-display\" data-tex=\"D_{\\mathrm{KL}}(P || Q)\n= \\sum_{x \\in \\mathcal{X}} P(x)\\log\\frac{P(x)}{Q(x)}\">\\[D_{\\mathrm{KL}}(P || Q)\n= \\sum_{x \\in \\mathcal{X}} P(x)\\log\\frac{P(x)}{Q(x)}\\]</div>\n<p>In this formula, <span class=\"math-inline\" data-tex=\"x\">\\(x\\)</span> is the token actually sampled at a given position in the teacher's trajectory. Since we're using the teacher for rollouts, this is an off-policy approach. The loss is then summed over the sampled tokens in that rollout. <span class=\"math-inline\" data-tex=\"P(x)\">\\(P(x)\\)</span> is the probability predicted by the teacher model for that token given a specific prefix. <span class=\"math-inline\" data-tex=\"Q(x)\">\\(Q(x)\\)</span> is the probability predicted by the student model for the exact same prefix.</p>\n<p>In other words, the teacher's predicted probability acts as the weight in the sum we want to minimize. Suppose, for example, that at a given prefix, the teacher's distribution looks like this:</p>\n<table>\n<thead>\n<tr>\n<th>Token</th>\n<th>Teacher (P(x))</th>\n<th>Meaning</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>A</td>\n<td>0.499</td>\n<td>valid reasoning path</td>\n</tr>\n<tr>\n<td>B</td>\n<td>0.499</td>\n<td>another valid reasoning path</td>\n</tr>\n<tr>\n<td>C</td>\n<td>0.002</td>\n<td>bad / weird continuation</td>\n</tr>\n</tbody>\n</table>\n<p>Now let's compare two possible students. Student 1 covers both good options and imitates the teacher's output almost exactly. Student 2 simply commits to a single good option:</p>\n<table>\n<thead>\n<tr>\n<th>Token</th>\n<th>Student (Q(x))</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td>A</td>\n<td>0.997</td>\n</tr>\n<tr>\n<td>B</td>\n<td>0.001</td>\n</tr>\n<tr>\n<td>C</td>\n<td>0.002</td>\n</tr>\n</tbody>\n</table>\n<p>Student 2 is basically saying, \"I know the teacher likes both A and B, but I'll mostly just use A.\" It doesn't perfectly match the full teacher distribution, but it still confidently chooses a valid, teacher-approved token.</p>\n<p>You can read Forward KL as asking: \"For every token the teacher likes, does the student also assign probability mass there?\" This happens because the multiplier in our equation is <span class=\"math-inline\" data-tex=\"P(x)\">\\(P(x)\\)</span>, the teacher's probability. Intuitively, we're teaching the student not to ignore any important options the teacher considers valid—Student 2 from the example above would be punished.</p>\n<p>This is exactly why Forward KL is known as a <strong>mode-covering</strong> objective. It forces the student to cover all the major modes of the teacher's distribution. If the teacher uses several plausible reasoning styles, Forward KL pushes the student to preserve every single one of them.</p>\n<p>Reverse KL is, as the name suggests, reversed:</p>\n<div class=\"math-display\" data-tex=\"D_{\\mathrm{KL}}(Q || P)\n= \\sum_{x \\in \\mathcal{X}} Q(x)\\log\\frac{Q(x)}{P(x)}\">\\[D_{\\mathrm{KL}}(Q || P)\n= \\sum_{x \\in \\mathcal{X}} Q(x)\\log\\frac{Q(x)}{P(x)}\\]</div>\n<p>The critical difference here is that <span class=\"math-inline\" data-tex=\"x\">\\(x\\)</span> is the token actually sampled at a position in the student's trajectory.</p>\n<p>You can read this one as: \"For every token the student wants to use, does the teacher also approve of it?\" This makes sense because our multiplier is now <span class=\"math-inline\" data-tex=\"Q(x)\">\\(Q(x)\\)</span>, the student's probability.</p>\n<p>If the student assigns zero probability to token B, it won't be penalized at all, even if the teacher thinks B is a perfectly valid option. However, Reverse KL heavily penalizes the student if it puts probability mass on a bad token like C, when the teacher considers <span class=\"math-inline\" data-tex=\"P(C)\">\\(P(C)\\)</span> to be tiny.</p>\n<p>This is why Reverse KL is called <strong>mode-seeking</strong>. It tends to commit to one high-probability mode instead of spreading its probability mass over all available modes. In other words, it tells the student: \"Pick something the teacher likes. You don't have to cover every possible thing they like, just pick one. But whatever you do, don't put mass on tokens the teacher deems unlikely.”</p>\n<p>The 2023 <a href=\"https://arxiv.org/abs/2306.08543\">MiniLLM</a> paper provides a good empirical validation of this concept using a toy experiment setup:</p>\n<figure>\n<img src=\"content/notes/media/045-reverse-kl-opd-minillm-validation.png\" alt=\"MiniLLM paper empirical validation\">\n</figure>\n\n<p>Real LLM experiments in that same paper confirm this behavior as well. We also saw this recently in the <a href=\"https://arxiv.org/abs/2505.09388\">Qwen 3 paper</a>:</p>\n<figure>\n<img src=\"content/notes/media/045-reverse-kl-opd-qwen3-results.png\" alt=\"Qwen 3 paper results\">\n</figure>\n\n<p>The Qwen team reported that starting from the exact same off-policy distilled 8B checkpoint, on-policy distillation outperformed direct RL while using roughly one-tenth of the GPU hours.</p>\n<p>The exact reason why OPD works this well is still an ongoing topic of discussion. Will Brown from Prime Intellect wrote an interesting piece on it <a href=\"https://x.com/willccbb/status/2050038277454143918\">here</a>.</p>" |
| }, |
| "teacher-hidden-state-cache": { |
| "page": 34, |
| "type": "Distillation note", |
| "title": "Teacher hidden-state caching", |
| "rects": [ |
| { |
| "left": 72.19, |
| "top": 51.08, |
| "width": 16.18, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 52.69, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.65, |
| "top": 54.3, |
| "width": 4.43, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>This might look like a tiny optimization you could easily skip. But let's run the numbers to see how much memory it actually saves.</p>\n<p>To figure this out, we need the bit precision for the teacher's last-layer hidden states and materialized logits. The paper leaves this out, so let's keep it simple and assume standard 2-byte floats (BF16/FP16). This gives us the following memory footprint per teacher, per sequence:</p>\n<ul>\n<li><span class=\"math-inline\" data-tex=\"M_{\\text{logits}} = \\text{Seq\\_len} \\times |\\text{Vocab}| \\times 2\">\\(M_{\\text{logits}} = \\text{Seq\\_len} \\times |\\text{Vocab}| \\times 2\\)</span></li>\n<li><span class=\"math-inline\" data-tex=\"M_{\\text{hidden}} = \\text{Seq\\_len} \\times \\text{dim} \\times 2\">\\(M_{\\text{hidden}} = \\text{Seq\\_len} \\times \\text{dim} \\times 2\\)</span></li>\n</ul>\n<p>Let's assume a context length of 64k—a fairly standard size for agentic rollouts. For DeepSeek-V4 Pro <span class=\"math-inline\" data-tex=\"|\\text{Vocab}| = 131,072\">\\(|\\text{Vocab}| = 131,072\\)</span> and <span class=\"math-inline\" data-tex=\"d = 7168\">\\(d = 7168\\)</span>, we'd need 16 GB to store the logits versus just 0.875 GB for the hidden states. That’s a difference of over 15 GB per sequence, per teacher.</p>\n<p>(And yes, if they used FP8 instead of BF16/FP16, the absolute numbers would be cut in half, but the savings ratio stays exactly the same).</p>\n<p>It's also worth noting that hidden states (and especially output logits) are not KV cache entries. They don't get compressed into chunks of 4 or 128 tokens, so we can't rely on those massive memory savings here.</p>\n<p>DeepSeek mentions using over ten teacher models. That means we are saving over 150 GB per sequence. That number sounds so ridiculous I actually had to double-check my math.</p>" |
| }, |
| "simpleqa-gap": { |
| "page": 38, |
| "type": "Eval note", |
| "title": "The SimpleQA-Verified situation", |
| "rects": [ |
| { |
| "left": 41.07, |
| "top": 71.45, |
| "width": 47.53, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 73.06, |
| "width": 76.59, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.65, |
| "top": 74.67, |
| "width": 15.6, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>DeepSeek-V4 Pro crushing Kimi K2.6 here feels a bit off. Since Kimi is also a huge 1-trillion parameter model trained on over 30T tokens, the performance gap on SimpleQA seems disproportionately large. While I don't doubt V4's underlying quality, this result points toward overfitting or a failure to properly filter out the eval data from their pretraining set.</p>\n<p>That said, I'm not saying DeepSeek is faking it. V4 performs legitimately well on <a href=\"https://artificialanalysis.ai/evaluations/omniscience\">AA-Omniscience</a>—a newer benchmark designed to measure a model's breadth of knowledge—where it still manages to outperform Kimi:</p>\n<figure>\n<img src=\"content/notes/media/047-simpleqa-gap-aa-omniscience.png\" alt=\"AA-Omniscience benchmark comparison\">\n</figure>" |
| }, |
| "agentic-reasoning-effort": { |
| "page": 39, |
| "type": "Eval note", |
| "title": "Agentic effort levels", |
| "rects": [ |
| { |
| "left": 64.45, |
| "top": 37.95, |
| "width": 23.95, |
| "height": 13.35 |
| } |
| ], |
| "html": "<p>Looking at the table, there isn't a significant difference between the <code>high</code> and <code>max</code> settings on DeepSeek-V4 Pro for agentic tasks. While diminishing returns are expected at the highest reasoning tiers, the real issue is the complete lack of <code>medium</code> or <code>low</code> options.</p>\n<p>Just look at how GPT-5.5 scales across a suite of benchmarks. Having these lower tiers gives you way more control:</p>\n<figure>\n<img src=\"content/notes/media/048-agentic-reasoning-effort-benchmarks.png\" alt=\"GPT-5.5 benchmark scores across reasoning efforts\">\n</figure>\n\n<p>Sure, GPT-5.5's <code>xhigh</code> and <code>high</code> settings are quite similar as well. However, the lower tiers are important for balancing quality, API costs, and response times. You can see how the reasoning chain lengths scale across different configurations:</p>\n<figure>\n<img src=\"content/notes/media/048-agentic-reasoning-effort-chain-lengths.png\" alt=\"Reasoning chain length comparison\">\n</figure>\n\n<p>It looks like we'll have to wait for DeepSeek-V4.1+ to introduce <code>medium</code> / <code>low</code> reasoning efforts before we see any meaningful differentiation. I’d say <code>medium + max</code> is an ideal combo.</p>" |
| }, |
| "rd-coding-benchmark": { |
| "page": 44, |
| "type": "Eval note", |
| "title": "Coding R&D benchmarks", |
| "rects": [ |
| { |
| "left": 63.27, |
| "top": 20.9, |
| "width": 25.09, |
| "height": 1.65 |
| }, |
| { |
| "left": 11.7, |
| "top": 22.51, |
| "width": 62.69, |
| "height": 1.65 |
| } |
| ], |
| "html": "<p>DeepSeek aside, this is an insane amount of progress on real-world engineering tasks. Sonnet 4.5 came out on September 29, 2025, and Opus 4.6 dropped on February 5, 2026. In just four months, the success rate jumped from 50% to 80%.</p>\n<p>And keep in mind, these aren't trivial problems. We're talking about complex R&D tasks typically handled by a top-tier engineering team.</p>\n<p>It's only going to accelerate from here 🚀</p>" |
| } |
| }; |
| window.annotationBuildInfo = {"source": "content/notes", "pdf": "/Users/seeall/Downloads/DeepSeek_V4.pdf", "notes": 50}; |
|
|
