LaTeX tab: collapse to 2 blocks; expand ablations with per-split table and commentary

app.py

@@ -329,9 +329,22 @@ LATEX_TABLE_UNDERSIZED = r"""% tab:undersized-wins — SoRL vs SFT on undersized
 \end{table}
 """
 
-LATEX_APPENDIX = r"""% ── Appendix: Arithmetic interpretability — Findings #2–5 ──────────────────
-% Requires: tcolorbox, booktabs, xcolor, multirow
-% Put \providecommand{\sorl}{\textsc{DLR}} and \providecommand{\sft}{\textsc{SFT}} in preamble.
+LATEX_APPENDIX = r"""% ═══════════════════════════════════════════════════════════════════════════
+% APPENDIX — Arithmetic case study (Findings #2–5 + training details)
+% ═══════════════════════════════════════════════════════════════════════════
+% Paste this entire block into your appendix .tex file.
+%
+% Required packages:
+%   \usepackage{tcolorbox}
+%   \usepackage{booktabs}
+%   \usepackage{xcolor}
+%   \usepackage{multirow}
+%   \usepackage{enumitem}   % for [nosep]
+%
+% Required macros (put in preamble if not already defined):
+%   \providecommand{\sorl}{\textsc{DLR}}
+%   \providecommand{\sft}{\textsc{SFT}}
+% ═══════════════════════════════════════════════════════════════════════════
 
 % ── Finding box macro (put in preamble) ──────────────────────────────────────
 % \usepackage{tcolorbox}
@@ -357,53 +370,90 @@ Experimental code: \texttt{arithmetic/experiments/} (all results reproducible).
 \label{app:causal}
 
 To confirm that \sorl{} tokens are causally necessary (not merely correlated
-with correct outputs), we run three intervention conditions:
+with correct outputs), we run three intervention conditions on model
+\texttt{2L/1H/128d} (100K), evaluated across 2{,}600 held-out problems:
 
 \begin{itemize}[nosep]
-  \item \textbf{Knockout}: replace every abstraction token with a single
-    fixed \texttt{[UNK]} embedding.
   \item \textbf{Shuffle}: randomly permute all abstraction tokens within
-    each sequence (preserving the token identities but breaking positional
-    assignment).
-  \item \textbf{Random}: replace each abstraction token with a token drawn
-    uniformly at random from the codebook.
+    each sequence (token identities preserved; positional assignment destroyed).
+  \item \textbf{Random}: replace each token with a draw uniform over the
+    30-token codebook (identity and position both destroyed).
+  \item \textbf{Knockout}: replace every token with a fixed \texttt{[UNK]}
+    embedding (strongest intervention; removes all information).
 \end{itemize}
 
+Table~\ref{tab:ablation-splits} shows per-split accuracy under each condition.
+
 \begin{table}[h]
   \centering\small
-  \begin{tabular}{lrr}
+  \begin{tabular}{llrrrr}
     \toprule
-    Condition & Accuracy & Drop \\
+    Family & Split & Baseline & Shuffle & Random & Knockout \\
+    \midrule
+    \multirow{4}{*}{\textit{Addition (easy)}} & S0 (no carry) & 100\% & 24\% & 28\% & 0\% \\
+     & S1 & 100\% & 17\% &  9\% & 0\% \\
+     & S2 & 100\% & 22\% & 10\% & 0\% \\
+     & random & 100\% & 26\% &  8\% & 0\% \\
+    \midrule
+    \multirow{4}{*}{\textit{Addition cascade (hard)}} & C3 (3-deep) & 96\% & 28\% & 14\% & 0\% \\
+     & C4 (4-deep) & 99\% & 25\% & 13\% & 0\% \\
+     & C5 (5-deep) & 99\% & 23\% & 19\% & 0\% \\
+     & C6 (6-deep) & 97\% & 27\% & 15\% & 0\% \\
+    \midrule
+    \multirow{1}{*}{\textit{Subtraction (easy)}} & random & 100\% & 46\% & 12\% & 0\% \\
+    \midrule
+    \multirow{3}{*}{\textit{Subtraction cascade (hard)}} & M3 (3-deep borrow) & 100\% & 22\% &  1\% & 0\% \\
+     & M4 (4-deep borrow) &  85\% &  6\% &  0\% & 0\% \\
+     & M5 (5-deep borrow) &  57\% &  3\% &  0\% & 2\% \\
     \midrule
-    Baseline (correct tokens) & 95.5\% & --- \\
-    Shuffle                   & 26.6\% & $-68.9$\,pp \\
-    Random                    & 12.3\% & $-83.2$\,pp \\
-    Knockout                  &  0.1\% & $-95.4$\,pp \\
+    \multicolumn{2}{l}{\textbf{Overall}} & \textbf{95.5\%} & 26.6\% & 12.3\% & 0.1\% \\
     \bottomrule
   \end{tabular}
-  \caption{Causal ablation on \texttt{2L/1H/128d} (2{,}600 examples).
-    Even shuffling — which preserves the token distribution but breaks
-    positional assignment — collapses accuracy by 69\,pp.}
-  \label{tab:causal-ablation}
+  \caption{Per-split causal ablation (\texttt{2L/1H/128d}, 100K training examples).
+    \textbf{Shuffle} preserves token identity but destroys positional assignment;
+    \textbf{Random} destroys both; \textbf{Knockout} removes all token information.
+    Generated by \texttt{paper/results/result\_ablation\_splits/run.py}.}
+  \label{tab:ablation-splits}
 \end{table}
 
-The 95\,pp knockout drop confirms that the model has offloaded the
-arithmetic computation onto its abstraction tokens: without them,
-it cannot solve the task. Shuffle being worse than random may seem
-counterintuitive; it reflects that wrong tokens at specific positions
-(e.g.\ a cascade position receiving a simple-add token) cause systematic
-one-off errors, while random tokens sometimes happen to match.
+\paragraph{Commentary.}
+Knockout reduces accuracy to $\leq$2\% on every split, confirming that
+the model has offloaded computation into the routing tokens.
+Three patterns are notable:
+
+\begin{itemize}[nosep]
+  \item \textbf{Shuffle $>$ Random on easy splits.}
+    On addition S0 (no carry), shuffle yields 24\% vs.\ random's 28\%;
+    the gap is small and reflects that no single position is critical —
+    any token in any position is roughly as bad as another.
+  \item \textbf{Shuffle $>$ Random on cascade splits (C3--C6).}
+    On 4--6-deep carry cascades, shuffle (23--28\%) consistently
+    outperforms random (13--19\%).
+    When tokens are shuffled, a cascade position receives a token
+    from another cascade position — a wrong token but from the
+    ``right family'', producing a systematic one-off carry error.
+    Random tokens provide no structural signal at all, making
+    cascade resolution impossible.
+  \item \textbf{Borrow cascades are uniquely sensitive.}
+    Sub-M4 (4-deep borrow) drops from 85\% baseline to 6\% under
+    shuffle and 0\% under random — a 79\,pp collapse from shuffle
+    alone. Sub-M5 (5-deep borrow) is hardest even at baseline (57\%),
+    and all ablations reduce it to $\leq$3\%, showing that deep
+    borrow cascades are the single hardest regime and that \sorl{}
+    tokens are essential for solving them.
+\end{itemize}
 
 \begin{tcolorbox}[colback=gray!6, colframe=gray!40,
   fonttitle=\bfseries\small, title={Finding \#2},
   left=5pt, right=5pt, top=4pt, bottom=4pt]
 \small
-\sorl{} abstraction tokens are causally necessary for correct six-digit
-arithmetic: replacing all tokens with a fixed mask reduces accuracy from
-95.5\% to 0.1\%, and shuffling tokens within a sequence (preserving
-token identity but destroying positional assignment) drops accuracy
-by 69\,pp. The model has externalised the carry/borrow computation
-into its routing tokens.
+\sorl{} abstraction tokens are causally necessary: knockout collapses
+accuracy from 95.5\% to 0.1\% overall, and to $\leq$3\% on the hardest
+borrow-cascade splits (M4--M5).
+Shuffle (identity-preserving, position-destroying) is more harmful than
+random on cascade splits — wrong-position tokens from the same structural
+family cause systematic carry errors, while random tokens cause broader
+incoherence.
 \end{tcolorbox}
 
 % ─────────────────────────────────────────────────────────────────────────────
@@ -1205,23 +1255,14 @@ hidden activations — but here it is readable directly from the token sequence.
         # ── Tab 3: LaTeX Scratchpad ──
         with gr.TabItem("LaTeX"):
             gr.Markdown("### LaTeX Scratchpad — copy sections directly into Overleaf")
-            gr.Markdown("Each block is a self-contained chunk. Click the copy icon top-right of each box.")
+            gr.Markdown("Two blocks: **main body** (setup + Finding #1 + forward ref) and **appendix** (Findings #2–5, all tables, training details). Copy each into the appropriate place in your paper.")
 
-            gr.Markdown("#### § Arithmetic setup + Quirke subtask definitions")
+            gr.Markdown("#### § Main body — §Arithmetic setup (paste into paper body)")
             gr.Code(value=LATEX_ARITHMETIC_SETUP, label="arithmetic_setup.tex",
                     language=None, interactive=False)
 
-            gr.Markdown("#### § Results table — SoRL vs baseline on undersized architectures")
-            gr.Markdown("Requires: `\\usepackage{booktabs}`, `\\usepackage{xcolor}`.")
-            gr.Code(value=LATEX_TABLE_UNDERSIZED, label="tab_undersized_wins.tex",
-                    language=None, interactive=False)
-
-            gr.Markdown("#### § Carry-cascade example figure (TikZ)")
-            gr.Markdown("Requires: `\\usepackage{tikz}`, `\\usetikzlibrary{matrix}`, `\\usepackage{xcolor}`, and `\\providecommand{\\sorl}{\\textsc{DLR}}`.")
-            gr.Code(value=LATEX_FIGURE_EXAMPLE, label="fig_arithmetic_example.tex",
-                    language=None, interactive=False)
-
-            gr.Markdown("#### § Appendix — full experimental details")
+            gr.Markdown("#### § Appendix — full analysis (paste into appendix)")
+            gr.Markdown("Includes: per-split ablation table, token profiles, guided computation, Quirke analogy, training details. Requires `tcolorbox`, `booktabs`, `xcolor`, `multirow`, `enumitem`.")
             gr.Code(value=LATEX_APPENDIX, label="appendix_arithmetic.tex",
                     language=None, interactive=False)