The Lie Algebra of Neural Network Training

Linear neural networks under SGD-with-momentum produce a richer Poisson algebra at gradient-product coupling than any pairwise potential in our 16-potential survey, and split into seven distinct universality classes overall.

Train a 3-layer linear network with SGD and momentum. The training dynamics form a Hamiltonian system — weights are coordinates, velocity buffers are momenta, and the loss landscape defines pairwise interactions between layers. Compute the Lie algebra generated by these interactions via iterated Poisson brackets.

You get [3, 6, 17, 119].

Now replace the network with three gravitating bodies — stars, atoms, black holes — interacting via $1\mathrm{/}r1/r$. You get [3, 6, 17, 116].

The neural network produces 3 extra generators. It breaks a universality that holds across every pairwise potential $V(r)V(r)$ in our 16-potential physical survey — from $1\mathrm{/}r1/r$ through $r^{10}r^\{10\}$, $\log r\backslash log\; r$, Yukawa $e^{-\mu r}\mathrm{/}re^\{-\backslash mu\; r\}/r$, and the Calogero-Moser $1\mathrm{/}{r}^{2}1/r^2$ — and across all spatial dimensions, mass ratios, and charge configurations tested. (Other neural couplings instead produce smaller algebras; see the seven-class table below.)

This article introduces bshepp/pairwise-poisson-algebras — the first systematic computation of Poisson bracket Lie algebras for neural network training dynamics. The dataset contains 993 rows across 13 Parquet tables, including a dedicated neural_algebras split with 21 configurations sweeping network depth (L=2..5), width (k=1..3), 12 coupling types, loss function, and activation function — revealing seven distinct neural universality classes at L=3, versus the single physical class at [3, 6, 17, 116].

Table of Contents

• The Lie Algebra of Neural Network Training
  • Table of Contents
  • SGD as a Hamiltonian System
    • Coupling Types
  • Seven Universality Classes in Neural Algebras
    • The structure of the extras
  • Width Invariance
  • Loss Function: Partial Invariance
  • Activation Invariance
  • Depth Scaling: L=3 Is an Accidental Match
  • The Physical Benchmark: 116 vs 119
  • Dataset Overview
  • Quick Start
  • Open Problems
  • Citation
  • Links

SGD as a Hamiltonian System

Consider a linear network $f(x)=w_3{w}_2{w}_{1}xf(x)\; =\; w\_3\; w\_2\; w\_1\; x$ trained on a single data point $(x,t)=(1,1)(x,\; t)\; =\; (1,\; 1)$ with MSE loss:

$$L=\frac{1}{2}(w_1{w}_2{w}_3-1{)}^{2}L\; =\; \backslash frac\{1\}\{2\}(w\_1\; w\_2\; w\_3\; -\; 1)^2$$

SGD with momentum updates weights $w_{i}w\_i$ and velocity buffers $v_{i}v\_i$:

$$v_i\leftarrow \mu v_i-\eta \frac{\mathrm{\partial }L}{\mathrm{\partial }{w}_i},w_i\leftarrow w_i+v_{i}v\_i\; \backslash leftarrow\; \backslash mu\; v\_i\; -\; \backslash eta\; \backslash frac\{\backslash partial\; L\}\{\backslash partial\; w\_i\},\; \backslash quad\; w\_i\; \backslash leftarrow\; w\_i\; +\; v\_i$$

In the continuous-time limit, this is Hamilton's equations with $w_{i}w\_i$ as positions and $v_{i}v\_i$ as conjugate momenta. The phase space is $2L2L$-dimensional for an $LL$-layer network.

The key insight: the loss couples all weights simultaneously, but we can decompose this into pairwise interactions between weight layers — exactly as gravitational N-body dynamics decomposes into pairwise forces. Each pair $(i,j)(i,\; j)$ gets a Hamiltonian:

$$H_{ij}=\frac{v_i^2}{2}+\frac{v_j^2}{2}+V_{ij}(w_1,\dots ,w_L)H\_\{ij\}\; =\; \backslash frac\{v\_i^2\}\{2\}\; +\; \backslash frac\{v\_j^2\}\{2\}\; +\; V\_\{ij\}(w\_1,\; \backslash ldots,\; w\_L)$$

where $V_{ij}V\_\{ij\}$ captures how layers $ii$ and $jj$ interact through the loss. The choice of $V_{ij}V\_\{ij\}$ defines the coupling type.

Coupling Types

The dataset sweeps twelve ways to extract pairwise interactions from the loss. Here are the four canonical ones; the full set is summarized in the next section's universality table.

The remaining eight (gradient_sum, gradient_abs, hessian_plain, hessian_full, directional, natural_gradient, loss_power, gradient_cubic) probe variations in symmetrization, normalization, and degree — and as we will see, every variation that changes the polynomial structure of $V_{ij}V\_\{ij\}$ at high degree produces a new universality class.

The Lie algebra is then generated by all $\{H_{ij},H_{kl}\}\backslash lbrace\; H\_\{ij\},\; H\_\{kl\}\backslash rbrace$ under the canonical Poisson bracket, exactly as in the gravitational N-body problem.

Seven Universality Classes in Neural Algebras

The headline discovery: different coupling types produce different algebras. A sweep of 12 pairwise couplings and 3 loss functions reveals at least seven distinct universality classes at L=3.

The physical universality class at N=3 is [3, 6, 17, 116] — it sits between classes A (119) and B (115), and above all others in level-3 dimension. Classes A–F all match physics at levels 0–2 but diverge at level 3. Class G diverges already at level 1, producing only 5 independent generators (vs 6 in all other classes).

A key methodological note: the cleaner Fisher formulation $V_{ij}=g_i{g}_j(1-L)V\_\{ij\}\; =\; g\_i\; g\_j\; (1\; -\; L)$ used here places Fisher in Class A; an earlier polynomial-truncated variant landed in its own class. Small choices in how the coupling polynomial is normalized — what gets divided by what, where the loss enters multiplicatively — can move a coupling between classes. The 12-coupling sweep is designed to map this fine-grained landscape rather than treat it as noise.

The structure of the extras

The neural Class A algebra [3, 6, 17, 119] and the physical algebra [3, 6, 17, 116] have identical dimension counts at levels 0, 1, 2 but evaluate to completely different polynomials in phase space. In a shared 6D phase space (3 weights vs 3 positions), the rank of the concatenated 156+156 matrix equals 116 + 119 = 235 — every physical generator and every neural generator contributes an independent direction. The comparison is therefore structural (count vs count) rather than literal subalgebra.

Polynomial-degree analysis reveals the source of the "3 extra":

• Physical $(q-q{)}^4(q-q)^4$ algebra: level-3 generators at degrees 6, 8, 10 with ranks 38, 95, 116. The top-degree (10) stratum contributes $116-95=21116-95\; =\; 21$ new dimensions from 21 candidates (no syzygies).
• Neural gradient algebra: level-3 generators at degrees 18, 26, 34 with ranks 32, 92, 119. The top-degree (34) stratum contributes $119-92=27119-92\; =\; 27$ new dimensions from 30 candidates (3 syzygies).

The 3 extra neural directions at level 3 are specifically at the highest polynomial-degree stratum, where neural generators have 3 fewer linear dependencies than their physical counterparts.

Width Invariance

Does the dimension of weight space matter? We test networks with scalar weights $k=1k=1$, 2D weights $k=2k=2$, and 3D weights $k=3k=3$:

The algebra is independent of weight dimension. This is the neural network analog of a remarkable property in physics: the pairwise Poisson algebra for $NN$ gravitating bodies is independent of the spatial dimension $dd$. Gravity in 1D, 2D, and 3D all produce [3, 6, 17, 116]. Neural networks with scalar, vector, and matrix weights all produce [3, 6, 17, ...].

Loss Function: Partial Invariance

Does the loss function matter? We test gradient-product coupling with three loss functions:

† The "L1" row uses a smooth quadratic proxy rather than the non-differentiable $\mid f-t\mid \backslash lvert\; f-t\; \backslash rvert$ itself; the polynomial structure of the coupling is what enters the algebra computation.

MSE and L1 give the identical algebra; cross-entropy does not. The nonlinear $z^{4}z^4$ term in the cross-entropy Taylor expansion contributes new monomials that change the level-3 structure dramatically, dropping from 119 to 62 independent generators — an entirely new universality class.

The MSE/L1 case is the neural analog of potential universality: quadratic losses behave like generic polynomial potentials $r^{n}r^n$ with $n\ge 4n\; \backslash ge\; 4$. Cross-entropy is the neural analog of an exceptional potential like $r^{3}r^3$ — it has enough special polynomial structure to produce its own class.

Activation Invariance

Does nonlinearity matter? We test gradient-product coupling with linear, tanh (Taylor truncation), and ReLU (softplus approximation) activations:

The algebra is independent of activation function through level 2. The Taylor approximations used here capture the leading nonlinear corrections; whether deeper levels also match is an open question requiring higher-order expansions or numerical methods.

Depth Scaling: L=3 Is an Accidental Match

As network depth increases, the neural algebra diverges from the physical N-body algebra at progressively earlier levels:

The L=3 match at levels 0-2 is essentially accidental. At L=3 the neural and physical algebras both have the same level-0, level-1, and level-2 counts (3, 6, 17), and the divergence appears only at level 3 as the 3 extra generators. But for L >= 4, the algebras diverge at level 1 already — neural L=4 produces 14 new generators at level 1 versus 8 new for physical N=4.

The extras at level 1 follow an exact pattern:

$${\text{extras}}_{L,1}=\left(\genfrac{}{}{0px}{}{L}{2}\right)(L-3)\backslash text\{extras\}\_\{L,1\}\; =\; \backslash binom\{L\}\{2\}(L\; -\; 3)$$

This gives 0, 6, 20 for $L=3,4,5L\; =\; 3,\; 4,\; 5$ — matching the observed data. The formula vanishes at $L=3L=3$, explaining why level 1 matches physics there but nowhere else.

For comparison, the physical N-body problem with $NN$ bodies in 1D has closed-form scaling formulas:

$$d_0(N)=\left(\genfrac{}{}{0px}{}{N}{2}\right),d_1(N)=\frac{N(3N-5)}{2},d_2(N)=\frac{N(4N^2-9N+3)}{2}d\_0(N)\; =\; \backslash binom\{N\}\{2\},\; \backslash quad\; d\_1(N)\; =\; \backslash frac\{N(3N-5)\}\{2\},\; \backslash quad\; d\_2(N)\; =\; \backslash frac\{N(4N^2\; -\; 9N\; +\; 3)\}\{2\}$$

Whether the neural depth scaling follows similar closed-form formulas for $L\ge 4L\; \backslash ge\; 4$ is an open question, though the linear formula for level-1 extras is now established.

The Physical Benchmark: 116 vs 119

The physical universality class at $N=3N\; =\; 3$ gives [3, 6, 17, 116] for:

• Newtonian gravity $1\mathrm{/}r1/r$
• Coulomb potential $1\mathrm{/}r1/r$ (identical)
• Calogero-Moser $1\mathrm{/}{r}^{2}1/r^2$
• Dipole-dipole $1\mathrm{/}{r}^{3}1/r^3$
• Logarithmic vortex $\log r\backslash log\; r$
• Yukawa nuclear force $e^{-\mu r}\mathrm{/}re^\{-\backslash mu\; r\}/r$
• GUE eigenvalue dynamics (random matrix theory)
• All polynomial $r^{n}r^n$ with $n\ge 4n\; \backslash ge\; 4$
• 576 exponent values across the continuous landscape
• 20 named physical systems (helium to triple black holes)
• All mass ratios from 1 to ${10}^{10}10^\{10\}$
• All charge configurations tested

The neural gradient-product coupling gives [3, 6, 17, 119]. The structure of the 3-dimensional "gap" has been characterized by polynomial-degree stratification:

The 3 extra generators appear specifically at the top-degree stratum of level 3. Physics has 21 independent top-degree generators out of 21 candidates (no syzygies). Neural has 27 independent top-degree generators out of 30 candidates (only 3 syzygies). Neural simply has 3 fewer linear relations at the polynomial frontier.

The dimension counts are identical at levels 0, 1, 2; the neural and physical generators occupy entirely different polynomial subspaces (the combined rank is $116+119=235116+119=235$), so this is a structural-count rather than literal-containment relation.

Dataset Overview

The full dataset contains 993 rows across 13 Parquet tables:

Quick Start

from datasets import load_dataset

# Load neural network algebras
ds = load_dataset("bshepp/pairwise-poisson-algebras", "neural_algebras")
df = ds["train"].to_pandas()

# Compare coupling types
for _, row in df[df["n_layers"] == 3].iterrows():
    print(f"{row['coupling_type']:12s} {row['loss_function']:15s} "
          f"{row['activation']:12s} -> {row['dimension_sequence']}")

# Compare neural vs physical dimension sequences
import json

# Neural algebras
nn = load_dataset("bshepp/pairwise-poisson-algebras", "neural_algebras")["train"].to_pandas()
gradient_l3 = nn[(nn["n_layers"] == 3) & (nn["coupling_type"] == "gradient")
                  & (nn["activation"] == "linear") & (nn["loss_function"] == "mse")]
nn_dims = json.loads(gradient_l3.iloc[0]["dimension_sequence"])

# Physical systems (helium has the universal physical algebra [3, 6, 17, 116])
phys = load_dataset("bshepp/pairwise-poisson-algebras", "physical_systems")["train"].to_pandas()
gravity_dims = json.loads(phys[phys["system_name"] == "helium"].iloc[0]["dimension_sequence"])

print(f"Neural (gradient):  {nn_dims}")
print(f"Physics (helium):   {gravity_dims}")
print(f"Extra generators:   {nn_dims[3] - gravity_dims[3]}")

# Discover universality classes
from collections import defaultdict
classes = defaultdict(list)
for _, row in nn.iterrows():
    dims = row["dimension_sequence"]
    classes[dims].append(f"{row['coupling_type']} / {row['loss_function']} / {row['activation']}")

for dims, configs in sorted(classes.items()):
    print(f"\n{dims}:")
    for c in configs:
        print(f"  {c}")

Open Problems

1. Exact neural rank for the remaining classes at level 3: The gradient-product [3, 6, 17, 119] result has been verified exactly over $\mathbb{Q}\backslash mathbb\{Q\}$ (neural/nn_poisson.py). The other six classes currently rely on numerical SVD over 600+ phase-space samples, with a singular-value gap of $\sim 5\times {10}^8\backslash sim\; 5\; \backslash times\; 10^8$ at the cutoff, which is strong but not symbolic. Confirming each class with exact rational arithmetic would close the loop.

2. Does activation break universality at level 3? Linear, tanh (Taylor), and ReLU (softplus) all give [3, 6, 17] through level 2. The linear case extends to 119 at level 3; verifying tanh and ReLU at level 3 is computationally difficult due to polynomial blow-up.

3. The seven classes: What operation on couplings governs the class? Hessian vs gradient vs gradient_sum are all natural choices, yet they produce 47, 119, 111. Is there a classification theorem for neural universality classes analogous to the physical $r^{n}r^n$ classification?

4. Depth scaling formulas: The extras-at-level-1 formula $\left(\genfrac{}{}{0px}{}{L}{2}\right)(L-3)\backslash binom\{L\}\{2\}(L-3)$ is empirical and exact for $L=3,4,5L=3,\; 4,\; 5$. A closed form for the full dimension sequence as a function of $LL$ (and for level 2, 3 extras) would generalize the physical scaling laws.

5. Nonlinear networks: Full (non-Taylor) nonlinear activations create non-polynomial dynamics. Can the algebra be computed for genuine ReLU or sigmoid networks via asymptotic or numerical methods?

6. The 3 extra generators at L=3: They appear at the highest polynomial-degree stratum (degree 34), where neural has 3 fewer syzygies than physics at degree 10. What is the algebraic origin of these missing relations?

7. Converse question: Are any of the seven classes (119, 115, 111, 104, 87, 47) producible by some physical potential? Or do they define genuinely new algebraic structures with no physical realization? The log-gas/GUE algebra matches physical 116 — are there other "hidden" physical systems in these classes?

8. Connection to loss landscape geometry: The coupling type determines the universality class. Gradient-product measures force co-alignment; Hessian measures curvature; directional couples to velocity. How does this connect to saddle-point structure, mode connectivity, and the edge of stability in real networks?

9. Wider networks (matrix weights): Width-1, 2, 3 scalar weights all give the same algebra at L=3. Does this extend to full matrix-valued weight layers in realistic architectures, or does the symmetry break at matrix widths where weight-space ordering becomes non-trivial?

Citation

@dataset{sheppard2026poisson,
  title={Pairwise Poisson Algebras of the N-Body Problem},
  author={Sheppard, Brian},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/bshepp/pairwise-poisson-algebras}
}

Links

• Dataset: bshepp/pairwise-poisson-algebras
• Code: github.com/bshepp/3body-poisson-algebra
• Physics-focused article: hf_article_physics.md — the full physical universality story across 16 potentials, 576-exponent landscape, GUE connection, and the three exceptions