README.md

---
license: mit
---
CEAU: Learning Algebraic Representations for Cubic Equation Solving

CEAU is a research neural network designed to solve cubic equations with three real roots while enabling direct analysis of the mathematical structures that emerge inside the model.

The original project started from a much simpler question: can a neural network learn to solve quadratic equations without using a symbolic solver, a closed-form root formula, or an external calculator?

Once the network could perform the task, a more interesting question emerged:

What has the network actually learned internally?

The project was then extended to cubic equations,

$$
ax^3+bx^2+cx+d=0,
$$

with training examples constructed so that all three real roots were known exactly.

The current experiments suggest that the trained model develops internal representations correlated with algebraic quantities associated with the classical reduction of a cubic equation, including

$$
p,\quad q,\quad r,
$$

$$
P=q-\frac{p^2}{3},
$$

$$
Q=\frac{2p^3}{27}-\frac{pq}{3}+r,
$$

$$
U=\sqrt{-\frac{P}{3}},
$$

and, particularly persistently,

$$
\frac{Q}{U^3}.
$$

These observations are interpretability results. They do not establish that the network independently rediscovered Cardano's formula.

Model Summary

The current best checkpoint is a 6-layer cubic solver with approximately 4.49 million parameters.

The model receives representations of the coefficients of a cubic equation and predicts its three real roots.

Its central architectural component is a custom CubicLayer rather than a standard linear layer followed by a conventional activation.

The layer first computes a learned affine projection,

$$
z=Wx+b,
$$

and then applies a learned cubic polynomial transformation,

$$
Az^3+Bz^2+Cz+D.
$$

This provides the network with an explicit mechanism for constructing polynomial nonlinear transformations of learned projections.

The experiments also showed that simply increasing model size does not necessarily improve performance. A much larger eight-layer model with roughly 20.7 million parameters performed worse than the more compact model used for the main experiments.
Mathematical Formulation

The task is to solve

$$
ax^3+bx^2+cx+d=0
$$

for three real roots.

Instead of generating arbitrary coefficients and then solving each equation numerically to obtain targets, the dataset is constructed in the opposite direction.

Three real roots

$$
x_1,x_2,x_3
$$

are sampled first, and the coefficients are generated using Vieta's relations:

$$
b=-a(x_1+x_2+x_3),
$$

$$
c=a(x_1x_2+x_1x_3+x_2x_3),
$$

$$
d=-ax_1x_2x_3.
$$

This guarantees that every generated example corresponds to a mathematically consistent cubic equation whose exact roots are known by construction.

This formulation is important for the interpretability experiments because the network is trained on a controlled mathematical distribution rather than arbitrary coefficient-target pairs.

Dataset

The dataset is synthetic.

For each example:

three real roots are sampled;
a non-zero leading coefficient is selected;
the remaining coefficients are constructed using Vieta's relations;
the known roots are used as the target.

The current experiment focuses on cubic equations with three real roots.

This restriction was intentional: it creates a controlled environment in which the network can be studied while the underlying algebraic structure is known.

The current report does not establish model behavior outside this three-real-root regime.

CubicLayer

The main architectural idea is the custom CubicLayer.

Conceptually, the layer performs

z = W @ x + b
y = A * z**3 + B * z**2 + C * z + D

where the polynomial coefficients are learned parameters.

In mathematical notation,

$$
z=Wx+b,
$$

followed by

$$
y=Az^3+Bz^2+Cz+D.
$$

This differs from a conventional Linear + GELU architecture because the nonlinear transformation is explicitly polynomial.

That makes the architecture particularly interesting for a study of whether hidden activations align with algebraic structures.

The main checkpoint contains six cubic stages and has approximately 4.49 million parameters.
Evaluation

The best checkpoint achieved the following results on an independent test:

Metric	Result
Parameters	~4.49M
MAE	0.221891
RMSE	0.382195

Earlier evaluation produced:

$$
MAE=0.2157
$$

on a saved test evaluation, while a fresh independent evaluation produced

$$
MAE=0.2219
$$

and

$$
RMSE=0.3822.
$$

A previous baseline achieved

$$
MAE=0.2425.
$$

An important observation was that the smaller model outperformed the much larger model tested earlier. This suggests that simply increasing parameter count was not sufficient to improve the task in this experimental setup.

Interpretability Analysis

After training the model, a separate analyzer was used to inspect hidden activations.

The analyzer did not train the network further. Instead, it loaded a checkpoint and examined the activations produced by individual neurons across the cubic layers.

The central question was:

Are the hidden layers merely implementing an opaque nonlinear mapping from coefficients to roots, or are they constructing intermediate representations that resemble meaningful mathematical quantities?

The analysis compared neuron activations with algebraically meaningful candidate structures.

Observed Representation Structure

The most interesting result was a qualitative progression across the network.

First Cubic Layer

In the first cubic layer, individual neurons were found to be strongly aligned with the original variables.

One neuron reached approximately

$$
|corr|\approx0.9992
$$

with the representation corresponding to (p).

Other neurons showed similarly strong correspondence with (q) and (r).

These relationships were also observed on independent probe datasets.

This suggests that the early layer forms relatively clean representations of the original mathematical variables.

Second Cubic Layer

The second layer was more interesting.

Here, nonlinear algebraic structures began to appear:

$$
P=q-\frac{p^2}{3},
$$

$$
Q=\frac{2p^3}{27}-\frac{pq}{3}+r,
$$

and

$$
U=\sqrt{-\frac{P}{3}}.
$$

Representative neurons reached approximately

$$
R^2\approx0.969
$$

and

$$
R^2\approx0.968
$$

with candidate mathematical quantities.

The same general behavior appeared on independent probes.

Third to Sixth Cubic Layers

The most persistent structure found in the deeper layers was

$$
\frac{Q}{U^3}.
$$

Several neurons in the third layer showed correlations around 0.80 with this quantity.

In the fourth layer, one strong neuron reached approximately

$$
|corr|\approx0.848.
$$

Related representations also appeared in later layers.

The resulting picture looked like a rough computational hierarchy:

$$
p,q,r
$$

↓

$$
P,U,Q
$$

↓

$$
\frac{Q}{U^3}
$$

The same family of structures appeared across different layers and independent probe datasets rather than being tied to a single neuron or a single random sample.

Blind Symbolic Distillation

The next step was to ask whether the network's outputs themselves could be described by relatively small algebraic expressions.

Rather than providing the model with a known closed-form solution, the analysis attempted to fit compact symbolic expressions directly to the outputs of the trained network.

For the middle root (x_2), one particularly stable approximation was

$$
x_2\approx
0.0873
-0.3367p
+0.0004349Q
+3.8113\frac{Q}{U^3}.
$$

On validation, this approximation reached approximately

$$
R^2\approx0.9914.
$$

The result remained similar across five independent probe datasets.

For (x_1) and (x_3), the fit was somewhat weaker:

$$
R^2\approx0.937.
$$

However, similar algebraic ingredients appeared again, including

$$
p,\quad pq,\quad U.
$$

This was important because the same structures were not only visible in hidden activations; they also appeared when attempting to distill the behavior of the trained network into compact algebraic expressions.

Centering the Roots

The next experiment explicitly removed the translational shift associated with the quadratic term.

Instead of working directly with (x), the analysis considered

$$
y=x+\frac{p}{3}.
$$

After centering the roots, the quality of the symbolic approximations increased substantially.

The validation (R^2) for centered roots reached approximately

$$
0.978-0.984.
$$

Stable expressions involving (U), (Q), and (Q/U^3) repeatedly appeared.

One example was

$$
y\approx
-1.64U
-0.00022Q
-1.9\frac{Q}{U^3}.
$$

The fitted coefficients remained similar across different probe datasets.

The centering experiment therefore made the algebraic structure easier to capture and inspect.

Does This Mean the Network Discovered Cardano's Formula?

Not proven.

The observed internal representations show a strong resemblance to mathematical structures used in the analytical treatment of cubic equations.

In particular, the network exhibits representations associated with

$$
P=q-\frac{p^2}{3},
$$

$$
Q=\frac{2p^3}{27}-\frac{pq}{3}+r,
$$

$$
U=\sqrt{-\frac{P}{3}},
$$

and especially

$$
\frac{Q}{U^3}.
$$

However, an important limitation remains.

Some of these quantities were explicitly included in the candidate library used for the symbolic analysis.

Therefore, the current experiment does not demonstrate that the network independently invented Cardano's formula from scratch.

A more precise interpretation is:

The trained network contains stable internal representations that are statistically aligned with important algebraic structures used in the analytical solution of cubic equations.

This is evidence for mathematically meaningful internal structure, but it is not proof of autonomous symbolic discovery.

Ablation Study

The analysis also investigated whether the discovered representations were actually involved in the network's computation.

When selected neurons from the sixth layer were removed, the MAE increased dramatically to approximately

$$
5.78.
$$

The fifth layer also showed a substantial change, while earlier layers produced different levels of sensitivity.

These results do not prove that a particular neuron implements Cardano's formula.

They do, however, provide evidence that some of the identified internal representations are connected to the computation actually used by the network rather than being purely accidental correlations.

Error Behaviour Near Repeated Roots

Another clear pattern appeared when prediction error was analyzed as a function of the distance between the true roots.

Minimum root gap	MAE
0–0.5	0.715
0.5–1	0.616
1–2	0.428
2–5	0.323
5–10	0.218
≥10	0.115

The correlation between error and minimum root separation was

$$
-0.4355.
$$

Thus, in the current experiment, the network has substantially greater difficulty when roots become close to one another.

This regime is also mathematically interesting because small perturbations in the coefficients can lead to larger changes in the individual roots when the roots approach each other.

Limitations

Several limitations are important when interpreting the current results.

Correlation Is Not Proof of Symbolic Computation

A strong correlation between a neuron and a mathematical expression does not prove that the neuron explicitly computes that expression.

Candidate Feature Bias

Some of the most interesting quantities, including (P), (Q), (U), and (Q/U^3), were already included in the candidate feature library.

This means the current symbolic analysis is not a completely blind discovery experiment.

Restricted Data Distribution

The experiments focus on synthetic cubic equations with three real roots.

Generalization beyond the experimental distribution has not yet been established.

Preliminary Evaluation Protocol

The project used multiple independently generated evaluations and probes, but a stronger reproducibility protocol would require fixed train, validation, and test datasets together with fixed random seeds.

Interpretability Remains Exploratory

The current analysis relies heavily on neuron-level correlations and targeted symbolic fitting.

A stronger mechanistic interpretation would require systematic symbolic regression, reproducibility across independently trained models, and causal interventions.

No Claim of Autonomous Cardano Discovery

The current evidence supports a connection between learned representations and the algebra of cubic solving.

It does not establish that the network independently rediscovered Cardano's method.

Future Research

The most important next experiment is to make the symbolic analysis genuinely blind.

Instead of providing the analyzer with

$$
P,\quad Q,\quad U,\quad \Delta,\quad \frac{Q}{U^3},
$$

the candidate library should contain only the original

$$
p,q,r
$$

and a controlled set of elementary operations.

The key question is:

Can the analysis rediscover the same algebraic structures without being told what to look for?

If the answer is yes, the evidence for internally emerging mathematical structure would become substantially stronger.

Other useful directions include:

fixed reproducible train/validation/test splits;
multiple random seeds;
standard MLP baselines;
ablations of the cubic transformation;
systematic neuron-level symbolic regression;
causal interventions on candidate neurons;
out-of-distribution evaluation;
larger root and coefficient ranges;
representation-stability analysis across independently trained models;
comparison with conventional polynomial architectures;
extension to higher-degree polynomial equations.
Project Evolution

The project evolved approximately as follows:

quadratic equation

↓

neural solver

↓

custom polynomial / cubic layer

↓

cubic equations

↓

depth and model-size experiments

↓

~4.49M-parameter model

↓

hidden-neuron analysis

↓

algebraic feature search

↓

stable (P,Q,U,Q/U^3)-related representations

↓

blind symbolic distillation

↓

hypothesis connecting learned structure to cubic-solving algebra

The project started from a practical question:

“Can a neural network solve cubic equations?”

It ended with a more fundamental one:

“Can a neural network develop internal mathematical representations that resemble those used in an analytical derivation?”

The current evidence suggests that such representations exist in the trained model.

Whether they arise autonomously, reproducibly, and causally remains an open research question.

Current Project Status

Research prototype — experimental

Best current checkpoint:

CEAU — 6-layer cubic solver

Parameters:

~4.49M

Independent test:

$$
MAE=0.221891
$$

$$
RMSE=0.382195
$$

Observed internal representations are consistently associated with

$$
p,q,r,
$$

$$
P=q-\frac{p^2}{3},
$$

$$
Q=\frac{2p^3}{27}-\frac{pq}{3}+r,
$$

$$
U=\sqrt{-\frac{P}{3}},
$$

and

$$
\frac{Q}{U^3}.
$$

At this stage, CEAU is no longer just an experiment in the form “a neural network solves an equation”.

It is an experimental platform for studying whether neural networks can develop internal representations aligned with mathematical structure.

🤖AI
import torch as t
import torch.nn as nn
import torch.optim as opt
from torch.utils.data import DataLoader, Dataset

# =========================================================
# CUBIC LAYER
# =========================================================

class CubicLayer(nn.Module):
    """
    z = W x + b

    y = A*z^3 + B*z^2 + C*z + D
    """

    def __init__(self, in_features, out_features):
        super().__init__()

        self.linear = nn.Linear(
            in_features,
            out_features
        )

        # Trainable cubic coefficients
        self.A = nn.Parameter(
            t.ones(out_features) * 0.01
        )

        self.B = nn.Parameter(
            t.ones(out_features)
        )

        self.C = nn.Parameter(
            t.zeros(out_features)
        )

        self.D = nn.Parameter(
            t.zeros(out_features)
        )

    def forward(self, x):
        z = self.linear(x)

        return (
            self.A * z**3
            + self.B * z**2
            + self.C * z
            + self.D
        )


# =========================================================
# CEAU / 4.49M MODEL
# =========================================================

class CEAU(nn.Module):
    """
    CEAU cubic solver.

    Architecture:

        3
        ↓
        CubicLayer(3, 700)
        ↓ LayerNorm + GELU
        900
        ↓
        1100
        ↓
        1100
        ↓
        900
        ↓
        700
        ↓
        Linear
        3

    Six CubicLayer blocks.
    """

    def __init__(self):
        super().__init__()

        self.net = nn.Sequential(

            # -------------------------------------------------
            # LAYER 1
            # -------------------------------------------------
            CubicLayer(3, 700),
            nn.LayerNorm(700),
            nn.GELU(),

            # -------------------------------------------------
            # LAYER 2
            # -------------------------------------------------
            CubicLayer(700, 900),
            nn.LayerNorm(900),
            nn.GELU(),

            # -------------------------------------------------
            # LAYER 3
            # -------------------------------------------------
            CubicLayer(900, 1100),
            nn.LayerNorm(1100),
            nn.GELU(),

            # -------------------------------------------------
            # LAYER 4
            # -------------------------------------------------
            CubicLayer(1100, 1100),
            nn.LayerNorm(1100),
            nn.GELU(),

            # -------------------------------------------------
            # LAYER 5
            # -------------------------------------------------
            CubicLayer(1100, 900),
            nn.LayerNorm(900),
            nn.GELU(),

            # -------------------------------------------------
            # LAYER 6
            # -------------------------------------------------
            CubicLayer(900, 700),
            nn.LayerNorm(700),
            nn.GELU(),

            # -------------------------------------------------
            # OUTPUT
            # -------------------------------------------------
            nn.Linear(700, 3),
        )

    def forward(self, x):
        return self.net(x)


# =========================================================
# CREATE MODEL
# =========================================================

device = t.device(
    "cuda" if t.cuda.is_available() else "cpu"
)

model = CEAU().to(device)

print(model)

print(
    "Parameters:",
    sum(p.numel() for p in model.parameters())
)
Download weights in another file