Q²-Fetus
Q²-Fetus is a 10.7M submission to BabyLM 2026, nearly identical to GPT-2-Small except for use of the actual GeLu function as opposed to a sigmoid approximation, along with reduced parameter count, dimensions and head count to fit in a small package, with just one twist
How it works
Q²-Fetus uses a special type of FFN (Feed forward network): a quadratic FFN
Unlike other approaches like linear FFNs (most transformers nowadays), which multiply a weight matrix by the inputs, then add biases, or pairwise quadratic FFNs (e.g Quadranet), which simulate pairwise interactions using a low-rank approximation, Q² is purely a diagonal quadratic: it adds a second weight matrix, multiplied by the inputs squared.
I believe this can reach greater performance due to its greater expressiveness: a single quadratic layer could approximate curves that take linear layers more layers. However, the downside is that the extra set of weights effectively doubles the parameter count, meaning you must make a tradeoff between doubling parameters or quadratic networks.
Fortunately, we believe the tradeoff is in favor of quadratics, as this model tops all auto-regressive models including the 13x larger GPT-2 baselines, while trailing behind the BERT bidirectional context models which are 3x larger.¹
For more info about this, see my repo dedicated to a C++ framework for training quadratic layers: https://github.com/supastishn/quadratic-neuralnets
¹We note that BERT has an inherent advantage on these types of benchmarks due to its bidirectional context, making it easier to get a higher score. I plan to make a BERT version for comparison in the near future, if compute allows for it.
Hyperparameters:
Training Setup:
The model was trained for around 1 hour on an Nvidia T4 GPU.
Limitations:
Due to compute restraints, I was unable to:
Optimize hyperparameters: I only just picked a good-enough LR, and in fact the reason I tested this LR was because of an accident! With Bayesian optimization or coordinate search I expect to be able to get better performance.
Use full determinism: To save on GPU hours, compilation and non deterministic algorithms were used. This causes variation between benchmarks, while it is mostly unnoticeable with ±2% in most benchmarks, Entity Trackin in particular varies quite a lot, and in our runs has varied between 15 and 45 percent.
Test different seeds: Since I can't run too many runs, I only tested on one seed (42). While the non-deterministic algorithms used in effect simulate different seeds, it is still not perfect.
Next Steps:
I fortunately have just enough to be able to train a larger BERT version of the model, which should hopefully give us a 1:1 comparison with the top-of-the-leaderboard BERT models.
Notes
To train, a custom forward pass is needed. Due to this, you must enable use_remote_code=True in the benchmark python files.