Saccadic | IRIS Training Observations
👁
A closer look inside neural network training with IRIS
AI & ML interests
Training diagnostics
Recent Activity
Saccadic-AI
updated a
Space 4 days ago
Saccadic-AI
posted an update 4 days ago
Post
4,185
One neural network. Five mathematical operations. Different learning times.
We trained a small mixture-of-experts transformer on addition, subtraction, multiplication, division, and exponentiation modulo 113.
The operations didn’t generalize together. Multiplication and division reached 90% held-out accuracy first, followed by exponentiation, subtraction, and finally addition.
Multiplication before addition—the reverse of what we expect from a schoolchild.
The accuracy chart in the linked article shows why a single overall accuracy number can hide an interesting learning story.
Measured with IRIS from Saccadic LLC. More figures and analysis: https://huggingface.co/blog/Saccadic-AI/accuracy-plateaued-the-model-kept-changing
Have you observed a similar ordering in multi-task grokking experiments?
We trained a small mixture-of-experts transformer on addition, subtraction, multiplication, division, and exponentiation modulo 113.
The operations didn’t generalize together. Multiplication and division reached 90% held-out accuracy first, followed by exponentiation, subtraction, and finally addition.
Multiplication before addition—the reverse of what we expect from a schoolchild.
The accuracy chart in the linked article shows why a single overall accuracy number can hide an interesting learning story.
Measured with IRIS from Saccadic LLC. More figures and analysis: https://huggingface.co/blog/Saccadic-AI/accuracy-plateaued-the-model-kept-changing
Have you observed a similar ordering in multi-task grokking experiments?
- 5 replies
Saccadic-AI
published a
Space 9 days ago