File size: 3,183 Bytes
8eaa96d 0845a98 2acaf75 8eaa96d 0845a98 2acaf75 0845a98 2acaf75 0845a98 2acaf75 0845a98 08d8108 0845a98 08d8108 0845a98 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 08d8108 2acaf75 edb481f 2acaf75 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
language:
- en
pipeline_tag: text-generation
tags:
- mathematics
- modular-arithmetic
- grokking
- scratchpad
- length-generalization
- transformers
license: cc-by-4.0
---
# SAIR Modular Arithmetic Challenge: Abacus Generalization
[](https://github.com/Amey-Thakur/SAIR-MODULAR-ARITHMETIC-CHALLENGE)
[](https://creativecommons.org/licenses/by/4.0/)
## Overview
This repository hosts the weights for the **SAIR Modular Arithmetic Challenge**. The model is an autoregressive transformer built to solve exact modular multiplication `(A × B) mod P` organically, without hardcoded arithmetic logic, symbolic parsers, or external computation modules.
Using state-machine decoding, this research targets the learnability wall of transformer mathematics, achieving theoretical infinite length generalization.
---
## Architectural Methods
To bypass the spatial limitations of standard transformers, this model uses three methods:
### 1. Abacus Significance Embeddings
Standard transformers track coordinate positions. This model strips coordinate embeddings, replacing them with Mathematical Significance Injections. Digits route based on their place-value, ensuring 1024-bit primes process through the exact same logic gates as 16-bit primes.
### 2. Algorithmic Scratchpads (Bit-Serial Decoding)
The network operates as a recurrent state machine. By forcing the model to generate intermediate computational traces autoregressively, the network allocates computation proportionally to integer complexity, mimicking a Turing machine tape.
### 3. Grokking Phase Transitions
The weights deployed here were captured after the grokking phase transition. The model trained through thousands of delayed gradient steps beyond the initial validation plateau with extreme weight decay, forcing the network to collapse memorization circuits into sparse mathematical algorithms.
---
## Inference Format
This model requires a specific execution format.
If prompted with an equation, it sequentially emits the step-by-step logic trace before terminating natively with the final matrix node.
**Input Format**: Character-level ASCII equations.
```text
123*456
```
**Output Format**: State machine algorithmic trace terminating in `ANS=`.
```text
(scratchpad logic...) ANS=56088<EOS>
```
---
## Technical Specifications
- **Architecture Layer**: Bit-Serial Autoregressive Transformer
- **Embedding Mechanism**: Significance Input Injection (Abacus)
- **Tokenization**: Discrete Character-level (`Base10Tokenizer`)
- **Framework Ecosystem**: PyTorch 2.0+
---
## Citations and Laboratory
This model is an isolated artifact. The complete research laboratory - containing the synthetic data generators, the training loops, the sandbox validators, and the PyTorch implementations - is open-source.
**Official Research Repository**:
[SAIR-MODULAR-ARITHMETIC-CHALLENGE](https://github.com/Amey-Thakur/SAIR-MODULAR-ARITHMETIC-CHALLENGE)
*Prepared by Amey Thakur for the SAIR Foundation AI Benchmark.*
|