updating readme and model card
Browse files
README.md
CHANGED
|
@@ -1,3 +1,92 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
tags:
|
| 5 |
+
- transformer
|
| 6 |
+
- arithmetic
|
| 7 |
+
- multiplication
|
| 8 |
+
- algorithmic-reasoning
|
| 9 |
+
- latent-algorithm-learning
|
| 10 |
---
|
| 11 |
+
|
| 12 |
+
# Lattice Multiplication Transformer
|
| 13 |
+
|
| 14 |
+
[Interactive Demo](https://huggingface.co/spaces/davidandroid/lattice-multiplication-transformer-demo) ·
|
| 15 |
+
[Source Code](https://github.com/he-man-david/learning-multiplication-with-lattice-representation-using-transformer)
|
| 16 |
+
|
| 17 |
+
This repository contains the final checkpoint for an approximately 11.7-million-parameter model studying latent algorithm learning through an algorithm-aligned input representation.
|
| 18 |
+
|
| 19 |
+
Multi-digit multiplication is used as a case study. A learned frontend transforms two operands into an N × N lattice of digit-pair representations, which is passed to a standard Transformer encoder-decoder. The model directly generates the final product without scratchpads, carry labels, local-product labels, diagonal sums, or other intermediate supervision.
|
| 20 |
+
|
| 21 |
+
## Results
|
| 22 |
+
|
| 23 |
+
The final checkpoint was trained for 600,000 optimization steps.
|
| 24 |
+
|
| 25 |
+
- 19,955 correct products across 20,000 square-length evaluation examples from 1 × 1 through 20 × 20
|
| 26 |
+
- 99.775% aggregate exact-match accuracy across those evaluations
|
| 27 |
+
- 100% exact-match accuracy from 1 × 1 through 15 × 15
|
| 28 |
+
- 96.5% exact-match accuracy and approximately 99.90% token accuracy at 20 × 20
|
| 29 |
+
|
| 30 |
+
## Included Files
|
| 31 |
+
|
| 32 |
+
```text
|
| 33 |
+
final_lattice_multiplication__11m.pt
|
| 34 |
+
lattice_multiplication_transformer.py
|
| 35 |
+
tokenizer.py
|
| 36 |
+
config.json
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
The checkpoint contains the model state, optimizer state, curriculum state, training history, and random-number-generator states. Inference requires only `checkpoint["model_state_dict"]`.
|
| 40 |
+
|
| 41 |
+
## Loading the Checkpoint
|
| 42 |
+
|
| 43 |
+
```python
|
| 44 |
+
import json
|
| 45 |
+
import sys
|
| 46 |
+
from pathlib import Path
|
| 47 |
+
|
| 48 |
+
import torch
|
| 49 |
+
from huggingface_hub import snapshot_download
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
model_directory = Path(
|
| 53 |
+
snapshot_download(
|
| 54 |
+
repo_id="davidandroid/lattice-multiplication-transformer"
|
| 55 |
+
)
|
| 56 |
+
)
|
| 57 |
+
|
| 58 |
+
sys.path.insert(0, str(model_directory))
|
| 59 |
+
|
| 60 |
+
from lattice_multiplication_transformer import (
|
| 61 |
+
LatticeMultiplicationLoopTransformer,
|
| 62 |
+
)
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
with open(
|
| 66 |
+
model_directory / "config.json",
|
| 67 |
+
encoding="utf-8",
|
| 68 |
+
) as config_file:
|
| 69 |
+
model_config = json.load(config_file)
|
| 70 |
+
|
| 71 |
+
model = LatticeMultiplicationLoopTransformer(
|
| 72 |
+
**model_config
|
| 73 |
+
)
|
| 74 |
+
|
| 75 |
+
checkpoint = torch.load(
|
| 76 |
+
model_directory / "final_lattice_multiplication__11m.pt",
|
| 77 |
+
map_location="cpu",
|
| 78 |
+
weights_only=False,
|
| 79 |
+
)
|
| 80 |
+
|
| 81 |
+
model.load_state_dict(
|
| 82 |
+
checkpoint["model_state_dict"]
|
| 83 |
+
)
|
| 84 |
+
|
| 85 |
+
model.eval()
|
| 86 |
+
```
|
| 87 |
+
|
| 88 |
+
## Limitations
|
| 89 |
+
|
| 90 |
+
The model was trained on operands containing at most 20 digits. It does not achieve reliable exact-match length generalization beyond that range, even though token accuracy remains higher immediately beyond the training boundary.
|
| 91 |
+
|
| 92 |
+
This is a research model for studying latent algorithm learning, not a replacement for deterministic multiplication software.
|