davidandroid commited on
Commit
07aafa5
·
verified ·
1 Parent(s): fa4fefd

updating readme and model card

Browse files
Files changed (1) hide show
  1. README.md +89 -0
README.md CHANGED
@@ -1,3 +1,92 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ library_name: pytorch
4
+ tags:
5
+ - transformer
6
+ - arithmetic
7
+ - multiplication
8
+ - algorithmic-reasoning
9
+ - latent-algorithm-learning
10
  ---
11
+
12
+ # Lattice Multiplication Transformer
13
+
14
+ [Interactive Demo](https://huggingface.co/spaces/davidandroid/lattice-multiplication-transformer-demo) ·
15
+ [Source Code](https://github.com/he-man-david/learning-multiplication-with-lattice-representation-using-transformer)
16
+
17
+ This repository contains the final checkpoint for an approximately 11.7-million-parameter model studying latent algorithm learning through an algorithm-aligned input representation.
18
+
19
+ Multi-digit multiplication is used as a case study. A learned frontend transforms two operands into an N × N lattice of digit-pair representations, which is passed to a standard Transformer encoder-decoder. The model directly generates the final product without scratchpads, carry labels, local-product labels, diagonal sums, or other intermediate supervision.
20
+
21
+ ## Results
22
+
23
+ The final checkpoint was trained for 600,000 optimization steps.
24
+
25
+ - 19,955 correct products across 20,000 square-length evaluation examples from 1 × 1 through 20 × 20
26
+ - 99.775% aggregate exact-match accuracy across those evaluations
27
+ - 100% exact-match accuracy from 1 × 1 through 15 × 15
28
+ - 96.5% exact-match accuracy and approximately 99.90% token accuracy at 20 × 20
29
+
30
+ ## Included Files
31
+
32
+ ```text
33
+ final_lattice_multiplication__11m.pt
34
+ lattice_multiplication_transformer.py
35
+ tokenizer.py
36
+ config.json
37
+ ```
38
+
39
+ The checkpoint contains the model state, optimizer state, curriculum state, training history, and random-number-generator states. Inference requires only `checkpoint["model_state_dict"]`.
40
+
41
+ ## Loading the Checkpoint
42
+
43
+ ```python
44
+ import json
45
+ import sys
46
+ from pathlib import Path
47
+
48
+ import torch
49
+ from huggingface_hub import snapshot_download
50
+
51
+
52
+ model_directory = Path(
53
+ snapshot_download(
54
+ repo_id="davidandroid/lattice-multiplication-transformer"
55
+ )
56
+ )
57
+
58
+ sys.path.insert(0, str(model_directory))
59
+
60
+ from lattice_multiplication_transformer import (
61
+ LatticeMultiplicationLoopTransformer,
62
+ )
63
+
64
+
65
+ with open(
66
+ model_directory / "config.json",
67
+ encoding="utf-8",
68
+ ) as config_file:
69
+ model_config = json.load(config_file)
70
+
71
+ model = LatticeMultiplicationLoopTransformer(
72
+ **model_config
73
+ )
74
+
75
+ checkpoint = torch.load(
76
+ model_directory / "final_lattice_multiplication__11m.pt",
77
+ map_location="cpu",
78
+ weights_only=False,
79
+ )
80
+
81
+ model.load_state_dict(
82
+ checkpoint["model_state_dict"]
83
+ )
84
+
85
+ model.eval()
86
+ ```
87
+
88
+ ## Limitations
89
+
90
+ The model was trained on operands containing at most 20 digits. It does not achieve reliable exact-match length generalization beyond that range, even though token accuracy remains higher immediately beyond the training boundary.
91
+
92
+ This is a research model for studying latent algorithm learning, not a replacement for deterministic multiplication software.