diff --git a/.gitattributes b/.gitattributes index a6344aac8c09253b3b630fb776ae94478aa0275b..9f38eb4cf7be7099b673b685aae35fd6a032fe7d 100644 --- a/.gitattributes +++ b/.gitattributes @@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text *.zip filter=lfs diff=lfs merge=lfs -text *.zst filter=lfs diff=lfs merge=lfs -text *tfevents* filter=lfs diff=lfs merge=lfs -text +figures/fig2_frontier.png filter=lfs diff=lfs merge=lfs -text diff --git a/README.md b/README.md index 69ae63d0f727b31b8f2debbf6fa54fb5b37e66c1..2224a96aa1c8589f0a6fdb3aa8af9345d0a5db2d 100644 --- a/README.md +++ b/README.md @@ -5,14 +5,121 @@ tags: [masked-diffusion, discrete-diffusion, length-generalization, positional-e # Length Generalization in Masked Diffusion Language Models -From-scratch masked diffusion language models (MDLM-style absorbing state) and -matched autoregressive baselines, trained on arithmetic and algorithmic tasks and -evaluated far beyond the lengths seen in training. +From-scratch masked diffusion language models (MDLM-style absorbing-state +diffusion) and matched autoregressive baselines, trained on multi-digit addition +and evaluated on operand lengths never seen in training. -**No pretrained weights or tokenizers are used anywhere.** +**No pretrained weights or tokenizers are used anywhere.** Every model is trained +from random initialization with a symbol-level vocabulary built from the data. -Code: https://github.com/GIND123/MSML612 +## What is here + +- `figures/` — plots and the summary table +- `results/` — one JSON per run (config + accuracy on every test length) +- `weights/` — checkpoints (~10.7M parameters each) +- `src/` — full training, evaluation and plotting code + +## Method + +**Significance-aligned position ids.** Instead of numbering tokens by sequence +index, every digit is numbered by its place value, so digits that must be +combined share an id: + +``` + 4 7 + 8 5 = 1 3 2 + 2 1 0 2 1 0 3 2 1 +``` + +The rule "combine equal ids, carry into id+1" does not depend on operand length, +which is what allows extrapolation. A random per-example offset is added to all +ids so the model keys on relative place value and encounters large ids during +training. + +## Findings + +1. **Positional encodings do not transfer between the two architectures.** + Removing positional information entirely, and sinusoidal encoding, both give + 100% in-distribution accuracy for autoregressive models and **0% for masked + diffusion** — the diffusion model cannot learn the task at all (3 seeds each). + A diffusion model attends bidirectionally and so has no implicit order to + fall back on. + +2. **Length-generalization accuracy has very high seed variance** on this task + (individual seeds span 3%–68% for one configuration), so single-seed + comparisons in this setting are unreliable. ## Results -_(pending)_ +# Results + +## Table 1 — Addition: exact match at ONE forward pass + +The whole answer committed in a single pass. Depth scaled as log2(digits); learning rate 1e-4 with 4000 warmup steps, which matters - see Table 4. + +| digits | MDLM baseline | full method | seeds >90% (base → ours) | +|---|---|---|---| +| **8** | 100.0 [100,100] | 100.0 [100,100] | 4/4 → 4/4 | +| **12** | 76.7 [30,100] | 100.0 [100,100] | 3/4 → 4/4 | +| **16** | 25.0 [0,75] | 75.0 [25,100] | 1/4 → 3/4 | +| **24** | 0.0 [0,0] | 0.0 [0,0] | 0/4 → 0/4 | +| **32** | 0.0 [0,0] | – | 0/4 → – | + +## Table 2 — Morphological inflection: where the problem does NOT arise + +Answers are unique but only ~10 characters with shallow, local dependencies. One pass already suffices for the plain baseline, so there is no gap for any method to close. This is the boundary the account predicts, and it is reported as such. + +| language | 1 pass | 32 passes | gap | +|---|---|---|---| +| english | 95.5 | 95.1 | -0.4 | +| english | 95.0 | 95.2 | +0.2 | +| english | 95.3 | 95.2 | -0.1 | +| finnish | 50.9 | 55.8 | +4.9 | +| finnish | 48.6 | 56.6 | +8.0 | +| georgian | 94.3 | 94.1 | -0.2 | +| georgian | 95.4 | 95.3 | -0.1 | +| german | 78.9 | 79.3 | +0.4 | +| german | 75.6 | 79.5 | +3.9 | +| german | 77.2 | 78.3 | +1.0 | +| navajo | 29.5 | 33.8 | +4.3 | +| navajo | 31.4 | 40.2 | +8.9 | +| russian | 72.7 | 72.0 | -0.7 | +| russian | 69.1 | 71.7 | +2.6 | +| spanish | 84.9 | 84.7 | -0.2 | +| spanish | 83.9 | 84.1 | +0.1 | +| turkish | 80.2 | 81.2 | +1.0 | +| turkish | 77.5 | 82.2 | +4.7 | +| turkish | 76.4 | 81.8 | +5.4 | + +## Table 3 — Non-unique answers: the limit no training removes + +On probes where a group must agree but the value is free, the likelihood-optimal marginal is uniform over the valid values, so two independent draws agree with probability 1/V. Sweeping an entropy penalty designed to break that symmetry over 66 runs: weak settings changed nothing, strong settings destroyed the model, failing even at fully sequential decoding. Succeeding would require abandoning the data distribution. + +## Table 4 — Why earlier runs appeared to fail (10-digit addition) + +Every apparent failure at 10+ digits was a learning-rate artifact, not a limit of the method or of depth. + +| layers | lr | warmup | mean @1 pass | seeds >90% | +|---|---|---|---|---| +| 6 | 5e-05 | 500 | 100.0 | 3/3 | +| 6 | 5e-05 | 4000 | 100.0 | 3/3 | +| 6 | 0.0001 | 500 | 100.0 | 3/3 | +| 6 | 0.0001 | 4000 | 100.0 | 3/3 | +| 6 | 0.0003 | 500 | 70.6 | 2/3 | +| 6 | 0.0003 | 4000 | 100.0 | 3/3 | +| 10 | 5e-05 | 500 | 100.0 | 3/3 | +| 10 | 5e-05 | 4000 | 100.0 | 3/3 | +| 10 | 0.0001 | 500 | 100.0 | 3/3 | +| 10 | 0.0001 | 4000 | 100.0 | 3/3 | +| 10 | 0.0003 | 500 | 47.3 | 1/3 | +| 10 | 0.0003 | 4000 | 100.0 | 3/3 | + + + +## Setup + +6 layers, 384 hidden dimension, 6 heads, ~10.7M parameters. AdamW, learning rate +1e-4, cosine schedule, bfloat16, effective batch 256. Trained on operands of 1-5 +digits; evaluated at 5, 6, 7, 8, 10 and 12 digits. Exact-match accuracy — the +entire answer must be correct. + +Trained on the University of Maryland Zaratan cluster (A100 MIG slices). diff --git a/figures/beat_d12.json b/figures/beat_d12.json new file mode 100644 index 0000000000000000000000000000000000000000..e2fe0364fb9796b8e5bec8d4d380715f76460791 --- /dev/null +++ b/figures/beat_d12.json @@ -0,0 +1,54 @@ +{ + "digits": 12, + "inference_time_on_baseline": { + "confidence": [ + 0.766875, + 0.764375, + 0.7637499999999999, + 0.763125, + 0.7625 + ], + "entropy": [ + 0.766875, + 0.764375, + 0.7637499999999999, + 0.763125, + 0.7625 + ], + "entropy-gated": [ + 0.766875, + 0.764375, + 0.7637499999999999, + 0.7618750000000001, + 0.7625 + ], + "margin": [ + 0.766875, + 0.764375, + 0.7637499999999999, + 0.763125, + 0.7618750000000001 + ], + "random": [ + 0.766875, + 0.771875, + 0.76875, + 0.7675000000000001, + 0.7637499999999999 + ] + }, + "ours": { + "confidence": [ + 1.0, + 1.0, + 1.0, + 0.999375, + 0.999375 + ] + }, + "summary": { + "best_inference_1pass": 0.766875, + "best_inference_any_budget": 0.771875, + "ours_1pass": 1.0 + } +} \ No newline at end of file diff --git a/figures/beat_d16.json b/figures/beat_d16.json new file mode 100644 index 0000000000000000000000000000000000000000..4496d558565f370f8f0cfcd1c550950f78924e42 --- /dev/null +++ b/figures/beat_d16.json @@ -0,0 +1,54 @@ +{ + "digits": 16, + "inference_time_on_baseline": { + "confidence": [ + 0.25, + 0.249375, + 0.25, + 0.25, + 0.25 + ], + "entropy": [ + 0.25, + 0.249375, + 0.25, + 0.25, + 0.25 + ], + "entropy-gated": [ + 0.25, + 0.249375, + 0.25, + 0.25, + 0.25 + ], + "margin": [ + 0.25, + 0.25, + 0.25, + 0.25, + 0.25 + ], + "random": [ + 0.25, + 0.25, + 0.25, + 0.249375, + 0.25 + ] + }, + "ours": { + "confidence": [ + 0.75, + 0.75, + 0.75, + 0.75, + 0.749375 + ] + }, + "summary": { + "best_inference_1pass": 0.25, + "best_inference_any_budget": 0.25, + "ours_1pass": 0.75 + } +} \ No newline at end of file diff --git a/figures/fig1_accuracy_vs_passes.png b/figures/fig1_accuracy_vs_passes.png new file mode 100644 index 0000000000000000000000000000000000000000..c425772df2e8d98aea6815e5c4449cb227614a6e Binary files /dev/null and b/figures/fig1_accuracy_vs_passes.png differ diff --git a/figures/fig1_addition_frontier.png b/figures/fig1_addition_frontier.png new file mode 100644 index 0000000000000000000000000000000000000000..59ba319f5165e724a528b1dea0a2d85816a5454c Binary files /dev/null and b/figures/fig1_addition_frontier.png differ diff --git a/figures/fig2_accuracy_vs_passes.png b/figures/fig2_accuracy_vs_passes.png new file mode 100644 index 0000000000000000000000000000000000000000..3fcd50b84b301de4522b36ba2643981ade66ebce Binary files /dev/null and b/figures/fig2_accuracy_vs_passes.png differ diff --git a/figures/fig2_frontier.png b/figures/fig2_frontier.png new file mode 100644 index 0000000000000000000000000000000000000000..401a58f849ae4a489ba9093649e49244ec1abf26 --- /dev/null +++ b/figures/fig2_frontier.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:30c7c04fef6c26a8143cf4fcfce6a03f2b4a52ba987df807ca4d7cee749e19b5 +size 102024 diff --git a/figures/summary.md b/figures/summary.md new file mode 100644 index 0000000000000000000000000000000000000000..00e583d458aeead47cf9c84e4033985977524f85 --- /dev/null +++ b/figures/summary.md @@ -0,0 +1,63 @@ +# Results + +## Table 1 — Addition: exact match at ONE forward pass + +The whole answer committed in a single pass. Depth scaled as log2(digits); learning rate 1e-4 with 4000 warmup steps, which matters - see Table 4. + +| digits | MDLM baseline | full method | seeds >90% (base → ours) | +|---|---|---|---| +| **8** | 100.0 [100,100] | 100.0 [100,100] | 4/4 → 4/4 | +| **12** | 76.7 [30,100] | 100.0 [100,100] | 3/4 → 4/4 | +| **16** | 25.0 [0,75] | 75.0 [25,100] | 1/4 → 3/4 | +| **24** | 0.0 [0,0] | 0.0 [0,0] | 0/4 → 0/4 | +| **32** | 0.0 [0,0] | – | 0/4 → – | + +## Table 2 — Morphological inflection: where the problem does NOT arise + +Answers are unique but only ~10 characters with shallow, local dependencies. One pass already suffices for the plain baseline, so there is no gap for any method to close. This is the boundary the account predicts, and it is reported as such. + +| language | 1 pass | 32 passes | gap | +|---|---|---|---| +| english | 95.5 | 95.1 | -0.4 | +| english | 95.0 | 95.2 | +0.2 | +| english | 95.3 | 95.2 | -0.1 | +| finnish | 50.9 | 55.8 | +4.9 | +| finnish | 48.6 | 56.6 | +8.0 | +| georgian | 94.3 | 94.1 | -0.2 | +| georgian | 95.4 | 95.3 | -0.1 | +| german | 78.9 | 79.3 | +0.4 | +| german | 75.6 | 79.5 | +3.9 | +| german | 77.2 | 78.3 | +1.0 | +| navajo | 29.5 | 33.8 | +4.3 | +| navajo | 31.4 | 40.2 | +8.9 | +| russian | 72.7 | 72.0 | -0.7 | +| russian | 69.1 | 71.7 | +2.6 | +| spanish | 84.9 | 84.7 | -0.2 | +| spanish | 83.9 | 84.1 | +0.1 | +| turkish | 80.2 | 81.2 | +1.0 | +| turkish | 77.5 | 82.2 | +4.7 | +| turkish | 76.4 | 81.8 | +5.4 | + +## Table 3 — Non-unique answers: the limit no training removes + +On probes where a group must agree but the value is free, the likelihood-optimal marginal is uniform over the valid values, so two independent draws agree with probability 1/V. Sweeping an entropy penalty designed to break that symmetry over 66 runs: weak settings changed nothing, strong settings destroyed the model, failing even at fully sequential decoding. Succeeding would require abandoning the data distribution. + +## Table 4 — Why earlier runs appeared to fail (10-digit addition) + +Every apparent failure at 10+ digits was a learning-rate artifact, not a limit of the method or of depth. + +| layers | lr | warmup | mean @1 pass | seeds >90% | +|---|---|---|---|---| +| 6 | 5e-05 | 500 | 100.0 | 3/3 | +| 6 | 5e-05 | 4000 | 100.0 | 3/3 | +| 6 | 0.0001 | 500 | 100.0 | 3/3 | +| 6 | 0.0001 | 4000 | 100.0 | 3/3 | +| 6 | 0.0003 | 500 | 70.6 | 2/3 | +| 6 | 0.0003 | 4000 | 100.0 | 3/3 | +| 10 | 5e-05 | 500 | 100.0 | 3/3 | +| 10 | 5e-05 | 4000 | 100.0 | 3/3 | +| 10 | 0.0001 | 500 | 100.0 | 3/3 | +| 10 | 0.0001 | 4000 | 100.0 | 3/3 | +| 10 | 0.0003 | 500 | 47.3 | 1/3 | +| 10 | 0.0003 | 4000 | 100.0 | 3/3 | + diff --git a/results/g10-L10-lr1e-4-w4000-s0.json b/results/g10-L10-lr1e-4-w4000-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..b9d03bd02584b803e1ac89b52a322c238e493c0d --- /dev/null +++ b/results/g10-L10-lr1e-4-w4000-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr1e-4-w4000-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 0.7025, + "3": 0.7324999999999999, + "4": 0.8325, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.24793006579081, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr1e-4-w4000-s1.json b/results/g10-L10-lr1e-4-w4000-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..57598ae213a17a12086520723d2627bd00c35ed2 --- /dev/null +++ b/results/g10-L10-lr1e-4-w4000-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr1e-4-w4000-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.504784576098125, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr1e-4-w4000-s2.json b/results/g10-L10-lr1e-4-w4000-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..017718c5f3946a4fecadb8846dc047f3b63e5109 --- /dev/null +++ b/results/g10-L10-lr1e-4-w4000-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr1e-4-w4000-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.34981673558553, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr1e-4-w500-s0.json b/results/g10-L10-lr1e-4-w500-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..2bf05fa22a6edf7b0587c59f2ad63c99e9eefdb7 --- /dev/null +++ b/results/g10-L10-lr1e-4-w500-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr1e-4-w500-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.456493131319682, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr1e-4-w500-s1.json b/results/g10-L10-lr1e-4-w500-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..bf7f54171fa85d034a4211e64031c0f049961b45 --- /dev/null +++ b/results/g10-L10-lr1e-4-w500-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr1e-4-w500-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.369545900821684, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr1e-4-w500-s2.json b/results/g10-L10-lr1e-4-w500-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..faf8ce66563ad84756ae53803ee77f0f6075f650 --- /dev/null +++ b/results/g10-L10-lr1e-4-w500-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr1e-4-w500-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.33252931833267, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr3e-4-w4000-s0.json b/results/g10-L10-lr3e-4-w4000-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..df3d56e9537b51d4ce14c8d280a6a1efda11cd93 --- /dev/null +++ b/results/g10-L10-lr3e-4-w4000-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr3e-4-w4000-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.36896385749181, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr3e-4-w4000-s1.json b/results/g10-L10-lr3e-4-w4000-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..fbcb609c2cd55f47bfa11dcadb7d9a7a430015d6 --- /dev/null +++ b/results/g10-L10-lr3e-4-w4000-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr3e-4-w4000-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.307572778066, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr3e-4-w4000-s2.json b/results/g10-L10-lr3e-4-w4000-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..a775a0ad6cbccf03e631afec5192740d27d34b7e --- /dev/null +++ b/results/g10-L10-lr3e-4-w4000-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr3e-4-w4000-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.570110074679057, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr3e-4-w500-s0.json b/results/g10-L10-lr3e-4-w500-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..d274b30347f35023809391c829b6eabfcd906aa9 --- /dev/null +++ b/results/g10-L10-lr3e-4-w500-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr3e-4-w500-s0" + }, + "accuracy_by_passes": { + "1": 0.005, + "2": 0.0075, + "3": 0.0075, + "4": 0.015000000000000001, + "6": 0.0075, + "8": 0.005, + "12": 0.005, + "16": 0.005 + }, + "minutes": 18.132507447401682, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr3e-4-w500-s1.json b/results/g10-L10-lr3e-4-w500-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..d69e89cf8c9ae2d737fcf445859c3860a70418b3 --- /dev/null +++ b/results/g10-L10-lr3e-4-w500-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr3e-4-w500-s1" + }, + "accuracy_by_passes": { + "1": 0.41500000000000004, + "2": 0.39249999999999996, + "3": 0.355, + "4": 0.3575, + "6": 0.36750000000000005, + "8": 0.36750000000000005, + "12": 0.36750000000000005, + "16": 0.36750000000000005 + }, + "minutes": 18.539059253533683, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr3e-4-w500-s2.json b/results/g10-L10-lr3e-4-w500-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..1970cb3fcf5c18f8a76dbc9a270aa2ddf70990e6 --- /dev/null +++ b/results/g10-L10-lr3e-4-w500-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr3e-4-w500-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.266150387128196, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr5e-5-w4000-s0.json b/results/g10-L10-lr5e-5-w4000-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..3e32769232c52564fd8619774c3369c0fe90d596 --- /dev/null +++ b/results/g10-L10-lr5e-5-w4000-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr5e-5-w4000-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.333289953072867, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr5e-5-w4000-s1.json b/results/g10-L10-lr5e-5-w4000-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..b78d8f1af47df4c56cc5d5f8c8a6cf72ee0008ab --- /dev/null +++ b/results/g10-L10-lr5e-5-w4000-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr5e-5-w4000-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.333352772394814, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr5e-5-w4000-s2.json b/results/g10-L10-lr5e-5-w4000-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..85b815bd8f5f76beb31c8d11e13351758d856178 --- /dev/null +++ b/results/g10-L10-lr5e-5-w4000-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr5e-5-w4000-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.16126123269399, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr5e-5-w500-s0.json b/results/g10-L10-lr5e-5-w500-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..6db94f0dc245d4937b1c4201c7102426ee3b92b6 --- /dev/null +++ b/results/g10-L10-lr5e-5-w500-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr5e-5-w500-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.2984050989151, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr5e-5-w500-s1.json b/results/g10-L10-lr5e-5-w500-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..4352ab86f528f02e5fe08e309f5e6a2f06a75d39 --- /dev/null +++ b/results/g10-L10-lr5e-5-w500-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr5e-5-w500-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.280582133928934, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L10-lr5e-5-w500-s2.json b/results/g10-L10-lr5e-5-w500-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..79cfdf8194bc000fe4b32a126dfd26e6642456ff --- /dev/null +++ b/results/g10-L10-lr5e-5-w500-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L10-lr5e-5-w500-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 18.434220337867735, + "compute": { + "params": 17740800, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr1e-4-w4000-s0.json b/results/g10-L6-lr1e-4-w4000-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..3d23495af3e0c62ed34d333b9d867a930edd8486 --- /dev/null +++ b/results/g10-L6-lr1e-4-w4000-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr1e-4-w4000-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.732448327541352, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr1e-4-w4000-s1.json b/results/g10-L6-lr1e-4-w4000-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..323cc3d6c66622fe9357016125a4343652c3a63e --- /dev/null +++ b/results/g10-L6-lr1e-4-w4000-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr1e-4-w4000-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.891742567221323, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr1e-4-w4000-s2.json b/results/g10-L6-lr1e-4-w4000-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..32298822e0fb7b9e9d14cc3928fda3be9b6cef2a --- /dev/null +++ b/results/g10-L6-lr1e-4-w4000-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr1e-4-w4000-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.605476510524749, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr1e-4-w500-s0.json b/results/g10-L6-lr1e-4-w500-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..8995c35f7e954a03ced96c9ff43a49ecf6ab5e68 --- /dev/null +++ b/results/g10-L6-lr1e-4-w500-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr1e-4-w500-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.966393812497456, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr1e-4-w500-s1.json b/results/g10-L6-lr1e-4-w500-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..a81a0794da97f080748070a1108e4e22ff66a01c --- /dev/null +++ b/results/g10-L6-lr1e-4-w500-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr1e-4-w500-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.829342067241669, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr1e-4-w500-s2.json b/results/g10-L6-lr1e-4-w500-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..f2101d4b0a91f3ffb63d7d9241d9c70d47710b38 --- /dev/null +++ b/results/g10-L6-lr1e-4-w500-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0001, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr1e-4-w500-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.929011670748393, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr3e-4-w4000-s0.json b/results/g10-L6-lr3e-4-w4000-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..caa7f77a42c8645365e9d93251fecf0169b99486 --- /dev/null +++ b/results/g10-L6-lr3e-4-w4000-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr3e-4-w4000-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.790480820337931, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr3e-4-w4000-s1.json b/results/g10-L6-lr3e-4-w4000-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..c778d59a73c490f5a6c0f4afbef48b6f1b99306b --- /dev/null +++ b/results/g10-L6-lr3e-4-w4000-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr3e-4-w4000-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.863731988271077, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr3e-4-w4000-s2.json b/results/g10-L6-lr3e-4-w4000-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..1a694296348e63c4a2648229eb15eae6707b2b51 --- /dev/null +++ b/results/g10-L6-lr3e-4-w4000-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr3e-4-w4000-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.79504845937093, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr3e-4-w500-s0.json b/results/g10-L6-lr3e-4-w500-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..9ff0947eaac205279dab49e79e6dae47a9cb216f --- /dev/null +++ b/results/g10-L6-lr3e-4-w500-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr3e-4-w500-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.876898606618246, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr3e-4-w500-s1.json b/results/g10-L6-lr3e-4-w500-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..5d0df9a2110ef0a34456b4638e065aa3f08f3ab4 --- /dev/null +++ b/results/g10-L6-lr3e-4-w500-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr3e-4-w500-s1" + }, + "accuracy_by_passes": { + "1": 0.1175, + "2": 0.0925, + "3": 0.085, + "4": 0.09000000000000001, + "6": 0.09000000000000001, + "8": 0.09000000000000001, + "12": 0.09000000000000001, + "16": 0.09000000000000001 + }, + "minutes": 11.782170780499776, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr3e-4-w500-s2.json b/results/g10-L6-lr3e-4-w500-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..95d742fa5eefe2f088361f60ba787e9011871a91 --- /dev/null +++ b/results/g10-L6-lr3e-4-w500-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr3e-4-w500-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.835885870456696, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr5e-5-w4000-s0.json b/results/g10-L6-lr5e-5-w4000-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..fba0e0403ecc81c2d698084a2c1e4ce57c67e8d4 --- /dev/null +++ b/results/g10-L6-lr5e-5-w4000-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr5e-5-w4000-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.80445229212443, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr5e-5-w4000-s1.json b/results/g10-L6-lr5e-5-w4000-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..15311315f01e64839683f95b0a00565244d1d971 --- /dev/null +++ b/results/g10-L6-lr5e-5-w4000-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr5e-5-w4000-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.78053449789683, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr5e-5-w4000-s2.json b/results/g10-L6-lr5e-5-w4000-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..0774b01c45c0ea985cdc8e042f59dd95d8e8b4b5 --- /dev/null +++ b/results/g10-L6-lr5e-5-w4000-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr5e-5-w4000-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 0.9975, + "3": 0.9975, + "4": 0.9975, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.812646587689718, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr5e-5-w500-s0.json b/results/g10-L6-lr5e-5-w500-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..e5162a5bf9fcedf777d7f76d7d744214780bc892 --- /dev/null +++ b/results/g10-L6-lr5e-5-w500-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr5e-5-w500-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.766953949133555, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr5e-5-w500-s1.json b/results/g10-L6-lr5e-5-w500-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..e704617619e7494a7fe0a8e89439cf465b5806e3 --- /dev/null +++ b/results/g10-L6-lr5e-5-w500-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr5e-5-w500-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.684520812829335, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g10-L6-lr5e-5-w500-s2.json b/results/g10-L6-lr5e-5-w500-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..2f12aaae59639299518986aa0b51f3f918e37a6f --- /dev/null +++ b/results/g10-L6-lr5e-5-w500-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 40000, + "lr": 5e-05, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g10-L6-lr5e-5-w500-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 11.703336783250172, + "compute": { + "params": 10649088, + "train_tokens": 174080000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d12-full-s0.json b/results/g11-d12-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..b9edb2cb8c47713fe9f73ffff6fb71d799989075 --- /dev/null +++ b/results/g11-d12-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-full-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.25991241534551, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d12-full-s1.json b/results/g11-d12-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..e95548d5a73b6ff7695ac458e6c1a1f46ae2f021 --- /dev/null +++ b/results/g11-d12-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-full-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 0.9975, + "12": 0.9975, + "16": 0.9975 + }, + "minutes": 22.634450801213582, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d12-full-s2.json b/results/g11-d12-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..3cc11f517ce6dd7c446fa12c9371ac6b269060fe --- /dev/null +++ b/results/g11-d12-full-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-full-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.555700461069744, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d12-full-s3.json b/results/g11-d12-full-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..7357d3b0ce5e13364241e3de4ec7a75da2b23ffc --- /dev/null +++ b/results/g11-d12-full-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-full-s3" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.571916433175407, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d12-mdlm-s0.json b/results/g11-d12-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..9e11a916c35e1debebf8033738c5db0c9938c4b7 --- /dev/null +++ b/results/g11-d12-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0675, + "2": 0.0575, + "3": 0.055, + "4": 0.055, + "6": 0.0625, + "8": 0.0525, + "12": 0.05, + "16": 0.05 + }, + "minutes": 12.725552813212078, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d12-mdlm-s1.json b/results/g11-d12-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..10314b6c250eaf008c3f11f3f6387b2d3dad55b2 --- /dev/null +++ b/results/g11-d12-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.798799379666645, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d12-mdlm-s2.json b/results/g11-d12-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..1c56eeb60af386682e677bb452ccae19edf16c0d --- /dev/null +++ b/results/g11-d12-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.69221136967341, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d12-mdlm-s3.json b/results/g11-d12-mdlm-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..c896826e5a18386d3432515b807c8d59f7df21a9 --- /dev/null +++ b/results/g11-d12-mdlm-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d12-mdlm-s3" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.729625848929087, + "compute": { + "params": 14194944, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d16-full-s0.json b/results/g11-d16-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..ae59ff49398382098d8ebe04afed866a8e8efb13 --- /dev/null +++ b/results/g11-d16-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-full-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 27.694663234551747, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d16-full-s1.json b/results/g11-d16-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..fec7bb95732e561aa75911088eeaf48949e4c873 --- /dev/null +++ b/results/g11-d16-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-full-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 0.9975, + "16": 0.9975 + }, + "minutes": 28.009663645426432, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d16-full-s2.json b/results/g11-d16-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..772e254191952ec18522b668d03e7ac92fca9966 --- /dev/null +++ b/results/g11-d16-full-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-full-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 27.92376529375712, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d16-full-s3.json b/results/g11-d16-full-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..f51b2ecb808b130c223615399a6e8edd4dc6dece --- /dev/null +++ b/results/g11-d16-full-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-full-s3" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 27.92292259136836, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d16-mdlm-s0.json b/results/g11-d16-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..2719d25b5c45303a5a3f13fd1f10eba42133f1f1 --- /dev/null +++ b/results/g11-d16-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 15.346402597427367, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d16-mdlm-s1.json b/results/g11-d16-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..c1dea8dbf7f04aa2aea3fa588b6a3542757dc110 --- /dev/null +++ b/results/g11-d16-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 0.9975, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 15.58473383585612, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d16-mdlm-s2.json b/results/g11-d16-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..f79576309785c87caa7a4c8739c99ddd61df10e1 --- /dev/null +++ b/results/g11-d16-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 15.94875235557556, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d16-mdlm-s3.json b/results/g11-d16-mdlm-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..f2f3aa63e4f69b51ed63bf83f5165ae8ff26750c --- /dev/null +++ b/results/g11-d16-mdlm-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d16-mdlm-s3" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 15.960274545351664, + "compute": { + "params": 17740800, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d24-full-s0.json b/results/g11-d24-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..0391fc0f891ed7107dec0a55782b13b655331767 --- /dev/null +++ b/results/g11-d24-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 34.43513007164002, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d24-full-s1.json b/results/g11-d24-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..7bcb542d87c27c46c3521c3fa28c70b676fbedf4 --- /dev/null +++ b/results/g11-d24-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 34.662449518839516, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d24-full-s2.json b/results/g11-d24-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..7532de93544b305bb392042054704be8fc047a89 --- /dev/null +++ b/results/g11-d24-full-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-full-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 34.210229539871214, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d24-full-s3.json b/results/g11-d24-full-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..e474d5de75c490bca9525e2cb2001d8d8f42d8fe --- /dev/null +++ b/results/g11-d24-full-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-full-s3" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 34.53418607711792, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d24-mdlm-s0.json b/results/g11-d24-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..f498e3171eb3be4b0690e1ed57dd68f9e56490fe --- /dev/null +++ b/results/g11-d24-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 18.136019909381865, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d24-mdlm-s1.json b/results/g11-d24-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..a7bbdf7365ce17831b0f8e38391bcc326f7087e9 --- /dev/null +++ b/results/g11-d24-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 18.02810135682424, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d24-mdlm-s2.json b/results/g11-d24-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..da64be0153b14bdf448b96819df1afa5ce631d82 --- /dev/null +++ b/results/g11-d24-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 17.997650043169656, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d24-mdlm-s3.json b/results/g11-d24-mdlm-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..ff9af61f40645f1dfde832cdad1f805cb85ebfe7 --- /dev/null +++ b/results/g11-d24-mdlm-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 24, + "seq_len": 76, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 10, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d24-mdlm-s3" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 18.390210171540577, + "compute": { + "params": 17740800, + "train_tokens": 583680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d32-mdlm-s0.json b/results/g11-d32-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..19ae4f369d171187109dfa6de5dd51ec93807adb --- /dev/null +++ b/results/g11-d32-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 32, + "seq_len": 100, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 12, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d32-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 26.76668523947398, + "compute": { + "params": 21286656, + "train_tokens": 768000000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d32-mdlm-s1.json b/results/g11-d32-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..d4ad8267425a99fa3f1b45b836d2c9d836db8264 --- /dev/null +++ b/results/g11-d32-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 32, + "seq_len": 100, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 12, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d32-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 27.06659528017044, + "compute": { + "params": 21286656, + "train_tokens": 768000000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d32-mdlm-s2.json b/results/g11-d32-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..51ba9d2bcddd1ec45f425e0a1b6a4b6d7b5c1d6c --- /dev/null +++ b/results/g11-d32-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 32, + "seq_len": 100, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 12, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d32-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 27.03916134039561, + "compute": { + "params": 21286656, + "train_tokens": 768000000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d32-mdlm-s3.json b/results/g11-d32-mdlm-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..ae0b5e84316ec70fcf4fc9ac52270953bdb5a41f --- /dev/null +++ b/results/g11-d32-mdlm-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 32, + "seq_len": 100, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 12, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d32-mdlm-s3" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 26.78021657069524, + "compute": { + "params": 21286656, + "train_tokens": 768000000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d8-full-s0.json b/results/g11-d8-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..3869a7c88e9005222579395009178749a1fea1e8 --- /dev/null +++ b/results/g11-d8-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-full-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.0401043176651, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d8-full-s1.json b/results/g11-d8-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..da6508f4b7f3ca32c483a46541e183f3e9eff20f --- /dev/null +++ b/results/g11-d8-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-full-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.53541429042816, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d8-full-s2.json b/results/g11-d8-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..3d4e187c709ad19eade07ddf0509d3879554fbc7 --- /dev/null +++ b/results/g11-d8-full-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-full-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.313481549421947, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d8-full-s3.json b/results/g11-d8-full-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..0557dddbd5c06a0052efcae983175454bda9777b --- /dev/null +++ b/results/g11-d8-full-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-full-s3" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 22.55289461215337, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g11-d8-mdlm-s0.json b/results/g11-d8-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..923b593109a3ee970f4a3b02205b5942b219bc23 --- /dev/null +++ b/results/g11-d8-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.785103106498719, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d8-mdlm-s1.json b/results/g11-d8-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..2c519bba0da1c488d50281b4b97b8606fe19f8f5 --- /dev/null +++ b/results/g11-d8-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.69441944360733, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d8-mdlm-s2.json b/results/g11-d8-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..eefe867b73b303a32a1d1fd7fcc62ac9e5297324 --- /dev/null +++ b/results/g11-d8-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.605760153134664, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g11-d8-mdlm-s3.json b/results/g11-d8-mdlm-s3.json new file mode 100644 index 0000000000000000000000000000000000000000..8c82b42401ec56fddf4c3bfd0331da37e6c27cb7 --- /dev/null +++ b/results/g11-d8-mdlm-s3.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 800000, + "n_eval": 400, + "eval_bs": 100, + "d": 384, + "layers": 8, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0001, + "warmup": 4000, + "seed": 3, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g11-d8-mdlm-s3" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 12.767644548416138, + "compute": { + "params": 14194944, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g12-english-full.json b/results/g12-english-full.json new file mode 100644 index 0000000000000000000000000000000000000000..067149d49a3d20faa9b87e73b307b5ddb4bb2f63 --- /dev/null +++ b/results/g12-english-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "english", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-english-full" + }, + "accuracy_by_passes": { + "1": 0.9551532451923077, + "2": 0.9514723557692308, + "4": 0.9485426682692308, + "8": 0.9514723557692308, + "16": 0.9514723557692308, + "32": 0.9514723557692308, + "ar": 0.07684795673076923 + }, + "minutes": 4.2856955687205, + "compute": { + "params": 4760576, + "answer_len_mean": 10.067000389099121 + } +} \ No newline at end of file diff --git a/results/g12-english-mdlm.json b/results/g12-english-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..99c4c30ecf6a9985319aa27c0bc9135293dbe9cb --- /dev/null +++ b/results/g12-english-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "english", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-english-mdlm" + }, + "accuracy_by_passes": { + "1": 0.9499699519230769, + "2": 0.9477914663461539, + "4": 0.9509465144230769, + "8": 0.9507211538461539, + "16": 0.9516977163461539, + "32": 0.9516977163461539, + "ar": 0.07782451923076923 + }, + "minutes": 2.4468347986539203, + "compute": { + "params": 4760576, + "answer_len_mean": 10.067000389099121 + } +} \ No newline at end of file diff --git a/results/g12-finnish-full.json b/results/g12-finnish-full.json new file mode 100644 index 0000000000000000000000000000000000000000..b83b05b06ee140e99cd51d88a78b6713385639c4 --- /dev/null +++ b/results/g12-finnish-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "finnish", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-finnish-full" + }, + "accuracy_by_passes": { + "1": 0.5093256353021978, + "2": 0.511461195054945, + "4": 0.5276978880494505, + "8": 0.5495793269230769, + "16": 0.5583683894230769, + "32": 0.5583683894230769, + "ar": 0.05099587912087912 + }, + "minutes": 4.342159048716227, + "compute": { + "params": 4769792, + "answer_len_mean": 15.12968635559082 + } +} \ No newline at end of file diff --git a/results/g12-finnish-mdlm.json b/results/g12-finnish-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..f5697ff98493966572748f7e094f3c08aba141e5 --- /dev/null +++ b/results/g12-finnish-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "finnish", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-finnish-mdlm" + }, + "accuracy_by_passes": { + "1": 0.4858237465659341, + "2": 0.5041959993131868, + "4": 0.5329133756868132, + "8": 0.545640882554945, + "16": 0.5636160714285714, + "32": 0.5655691964285714, + "ar": 0.05392556662087912 + }, + "minutes": 2.445442732175191, + "compute": { + "params": 4769792, + "answer_len_mean": 15.12968635559082 + } +} \ No newline at end of file diff --git a/results/g12-georgian-full.json b/results/g12-georgian-full.json new file mode 100644 index 0000000000000000000000000000000000000000..97c7b4e56d9ed8ac889c74320721f24557ee3f98 --- /dev/null +++ b/results/g12-georgian-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "georgian", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-georgian-full" + }, + "accuracy_by_passes": { + "1": 0.9432091346153846, + "2": 0.9434344951923077, + "4": 0.9402794471153846, + "8": 0.9424579326923077, + "16": 0.9412560096153846, + "32": 0.9412560096153846, + "ar": 0.1258263221153846 + }, + "minutes": 4.345291066169739, + "compute": { + "params": 4766208, + "answer_len_mean": 9.932000160217285 + } +} \ No newline at end of file diff --git a/results/g12-georgian-mdlm.json b/results/g12-georgian-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..daea15da2d9c5232d07eac0f9d3758c9afeed057 --- /dev/null +++ b/results/g12-georgian-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "georgian", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-georgian-mdlm" + }, + "accuracy_by_passes": { + "1": 0.9539513221153846, + "2": 0.9522235576923077, + "4": 0.9502704326923077, + "8": 0.9532001201923077, + "16": 0.9532001201923077, + "32": 0.9532001201923077, + "ar": 0.1258263221153846 + }, + "minutes": 2.4396663467089335, + "compute": { + "params": 4766208, + "answer_len_mean": 9.932000160217285 + } +} \ No newline at end of file diff --git a/results/g12-german-full.json b/results/g12-german-full.json new file mode 100644 index 0000000000000000000000000000000000000000..96d815b6153822c7afcbea058e869622663a17a5 --- /dev/null +++ b/results/g12-german-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "german", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-german-full" + }, + "accuracy_by_passes": { + "1": 0.7893359375, + "2": 0.792265625, + "4": 0.7903125, + "8": 0.79296875, + "16": 0.79296875, + "32": 0.79296875, + "ar": 0.0776171875 + }, + "minutes": 4.302863411108652, + "compute": { + "params": 4776448, + "answer_len_mean": 10.743975639343262 + } +} \ No newline at end of file diff --git a/results/g12-german-mdlm.json b/results/g12-german-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..816a6d0d2f1a730879203b48c74c744ecb1a9255 --- /dev/null +++ b/results/g12-german-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "german", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-german-mdlm" + }, + "accuracy_by_passes": { + "1": 0.7555859375, + "2": 0.7710546875, + "4": 0.7808203125, + "8": 0.7911328125, + "16": 0.7950390625, + "32": 0.7950390625, + "ar": 0.076640625 + }, + "minutes": 2.451639501253764, + "compute": { + "params": 4776448, + "answer_len_mean": 10.743975639343262 + } +} \ No newline at end of file diff --git a/results/g12-navajo-full.json b/results/g12-navajo-full.json new file mode 100644 index 0000000000000000000000000000000000000000..f0d48cae6ed01231195dfdb00a4093fad559a388 --- /dev/null +++ b/results/g12-navajo-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "navajo", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-navajo-full" + }, + "accuracy_by_passes": { + "1": 0.29529747596153844, + "2": 0.30814302884615385, + "4": 0.3108473557692308, + "8": 0.33743990384615385, + "16": 0.33841646634615385, + "32": 0.33841646634615385, + "ar": 0.06332632211538461 + }, + "minutes": 4.333148336410522, + "compute": { + "params": 4765696, + "answer_len_mean": 10.005999565124512 + } +} \ No newline at end of file diff --git a/results/g12-navajo-mdlm.json b/results/g12-navajo-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..22826c3da3097074c7634d8a0d66c6a753e25b5f --- /dev/null +++ b/results/g12-navajo-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "navajo", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-navajo-mdlm" + }, + "accuracy_by_passes": { + "1": 0.3137019230769231, + "2": 0.3273737980769231, + "4": 0.37169471153846156, + "8": 0.4004657451923077, + "16": 0.4024188701923077, + "32": 0.4024188701923077, + "ar": 0.08473557692307693 + }, + "minutes": 2.4314921736717223, + "compute": { + "params": 4765696, + "answer_len_mean": 10.005999565124512 + } +} \ No newline at end of file diff --git a/results/g12-russian-full.json b/results/g12-russian-full.json new file mode 100644 index 0000000000000000000000000000000000000000..1a54f4f65fcfd9537b754d954e6b9420ab14508a --- /dev/null +++ b/results/g12-russian-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "russian", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-russian-full" + }, + "accuracy_by_passes": { + "1": 0.7271740845959596, + "2": 0.7232678345959596, + "4": 0.7213147095959596, + "8": 0.7218868371212122, + "16": 0.7199337121212122, + "32": 0.7199337121212122, + "ar": 0.03911182133838384 + }, + "minutes": 4.282116937637329, + "compute": { + "params": 4770304, + "answer_len_mean": 11.276381492614746 + } +} \ No newline at end of file diff --git a/results/g12-russian-mdlm.json b/results/g12-russian-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..654c21787fbfd73555a23d3e7d44011c2631c511 --- /dev/null +++ b/results/g12-russian-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "russian", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-russian-mdlm" + }, + "accuracy_by_passes": { + "1": 0.6910412720959596, + "2": 0.7068339646464646, + "4": 0.7132161458333333, + "8": 0.7106908933080809, + "16": 0.7165502683080809, + "32": 0.7165502683080809, + "ar": 0.035896070075757576 + }, + "minutes": 2.463564932346344, + "compute": { + "params": 4770304, + "answer_len_mean": 11.276381492614746 + } +} \ No newline at end of file diff --git a/results/g12-spanish-full.json b/results/g12-spanish-full.json new file mode 100644 index 0000000000000000000000000000000000000000..14da780320d3d4db5084d0ac850d746355f2de97 --- /dev/null +++ b/results/g12-spanish-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "spanish", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-spanish-full" + }, + "accuracy_by_passes": { + "1": 0.8490835336538461, + "2": 0.8490835336538461, + "4": 0.8442007211538461, + "8": 0.8459284855769231, + "16": 0.8471304086538461, + "32": 0.8471304086538461, + "ar": 0.12176983173076923 + }, + "minutes": 4.297809946537018, + "compute": { + "params": 4763136, + "answer_len_mean": 11.312000274658203 + } +} \ No newline at end of file diff --git a/results/g12-spanish-mdlm.json b/results/g12-spanish-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..6801d3b6afca32e8c2a3bb725e80243c31597718 --- /dev/null +++ b/results/g12-spanish-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "spanish", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-spanish-mdlm" + }, + "accuracy_by_passes": { + "1": 0.8393179086538461, + "2": 0.8310546875, + "4": 0.8310546875, + "8": 0.8373647836538461, + "16": 0.8407451923076923, + "32": 0.8407451923076923, + "ar": 0.1229717548076923 + }, + "minutes": 2.442912705739339, + "compute": { + "params": 4763136, + "answer_len_mean": 11.312000274658203 + } +} \ No newline at end of file diff --git a/results/g12-turkish-full.json b/results/g12-turkish-full.json new file mode 100644 index 0000000000000000000000000000000000000000..3a2f43ea8175756efd338dda7946cfe013ea2a3c --- /dev/null +++ b/results/g12-turkish-full.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "turkish", + "size": "medium", + "mode": "full", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-turkish-full" + }, + "accuracy_by_passes": { + "1": 0.8020938340053764, + "2": 0.7859753024193549, + "4": 0.7945228494623656, + "8": 0.8103998655913979, + "16": 0.8119854670698925, + "32": 0.8119854670698925, + "ar": 0.10095346102150538 + }, + "minutes": 4.304969179630279, + "compute": { + "params": 4772864, + "answer_len_mean": 13.684530258178711 + } +} \ No newline at end of file diff --git a/results/g12-turkish-mdlm.json b/results/g12-turkish-mdlm.json new file mode 100644 index 0000000000000000000000000000000000000000..443592a4b3a93c623406da85a156281a60880eae --- /dev/null +++ b/results/g12-turkish-mdlm.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "turkish", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 15000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g12-turkish-mdlm" + }, + "accuracy_by_passes": { + "1": 0.7749915994623656, + "2": 0.7828040994623656, + "4": 0.8090557795698925, + "8": 0.8204070060483871, + "16": 0.8217510920698925, + "32": 0.8217510920698925, + "ar": 0.09607064852150538 + }, + "minutes": 2.4770234982172648, + "compute": { + "params": 4772864, + "answer_len_mean": 13.684530258178711 + } +} \ No newline at end of file diff --git a/results/g6-d10-distill-s0.json b/results/g6-d10-distill-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..415d8d34a271907402c92b202e53b45c947a3f58 --- /dev/null +++ b/results/g6-d10-distill-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-distill-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.779380659262339, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d10-distill-s1.json b/results/g6-d10-distill-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..ce3083b189debb23bbbc11786f06e1eb864c967e --- /dev/null +++ b/results/g6-d10-distill-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-distill-s1" + }, + "accuracy_by_passes": { + "1": 0.248, + "2": 0.22600000000000003, + "3": 0.202, + "4": 0.184, + "6": 0.10200000000000001, + "8": 0.10200000000000001, + "12": 0.10200000000000001, + "16": 0.10200000000000001 + }, + "minutes": 8.710233815511067, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d10-distill-s2.json b/results/g6-d10-distill-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..54521763e9067cf59c4a01ffdd30bf72730b9405 --- /dev/null +++ b/results/g6-d10-distill-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-distill-s2" + }, + "accuracy_by_passes": { + "1": 0.9800000000000001, + "2": 0.958, + "3": 0.944, + "4": 0.9259999999999999, + "6": 0.9099999999999999, + "8": 0.9099999999999999, + "12": 0.9099999999999999, + "16": 0.9099999999999999 + }, + "minutes": 8.797441339492797, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d10-full-s0.json b/results/g6-d10-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..70708a73cec286236e55a9fb66019bd3b7fb608d --- /dev/null +++ b/results/g6-d10-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-full-s0" + }, + "accuracy_by_passes": { + "1": 0.998, + "2": 1.0, + "3": 0.998, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 8.695208617051442, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d10-full-s1.json b/results/g6-d10-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..701e8f07387820d79e98edd76b3123e85eb991dd --- /dev/null +++ b/results/g6-d10-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-full-s1" + }, + "accuracy_by_passes": { + "1": 0.018, + "2": 0.019999999999999997, + "3": 0.014000000000000002, + "4": 0.014000000000000002, + "6": 0.02, + "8": 0.02, + "12": 0.02, + "16": 0.02 + }, + "minutes": 8.758371651172638, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d10-full-s2.json b/results/g6-d10-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..484733daa45c31e6a35b235d84fdeb46695f417b --- /dev/null +++ b/results/g6-d10-full-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-full-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.786107873916626, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d10-matched-s0.json b/results/g6-d10-matched-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..19750381694cf3550a425e1fb3086db33a160f94 --- /dev/null +++ b/results/g6-d10-matched-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-matched-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.058370518684387, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d10-matched-s1.json b/results/g6-d10-matched-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..4ea6a910f886e1e1ee0e2f9127b23da0659a882d --- /dev/null +++ b/results/g6-d10-matched-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-matched-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.011176633834839, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d10-matched-s2.json b/results/g6-d10-matched-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..07165bd4283e20414a4a00e87a7329060604988c --- /dev/null +++ b/results/g6-d10-matched-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-matched-s2" + }, + "accuracy_by_passes": { + "1": 0.01, + "2": 0.008, + "3": 0.012, + "4": 0.014000000000000002, + "6": 0.006, + "8": 0.008, + "12": 0.008, + "16": 0.008 + }, + "minutes": 5.012523424625397, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d10-mdlm-s0.json b/results/g6-d10-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..af2f88bb5e1a12318c75cb52668f88ee3b979f76 --- /dev/null +++ b/results/g6-d10-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.107721225420634, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d10-mdlm-s1.json b/results/g6-d10-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..cc051c77008c3dd24ba958068c6be9d963b1588d --- /dev/null +++ b/results/g6-d10-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.079894876480102, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d10-mdlm-s2.json b/results/g6-d10-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..15477dc62e8feab80d6b6faf3e5b108fc6d64b72 --- /dev/null +++ b/results/g6-d10-mdlm-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d10-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.0564715107282, + "compute": { + "params": 10649088, + "train_tokens": 130560000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d12-distill-s0.json b/results/g6-d12-distill-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..5a1d87cf7dd63dba02603cf1e10e4a3360f9fc46 --- /dev/null +++ b/results/g6-d12-distill-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-distill-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.825406022866567, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d12-distill-s1.json b/results/g6-d12-distill-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..80a3c49d701e991583a52423e1699bf81152cf77 --- /dev/null +++ b/results/g6-d12-distill-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-distill-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.928057714303334, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d12-distill-s2.json b/results/g6-d12-distill-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..8cb3dd09c1ddbe45f074e98d9940f85346bf61b1 --- /dev/null +++ b/results/g6-d12-distill-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-distill-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.977498888969421, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d12-full-s0.json b/results/g6-d12-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..1d48211605b546a0982515e0c8c3ef86fd23915a --- /dev/null +++ b/results/g6-d12-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-full-s0" + }, + "accuracy_by_passes": { + "1": 0.79, + "2": 0.696, + "3": 0.688, + "4": 0.6799999999999999, + "6": 0.6519999999999999, + "8": 0.634, + "12": 0.628, + "16": 0.628 + }, + "minutes": 8.752611108620961, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d12-full-s1.json b/results/g6-d12-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..485fc28ff38595f869900ed481f7170c8c47d391 --- /dev/null +++ b/results/g6-d12-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.955190853277843, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d12-full-s2.json b/results/g6-d12-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..761b56d56063137e9dbdebcb9029945e7f564f63 --- /dev/null +++ b/results/g6-d12-full-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-full-s2" + }, + "accuracy_by_passes": { + "1": 0.086, + "2": 0.08399999999999999, + "3": 0.066, + "4": 0.064, + "6": 0.044, + "8": 0.052000000000000005, + "12": 0.044, + "16": 0.044 + }, + "minutes": 8.969864714145661, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d12-matched-s0.json b/results/g6-d12-matched-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..1d925f46fb520e68a82b1df857b0a9ab4569426d --- /dev/null +++ b/results/g6-d12-matched-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-matched-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 4.976008685429891, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d12-matched-s1.json b/results/g6-d12-matched-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..e5993b75630705589a0936c2817bfb6ea68d72ff --- /dev/null +++ b/results/g6-d12-matched-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-matched-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 4.9875031073888145, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d12-matched-s2.json b/results/g6-d12-matched-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..3eab65e50ae086283232555bc97e01fb9bae9885 --- /dev/null +++ b/results/g6-d12-matched-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-matched-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.047432001431783, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d12-mdlm-s0.json b/results/g6-d12-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..2ccd7f57e783911790ee25daf23db8f3510dd502 --- /dev/null +++ b/results/g6-d12-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.098425841331482, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d12-mdlm-s1.json b/results/g6-d12-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..eefa3006948c469e4cd1856f4450d1c56dffb811 --- /dev/null +++ b/results/g6-d12-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.057489677270254, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d12-mdlm-s2.json b/results/g6-d12-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..1450a048acfc82b0e15c23197d4f6e825f8eaf94 --- /dev/null +++ b/results/g6-d12-mdlm-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d12-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.111955551306407, + "compute": { + "params": 10649088, + "train_tokens": 153600000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d14-distill-s0.json b/results/g6-d14-distill-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..b0ea2573d6098193fb98b9617d2b7338e35c8720 --- /dev/null +++ b/results/g6-d14-distill-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-distill-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.88578055302302, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d14-distill-s1.json b/results/g6-d14-distill-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..3b86b435493987820528d120e236d5d23c2a8990 --- /dev/null +++ b/results/g6-d14-distill-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-distill-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.98511467774709, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d14-distill-s2.json b/results/g6-d14-distill-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..ba39ba085971003c1a2c880867b8afb90c86224a --- /dev/null +++ b/results/g6-d14-distill-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-distill-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.918712143103281, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d14-full-s0.json b/results/g6-d14-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..43ca17a84ddf57a1ff0dee2fc5463f4d0bc624e1 --- /dev/null +++ b/results/g6-d14-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.931685078144074, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d14-full-s1.json b/results/g6-d14-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..912a387e17201d42bd0b05642774b06bbbe4ea3a --- /dev/null +++ b/results/g6-d14-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.95813860098521, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d14-full-s2.json b/results/g6-d14-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..59e618901f32f627d9d2e215db501cbd8fdd9819 --- /dev/null +++ b/results/g6-d14-full-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-full-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.9239621758461, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d14-matched-s0.json b/results/g6-d14-matched-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..a25d3ccd09f8a52aa35f7e144231f544600421a9 --- /dev/null +++ b/results/g6-d14-matched-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-matched-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.087729636828104, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d14-matched-s1.json b/results/g6-d14-matched-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..f176c5d5021e265c56b569ddb05b160a793bfa7b --- /dev/null +++ b/results/g6-d14-matched-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-matched-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.070337331295013, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d14-matched-s2.json b/results/g6-d14-matched-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..ac36bde2c77789dae34b08624e7048910dcc8958 --- /dev/null +++ b/results/g6-d14-matched-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-matched-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.14624871412913, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d14-mdlm-s0.json b/results/g6-d14-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..a9e4efa95b930b1c04cc38757a2e53eb18e53dc3 --- /dev/null +++ b/results/g6-d14-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.059604934851328, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d14-mdlm-s1.json b/results/g6-d14-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..af7c95601988b861d83aab65dccce59ef7d52e8f --- /dev/null +++ b/results/g6-d14-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.145091533660889, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d14-mdlm-s2.json b/results/g6-d14-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..0cecb486bad344f7a210cf6c7f2c000616ccd602 --- /dev/null +++ b/results/g6-d14-mdlm-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d14-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.115038243929545, + "compute": { + "params": 10649088, + "train_tokens": 176640000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d16-distill-s0.json b/results/g6-d16-distill-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..3cbdb74325d31b5ac4b86c4bb4afbb9e60f7e115 --- /dev/null +++ b/results/g6-d16-distill-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-distill-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.974436064561209, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d16-distill-s1.json b/results/g6-d16-distill-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..2aca7337724bc94e80e92999e0e98a40110736cb --- /dev/null +++ b/results/g6-d16-distill-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-distill-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.913798876603444, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d16-distill-s2.json b/results/g6-d16-distill-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..9944d6e2b5a6cc037b404e9156165d9597de2d29 --- /dev/null +++ b/results/g6-d16-distill-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "distill", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-distill-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.916855800151826, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d16-full-s0.json b/results/g6-d16-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..3bda6415ad0f75935d09574775bee38c7d1cca12 --- /dev/null +++ b/results/g6-d16-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 9.008466629187266, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d16-full-s1.json b/results/g6-d16-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..89271b4e6d3681b00ce2fae80999a28f10f74115 --- /dev/null +++ b/results/g6-d16-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.92288788954417, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d16-full-s2.json b/results/g6-d16-full-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..f2ef361d125502ebe4ef2924da75ab9375b8c0f0 --- /dev/null +++ b/results/g6-d16-full-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-full-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 8.8776793162028, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g6-d16-matched-s0.json b/results/g6-d16-matched-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..045fb73e9bd641e5aeacc1b99f240c7eaaba90d8 --- /dev/null +++ b/results/g6-d16-matched-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-matched-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.029508344332377, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d16-matched-s1.json b/results/g6-d16-matched-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..5773aafbe8a1f8f4bde2bdfedb4c18f357c09970 --- /dev/null +++ b/results/g6-d16-matched-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-matched-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.067178654670715, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d16-matched-s2.json b/results/g6-d16-matched-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..726a6908a352722f1a47132eaca95551c93104a7 --- /dev/null +++ b/results/g6-d16-matched-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "matched", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-matched-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.037939771016439, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d16-mdlm-s0.json b/results/g6-d16-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..f5018f0a8ecf04a6ea571a9230b94a31e9b4747c --- /dev/null +++ b/results/g6-d16-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.177178863684336, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d16-mdlm-s1.json b/results/g6-d16-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..af747aaaf425ee395d8f73a0e23e04deead9c285 --- /dev/null +++ b/results/g6-d16-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.105695509910584, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g6-d16-mdlm-s2.json b/results/g6-d16-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..b789bb82fa4b5c464dd1ceb25ee59a5f0a3ac36c --- /dev/null +++ b/results/g6-d16-mdlm-s2.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 300000, + "n_eval": 500, + "eval_bs": 100, + "d": 384, + "layers": 6, + "heads": 6, + "pe": "rope", + "bs": 128, + "steps": 30000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g6-d16-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 5.07826308409373, + "compute": { + "params": 10649088, + "train_tokens": 199680000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d12-L12-dm512-st60000-full-s0.json b/results/g7-d12-L12-dm512-st60000-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..3cc88b73b405ee27ecf2c3829761734783bdbfa9 --- /dev/null +++ b/results/g7-d12-L12-dm512-st60000-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L12-dm512-st60000-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 33.27085047165553, + "compute": { + "params": 37819392, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d12-L12-dm512-st60000-full-s1.json b/results/g7-d12-L12-dm512-st60000-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..028966b17663b3a5e29be708a96bdc82aa013472 --- /dev/null +++ b/results/g7-d12-L12-dm512-st60000-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L12-dm512-st60000-full-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 33.84639033079147, + "compute": { + "params": 37819392, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d12-L12-dm512-st60000-mdlm-s0.json b/results/g7-d12-L12-dm512-st60000-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..04efc4f2f4de10d36bd0669592646c6f42a0e9ff --- /dev/null +++ b/results/g7-d12-L12-dm512-st60000-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L12-dm512-st60000-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 18.751598886648814, + "compute": { + "params": 37819392, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d12-L12-dm512-st60000-mdlm-s1.json b/results/g7-d12-L12-dm512-st60000-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..9e37be2935050b7fc0f7be64c87bc340ef3332fa --- /dev/null +++ b/results/g7-d12-L12-dm512-st60000-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L12-dm512-st60000-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 18.882293991247813, + "compute": { + "params": 37819392, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st120000-full-s0.json b/results/g7-d12-L8-dm512-st120000-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..b6f4ca6ecbea28f88dffe82d7b6cb5167e5b2563 --- /dev/null +++ b/results/g7-d12-L8-dm512-st120000-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st120000-full-s0" + }, + "accuracy_by_passes": { + "1": 0.9975, + "2": 0.9975, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 45.572937782605486, + "compute": { + "params": 25218048, + "train_tokens": 614400000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st120000-full-s1.json b/results/g7-d12-L8-dm512-st120000-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..46c07593949c4e412c60613afc273d72d3821938 --- /dev/null +++ b/results/g7-d12-L8-dm512-st120000-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st120000-full-s1" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 46.11180016597112, + "compute": { + "params": 25218048, + "train_tokens": 614400000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st120000-mdlm-s0.json b/results/g7-d12-L8-dm512-st120000-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..2c9259a6b6a4637aa0c00fe535186bac434edf46 --- /dev/null +++ b/results/g7-d12-L8-dm512-st120000-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st120000-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.93002525965373, + "compute": { + "params": 25218048, + "train_tokens": 614400000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st120000-mdlm-s1.json b/results/g7-d12-L8-dm512-st120000-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..153e45d5cd01ef1e923f3d2b9f334025ae495a8f --- /dev/null +++ b/results/g7-d12-L8-dm512-st120000-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st120000-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.709957778453827, + "compute": { + "params": 25218048, + "train_tokens": 614400000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st60000-full-s0.json b/results/g7-d12-L8-dm512-st60000-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..68802e8db7bcb36ae36ce61950f8db4b24a9fdb4 --- /dev/null +++ b/results/g7-d12-L8-dm512-st60000-full-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st60000-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 22.985121659437816, + "compute": { + "params": 25218048, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st60000-full-s1.json b/results/g7-d12-L8-dm512-st60000-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..f21ae340242fad300d3a4f3274c329ebea6037b6 --- /dev/null +++ b/results/g7-d12-L8-dm512-st60000-full-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "full", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st60000-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0025, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 23.20198518037796, + "compute": { + "params": 25218048, + "train_tokens": 307200000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st60000-mdlm-s0.json b/results/g7-d12-L8-dm512-st60000-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..fb5b86ce138fd54cb1f14b0d7f4dfd9b179c9e2f --- /dev/null +++ b/results/g7-d12-L8-dm512-st60000-mdlm-s0.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st60000-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 13.141208012898764, + "compute": { + "params": 25218048, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d12-L8-dm512-st60000-mdlm-s1.json b/results/g7-d12-L8-dm512-st60000-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..a1b3e6e62d46b96a0e4fab34f75a8193e2e9b511 --- /dev/null +++ b/results/g7-d12-L8-dm512-st60000-mdlm-s1.json @@ -0,0 +1,38 @@ +{ + "args": { + "mode": "mdlm", + "K": 4, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d12-L8-dm512-st60000-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 13.044934821128845, + "compute": { + "params": 25218048, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm512-st60000-full-s0.json b/results/g7-d16-L12-dm512-st60000-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..908a5ed13e82dfb9ae05301a1478c22fea5fcad0 --- /dev/null +++ b/results/g7-d16-L12-dm512-st60000-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm512-st60000-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 39.177851510047915, + "compute": { + "params": 37819392, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm512-st60000-full-s1.json b/results/g7-d16-L12-dm512-st60000-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..9d339f59fdaf84c6b69cd737e6ffea796744764a --- /dev/null +++ b/results/g7-d16-L12-dm512-st60000-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm512-st60000-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 39.49690783818563, + "compute": { + "params": 37819392, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm512-st60000-mdlm-s0.json b/results/g7-d16-L12-dm512-st60000-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..b37ca9737668014706ccfeabb815a7964c437aba --- /dev/null +++ b/results/g7-d16-L12-dm512-st60000-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm512-st60000-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 20.990086873372395, + "compute": { + "params": 37819392, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm512-st60000-mdlm-s1.json b/results/g7-d16-L12-dm512-st60000-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..4cd9a709f9dc9bb2ed14106be6b6bd6ac72064a7 --- /dev/null +++ b/results/g7-d16-L12-dm512-st60000-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm512-st60000-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 20.754307957490287, + "compute": { + "params": 37819392, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm768-st120000-full-s0.json b/results/g7-d16-L12-dm768-st120000-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..184f102fdf9999e90df1ccbb4452ee9bd0d9b408 --- /dev/null +++ b/results/g7-d16-L12-dm768-st120000-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 768, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm768-st120000-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 110.15977214972177, + "compute": { + "params": 85040640, + "train_tokens": 798720000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm768-st120000-full-s1.json b/results/g7-d16-L12-dm768-st120000-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..98d283c17c7a07dbe88b28f49113ee14521b321f --- /dev/null +++ b/results/g7-d16-L12-dm768-st120000-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 768, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm768-st120000-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 110.63871949513754, + "compute": { + "params": 85040640, + "train_tokens": 798720000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm768-st120000-mdlm-s0.json b/results/g7-d16-L12-dm768-st120000-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..debc5a1b8045a1a0cb3fad8edfed3a1ad9ae1beb --- /dev/null +++ b/results/g7-d16-L12-dm768-st120000-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 768, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm768-st120000-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 60.15123441616694, + "compute": { + "params": 85040640, + "train_tokens": 798720000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d16-L12-dm768-st120000-mdlm-s1.json b/results/g7-d16-L12-dm768-st120000-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..dc0f1771a428f9e79922028c150bcf042d9c994d --- /dev/null +++ b/results/g7-d16-L12-dm768-st120000-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 768, + "layers": 12, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 120000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L12-dm768-st120000-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 60.09449849923452, + "compute": { + "params": 85040640, + "train_tokens": 798720000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d16-L8-dm512-st60000-full-s0.json b/results/g7-d16-L8-dm512-st60000-full-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..84edeb93d58c7c2f38355ad097e28ddbd1d7b040 --- /dev/null +++ b/results/g7-d16-L8-dm512-st60000-full-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L8-dm512-st60000-full-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 26.36857160727183, + "compute": { + "params": 25218048, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d16-L8-dm512-st60000-full-s1.json b/results/g7-d16-L8-dm512-st60000-full-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..4ef7a43d498a5f8e07b6dee76a38a58c5cfafb18 --- /dev/null +++ b/results/g7-d16-L8-dm512-st60000-full-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "full", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L8-dm512-st60000-full-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 26.923351856072745, + "compute": { + "params": 25218048, + "train_tokens": 399360000, + "fwd_per_step": 2 + } +} \ No newline at end of file diff --git a/results/g7-d16-L8-dm512-st60000-mdlm-s0.json b/results/g7-d16-L8-dm512-st60000-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..df857a33e9e91fbb0532b0d38a93d6ee84510d52 --- /dev/null +++ b/results/g7-d16-L8-dm512-st60000-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L8-dm512-st60000-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 14.354837755362192, + "compute": { + "params": 25218048, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g7-d16-L8-dm512-st60000-mdlm-s1.json b/results/g7-d16-L8-dm512-st60000-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..d7eb9c9c90f7e43e7e07b23bfc47abea29b23917 --- /dev/null +++ b/results/g7-d16-L8-dm512-st60000-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 4, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 8, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g7-d16-L8-dm512-st60000-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 14.260074738661448, + "compute": { + "params": 25218048, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d10-mdlm-s0.json b/results/g8-d10-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..0e40801ee7c186c2cc0829bf20dfd2b31c3a0942 --- /dev/null +++ b/results/g8-d10-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d10-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.088718926906587, + "compute": { + "params": 31518720, + "train_tokens": 261120000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d10-mdlm-s1.json b/results/g8-d10-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..4106c1b0a49186951096e13bcbc1751a71fc3687 --- /dev/null +++ b/results/g8-d10-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d10-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.067612473169962, + "compute": { + "params": 31518720, + "train_tokens": 261120000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d10-mdlm-s2.json b/results/g8-d10-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..bb189b6a41d4b4de15b845ddbd3579af6005d3a2 --- /dev/null +++ b/results/g8-d10-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d10-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.023622858524323, + "compute": { + "params": 31518720, + "train_tokens": 261120000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d10-prog-s0.json b/results/g8-d10-prog-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..be6190ef6eba9aa9ae1671ec9a8ad6dd34668395 --- /dev/null +++ b/results/g8-d10-prog-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d10-prog-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 24.809902067979177, + "compute": { + "params": 31518720, + "train_tokens": 261120000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d10-prog-s1.json b/results/g8-d10-prog-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..d01923b6b9a9ccbe406efbbd55cd2fd8d82de933 --- /dev/null +++ b/results/g8-d10-prog-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d10-prog-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.04687626759211, + "compute": { + "params": 31518720, + "train_tokens": 261120000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d10-prog-s2.json b/results/g8-d10-prog-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..e925693c070479d710b8f54aecbcdeb50d3e62da --- /dev/null +++ b/results/g8-d10-prog-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 10, + "seq_len": 34, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d10-prog-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.067701721191405, + "compute": { + "params": 31518720, + "train_tokens": 261120000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d12-mdlm-s0.json b/results/g8-d12-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..0920e65a306436d4e3e784012717181fc717be78 --- /dev/null +++ b/results/g8-d12-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d12-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.080562214056652, + "compute": { + "params": 31518720, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d12-mdlm-s1.json b/results/g8-d12-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..f7cf7c98494acb9a8e101cb94262f6a5e7185a93 --- /dev/null +++ b/results/g8-d12-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d12-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.080155086517333, + "compute": { + "params": 31518720, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d12-mdlm-s2.json b/results/g8-d12-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..8e73bec813e1777cd6cd4d8bc717e12c708168aa --- /dev/null +++ b/results/g8-d12-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d12-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.22305602232615, + "compute": { + "params": 31518720, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d12-prog-s0.json b/results/g8-d12-prog-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..5e4031de85689cef64fbfdb97cf8430f69610405 --- /dev/null +++ b/results/g8-d12-prog-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d12-prog-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.32835960785548, + "compute": { + "params": 31518720, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d12-prog-s1.json b/results/g8-d12-prog-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..e517e216db36aa25af4bda953c3bad45cbd27047 --- /dev/null +++ b/results/g8-d12-prog-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d12-prog-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.548884419600167, + "compute": { + "params": 31518720, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d12-prog-s2.json b/results/g8-d12-prog-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..e806a393d462762801bc3a62c8335090e19245fa --- /dev/null +++ b/results/g8-d12-prog-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 12, + "seq_len": 40, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d12-prog-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 25.20978350241979, + "compute": { + "params": 31518720, + "train_tokens": 307200000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d14-mdlm-s0.json b/results/g8-d14-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..9cd47518e9430b7f44597ccd3269fe79db1f1e7f --- /dev/null +++ b/results/g8-d14-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d14-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.765220610300698, + "compute": { + "params": 31518720, + "train_tokens": 353280000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d14-mdlm-s1.json b/results/g8-d14-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..32d3219655afdcf6f245b1e4e4000332ab5008cc --- /dev/null +++ b/results/g8-d14-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d14-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.982128580411274, + "compute": { + "params": 31518720, + "train_tokens": 353280000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d14-mdlm-s2.json b/results/g8-d14-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..1721de7efbf7e0610359e4db7e447078aefe9526 --- /dev/null +++ b/results/g8-d14-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d14-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 16.896530652046202, + "compute": { + "params": 31518720, + "train_tokens": 353280000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d14-prog-s0.json b/results/g8-d14-prog-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..9bd5a66a8e0f24de0ef14955bfe3bac8336fa1cd --- /dev/null +++ b/results/g8-d14-prog-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d14-prog-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 27.07018043200175, + "compute": { + "params": 31518720, + "train_tokens": 353280000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d14-prog-s1.json b/results/g8-d14-prog-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..8aa75c13f039b01f1fe9563e0d21f20726bcf999 --- /dev/null +++ b/results/g8-d14-prog-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d14-prog-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 27.127198421955107, + "compute": { + "params": 31518720, + "train_tokens": 353280000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d14-prog-s2.json b/results/g8-d14-prog-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..2343b2ff0e50ed62b6a44cc1b0bc21eff3869296 --- /dev/null +++ b/results/g8-d14-prog-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 14, + "seq_len": 46, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d14-prog-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 27.173766346772513, + "compute": { + "params": 31518720, + "train_tokens": 353280000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d16-mdlm-s0.json b/results/g8-d16-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..f67c984ff9f9a4ba2588d218e6ed64ce4bdd06ec --- /dev/null +++ b/results/g8-d16-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d16-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 17.51807384888331, + "compute": { + "params": 31518720, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d16-mdlm-s1.json b/results/g8-d16-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..eaeebfb3e9bcc68a3a20e613fed363e1f660a2e2 --- /dev/null +++ b/results/g8-d16-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d16-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 17.93566433986028, + "compute": { + "params": 31518720, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d16-mdlm-s2.json b/results/g8-d16-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..34c9146957131350cde7d6ceb081739ce35fee9c --- /dev/null +++ b/results/g8-d16-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d16-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 17.842382315794627, + "compute": { + "params": 31518720, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d16-prog-s0.json b/results/g8-d16-prog-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..337efa91f9472d288facd1b10d4a284092402426 --- /dev/null +++ b/results/g8-d16-prog-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d16-prog-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 29.578888940811158, + "compute": { + "params": 31518720, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d16-prog-s1.json b/results/g8-d16-prog-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..07b7075c6820f28021ef7e66661e508790edd1f5 --- /dev/null +++ b/results/g8-d16-prog-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d16-prog-s1" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 29.316549309094746, + "compute": { + "params": 31518720, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d16-prog-s2.json b/results/g8-d16-prog-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..054b427ea6c03e3740dab2f69ee02316690c06a2 --- /dev/null +++ b/results/g8-d16-prog-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 16, + "seq_len": 52, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d16-prog-s2" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 29.61462666193644, + "compute": { + "params": 31518720, + "train_tokens": 399360000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d8-mdlm-s0.json b/results/g8-d8-mdlm-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..c67549571efae9e65b8835e316c7f5ba95d7a311 --- /dev/null +++ b/results/g8-d8-mdlm-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d8-mdlm-s0" + }, + "accuracy_by_passes": { + "1": 0.0, + "2": 0.0, + "3": 0.0, + "4": 0.0, + "6": 0.0, + "8": 0.0, + "12": 0.0, + "16": 0.0 + }, + "minutes": 15.805170357227325, + "compute": { + "params": 31518720, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d8-mdlm-s1.json b/results/g8-d8-mdlm-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..23af162534b7a0114f94041acd35e68cd41e9657 --- /dev/null +++ b/results/g8-d8-mdlm-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d8-mdlm-s1" + }, + "accuracy_by_passes": { + "1": 0.0625, + "2": 0.045000000000000005, + "3": 0.052500000000000005, + "4": 0.05, + "6": 0.045, + "8": 0.0475, + "12": 0.0475, + "16": 0.0475 + }, + "minutes": 15.977666087945302, + "compute": { + "params": 31518720, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d8-mdlm-s2.json b/results/g8-d8-mdlm-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..dd1a7ab7c44c06540eee219f3cc0a9a29f12ae07 --- /dev/null +++ b/results/g8-d8-mdlm-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "mdlm", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d8-mdlm-s2" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 15.785936307907104, + "compute": { + "params": 31518720, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d8-prog-s0.json b/results/g8-d8-prog-s0.json new file mode 100644 index 0000000000000000000000000000000000000000..9344390c61e630a891994f204c0cedbbd59e1d74 --- /dev/null +++ b/results/g8-d8-prog-s0.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 0, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d8-prog-s0" + }, + "accuracy_by_passes": { + "1": 1.0, + "2": 1.0, + "3": 1.0, + "4": 1.0, + "6": 1.0, + "8": 1.0, + "12": 1.0, + "16": 1.0 + }, + "minutes": 24.830967692534127, + "compute": { + "params": 31518720, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d8-prog-s1.json b/results/g8-d8-prog-s1.json new file mode 100644 index 0000000000000000000000000000000000000000..9e0f8a6e961dfffea0f14816ae1a167125fd80ad --- /dev/null +++ b/results/g8-d8-prog-s1.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 1, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d8-prog-s1" + }, + "accuracy_by_passes": { + "1": 0.995, + "2": 0.97, + "3": 0.955, + "4": 0.96, + "6": 0.9824999999999999, + "8": 0.9850000000000001, + "12": 0.9850000000000001, + "16": 0.9850000000000001 + }, + "minutes": 24.830564240614574, + "compute": { + "params": 31518720, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/g8-d8-prog-s2.json b/results/g8-d8-prog-s2.json new file mode 100644 index 0000000000000000000000000000000000000000..bb0f56999f1c4450f2da67e4e117f794da0a7a5c --- /dev/null +++ b/results/g8-d8-prog-s2.json @@ -0,0 +1,39 @@ +{ + "args": { + "mode": "prog", + "K_max": 16, + "K": 8, + "lam": 1.0, + "digits": 8, + "seq_len": 28, + "n_train": 500000, + "n_eval": 400, + "eval_bs": 100, + "d": 512, + "layers": 10, + "heads": 8, + "pe": "rope", + "bs": 128, + "steps": 60000, + "lr": 0.0003, + "warmup": 500, + "seed": 2, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/g8-d8-prog-s2" + }, + "accuracy_by_passes": { + "1": 0.005, + "2": 0.0125, + "3": 0.0125, + "4": 0.0075, + "6": 0.0075, + "8": 0.0075, + "12": 0.0075, + "16": 0.0075 + }, + "minutes": 24.766082608699797, + "compute": { + "params": 31518720, + "train_tokens": 215040000, + "fwd_per_step": 1 + } +} \ No newline at end of file diff --git a/results/gate-english.json b/results/gate-english.json new file mode 100644 index 0000000000000000000000000000000000000000..b1ba9f01dced8c3d8729a5fcc252515356cd6505 --- /dev/null +++ b/results/gate-english.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "english", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 12000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/gate-english" + }, + "accuracy_by_passes": { + "1": 0.9526742788461539, + "2": 0.9497445913461539, + "4": 0.9477914663461539, + "8": 0.9516977163461539, + "16": 0.9516977163461539, + "32": 0.9516977163461539, + "ar": 0.07782451923076923 + }, + "minutes": 2.1154829422632853, + "compute": { + "params": 4760576, + "answer_len_mean": 10.067000389099121 + } +} \ No newline at end of file diff --git a/results/gate-german.json b/results/gate-german.json new file mode 100644 index 0000000000000000000000000000000000000000..3c50c0003cdcbb353d6ce585c037d86138789cd0 --- /dev/null +++ b/results/gate-german.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "german", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 12000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/gate-german" + }, + "accuracy_by_passes": { + "1": 0.7721875, + "2": 0.7733203125, + "4": 0.7797265625, + "8": 0.780703125, + "16": 0.78265625, + "32": 0.78265625, + "ar": 0.07859375 + }, + "minutes": 2.1675578077634174, + "compute": { + "params": 4776448, + "answer_len_mean": 10.743975639343262 + } +} \ No newline at end of file diff --git a/results/gate-turkish.json b/results/gate-turkish.json new file mode 100644 index 0000000000000000000000000000000000000000..945d482112e053e13e47fe4b20ca60df30bf7f64 --- /dev/null +++ b/results/gate-turkish.json @@ -0,0 +1,36 @@ +{ + "args": { + "lang": "turkish", + "size": "medium", + "mode": "mdlm", + "seq_len": 48, + "d": 256, + "layers": 6, + "heads": 8, + "pe": "rope", + "bs": 64, + "steps": 12000, + "lr": 0.0001, + "warmup": 1000, + "K": 4, + "lam": 1.0, + "seed": 0, + "data_root": "/scratch/zt1/project/msml612/user/govind02/pcmd/data/sig", + "eval_bs": 128, + "out": "/scratch/zt1/project/msml612/user/govind02/pcmd/runs/gate-turkish" + }, + "accuracy_by_passes": { + "1": 0.7641234038978495, + "2": 0.7903750840053764, + "4": 0.8030703965053764, + "8": 0.8190629200268817, + "16": 0.8180863575268817, + "32": 0.8180863575268817, + "ar": 0.10290658602150538 + }, + "minutes": 2.1366139531135557, + "compute": { + "params": 4772864, + "answer_len_mean": 13.684530258178711 + } +} \ No newline at end of file diff --git a/src/audit.py b/src/audit.py new file mode 100644 index 0000000000000000000000000000000000000000..8f2f8d986d3d83f58c4e3658024a5cae5afc5415 --- /dev/null +++ b/src/audit.py @@ -0,0 +1,108 @@ +"""Numerical audit. Checks the things that would silently invalidate results. + +Written because two earlier projects produced confident, wrong numbers: an +answer-length leak that handed the model free information, and a learning-rate +artifact that made a working method look broken. Both were invisible in the +accuracy tables. +""" +import glob, json, os, sys +import numpy as np +import torch + +sys.path.insert(0, os.path.dirname(__file__)) +from data import build_addition, build_probe +from sigmorphon import load_language +import diffusion as dfn +from model import Transformer + +FAIL = [] + + +def check(name, ok, detail=""): + print(f" [{'PASS' if ok else 'FAIL'}] {name}" + (f" — {detail}" if detail else ""), flush=True) + if not ok: + FAIL.append(name) + + +print("\n=== 1. data integrity ===") +for d in (4, 8, 12, 16): + X, A, tok = build_addition(300, d, 3 * d + 4, seed=5, exact=True) + bad = 0 + for i in range(len(X)): + s = [tok.itos[t] for t in X[i] if t != tok.pad] + a = int("".join(s[: s.index("+")])) + b = int("".join(s[s.index("+") + 1: s.index("=")])) + got = int("".join(reversed(s[s.index("=") + 1:]))) + bad += (a + b != got) + check(f"{d}-digit addition arithmetic", bad == 0, f"{bad}/300 wrong") + # the answer region must not reveal its own length + check(f"{d}-digit answer mask is fixed-width", + len({int(x) for x in A.sum(1)}) <= 2, + f"{len({int(x) for x in A.sum(1)})} distinct widths") + +print("\n=== 2. the prompt is never masked (no label leakage) ===") +X, A, tok = build_addition(200, 8, 28, seed=7, exact=True) +Xt, At = torch.from_numpy(X), torch.from_numpy(A) +xt = torch.where(At, torch.full_like(Xt, tok.mask), Xt) +prompt_ok = ((xt != tok.mask) | At).all() +check("only answer positions are masked at decode time", bool(prompt_ok)) +eq_visible = all((xt[i] == tok.stoi["="]).any() for i in range(len(xt))) +check("the '=' separator stays visible", eq_visible) + +print("\n=== 3. metric cannot be gamed ===") +gold = Xt[:50] +am = At[:50] +perfect = dfn.cond_decode # placeholder to keep import used +from train_add import exact_match +check("identical prediction scores 1.0", exact_match(gold, gold, am, tok) == 1.0) +shifted = gold.clone() +shifted[am] = (shifted[am] + 1) % 10 + 4 +check("all-wrong prediction scores 0.0", exact_match(shifted, gold, am, tok) == 0.0) +const = gold.clone() +const[am] = tok.stoi["0"] +acc_const = exact_match(const, gold, am, tok) +check("constant-output prediction is not rewarded", acc_const < 0.05, f"{acc_const:.3f}") + +print("\n=== 4. probe tasks: ground truth valid, degenerate output rejected ===") +from train import probe_accuracy +from data import CharTokenizer +for task in ("copy", "agree", "parity"): + Xp, tk = build_probe(200, 64, 4, seed=1, task=task) + gt = probe_accuracy(Xp, task, 4, tk) + cst = probe_accuracy(np.full_like(Xp, 5), task, 4, tk) + check(f"{task}: ground truth valid", gt > 0.99, f"{gt:.2f}") + if task != "copy": + check(f"{task}: constant output rejected", cst < 0.01, f"{cst:.2f}") + +print("\n=== 5. inflection: no train/dev overlap ===") +try: + (Xtr, Atr), (Xdv, Adv), tok2 = load_language( + "english", os.environ.get("SIGROOT", "data/sig"), "medium", 48) + tr = {tuple(int(v) for v in r) for r in Xtr} + dv = {tuple(int(v) for v in r) for r in Xdv} + check("english train/dev disjoint", len(tr & dv) == 0, f"{len(tr & dv)} shared") + check("dev set is non-trivial", len(Xdv) > 200, f"{len(Xdv)} items") +except Exception as e: + check("inflection loader", False, f"{type(e).__name__}: {e}") + +print("\n=== 6. reported numbers reproduce from raw run files ===") +runs = os.environ.get("RUNS", "runs") +groups = {} +for f in glob.glob(f"{runs}/g11-*/result.json"): + r = json.load(open(f)) + a = r["args"] + groups.setdefault((a["digits"], a["mode"]), []).append(r["accuracy_by_passes"]["1"]) +bad = [k for k, v in groups.items() if len(v) < 2] +check("every reported group has >=2 seeds", not bad, f"thin groups: {bad[:3]}") +oor = [] +for f in glob.glob(f"{runs}/*/result.json"): + r = json.load(open(f)) + for k, v in (r.get("accuracy_by_passes") or {}).items(): + if not (0.0 <= v <= 1.0): + oor.append((os.path.basename(os.path.dirname(f)), k, v)) +check("all accuracies within [0,1]", not oor, str(oor[:3])) + +print(f"\n=== AUDIT: {len(FAIL)} failure(s) ===") +for f in FAIL: + print(" !", f) +sys.exit(1 if FAIL else 0) diff --git a/src/beat_baselines.py b/src/beat_baselines.py new file mode 100644 index 0000000000000000000000000000000000000000..78ec3306d8d2a5e524b1874b0e8a46f11fb2a327 --- /dev/null +++ b/src/beat_baselines.py @@ -0,0 +1,134 @@ +"""Head-to-head against inference-time decoding methods. + +This is the comparison that decides whether the paper's claim holds. Roughly a +dozen 2026 papers accelerate diffusion decoding by choosing *which* positions to +reveal together - confidence ranking, entropy gating, dependency-aware ordering. +All of them take the trained marginals as given and only reorder commits. + +The argument of this project is that no such method can close the gap, because +the marginals themselves were never trained for the one-pass regime. That is +testable directly: take a BASELINE model, decode it with every strategy at every +budget, and check whether any of them reaches the training-time method. + +For a fair fight each inference-time method is given its best configuration and +is allowed MORE passes than our method uses. +""" +import argparse, glob, json, os, sys +import numpy as np +import torch + +sys.path.insert(0, os.path.dirname(__file__)) +from data import build_addition +from model import Transformer +import diffusion as dfn +from train_add import exact_match + +p = argparse.ArgumentParser() +p.add_argument("--runs", default="runs") +p.add_argument("--digits", type=int, default=12) +p.add_argument("--n_eval", type=int, default=400) +p.add_argument("--eval_bs", type=int, default=100) +p.add_argument("--out", default="figures/beat_baselines.json") +a = p.parse_args() + +dev = "cuda" if torch.cuda.is_available() else "cpu" +SL = 3 * a.digits + 4 +X, A, tok = build_addition(a.n_eval, a.digits, SL, seed=90_000, exact=True) +X, A = torch.from_numpy(X), torch.from_numpy(A) + + +def load(run_dir): + cfg = json.load(open(os.path.join(run_dir, "result.json")))["args"] + m = Transformer(len(tok), cfg["d"], cfg["layers"], cfg["heads"], cfg["pe"], + causal=False, max_len=cfg["seq_len"] + 8).to(dev) + m.load_state_dict(torch.load(os.path.join(run_dir, "model.pt"), map_location=dev)) + m.eval() + return m, cfg + + +@torch.no_grad() +def run(model, steps, strategy, tau=None): + accs = [] + for i in range(0, len(X), a.eval_bs): + xb, ab = X[i:i + a.eval_bs].to(dev), A[i:i + a.eval_bs].to(dev) + x = torch.where(ab, torch.full_like(xb, tok.mask), xb) + n_ans = int(ab[0].sum()) + pred = None + for s in range(steps, 0, -1): + masked = (x == tok.mask) & ab + if not masked.any(): + break + logits = model(x) + probs = logits.softmax(-1) + conf, pred = probs.max(-1) + if strategy == "random": + score = torch.rand_like(conf) + elif strategy == "entropy": + ent = -(probs * probs.clamp_min(1e-9).log()).sum(-1) + score = -ent + elif strategy == "margin": + top2 = probs.topk(2, dim=-1).values + score = top2[..., 0] - top2[..., 1] + else: + score = conf + score = score.masked_fill(~masked, -1e9) + left = int(n_ans * (s - 1) / steps) + for b in range(x.shape[0]): + k = max(0, int(masked[b].sum()) - left) + if strategy == "entropy" and tau is not None and k > 0: + ent_b = -(probs[b] * probs[b].clamp_min(1e-9).log()).sum(-1) + allowed = int(((ent_b < tau) & masked[b]).sum()) + k = max(1, min(k, allowed)) if allowed else 1 + if k: + idx = score[b].topk(k).indices + x[b, idx] = pred[b, idx] + rem = (x == tok.mask) & ab + if rem.any() and pred is not None: + x[rem] = pred[rem] + accs.append(exact_match(x.cpu(), xb.cpu(), ab.cpu(), tok)) + return float(np.mean(accs)) + + +base_dirs = sorted(glob.glob(f"{a.runs}/g11-d{a.digits}-mdlm-s*")) +ours_dirs = sorted(glob.glob(f"{a.runs}/g11-d{a.digits}-full-s*")) +if not base_dirs or not ours_dirs: + sys.exit(f"need both baseline and method runs for {a.digits} digits") + +results = {"digits": a.digits, "inference_time_on_baseline": {}, "ours": {}} + +print(f"\n=== {a.digits}-digit addition: can any inference-time method rescue " + f"the baseline? ===") +print(f"{'strategy':<22}{'1 pass':>9}{'2':>8}{'4':>8}{'8':>8}{'16':>8}") +for strat, tau in [("confidence", None), ("entropy", None), ("entropy-gated", 0.5), + ("margin", None), ("random", None)]: + row = [] + for steps in (1, 2, 4, 8, 16): + vals = [] + for d in base_dirs: + m, _ = load(d) + vals.append(run(m, steps, "entropy" if strat == "entropy-gated" else strat, tau)) + row.append(float(np.mean(vals))) + results["inference_time_on_baseline"][strat] = row + print(f"{strat:<22}" + "".join(f"{v*100:8.1f}" for v in row)) + +row = [] +for steps in (1, 2, 4, 8, 16): + vals = [] + for d in ours_dirs: + m, _ = load(d) + vals.append(run(m, steps, "confidence")) + row.append(float(np.mean(vals))) +results["ours"]["confidence"] = row +print(f"{'OURS (training-time)':<22}" + "".join(f"{v*100:8.1f}" for v in row)) + +best_inf = max(v[0] for v in results["inference_time_on_baseline"].values()) +best_inf_any = max(max(v) for v in results["inference_time_on_baseline"].values()) +print(f"\nbest inference-time method at 1 pass : {best_inf*100:.1f}%") +print(f"best inference-time at ANY budget : {best_inf_any*100:.1f}%") +print(f"ours at 1 pass : {row[0]*100:.1f}%") +results["summary"] = {"best_inference_1pass": best_inf, + "best_inference_any_budget": best_inf_any, + "ours_1pass": row[0]} +os.makedirs(os.path.dirname(a.out) or ".", exist_ok=True) +json.dump(results, open(a.out, "w"), indent=2) +print(f"\nwrote {a.out}") diff --git a/src/collect.py b/src/collect.py new file mode 100644 index 0000000000000000000000000000000000000000..fbea377d2aa758d5b757b9a3bd96746506751dc0 --- /dev/null +++ b/src/collect.py @@ -0,0 +1,167 @@ +"""Aggregate every experiment into the paper's tables and figures. + +The organising variable across all three task families is DEPENDENCY DEPTH: + + shallow + unique (inflection) -> one pass already works, no method needed + deep + unique (addition) -> one pass fails, and the method fixes it + any + non-unique (probes) -> information-theoretic limit, unfixable +""" +import json, glob, os +import numpy as np +import matplotlib +matplotlib.use("Agg") +import matplotlib.pyplot as plt + +RUNS = os.environ.get("RUNS", "runs") +OUT = os.environ.get("OUT", "figures") +os.makedirs(OUT, exist_ok=True) + +R = [] +for f in sorted(glob.glob(f"{RUNS}/*/result.json")): + try: + r = json.load(open(f)) + except Exception: + continue + r["name"] = os.path.basename(os.path.dirname(f)) + R.append(r) +print(f"collected {len(R)} runs") + +MODE = {"mdlm": "MDLM baseline", "matched": "+ schedule matching", + "distill": "+ one-pass supervision", "full": "full method", + "prog": "parallelism curriculum"} +COL = {"mdlm": "#c0392b", "matched": "#e67e22", + "distill": "#2980b9", "full": "#1a5276", "prog": "#117a65"} + + +def runs(prefix, **kw): + out = [] + for r in R: + if not r["name"].startswith(prefix): + continue + a = r.get("args", {}) + if all(a.get(k) == v for k, v in kw.items()): + out.append(r) + return out + + +def boot(v, n=2000, seed=0): + if len(v) < 2: + return (np.mean(v) * 100,) * 2 + rng = np.random.default_rng(seed) + a = np.array(v) + bs = [rng.choice(a, len(a), replace=True).mean() for _ in range(n)] + return float(np.percentile(bs, 2.5) * 100), float(np.percentile(bs, 97.5) * 100) + + +md = ["# Results\n"] + +# ---- Table 1: the headline, addition at one pass --------------------------- +md.append("## Table 1 — Addition: exact match at ONE forward pass\n") +md.append("The whole answer committed in a single pass. Depth scaled as " + "log2(digits); learning rate 1e-4 with 4000 warmup steps, which " + "matters - see Table 4.\n") +md.append("| digits | MDLM baseline | full method | seeds >90% (base → ours) |") +md.append("|---|---|---|---|") +dig = sorted({r["args"]["digits"] for r in R + if r["name"].startswith("g11-") and "digits" in r.get("args", {})}) +for d in dig: + cells = [] + rel = [] + for m in ("mdlm", "full"): + rs = runs("g11-", digits=d, mode=m) + if not rs: + cells.append("–"); rel.append("–"); continue + v = [r["accuracy_by_passes"]["1"] for r in rs] + lo, hi = boot(v) + cells.append(f"{np.mean(v)*100:.1f} [{lo:.0f},{hi:.0f}]") + rel.append(f"{sum(x>0.9 for x in v)}/{len(v)}") + md.append(f"| **{d}** | {cells[0]} | {cells[1]} | {rel[0]} → {rel[1]} |") +md.append("") + +# ---- Figure 1: gap vs dependency depth ------------------------------------- +fig, ax = plt.subplots(figsize=(7, 4.3)) +for m in ("mdlm", "full"): + xs, mu, sd = [], [], [] + for d in dig: + rs = runs("g11-", digits=d, mode=m) + if not rs: + continue + v = [r["accuracy_by_passes"]["1"] for r in rs] + xs.append(d); mu.append(np.mean(v) * 100); sd.append(np.std(v) * 100) + if xs: + ax.errorbar(xs, mu, yerr=sd, marker="o", capsize=3, color=COL[m], label=MODE[m]) +ax.set_xlabel("operand digits (length of the carry chain)") +ax.set_ylabel("exact match at one pass (%)") +ax.set_title("The gap opens exactly where one forward pass stops sufficing") +ax.legend(); ax.grid(alpha=.3); ax.set_ylim(-3, 103) +fig.tight_layout(); fig.savefig(f"{OUT}/fig1_addition_frontier.png", dpi=160) + +# ---- Figure 2: accuracy vs passes ------------------------------------------ +show = [d for d in dig if d >= 12][:3] or dig[:3] +if show: + fig, axes = plt.subplots(1, len(show), figsize=(4 * len(show), 3.8), + sharey=True, squeeze=False) + grid = [1, 2, 3, 4, 6, 8, 12, 16] + for ax, d in zip(axes[0], show): + for m in ("mdlm", "full"): + rs = runs("g11-", digits=d, mode=m) + if not rs: + continue + mu = [np.mean([r["accuracy_by_passes"][str(p)] for r in rs]) * 100 for p in grid] + ax.plot(grid, mu, marker="o", ms=4, color=COL[m], label=MODE[m]) + ax.set_xscale("log", base=2); ax.set_xlabel("denoising passes") + ax.set_title(f"{d}-digit"); ax.grid(alpha=.3); ax.set_ylim(-3, 103) + axes[0][0].set_ylabel("exact match (%)"); axes[0][0].legend(fontsize=7) + fig.suptitle("Fewer passes = more parallelism") + fig.tight_layout(); fig.savefig(f"{OUT}/fig2_accuracy_vs_passes.png", dpi=160) + +# ---- Table 2: SIGMORPHON, the shallow-dependency boundary ------------------ +sig = [r for r in R if r["name"].startswith("gate-") or + (r.get("args", {}).get("lang") if isinstance(r.get("args"), dict) else None)] +sig = [r for r in sig if "lang" in r.get("args", {})] +if sig: + md.append("## Table 2 — Morphological inflection: where the problem does NOT arise\n") + md.append("Answers are unique but only ~10 characters with shallow, local " + "dependencies. One pass already suffices for the plain baseline, " + "so there is no gap for any method to close. This is the boundary " + "the account predicts, and it is reported as such.\n") + md.append("| language | 1 pass | 32 passes | gap |") + md.append("|---|---|---|---|") + for r in sorted(sig, key=lambda x: x["args"]["lang"]): + a = r["accuracy_by_passes"] + md.append(f"| {r['args']['lang']} | {a['1']*100:.1f} | {a['32']*100:.1f} | " + f"{(a['32']-a['1'])*100:+.1f} |") + md.append("") + +# ---- Table 3: the unfixable limit ------------------------------------------ +md.append("## Table 3 — Non-unique answers: the limit no training removes\n") +md.append("On probes where a group must agree but the value is free, the " + "likelihood-optimal marginal is uniform over the valid values, so two " + "independent draws agree with probability 1/V. Sweeping an entropy " + "penalty designed to break that symmetry over 66 runs: weak settings " + "changed nothing, strong settings destroyed the model, failing even at " + "fully sequential decoding. Succeeding would require abandoning the " + "data distribution.\n") + +# ---- Table 4: the optimisation confound ------------------------------------ +g10 = runs("g10-") +if g10: + md.append("## Table 4 — Why earlier runs appeared to fail (10-digit addition)\n") + md.append("Every apparent failure at 10+ digits was a learning-rate artifact, " + "not a limit of the method or of depth.\n") + md.append("| layers | lr | warmup | mean @1 pass | seeds >90% |") + md.append("|---|---|---|---|---|") + by = {} + for r in g10: + a = r["args"] + by.setdefault((a["layers"], a["lr"], a["warmup"]), []).append( + r["accuracy_by_passes"]["1"]) + for k in sorted(by): + v = by[k] + md.append(f"| {k[0]} | {k[1]} | {k[2]} | {np.mean(v)*100:.1f} | " + f"{sum(x>0.9 for x in v)}/{len(v)} |") + md.append("") + +open(f"{OUT}/summary.md", "w").write("\n".join(md) + "\n") +print("\n".join(md)) +print(f"\nwrote figures + summary to {OUT}") diff --git a/src/diffusion.py b/src/diffusion.py index 0c89bdad1472f3a3365506b5ff190432aa3ad15a..d3b4b2e2226fd0d49193d383ac9785703ce03e84 100644 --- a/src/diffusion.py +++ b/src/diffusion.py @@ -13,6 +13,7 @@ Every published fix reorders or gates commits at INFERENCE. The model has still never seen a partially-decoded state containing its own correlated mistakes. `self_corrupted_loss` changes that at training time. """ +import math import torch import torch.nn.functional as F @@ -432,3 +433,39 @@ def cond_decode(model, x0, amask, tok, steps, device): if rem.any() and pred is not None: x[rem] = pred[rem] return x + + +def pmd_progressive(model, x0, amask, tok, progress, K_max=16, lam_max=1.0, + anneal="log"): + """PARALLELISM CURRICULUM - the fix for the instability at long chains. + + Supervising the fully-masked one-pass prediction from step 0 is a very hard + objective for a long carry chain: at 8 digits every seed found it, at 12 most + collapsed to zero. The target is right but the optimisation is not reachable + from random initialisation in one jump. + + So we anneal along the parallelism axis instead. Early training is decoded + over many passes, where each pass reads the digits already written and the + chain unrolls cheaply. K then halves toward 1, each stage a small increment + from the last, while the one-pass term is ramped in rather than imposed. + This is progressive distillation's schedule applied to the number of + PARALLEL commits rather than the number of noise levels, and unlike prior + step-distillation work it runs inside pretraining from scratch. + + Addition's carry is a prefix scan, so a log-depth solution exists (carry + lookahead) - the curriculum is about making it discoverable, not about + adding capacity. + + progress in [0,1]; returns (loss, K_now, lam_now) so the schedule is logged. + """ + if anneal == "log": + # 16 -> 8 -> 4 -> 2 -> 1, equal time per halving + K_now = max(1, int(K_max / (2 ** int(progress * (math.log2(K_max) + 1e-9))))) + else: + K_now = max(1, int(round(K_max * (1 - progress) + 1 * progress))) + lam_now = lam_max * min(1.0, max(0.0, (progress - 0.25) / 0.5)) + + loss = cond_mdlm_loss(model, x0, amask, tok, "matched", K_now) + if lam_now > 0: + loss = loss + lam_now * sequential_distill_loss(model, x0, amask, tok) + return loss, K_now, lam_now diff --git a/src/sigmorphon.py b/src/sigmorphon.py new file mode 100644 index 0000000000000000000000000000000000000000..da2879422bd0271cf0bd37649958f09e15b8ae73 --- /dev/null +++ b/src/sigmorphon.py @@ -0,0 +1,109 @@ +"""CoNLL-SIGMORPHON 2017 morphological inflection. + +Format per line: lemma inflected form semicolon-separated tags +e.g. swim swam V;PST + +Why this is the right benchmark for one-pass decoding: + + * the answer is UNIQUE given lemma and tags, so every position's true marginal + is a point mass and there is no information-theoretic barrier to committing + the whole form in a single pass; + * forms are short (about ten characters) with shallow, local dependencies, so + unlike 16-digit addition the dependency chain fits comfortably inside one + forward pass; + * exact-match accuracy is the official metric - deterministic, no judges; + * it is a shared task with published baselines across 52 languages, so breadth + is measured across typologically diverse data rather than across seeds of + one configuration. +""" +import os +import numpy as np + +BASE = "https://raw.githubusercontent.com/sigmorphon/conll2017/master/all/task1" +SPECIALS = ["[PAD]", "[MASK]", "[SEP]", "[EOS]"] + + +def download(lang, split, size, root): + """split in {train, dev, test}; size in {low, medium, high} (train only).""" + fn = f"{lang}-{split}" + (f"-{size}" if split == "train" else "") + path = os.path.join(root, fn) + if not os.path.exists(path): + import urllib.request + os.makedirs(root, exist_ok=True) + urllib.request.urlretrieve(f"{BASE}/{fn}", path) + return path + + +def read(path): + out = [] + for line in open(path, encoding="utf-8"): + parts = line.rstrip("\n").split("\t") + if len(parts) == 3: + out.append((parts[0], parts[1], parts[2])) + return out + + +class InflectionTokenizer: + """Character vocabulary plus one symbol per morphological tag, built from + the training split only - no pretrained tokenizer anywhere.""" + + def __init__(self, rows): + chars, tags = set(), set() + for lemma, form, tag in rows: + chars |= set(lemma) | set(form) + tags |= set(tag.split(";")) + self.itos = SPECIALS + sorted(chars) + [f"<{t}>" for t in sorted(tags)] + self.stoi = {s: i for i, s in enumerate(self.itos)} + self.pad, self.mask = self.stoi["[PAD]"], self.stoi["[MASK]"] + self.sep, self.eos = self.stoi["[SEP]"], self.stoi["[EOS]"] + + def __len__(self): + return len(self.itos) + + def encode_src(self, lemma, tag): + ids = [self.stoi[c] for c in lemma if c in self.stoi] + ids += [self.stoi[f"<{t}>"] for t in tag.split(";") if f"<{t}>" in self.stoi] + return ids + + def encode_tgt(self, form): + return [self.stoi[c] for c in form if c in self.stoi] + [self.eos] + + def decode(self, ids): + out = [] + for i in ids: + s = self.itos[i] + if s == "[EOS]": + break + if s not in SPECIALS and not s.startswith("<"): + out.append(s) + return "".join(out) + + +def build(rows, tok, seq_len): + """One flat sequence: source, [SEP], then the answer region. + + The prompt stays visible throughout decoding and only the answer region is + masked, exactly as in the addition setup. + """ + X = np.full((len(rows), seq_len), tok.pad, dtype=np.int64) + A = np.zeros((len(rows), seq_len), dtype=bool) + kept = 0 + for lemma, form, tag in rows: + src = tok.encode_src(lemma, tag) + tgt = tok.encode_tgt(form) + if len(src) + 1 + len(tgt) > seq_len: + continue # drop the rare over-long item + ids = src + [tok.sep] + tgt + X[kept, : len(ids)] = ids + A[kept, len(src) + 1: len(ids)] = True + kept += 1 + return X[:kept], A[:kept] + + +def load_language(lang, root, size="medium", seq_len=48): + tr = read(download(lang, "train", size, root)) + dv = read(download(lang, "dev", None, root)) + tok = InflectionTokenizer(tr) + Xtr, Atr = build(tr, tok, seq_len) + Xdv, Adv = build(dv, tok, seq_len) + return (Xtr, Atr), (Xdv, Adv), tok diff --git a/src/summarize.py b/src/summarize.py new file mode 100644 index 0000000000000000000000000000000000000000..ba772dde9c38bd52bb49a220865c1b2a97e29a78 --- /dev/null +++ b/src/summarize.py @@ -0,0 +1,59 @@ +"""Terminal summary of every experiment, for quick status checks.""" +import glob, json, os, sys +import numpy as np + +RUNS = os.environ.get("RUNS", "runs") + + +def load(pattern): + out = [] + for f in glob.glob(f"{RUNS}/{pattern}/result.json"): + try: + r = json.load(open(f)) + except Exception: + continue + r["name"] = os.path.basename(os.path.dirname(f)) + out.append(r) + return out + + +print("=" * 68) +print("ADDITION — exact match at ONE forward pass (grid 11)") +print("=" * 68) +rows = {} +for r in load("g11-*"): + a = r["args"] + rows.setdefault((a["digits"], a["mode"]), []).append(r["accuracy_by_passes"]) +print(f"{'digits':>7} {'method':<8}{'1 pass':>9}{'16 pass':>9}{'seeds>90%':>11}") +for k in sorted(rows): + v = rows[k] + one = [x["1"] for x in v] + six = [x["16"] for x in v] + print(f"{k[0]:>7} {k[1]:<8}{np.mean(one)*100:8.1f} {np.mean(six)*100:8.1f} " + f"{sum(o > 0.9 for o in one):>6}/{len(one)}") + +print() +print("=" * 68) +print("INFLECTION — shallow dependencies, where no method is needed (grid 12)") +print("=" * 68) +rows = {} +for r in load("g12-*"): + a = r["args"] + rows[(a["lang"], a["mode"])] = r["accuracy_by_passes"] +print(f"{'language':<11}{'mode':<7}{'1 pass':>9}{'32 pass':>9}{'gap':>8}") +for k in sorted(rows): + v = rows[k] + one, many = v["1"] * 100, v["32"] * 100 + print(f"{k[0]:<11}{k[1]:<7}{one:8.1f} {many:8.1f} {many - one:+7.1f}") + +print() +print("=" * 68) +print("HEAD-TO-HEAD vs INFERENCE-TIME METHODS") +print("=" * 68) +for f in sorted(glob.glob("figures/beat_d*.json")) + sorted(glob.glob("../figures/beat_d*.json")): + b = json.load(open(f)) + s = b.get("summary", {}) + print(f"{b['digits']}-digit:") + for k, v in b["inference_time_on_baseline"].items(): + print(f" {k:<16} 1pass={v[0]*100:6.1f} best-any-budget={max(v)*100:6.1f}") + print(f" {'OURS':<16} 1pass={b['ours']['confidence'][0]*100:6.1f}") diff --git a/src/train_add.py b/src/train_add.py index 540bfd45855f946486c9d386aa924f59ee0e0515..eed162958dec4b82af10adf9ea62b9fd3e9f20ea 100644 --- a/src/train_add.py +++ b/src/train_add.py @@ -20,7 +20,9 @@ import diffusion as dfn def get_args(): p = argparse.ArgumentParser() - p.add_argument("--mode", choices=["mdlm", "matched", "distill", "full"], default="mdlm") + p.add_argument("--mode", choices=["mdlm", "matched", "distill", "full", "prog"], + default="mdlm", help="prog = parallelism curriculum (the method)") + p.add_argument("--K_max", type=int, default=16) p.add_argument("--K", type=int, default=8, help="parallelism the schedule is matched to") p.add_argument("--lam", type=float, default=1.0, help="one-pass supervision weight") p.add_argument("--digits", type=int, default=6) @@ -97,12 +99,18 @@ def main(): xb, ab = X[idx].to(dev), A[idx].to(dev) opt.zero_grad(set_to_none=True) with torch.autocast(**amp): - loss = dfn.pmd_loss(model, xb, ab, tok, a.mode, a.K, a.lam) + if a.mode == "prog": + loss, K_now, lam_now = dfn.pmd_progressive( + model, xb, ab, tok, step / max(1, a.steps), a.K_max, a.lam) + else: + loss = dfn.pmd_loss(model, xb, ab, tok, a.mode, a.K, a.lam) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step(); sched.step() if step % 2000 == 0: - print(f"step {step:6d} loss {loss.item():.4f} ({time.time()-t0:.0f}s)", flush=True) + extra = f" K={K_now} lam={lam_now:.2f}" if a.mode == "prog" else "" + print(f"step {step:6d} loss {loss.item():.4f}{extra} " + f"({time.time()-t0:.0f}s)", flush=True) acc = evaluate(model, a, tok, dev) n_par = model.n_params() diff --git a/src/train_sig.py b/src/train_sig.py new file mode 100644 index 0000000000000000000000000000000000000000..64c73d9d6b837567be25f8afccf163df4393ce1c --- /dev/null +++ b/src/train_sig.py @@ -0,0 +1,147 @@ +"""One-pass masked diffusion on SIGMORPHON morphological inflection. + +Autoregressive seq2seq needs |y| decoding steps, about ten per word. If a +diffusion model can commit the whole form in ONE forward pass at the same +exact-match accuracy, that is an order-of-magnitude inference saving on a task +with published baselines across 52 languages. + +Breadth here is measured across typologically diverse languages rather than +across seeds of a single configuration, which is the honest test of whether a +method generalises. +""" +import argparse, json, math, os, time +import numpy as np +import torch +import torch.nn.functional as F + +from sigmorphon import load_language +from model import Transformer +import diffusion as dfn + + +def get_args(): + p = argparse.ArgumentParser() + p.add_argument("--lang", default="english") + p.add_argument("--size", default="medium", choices=["low", "medium", "high"]) + p.add_argument("--mode", choices=["ar", "mdlm", "full", "prog"], default="mdlm", + help="ar = autoregressive baseline; full = ours") + p.add_argument("--seq_len", type=int, default=48) + p.add_argument("--d", type=int, default=256) + p.add_argument("--layers", type=int, default=6) + p.add_argument("--heads", type=int, default=8) + p.add_argument("--pe", default="rope") + p.add_argument("--bs", type=int, default=64) + p.add_argument("--steps", type=int, default=12000) + p.add_argument("--lr", type=float, default=1e-4) + p.add_argument("--warmup", type=int, default=1000) + p.add_argument("--K", type=int, default=4) + p.add_argument("--lam", type=float, default=1.0) + p.add_argument("--seed", type=int, default=0) + p.add_argument("--data_root", default="data/sig") + p.add_argument("--eval_bs", type=int, default=128) + p.add_argument("--out", default="runs/dev") + return p.parse_args() + + +PASSES = [1, 2, 4, 8, 16, 32] + + +def exact_match(pred, gold, amask): + ok = 0 + for p, g, a in zip(pred.tolist(), gold.tolist(), amask.tolist()): + pa = [p[i] for i in range(len(a)) if a[i]] + ga = [g[i] for i in range(len(a)) if a[i]] + ok += pa == ga + return ok / max(len(gold), 1) + + +@torch.no_grad() +def ar_decode(model, x0, amask, tok, device): + """Autoregressive baseline: fill the answer strictly left to right, one + position per forward pass. This is the |y|-step cost the method removes.""" + x = torch.where(amask, torch.full_like(x0, tok.mask), x0).clone() + order = amask[0].nonzero().flatten().tolist() + for j in order: + logits = model(x) + x[:, j] = torch.where(amask[:, j], logits[:, j].argmax(-1), x[:, j]) + return x + + +@torch.no_grad() +def evaluate(model, a, tok, Xdv, Adv, device): + model.eval() + out = {} + for steps in PASSES: + accs = [] + for i in range(0, len(Xdv), a.eval_bs): + xb = Xdv[i:i + a.eval_bs].to(device) + ab = Adv[i:i + a.eval_bs].to(device) + pred = dfn.cond_decode(model, xb, ab, tok, steps, device) + accs.append(exact_match(pred.cpu(), xb.cpu(), ab.cpu())) + out[str(steps)] = float(np.mean(accs)) + # full autoregressive decode: one pass per answer position + accs = [] + for i in range(0, len(Xdv), a.eval_bs): + xb = Xdv[i:i + a.eval_bs].to(device) + ab = Adv[i:i + a.eval_bs].to(device) + accs.append(exact_match(ar_decode(model, xb, ab, tok, device).cpu(), + xb.cpu(), ab.cpu())) + out["ar"] = float(np.mean(accs)) + model.train() + return out + + +def main(): + a = get_args() + torch.manual_seed(a.seed); np.random.seed(a.seed) + dev = "cuda" if torch.cuda.is_available() else "cpu" + os.makedirs(a.out, exist_ok=True) + + (Xtr, Atr), (Xdv, Adv), tok = load_language(a.lang, a.data_root, a.size, a.seq_len) + Xtr, Atr = torch.from_numpy(Xtr), torch.from_numpy(Atr) + Xdv, Adv = torch.from_numpy(Xdv), torch.from_numpy(Adv) + + model = Transformer(len(tok), a.d, a.layers, a.heads, a.pe, + causal=False, max_len=a.seq_len + 8).to(dev) + print(f"[{a.lang}/{a.size}/{a.mode}] params={model.n_params()/1e6:.2f}M " + f"vocab={len(tok)} train={len(Xtr)} dev={len(Xdv)}", flush=True) + + opt = torch.optim.AdamW(model.parameters(), lr=a.lr, weight_decay=0.01) + sched = torch.optim.lr_scheduler.LambdaLR( + opt, lambda s: min(1.0, (s + 1) / a.warmup) * + 0.5 * (1 + math.cos(math.pi * min(1.0, s / a.steps)))) + amp = dict(device_type="cuda", dtype=torch.bfloat16) if dev == "cuda" \ + else dict(device_type="cpu", enabled=False) + + t0 = time.time() + for step in range(a.steps): + idx = torch.randint(0, len(Xtr), (a.bs,)) + xb, ab = Xtr[idx].to(dev), Atr[idx].to(dev) + opt.zero_grad(set_to_none=True) + with torch.autocast(**amp): + if a.mode == "prog": + loss, _, _ = dfn.pmd_progressive(model, xb, ab, tok, + step / max(1, a.steps), 16, a.lam) + elif a.mode == "ar": + # same network, but supervised only on left-to-right prefixes + loss = dfn.cond_mdlm_loss(model, xb, ab, tok, "uniform") + else: + loss = dfn.pmd_loss(model, xb, ab, tok, a.mode, a.K, a.lam) + loss.backward() + torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) + opt.step(); sched.step() + if step % 2000 == 0: + print(f"step {step:6d} loss {loss.item():.4f} ({time.time()-t0:.0f}s)", flush=True) + + acc = evaluate(model, a, tok, Xdv, Adv, dev) + json.dump({"args": vars(a), "accuracy_by_passes": acc, + "minutes": (time.time() - t0) / 60, + "compute": {"params": model.n_params(), + "answer_len_mean": float(Adv.sum(1).float().mean())}}, + open(os.path.join(a.out, "result.json"), "w"), indent=2) + torch.save(model.state_dict(), os.path.join(a.out, "model.pt")) + print("FINAL", json.dumps(acc), flush=True) + + +if __name__ == "__main__": + main() diff --git a/weights/g10-L10-lr3e-4-w500-s0.pt b/weights/g10-L10-lr3e-4-w500-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..387f7c7d5574e24bca3c644351c50ca41767df42 --- /dev/null +++ b/weights/g10-L10-lr3e-4-w500-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:026318260ec96491ef1a95737554de4378eb9810270b135fa0053db2433f7077 +size 70992458 diff --git a/weights/g10-L6-lr5e-5-w500-s0.pt b/weights/g10-L6-lr5e-5-w500-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..0b03707b95e8fbd3e6f1449f10316eaf13050438 --- /dev/null +++ b/weights/g10-L6-lr5e-5-w500-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:35c6d2f646468516f8a7d80870c5431082e7a3207136cedd1c21701134625316 +size 42614414 diff --git a/weights/g2-agree-mdlm-s0.pt b/weights/g2-agree-mdlm-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..73460b6a01aaf999270d95f27b2de02bb4e57144 --- /dev/null +++ b/weights/g2-agree-mdlm-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:914b4b4c1757b5d014becb41b940da19a7ffaf62ce551ed1445d1c32142120e5 +size 42660494 diff --git a/weights/g3-gam0.0-elbo1.0-s0.pt b/weights/g3-gam0.0-elbo1.0-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..4adaaa82782481fd4ec318e54d3007eb701b428a --- /dev/null +++ b/weights/g3-gam0.0-elbo1.0-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:06e4701174342db4f8ba2b96ac50d57094416763cb9b52e3908aeed6570e2f10 +size 42660494 diff --git a/weights/g3-gam1.0-elbo0.2-s0.pt b/weights/g3-gam1.0-elbo0.2-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..117af27480440a0c847a710b6a6107d80c440347 --- /dev/null +++ b/weights/g3-gam1.0-elbo0.2-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4beffdff5d5f782e2448a5b06a137025e73b6f2d45e0cb684c693e4185e42bfb +size 42660494 diff --git a/weights/g6-d14-distill-s0.pt b/weights/g6-d14-distill-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..b540a516cdef095bced6f154f62f4d78655df2e1 --- /dev/null +++ b/weights/g6-d14-distill-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1b07de5de201251557c95f97fd63b77705e7d2ddfb461cf5cf2cde501499333b +size 42614414 diff --git a/weights/g6-d14-mdlm-s0.pt b/weights/g6-d14-mdlm-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..b90cf0ae9f56862b09e00b652844d0b054d73754 --- /dev/null +++ b/weights/g6-d14-mdlm-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ea9a9f47affccf7c846ba7ebdd855d552b8be7b4bbb5c3cf751b01f74260531f +size 42614414 diff --git a/weights/g7-d12-L8-dm512-st60000-mdlm-s0.pt b/weights/g7-d12-L8-dm512-st60000-mdlm-s0.pt new file mode 100644 index 0000000000000000000000000000000000000000..aa8ded6cbeb954253b56299fa434e6616a2da9d8 --- /dev/null +++ b/weights/g7-d12-L8-dm512-st60000-mdlm-s0.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:613ea458079c21530a883510636f831bea7d5c9786e920be98c28f477cc3ec1d +size 100895850