Add files using upload-large-folder tool
Browse files- README.md +4 -4
- checkpoint/README_REASSEMBLE.md +20 -12
README.md
CHANGED
|
@@ -40,7 +40,7 @@ m1_mix_submit/
|
|
| 40 |
βββ README.md # this file
|
| 41 |
βββ task_instructions.json # verbatim --global_task per task + scores
|
| 42 |
βββ checkpoint/
|
| 43 |
-
β βββ m1_mix_final_step50000.pt.part00 β¦
|
| 44 |
β βββ m1_mix_final_step50000.pt.sha256 # SHA-256 of the reassembled checkpoint
|
| 45 |
β βββ README_REASSEMBLE.md # how to cat the parts back together + verify
|
| 46 |
βββ norm_stats/
|
|
@@ -60,9 +60,9 @@ m1_mix_submit/
|
|
| 60 |
|
| 61 |
### About the checkpoint
|
| 62 |
|
| 63 |
-
> **Reassemble first.** The 15.3 GB checkpoint is uploaded as
|
| 64 |
-
> (`m1_mix_final_step50000.pt.part00β¦
|
| 65 |
-
>
|
| 66 |
>
|
| 67 |
> ```bash
|
| 68 |
> cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
|
|
|
|
| 40 |
βββ README.md # this file
|
| 41 |
βββ task_instructions.json # verbatim --global_task per task + scores
|
| 42 |
βββ checkpoint/
|
| 43 |
+
β βββ m1_mix_final_step50000.pt.part00 β¦ part08 # 15.3 GB full training ckpt, split into 9 parts (2Γ4 GB + 7Γβ€1 GB)
|
| 44 |
β βββ m1_mix_final_step50000.pt.sha256 # SHA-256 of the reassembled checkpoint
|
| 45 |
β βββ README_REASSEMBLE.md # how to cat the parts back together + verify
|
| 46 |
βββ norm_stats/
|
|
|
|
| 60 |
|
| 61 |
### About the checkpoint
|
| 62 |
|
| 63 |
+
> **Reassemble first.** The 15.3 GB checkpoint is uploaded as 9 byte-split parts
|
| 64 |
+
> (`m1_mix_final_step50000.pt.part00β¦08`) because the upload path capped single files and
|
| 65 |
+
> throttled per-window bytes. Concatenation reproduces the original **bit-for-bit**:
|
| 66 |
>
|
| 67 |
> ```bash
|
| 68 |
> cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
|
checkpoint/README_REASSEMBLE.md
CHANGED
|
@@ -1,16 +1,23 @@
|
|
| 1 |
# Reassembling the checkpoint
|
| 2 |
|
| 3 |
-
The full training checkpoint `m1_mix_final_step50000.pt` (15.3 GB) is uploaded as **
|
| 4 |
-
byte-split parts**
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
|
| 11 |
-
|
|
| 12 |
-
| `m1_mix_final_step50000.pt.
|
| 13 |
-
| `m1_mix_final_step50000.pt.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 14 |
|
| 15 |
## 1. Reassemble
|
| 16 |
|
|
@@ -18,7 +25,8 @@ checkpoint **bit-for-bit**.
|
|
| 18 |
cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
|
| 19 |
```
|
| 20 |
|
| 21 |
-
(`part??` expands to `part00 part01
|
|
|
|
| 22 |
|
| 23 |
## 2. Verify integrity
|
| 24 |
|
|
|
|
| 1 |
# Reassembling the checkpoint
|
| 2 |
|
| 3 |
+
The full training checkpoint `m1_mix_final_step50000.pt` (15.3 GB) is uploaded as **9
|
| 4 |
+
byte-split parts**. The upload path (an egress gateway) both capped single files at
|
| 5 |
+
~9.7 GB *and* throttled cumulative bytes per connection window, so the file is split into
|
| 6 |
+
small parts that each fit a window: the **first two parts are 4 GiB** (uploaded before the
|
| 7 |
+
throttle was understood) and the **remaining seven are β€1 GiB**. Concatenating all parts
|
| 8 |
+
in sorted order reproduces the checkpoint **bit-for-bit**.
|
| 9 |
+
|
| 10 |
+
| part | size |
|
| 11 |
+
|-----------------------------------------|------------|
|
| 12 |
+
| `m1_mix_final_step50000.pt.part00` | 4.0 GiB |
|
| 13 |
+
| `m1_mix_final_step50000.pt.part01` | 4.0 GiB |
|
| 14 |
+
| `m1_mix_final_step50000.pt.part02` | 1.0 GiB |
|
| 15 |
+
| `m1_mix_final_step50000.pt.part03` | 1.0 GiB |
|
| 16 |
+
| `m1_mix_final_step50000.pt.part04` | 1.0 GiB |
|
| 17 |
+
| `m1_mix_final_step50000.pt.part05` | 1.0 GiB |
|
| 18 |
+
| `m1_mix_final_step50000.pt.part06` | 1.0 GiB |
|
| 19 |
+
| `m1_mix_final_step50000.pt.part07` | 1.0 GiB |
|
| 20 |
+
| `m1_mix_final_step50000.pt.part08` | ~0.25 GiB |
|
| 21 |
|
| 22 |
## 1. Reassemble
|
| 23 |
|
|
|
|
| 25 |
cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
|
| 26 |
```
|
| 27 |
|
| 28 |
+
(`part??` expands to `part00 part01 β¦ part08` in sorted order β the command is unchanged
|
| 29 |
+
regardless of how many parts there are.)
|
| 30 |
|
| 31 |
## 2. Verify integrity
|
| 32 |
|