qiuly commited on
Commit
1dd1ae9
Β·
verified Β·
1 Parent(s): 0b45695

Add files using upload-large-folder tool

Browse files
Files changed (2) hide show
  1. README.md +4 -4
  2. checkpoint/README_REASSEMBLE.md +20 -12
README.md CHANGED
@@ -40,7 +40,7 @@ m1_mix_submit/
40
  β”œβ”€β”€ README.md # this file
41
  β”œβ”€β”€ task_instructions.json # verbatim --global_task per task + scores
42
  β”œβ”€β”€ checkpoint/
43
- β”‚ β”œβ”€β”€ m1_mix_final_step50000.pt.part00 … part03 # 15.3 GB full training ckpt, split into 4×≀4 GB parts
44
  β”‚ β”œβ”€β”€ m1_mix_final_step50000.pt.sha256 # SHA-256 of the reassembled checkpoint
45
  β”‚ └── README_REASSEMBLE.md # how to cat the parts back together + verify
46
  β”œβ”€β”€ norm_stats/
@@ -60,9 +60,9 @@ m1_mix_submit/
60
 
61
  ### About the checkpoint
62
 
63
- > **Reassemble first.** The 15.3 GB checkpoint is uploaded as 4 byte-split parts
64
- > (`m1_mix_final_step50000.pt.part00…03`) because the upload path capped single files at
65
- > ~9.7 GB. Concatenation reproduces the original **bit-for-bit**:
66
  >
67
  > ```bash
68
  > cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
 
40
  β”œβ”€β”€ README.md # this file
41
  β”œβ”€β”€ task_instructions.json # verbatim --global_task per task + scores
42
  β”œβ”€β”€ checkpoint/
43
+ β”‚ β”œβ”€β”€ m1_mix_final_step50000.pt.part00 … part08 # 15.3 GB full training ckpt, split into 9 parts (2Γ—4 GB + 7×≀1 GB)
44
  β”‚ β”œβ”€β”€ m1_mix_final_step50000.pt.sha256 # SHA-256 of the reassembled checkpoint
45
  β”‚ └── README_REASSEMBLE.md # how to cat the parts back together + verify
46
  β”œβ”€β”€ norm_stats/
 
60
 
61
  ### About the checkpoint
62
 
63
+ > **Reassemble first.** The 15.3 GB checkpoint is uploaded as 9 byte-split parts
64
+ > (`m1_mix_final_step50000.pt.part00…08`) because the upload path capped single files and
65
+ > throttled per-window bytes. Concatenation reproduces the original **bit-for-bit**:
66
  >
67
  > ```bash
68
  > cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
checkpoint/README_REASSEMBLE.md CHANGED
@@ -1,16 +1,23 @@
1
  # Reassembling the checkpoint
2
 
3
- The full training checkpoint `m1_mix_final_step50000.pt` (15.3 GB) is uploaded as **4
4
- byte-split parts** because the upload path capped single files at ~9.7 GB. The parts are a
5
- plain `split -b 4G` of the original file, so concatenating them in order reproduces the
6
- checkpoint **bit-for-bit**.
7
-
8
- | part | size |
9
- |-----------------------------------------|-----------|
10
- | `m1_mix_final_step50000.pt.part00` | 4.0 GiB |
11
- | `m1_mix_final_step50000.pt.part01` | 4.0 GiB |
12
- | `m1_mix_final_step50000.pt.part02` | 4.0 GiB |
13
- | `m1_mix_final_step50000.pt.part03` | ~2.25 GiB |
 
 
 
 
 
 
 
14
 
15
  ## 1. Reassemble
16
 
@@ -18,7 +25,8 @@ checkpoint **bit-for-bit**.
18
  cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
19
  ```
20
 
21
- (`part??` expands to `part00 part01 part02 part03` in sorted order.)
 
22
 
23
  ## 2. Verify integrity
24
 
 
1
  # Reassembling the checkpoint
2
 
3
+ The full training checkpoint `m1_mix_final_step50000.pt` (15.3 GB) is uploaded as **9
4
+ byte-split parts**. The upload path (an egress gateway) both capped single files at
5
+ ~9.7 GB *and* throttled cumulative bytes per connection window, so the file is split into
6
+ small parts that each fit a window: the **first two parts are 4 GiB** (uploaded before the
7
+ throttle was understood) and the **remaining seven are ≀1 GiB**. Concatenating all parts
8
+ in sorted order reproduces the checkpoint **bit-for-bit**.
9
+
10
+ | part | size |
11
+ |-----------------------------------------|------------|
12
+ | `m1_mix_final_step50000.pt.part00` | 4.0 GiB |
13
+ | `m1_mix_final_step50000.pt.part01` | 4.0 GiB |
14
+ | `m1_mix_final_step50000.pt.part02` | 1.0 GiB |
15
+ | `m1_mix_final_step50000.pt.part03` | 1.0 GiB |
16
+ | `m1_mix_final_step50000.pt.part04` | 1.0 GiB |
17
+ | `m1_mix_final_step50000.pt.part05` | 1.0 GiB |
18
+ | `m1_mix_final_step50000.pt.part06` | 1.0 GiB |
19
+ | `m1_mix_final_step50000.pt.part07` | 1.0 GiB |
20
+ | `m1_mix_final_step50000.pt.part08` | ~0.25 GiB |
21
 
22
  ## 1. Reassemble
23
 
 
25
  cat m1_mix_final_step50000.pt.part?? > m1_mix_final_step50000.pt
26
  ```
27
 
28
+ (`part??` expands to `part00 part01 … part08` in sorted order β€” the command is unchanged
29
+ regardless of how many parts there are.)
30
 
31
  ## 2. Verify integrity
32