Card update: training resumed 14 Nov from update 40,000 (agents 2fb783b0)
Browse files
TRAINING_CONFIGURATION.md
CHANGED
|
@@ -57,7 +57,9 @@ Inherited-count footprint: counter SHA256 `abb39b8ce019c1a18a19ccf0ef4c32977c0c5
|
|
| 57 |
|
| 58 |
## Current CPT execution receipt
|
| 59 |
|
| 60 |
-
**
|
|
|
|
|
|
|
| 61 |
|
| 62 |
**Historical launch receipt — 10 November 2026:** operators confirmed the 04:50 UTC start: fresh 1.07 hardware control within the 0.001 tolerance, pinned 1.08 weights/config converted to the training format, step-zero `INIT-FROM`, fresh optimizer, world size eight, c01/queue/CL validation identities and `PRE-FLIGHT OK`. The inherited 1.08 checkpoint step 1,709 is source metadata, **not the new continuation's initial training counter**.
|
| 63 |
|
|
@@ -75,7 +77,7 @@ The distinct [CPT Track run](https://track.fabryka.ai/run/71cf98ec-7c60-54a8-ac4
|
|
| 75 |
|
| 76 |
The older public `train_gpt_ref_r23.py` is **not the identity of this training implementation**. The inference-compatible architecture code packaged with a weight release and the executable used to train it are distinct artifacts; filenames alone do not establish identity.
|
| 77 |
|
| 78 |
-
The archived B200 recipe uses one eight-GPU node, PyTorch 2.8/CUDA 12.8 and DDP. It is a historical configuration reference, not a description of the
|
| 79 |
|
| 80 |
## Historical shared learning arguments
|
| 81 |
|
|
|
|
| 57 |
|
| 58 |
## Current CPT execution receipt
|
| 59 |
|
| 60 |
+
**Execution receipt — 15 November 2026, 20:02 UTC:** operators confirmed continued pretraining resumed from the full checkpoint at **update 42,800** on **8 × NVIDIA B300** (spot instance), with the same trainer and the same effective batch of **256 sequences × 4,096 tokens per update**. On the 8 logged points from update 42,810 to 42,880, the training loss matched the earlier RTX 4090 run within 0.0003 (mean difference −0.0002). Training is not finished.
|
| 61 |
+
|
| 62 |
+
**Execution receipt — 14 November 2026, 01:50 UTC:** operators confirmed continued pretraining resumed from **update 40,000** on **2 × NVIDIA RTX 4090**. Training updates from **40,001** retain the same effective batch of **256 sequences × 4,096 tokens per update**. Training is not finished. **Archive availability, 14 November 2026:** eight weights-only checkpoints cover updates 5,000–40,000; the published full-state recovery checkpoint remains at update 35,000. This availability does not extend benchmark coverage beyond update 35,000.
|
| 63 |
|
| 64 |
**Historical launch receipt — 10 November 2026:** operators confirmed the 04:50 UTC start: fresh 1.07 hardware control within the 0.001 tolerance, pinned 1.08 weights/config converted to the training format, step-zero `INIT-FROM`, fresh optimizer, world size eight, c01/queue/CL validation identities and `PRE-FLIGHT OK`. The inherited 1.08 checkpoint step 1,709 is source metadata, **not the new continuation's initial training counter**.
|
| 65 |
|
|
|
|
| 77 |
|
| 78 |
The older public `train_gpt_ref_r23.py` is **not the identity of this training implementation**. The inference-compatible architecture code packaged with a weight release and the executable used to train it are distinct artifacts; filenames alone do not establish identity.
|
| 79 |
|
| 80 |
+
The archived B200 recipe uses one eight-GPU node, PyTorch 2.8/CUDA 12.8 and DDP. It is a historical configuration reference, not a description of the later runtimes (two RTX 4090 from 14 November, eight B300 from 15 November). Compute uses bf16 on the verified hardware; `--dtype uint16` below describes **stored token IDs**, not model arithmetic.
|
| 81 |
|
| 82 |
## Historical shared learning arguments
|
| 83 |
|