| ======================================== |
| SYSTEM CONFIG |
| ======================================== |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| NVIDIA H200, 143771 MiB, 580.126.09 |
| PyTorch: 2.5.1+cu124 |
| ======================================== |
|
|
| **** 0.1: System deps (gcc) — Starts 2026-03-21 05:28:00 **** |
| gcc installed |
| **** 0.1: System deps (gcc) — Complete 05:28:06 (6s) **** |
|
|
|
|
| **** 0.2: PyTorch upgrade (cu126) — Starts 2026-03-21 05:28:06 **** |
| ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. |
| torchaudio 2.5.1+cu124 requires torch==2.5.1, but you have torch 2.10.0+cu126 which is incompatible. |
| WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. |
| **** 0.2: PyTorch upgrade (cu126) — Complete 05:29:06 (60s) **** |
|
|
|
|
| **** 0.3: Python deps — Starts 2026-03-21 05:29:06 **** |
| WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. |
| **** 0.3: Python deps — Complete 05:29:17 (11s) **** |
|
|
|
|
| **** 0.4: FA3 kernel install — Starts 2026-03-21 05:29:17 **** |
| FA3 variant: torch210-cxx11-cu126-x86_64-linux |
|
Fetching 4 files: 0%| | 0/4 [00:00<?, ?it/s]
Fetching 4 files: 25%|██▌ | 1/4 [00:03<00:11, 3.92s/it]
Fetching 4 files: 100%|██████████| 4/4 [00:03<00:00, 1.02it/s] |
| FA3 installed to /opt/conda/lib/python3.11/site-packages/flash_attention_3 |
| libcudart.so.12 registered in ldconfig |
| **** 0.4: FA3 kernel install — Complete 05:29:22 (5s) **** |
|
|
|
|
| **** 0.5: Verification — Starts 2026-03-21 05:29:22 **** |
| PyTorch: 2.10.0+cu126 |
| CUDA: 12.6 |
| GPU: 8x NVIDIA H200 (150 GB, SM 90) |
| FA3: OK (torch.Size([1, 64, 16, 64])) |
| /opt/conda/lib/python3.11/site-packages/torch/_inductor/compile_fx.py:321: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance. |
| warnings.warn( |
| compile: OK |
| tiktoken: OK |
| wandb: OK |
|
|
| === Setup Complete === |
| **** 0.5: Verification — Complete 05:29:32 (10s) **** |
|
|
|
|
| **** 0.6: Pre-flight validation — Starts 2026-03-21 05:29:32 **** |
| WARNING: FA3 load via 'kernels' failed on Hopper SM90: No module named 'kernels' |
| WARNING: train.py will attempt direct import from flash_attention_3 package |
| FA3 patched for preflight |
| Driver: 580.126.09 (>= 560 OK) |
| FATAL: wandb not configured. Run: source .env && wandb login $WANDB_API_KEY |
|
|
