========================================
  SYSTEM CONFIG
========================================
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
NVIDIA H200, 143771 MiB, 580.126.09
  PyTorch: 2.5.1+cu124
========================================

**** 0.1: System deps (gcc) — Starts 2026-03-21 05:28:00 ****
gcc installed
**** 0.1: System deps (gcc) — Complete 05:28:06 (6s) ****


**** 0.2: PyTorch upgrade (cu126) — Starts 2026-03-21 05:28:06 ****
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchaudio 2.5.1+cu124 requires torch==2.5.1, but you have torch 2.10.0+cu126 which is incompatible.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
**** 0.2: PyTorch upgrade (cu126) — Complete 05:29:06 (60s) ****


**** 0.3: Python deps — Starts 2026-03-21 05:29:06 ****
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
**** 0.3: Python deps — Complete 05:29:17 (11s) ****


**** 0.4: FA3 kernel install — Starts 2026-03-21 05:29:17 ****
FA3 variant: torch210-cxx11-cu126-x86_64-linux

Fetching 4 files:   0%|          | 0/4 [00:00<?, ?it/s]
Fetching 4 files:  25%|██▌       | 1/4 [00:03<00:11,  3.92s/it]
Fetching 4 files: 100%|██████████| 4/4 [00:03<00:00,  1.02it/s]
FA3 installed to /opt/conda/lib/python3.11/site-packages/flash_attention_3
libcudart.so.12 registered in ldconfig
**** 0.4: FA3 kernel install — Complete 05:29:22 (5s) ****


**** 0.5: Verification — Starts 2026-03-21 05:29:22 ****
PyTorch:  2.10.0+cu126
CUDA:     12.6
GPU:      8x NVIDIA H200 (150 GB, SM 90)
FA3:      OK (torch.Size([1, 64, 16, 64]))
/opt/conda/lib/python3.11/site-packages/torch/_inductor/compile_fx.py:321: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
  warnings.warn(
compile:  OK
tiktoken: OK
wandb:    OK

=== Setup Complete ===
**** 0.5: Verification — Complete 05:29:32 (10s) ****


**** 0.6: Pre-flight validation — Starts 2026-03-21 05:29:32 ****
WARNING: FA3 load via 'kernels' failed on Hopper SM90: No module named 'kernels'
WARNING: train.py will attempt direct import from flash_attention_3 package
FA3 patched for preflight
  Driver: 580.126.09 (>= 560 OK)
FATAL: wandb not configured. Run: source .env && wandb login $WANDB_API_KEY