Hugging Face's logo

Lakesenberg
/
nemotron-lora-work

Safetensors
Model card Files
xet
 Community
nemotron-lora-work / run.log
Lakesenberg's picture
Lakesenberg
Upload folder using huggingface_hub
e5385e2 verified
Raw
History Blame Contribute Delete
239 kB
=== [Fri May 22 15:03:42 UTC 2026] Phase 0: setup_env.sh ===
Retrieving notices: - \ | / - \ | done
Channels:
 - conda-forge
Platform: linux-64
Collecting package metadata (repodata.json): - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done
Solving environment: | / done
==> WARNING: A newer version of conda exists. <==
 current version: 26.1.1
 latest version: 26.5.0
Please update conda by running
 $ conda update -n base -c conda-forge conda
## Package Plan ##
 environment location: /home/lakesenberg/miniforge3/envs/nemo
 added / updated specs:
 - python=3.11
The following packages will be downloaded:
 package | build
 ---------------------------|-----------------
 ca-certificates-2026.5.20 | hbd8a1cb_0 127 KB conda-forge
 libexpat-2.8.1 | hecca717_0 75 KB conda-forge
 libgcc-15.2.0 | he0feb66_19 1017 KB conda-forge
 libgcc-ng-15.2.0 | h69a702a_19 27 KB conda-forge
 libgomp-15.2.0 | he0feb66_19 590 KB conda-forge
 libsqlite-3.53.1 | h0c1763c_0 933 KB conda-forge
 libuuid-2.42.1 | h5347b49_0 39 KB conda-forge
 ncurses-6.6 | hdb14827_0 897 KB conda-forge
 pip-26.1.1 | pyh8b19718_0 1.1 MB conda-forge
 python-3.11.15 |hd63d673_0_cpython 29.5 MB conda-forge
 ------------------------------------------------------------
 Total: 34.3 MB
The following NEW packages will be INSTALLED:
 _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-20_gnu
bzip2 conda-forge/linux-64::bzip2-1.0.8-hda65f42_9
ca-certificates conda-forge/noarch::ca-certificates-2026.5.20-hbd8a1cb_0
ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.45.1-default_hbd61a6d_102
libexpat conda-forge/linux-64::libexpat-2.8.1-hecca717_0
libffi conda-forge/linux-64::libffi-3.5.2-h3435931_0
libgcc conda-forge/linux-64::libgcc-15.2.0-he0feb66_19
libgcc-ng conda-forge/linux-64::libgcc-ng-15.2.0-h69a702a_19
libgomp conda-forge/linux-64::libgomp-15.2.0-he0feb66_19
liblzma conda-forge/linux-64::liblzma-5.8.3-hb03c661_0
libnsl conda-forge/linux-64::libnsl-2.0.1-hb9d3cd8_1
libsqlite conda-forge/linux-64::libsqlite-3.53.1-h0c1763c_0
libuuid conda-forge/linux-64::libuuid-2.42.1-h5347b49_0
libxcrypt conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1
libzlib conda-forge/linux-64::libzlib-1.3.2-h25fd6f3_2
ncurses conda-forge/linux-64::ncurses-6.6-hdb14827_0
openssl conda-forge/linux-64::openssl-3.6.2-h35e630c_0
packaging conda-forge/noarch::packaging-26.2-pyhc364b38_0
pip conda-forge/noarch::pip-26.1.1-pyh8b19718_0
python conda-forge/linux-64::python-3.11.15-hd63d673_0_cpython
readline conda-forge/linux-64::readline-8.3-h853b02a_0
setuptools conda-forge/noarch::setuptools-82.0.1-pyh332efcf_0
tk conda-forge/linux-64::tk-8.6.13-noxft_h366c992_103
tzdata conda-forge/noarch::tzdata-2025c-hc9c84f9_1
wheel conda-forge/noarch::wheel-0.47.0-pyhd8ed1ab_0
zstd conda-forge/linux-64::zstd-1.5.7-hb78ec9c_6
Downloading and Extracting Packages: ...working... python-3.11.15 | 29.5 MB | | 0%
pip-26.1.1 | 1.1 MB | | 0% 
 libgcc-15.2.0 | 1017 KB | | 0% 
 libsqlite-3.53.1 | 933 KB | | 0% 
 ncurses-6.6 | 897 KB | | 0% 
 libgomp-15.2.0 | 590 KB | | 0% 
 ca-certificates-2026 | 127 KB | | 0% 
 libexpat-2.8.1 | 75 KB | | 0% 
 libuuid-2.42.1 | 39 KB | | 0% 
 libgcc-ng-15.2.0 | 27 KB | | 0% 
 pip-26.1.1 | 1.1 MB | ########## | 100% 
 ncurses-6.6 | 897 KB | ########## | 100%  python-3.11.15 | 29.5 MB | 1 | 1%
libgcc-15.2.0 | 1017 KB | ########## | 100% 
 libgcc-15.2.0 | 1017 KB | ########## | 100% 
 libsqlite-3.53.1 | 933 KB | ########## | 100% 
 libsqlite-3.53.1 | 933 KB | ########## | 100% 
 libgomp-15.2.0 | 590 KB | 2 | 3% 
 libgomp-15.2.0 | 590 KB | ########## | 100% 
 ca-certificates-2026 | 127 KB | #2 | 13% 
 libuuid-2.42.1 | 39 KB | #### | 41% 
 libuuid-2.42.1 | 39 KB | ########## | 100% 
 ca-certificates-2026 | 127 KB | ########## | 100% 
 libgcc-ng-15.2.0 | 27 KB | #####9 | 59% 
 libexpat-2.8.1 | 75 KB | ##1 | 21% 
 libgcc-ng-15.2.0 | 27 KB | ########## | 100% 
 libexpat-2.8.1 | 75 KB | ########## | 100%  python-3.11.15 | 29.5 MB | ##2 | 22%
libgcc-15.2.0 | 1017 KB | ########## | 100% 
 libsqlite-3.53.1 | 933 KB | ########## | 100%  python-3.11.15 | 29.5 MB | ####5 | 46%
libgomp-15.2.0 | 590 KB | ########## | 100% 
 libgomp-15.2.0 | 590 KB | ########## | 100% 
 libuuid-2.42.1 | 39 KB | ########## | 100% 
 libuuid-2.42.1 | 39 KB | ########## | 100% 
 ca-certificates-2026 | 127 KB | ########## | 100% 
 ca-certificates-2026 | 127 KB | ########## | 100% 
 libgcc-ng-15.2.0 | 27 KB | ########## | 100% 
 libgcc-ng-15.2.0 | 27 KB | ########## | 100%  python-3.11.15 | 29.5 MB | #######3 | 73%
libexpat-2.8.1 | 75 KB | ########## | 100% 
 libexpat-2.8.1 | 75 KB | ########## | 100% 
 pip-26.1.1 | 1.1 MB | ########## | 100% 
 pip-26.1.1 | 1.1 MB | ########## | 100%  python-3.11.15 | 29.5 MB | ########## | 100% python-3.11.15 | 29.5 MB | ########## | 100%
ncurses-6.6 | 897 KB | ########## | 100% 
 ncurses-6.6 | 897 KB | ########## | 100%  python-3.11.15 | 29.5 MB | ########## | 100%

 
 
 
 
 
 
 
  done
Preparing transaction: \ | done
Verifying transaction: - \ | / - \ | / - done
Executing transaction: | / - \ | / - \ | / - \ | / - \ | / - done
#
# To activate this environment, use
#
# $ conda activate nemo
#
# To deactivate an active environment, use
#
# $ conda deactivate
Requirement already satisfied: pip in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (26.1.1)
Requirement already satisfied: wheel in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.47.0)
Requirement already satisfied: packaging>=24.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from wheel) (26.2)
Looking in indexes: https://download.pytorch.org/whl/cu124
Collecting torch>=2.4
 Downloading torch-2.6.0%2Bcu124-cp311-cp311-linux_x86_64.whl.metadata (28 kB)
Collecting filelock (from torch>=2.4)
 Using cached filelock-3.29.0-py3-none-any.whl.metadata (2.0 kB)
Collecting typing-extensions>=4.10.0 (from torch>=2.4)
 Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting networkx (from torch>=2.4)
 Using cached networkx-3.6.1-py3-none-any.whl.metadata (6.8 kB)
Collecting jinja2 (from torch>=2.4)
 Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec (from torch>=2.4)
 Downloading fsspec-2026.4.0-py3-none-any.whl.metadata (10 kB)
Collecting nvidia-cuda-nvrtc-cu12==12.4.127 (from torch>=2.4)
 Using cached nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (24.6 MB)
Collecting nvidia-cuda-runtime-cu12==12.4.127 (from torch>=2.4)
 Using cached nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (883 kB)
Collecting nvidia-cuda-cupti-cu12==12.4.127 (from torch>=2.4)
 Using cached nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (13.8 MB)
Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch>=2.4)
 Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/664.8 MB ? eta -:--:--  ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.9/664.8 MB 338.4 MB/s eta 0:00:02  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.3/664.8 MB 322.8 MB/s eta 0:00:02  ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.0/664.8 MB 312.2 MB/s eta 0:00:02  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 229.4/664.8 MB 290.5 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━ 284.7/664.8 MB 287.7 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 341.3/664.8 MB 271.6 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 399.2/664.8 MB 270.1 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 456.9/664.8 MB 271.6 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 503.6/664.8 MB 277.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━ 551.3/664.8 MB 263.8 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 605.6/664.8 MB 262.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 664.8/664.8 MB 139.6 MB/s 0:00:03
[?25hCollecting nvidia-cublas-cu12==12.4.5.8 (from torch>=2.4)
 Using cached nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl (363.4 MB)
Collecting nvidia-cufft-cu12==11.2.1.3 (from torch>=2.4)
 Using cached nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl (211.5 MB)
Collecting nvidia-curand-cu12==10.3.5.147 (from torch>=2.4)
 Using cached nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl (56.3 MB)
Collecting nvidia-cusolver-cu12==11.6.1.9 (from torch>=2.4)
 Using cached nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl (127.9 MB)
Collecting nvidia-cusparse-cu12==12.3.1.170 (from torch>=2.4)
 Using cached nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl (207.5 MB)
Collecting nvidia-cusparselt-cu12==0.6.2 (from torch>=2.4)
 Using cached nvidia_cusparselt_cu12-0.6.2-py3-none-manylinux2014_x86_64.whl.metadata (6.8 kB)
Collecting nvidia-nccl-cu12==2.21.5 (from torch>=2.4)
 Using cached nvidia_nccl_cu12-2.21.5-py3-none-manylinux2014_x86_64.whl (188.7 MB)
Collecting nvidia-nvtx-cu12==12.4.127 (from torch>=2.4)
 Using cached nvidia_nvtx_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (99 kB)
Collecting nvidia-nvjitlink-cu12==12.4.127 (from torch>=2.4)
 Using cached nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB)
Collecting triton==3.2.0 (from torch>=2.4)
 Downloading triton-3.2.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.4 kB)
Collecting sympy==1.13.1 (from torch>=2.4)
 Using cached sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
Collecting mpmath<1.4,>=1.1.0 (from sympy==1.13.1->torch>=2.4)
 Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting MarkupSafe>=2.0 (from jinja2->torch>=2.4)
 Downloading markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB)
Downloading torch-2.6.0%2Bcu124-cp311-cp311-linux_x86_64.whl (768.5 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/768.5 MB ? eta -:--:--  ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.1/768.5 MB 98.5 MB/s eta 0:00:08  ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.4/768.5 MB 113.7 MB/s eta 0:00:07  ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.8/768.5 MB 126.8 MB/s eta 0:00:06  ━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.9/768.5 MB 132.4 MB/s eta 0:00:06  ━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.2/768.5 MB 140.0 MB/s eta 0:00:05  ━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.0/768.5 MB 133.9 MB/s eta 0:00:05  ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 188.2/768.5 MB 134.1 MB/s eta 0:00:05  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━ 221.0/768.5 MB 137.8 MB/s eta 0:00:04  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━ 250.6/768.5 MB 138.8 MB/s eta 0:00:04  ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━ 280.8/768.5 MB 143.9 MB/s eta 0:00:04  ━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 304.9/768.5 MB 143.2 MB/s eta 0:00:04  ━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ 336.3/768.5 MB 143.8 MB/s eta 0:00:04  ━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 363.9/768.5 MB 142.7 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━ 396.1/768.5 MB 140.9 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 425.5/768.5 MB 146.3 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 450.1/768.5 MB 144.5 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 478.7/768.5 MB 142.8 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 514.1/768.5 MB 145.6 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 548.4/768.5 MB 149.2 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 576.2/768.5 MB 149.4 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━ 607.4/768.5 MB 149.6 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 635.4/768.5 MB 150.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 663.5/768.5 MB 146.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 694.2/768.5 MB 149.1 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 724.6/768.5 MB 151.7 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 767.3/768.5 MB 157.6 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 768.5/768.5 MB 93.1 MB/s 0:00:06
[?25hUsing cached nvidia_cusparselt_cu12-0.6.2-py3-none-manylinux2014_x86_64.whl (150.1 MB)
Using cached sympy-1.13.1-py3-none-any.whl (6.2 MB)
Downloading triton-3.2.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (166.7 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/166.7 MB ? eta -:--:--  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.0/166.7 MB 159.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 66.1/166.7 MB 165.3 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━ 87.6/166.7 MB 162.4 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 95.7/166.7 MB 119.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 128.5/166.7 MB 127.8 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 158.6/166.7 MB 131.4 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.7/166.7 MB 118.9 MB/s 0:00:01
[?25hUsing cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Using cached typing_extensions-4.15.0-py3-none-any.whl (44 kB)
Using cached filelock-3.29.0-py3-none-any.whl (39 kB)
Downloading fsspec-2026.4.0-py3-none-any.whl (203 kB)
Using cached jinja2-3.1.6-py3-none-any.whl (134 kB)
Downloading markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (22 kB)
Using cached networkx-3.6.1-py3-none-any.whl (2.1 MB)
Installing collected packages: triton, nvidia-cusparselt-cu12, mpmath, typing-extensions, sympy, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, networkx, MarkupSafe, fsspec, filelock, nvidia-cusparse-cu12, nvidia-cudnn-cu12, jinja2, nvidia-cusolver-cu12, torch
[?25l  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  2/23 [mpmath]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  6/23 [nvidia-nvjitlink-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━  8/23 [nvidia-curand-cu12]  ━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━  8/23 [nvidia-curand-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 11/23 [nvidia-cuda-nvrtc-cu12]  ━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 11/23 [nvidia-cuda-nvrtc-cu12]  ━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━ 12/23 [nvidia-cuda-cupti-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━ 16/23 [fsspec]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/23 [torch]
[?25h Successfully installed MarkupSafe-3.0.3 filelock-3.29.0 fsspec-2026.4.0 jinja2-3.1.6 mpmath-1.3.0 networkx-3.6.1 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-cusparselt-cu12-0.6.2 nvidia-nccl-cu12-2.21.5 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.4.127 sympy-1.13.1 torch-2.6.0+cu124 triton-3.2.0 typing-extensions-4.15.0
Collecting transformers>=4.46
 Downloading transformers-5.9.0-py3-none-any.whl.metadata (33 kB)
Collecting peft>=0.12
 Downloading peft-0.19.1-py3-none-any.whl.metadata (15 kB)
Collecting accelerate>=0.34
 Using cached accelerate-1.13.0-py3-none-any.whl.metadata (19 kB)
Collecting safetensors
 Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
Collecting sentencepiece
 Downloading sentencepiece-0.2.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (10 kB)
Collecting datasets
 Downloading datasets-4.8.5-py3-none-any.whl.metadata (19 kB)
Collecting tqdm
 Using cached tqdm-4.67.3-py3-none-any.whl.metadata (57 kB)
Collecting huggingface_hub[cli]
 Downloading huggingface_hub-1.16.1-py3-none-any.whl.metadata (14 kB)
Collecting numpy>=1.17 (from transformers>=4.46)
 Downloading numpy-2.4.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: packaging>=20.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (26.2)
Collecting pyyaml>=5.1 (from transformers>=4.46)
 Downloading pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB)
Collecting regex>=2025.10.22 (from transformers>=4.46)
 Downloading regex-2026.5.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
Collecting tokenizers<=0.23.0,>=0.22.0 (from transformers>=4.46)
 Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Collecting typer (from transformers>=4.46)
 Downloading typer-0.25.1-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: filelock>=3.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (3.29.0)
Requirement already satisfied: fsspec>=2023.5.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (2026.4.0)
Collecting hf-xet<2.0.0,>=1.4.3 (from huggingface_hub[cli])
 Downloading hf_xet-1.5.0-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (4.9 kB)
Collecting httpx<1,>=0.23.0 (from huggingface_hub[cli])
 Using cached httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB)
Requirement already satisfied: typing-extensions>=4.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (4.15.0)
Collecting anyio (from httpx<1,>=0.23.0->huggingface_hub[cli])
 Downloading anyio-4.13.0-py3-none-any.whl.metadata (4.5 kB)
Collecting certifi (from httpx<1,>=0.23.0->huggingface_hub[cli])
 Downloading certifi-2026.5.20-py3-none-any.whl.metadata (2.5 kB)
Collecting httpcore==1.* (from httpx<1,>=0.23.0->huggingface_hub[cli])
 Using cached httpcore-1.0.9-py3-none-any.whl.metadata (21 kB)
Collecting idna (from httpx<1,>=0.23.0->huggingface_hub[cli])
 Downloading idna-3.16-py3-none-any.whl.metadata (6.4 kB)
Collecting h11>=0.16 (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub[cli])
 Using cached h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Collecting psutil (from peft>=0.12)
 Using cached psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (22 kB)
Requirement already satisfied: torch>=1.13.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from peft>=0.12) (2.6.0+cu124)
Collecting pyarrow>=21.0.0 (from datasets)
 Downloading pyarrow-24.0.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.0 kB)
Collecting dill<0.4.2,>=0.3.0 (from datasets)
 Downloading dill-0.4.1-py3-none-any.whl.metadata (10 kB)
Collecting pandas (from datasets)
 Downloading pandas-3.0.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (79 kB)
Collecting requests>=2.32.2 (from datasets)
 Downloading requests-2.34.2-py3-none-any.whl.metadata (4.8 kB)
Collecting xxhash (from datasets)
 Downloading xxhash-3.7.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (13 kB)
Collecting multiprocess<0.70.20 (from datasets)
 Downloading multiprocess-0.70.19-py311-none-any.whl.metadata (7.5 kB)
Collecting fsspec>=2023.5.0 (from huggingface_hub[cli])
 Downloading fsspec-2026.2.0-py3-none-any.whl.metadata (10 kB)
Collecting aiohttp!=4.0.0a0,!=4.0.0a1 (from fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Downloading aiohttp-3.13.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (8.1 kB)
WARNING: huggingface-hub 1.16.1 does not provide the extra 'cli'
Collecting aiohappyeyeballs>=2.5.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Downloading aiohappyeyeballs-2.6.2-py3-none-any.whl.metadata (5.9 kB)
Collecting aiosignal>=1.4.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Using cached aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB)
Collecting attrs>=17.3.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Using cached attrs-26.1.0-py3-none-any.whl.metadata (8.8 kB)
Collecting frozenlist>=1.1.1 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Downloading frozenlist-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (20 kB)
Collecting multidict<7.0,>=4.5 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Downloading multidict-6.7.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (5.3 kB)
Collecting propcache>=0.2.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Downloading propcache-0.5.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (16 kB)
Collecting yarl<2.0,>=1.17.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets)
 Downloading yarl-1.24.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (94 kB)
Collecting charset_normalizer<4,>=2 (from requests>=2.32.2->datasets)
 Downloading charset_normalizer-3.4.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB)
Collecting urllib3<3,>=1.26 (from requests>=2.32.2->datasets)
 Downloading urllib3-2.7.0-py3-none-any.whl.metadata (6.9 kB)
Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.6.1)
Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.1.6)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.3.1.170)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (0.6.2)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.2.0)
Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.13.0->peft>=0.12) (1.3.0)
Collecting click>=8.2.1 (from typer->transformers>=4.46)
 Downloading click-8.4.1-py3-none-any.whl.metadata (2.6 kB)
Collecting shellingham>=1.3.0 (from typer->transformers>=4.46)
 Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting rich>=13.8.0 (from typer->transformers>=4.46)
 Downloading rich-15.0.0-py3-none-any.whl.metadata (18 kB)
Collecting annotated-doc>=0.0.2 (from typer->transformers>=4.46)
 Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)
Collecting markdown-it-py>=2.2.0 (from rich>=13.8.0->typer->transformers>=4.46)
 Downloading markdown_it_py-4.2.0-py3-none-any.whl.metadata (7.4 kB)
Collecting pygments<3.0.0,>=2.13.0 (from rich>=13.8.0->typer->transformers>=4.46)
 Downloading pygments-2.20.0-py3-none-any.whl.metadata (2.5 kB)
Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers>=4.46)
 Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch>=1.13.0->peft>=0.12) (3.0.3)
Collecting python-dateutil>=2.8.2 (from pandas->datasets)
 Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas->datasets)
 Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Downloading transformers-5.9.0-py3-none-any.whl (10.8 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/10.8 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.8/10.8 MB 247.0 MB/s 0:00:00
[?25hDownloading huggingface_hub-1.16.1-py3-none-any.whl (668 kB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/668.2 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 668.2/668.2 kB 48.4 MB/s 0:00:00
[?25hDownloading hf_xet-1.5.0-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.5 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/4.5 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 140.5 MB/s 0:00:00
[?25hUsing cached httpx-0.28.1-py3-none-any.whl (73 kB)
Using cached httpcore-1.0.9-py3-none-any.whl (78 kB)
Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
Downloading peft-0.19.1-py3-none-any.whl (680 kB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/680.7 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 680.7/680.7 kB 27.1 MB/s 0:00:00
[?25hUsing cached accelerate-1.13.0-py3-none-any.whl (383 kB)
Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (507 kB)
Downloading sentencepiece-0.2.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.4 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.4 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 113.1 MB/s 0:00:00
[?25hDownloading datasets-4.8.5-py3-none-any.whl (528 kB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/529.0 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 529.0/529.0 kB 24.6 MB/s 0:00:00
[?25hDownloading dill-0.4.1-py3-none-any.whl (120 kB)
Downloading fsspec-2026.2.0-py3-none-any.whl (202 kB)
Downloading multiprocess-0.70.19-py311-none-any.whl (144 kB)
Using cached tqdm-4.67.3-py3-none-any.whl (78 kB)
Downloading aiohttp-3.13.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.8 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.8 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 91.1 MB/s 0:00:00
[?25hDownloading multidict-6.7.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (246 kB)
Downloading yarl-1.24.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (106 kB)
Downloading aiohappyeyeballs-2.6.2-py3-none-any.whl (15 kB)
Using cached aiosignal-1.4.0-py3-none-any.whl (7.5 kB)
Using cached attrs-26.1.0-py3-none-any.whl (67 kB)
Downloading frozenlist-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (231 kB)
Using cached h11-0.16.0-py3-none-any.whl (37 kB)
Downloading idna-3.16-py3-none-any.whl (74 kB)
Downloading numpy-2.4.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.9 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/16.9 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 356.4 MB/s 0:00:00
[?25hDownloading propcache-0.5.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (58 kB)
Downloading pyarrow-24.0.0-cp311-cp311-manylinux_2_28_x86_64.whl (48.8 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/48.8 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 38.0/48.8 MB 189.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.8/48.8 MB 156.9 MB/s 0:00:00
[?25hDownloading pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (806 kB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/806.6 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 806.6/806.6 kB 15.8 MB/s 0:00:00
[?25hDownloading regex-2026.5.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (799 kB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/799.8 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 799.8/799.8 kB 3.9 MB/s 0:00:00
[?25hDownloading requests-2.34.2-py3-none-any.whl (73 kB)
Downloading charset_normalizer-3.4.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (214 kB)
Downloading urllib3-2.7.0-py3-none-any.whl (131 kB)
Downloading certifi-2026.5.20-py3-none-any.whl (134 kB)
Downloading typer-0.25.1-py3-none-any.whl (58 kB)
Downloading annotated_doc-0.0.4-py3-none-any.whl (5.3 kB)
Downloading click-8.4.1-py3-none-any.whl (116 kB)
Downloading rich-15.0.0-py3-none-any.whl (310 kB)
Downloading pygments-2.20.0-py3-none-any.whl (1.2 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.2 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 84.7 MB/s 0:00:00
[?25hDownloading markdown_it_py-4.2.0-py3-none-any.whl (91 kB)
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB)
Downloading anyio-4.13.0-py3-none-any.whl (114 kB)
Downloading pandas-3.0.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (11.3 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/11.3 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 140.0 MB/s 0:00:00
[?25hUsing cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Using cached psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl (155 kB)
Downloading xxhash-3.7.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (193 kB)
Installing collected packages: xxhash, urllib3, tqdm, six, shellingham, sentencepiece, safetensors, regex, pyyaml, pygments, pyarrow, psutil, propcache, numpy, multidict, mdurl, idna, hf-xet, h11, fsspec, frozenlist, dill, click, charset_normalizer, certifi, attrs, annotated-doc, aiohappyeyeballs, yarl, requests, python-dateutil, multiprocess, markdown-it-py, httpcore, anyio, aiosignal, rich, pandas, httpx, aiohttp, typer, huggingface_hub, tokenizers, datasets, accelerate, transformers, peft
[?25l  ━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/47 [regex]  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  9/47 [pygments]  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  9/47 [pygments]  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  9/47 [pygments]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Attempting uninstall: fsspec
 ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Found existing installation: fsspec 2026.4.0
 ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Uninstalling fsspec-2026.4.0:
 ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Successfully uninstalled fsspec-2026.4.0
 ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  ━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━ 21/47 [dill]  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 28/47 [yarl]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 32/47 [markdown-it-py]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━ 36/47 [rich]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 39/47 [aiohttp]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 41/47 [huggingface_hub]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 41/47 [huggingface_hub]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 43/47 [datasets]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 44/47 [accelerate]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 46/47 [peft]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 46/47 [peft]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47/47 [peft]
[?25h Successfully installed accelerate-1.13.0 aiohappyeyeballs-2.6.2 aiohttp-3.13.5 aiosignal-1.4.0 annotated-doc-0.0.4 anyio-4.13.0 attrs-26.1.0 certifi-2026.5.20 charset_normalizer-3.4.7 click-8.4.1 datasets-4.8.5 dill-0.4.1 frozenlist-1.8.0 fsspec-2026.2.0 h11-0.16.0 hf-xet-1.5.0 httpcore-1.0.9 httpx-0.28.1 huggingface_hub-1.16.1 idna-3.16 markdown-it-py-4.2.0 mdurl-0.1.2 multidict-6.7.1 multiprocess-0.70.19 numpy-2.4.6 pandas-3.0.3 peft-0.19.1 propcache-0.5.2 psutil-7.2.2 pyarrow-24.0.0 pygments-2.20.0 python-dateutil-2.9.0.post0 pyyaml-6.0.3 regex-2026.5.9 requests-2.34.2 rich-15.0.0 safetensors-0.7.0 sentencepiece-0.2.1 shellingham-1.5.4 six-1.17.0 tokenizers-0.22.2 tqdm-4.67.3 transformers-5.9.0 typer-0.25.1 urllib3-2.7.0 xxhash-3.7.0 yarl-1.24.2
Collecting flash-attn
 Downloading flash_attn-2.8.3.tar.gz (8.4 MB)
[?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/8.4 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/8.4 MB 112.2 MB/s 0:00:00
[?25h Preparing metadata (pyproject.toml): started
 Preparing metadata (pyproject.toml): finished with status 'done'
Requirement already satisfied: torch in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from flash-attn) (2.6.0+cu124)
Collecting einops (from flash-attn)
 Using cached einops-0.8.2-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: filelock in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.29.0)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (4.15.0)
Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.6.1)
Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.1.6)
Requirement already satisfied: fsspec in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (2026.2.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.3.1.170)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (0.6.2)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127)
Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.2.0)
Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch->flash-attn) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch->flash-attn) (3.0.3)
Using cached einops-0.8.2-py3-none-any.whl (65 kB)
Building wheels for collected packages: flash-attn
 Building wheel for flash-attn (pyproject.toml): started
 Building wheel for flash-attn (pyproject.toml): finished with status 'done'
 Created wheel for flash-attn: filename=flash_attn-2.8.3-cp311-cp311-linux_x86_64.whl size=256022485 sha256=0abc62d04f28f140f4f76ab7cfd1d8ce24a69c6ab0cbace8d4ab99640b68dc0a
 Stored in directory: /home/lakesenberg/.cache/pip/wheels/42/31/1f/4b22dd7295b3cb064b8fa9038f6d58fb15c9571555b2d7c39c
Successfully built flash-attn
Installing collected packages: einops, flash-attn
[?25l  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 [flash-attn]
[?25h Successfully installed einops-0.8.2 flash-attn-2.8.3
[info] downloading nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 -> /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
Warning: `huggingface-cli` is deprecated and no longer works. Use `hf` instead.
Hint: `hf` is already installed! Use it directly.
Hint: Examples:
 hf auth login
 hf download unsloth/gemma-4-31B-it-GGUF
 hf upload my-cool-model . .
 hf models ls --search "gemma"
 hf repos ls --format json
 hf jobs run python:3.12 python -c 'print("Hello!")'
 hf --help
EXIT_CODE=1
=== RESTART Fri May 22 15:20:38 UTC 2026 ===
=== [Fri May 22 15:20:38 UTC 2026] RESTART: Phase 0 setup + model download ===
Requirement already satisfied: pip in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (26.1.1)
Requirement already satisfied: wheel in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.47.0)
Requirement already satisfied: packaging>=24.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from wheel) (26.2)
Looking in indexes: https://download.pytorch.org/whl/cu124
Requirement already satisfied: torch>=2.4 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (2.6.0+cu124)
Requirement already satisfied: filelock in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.29.0)
Requirement already satisfied: typing-extensions>=4.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (4.15.0)
Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.6.1)
Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.1.6)
Requirement already satisfied: fsspec in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (2026.2.0)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.3.1.170)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (0.6.2)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127)
Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.2.0)
Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch>=2.4) (1.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch>=2.4) (3.0.3)
Requirement already satisfied: transformers>=4.46 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (5.9.0)
Requirement already satisfied: peft>=0.12 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.19.1)
Requirement already satisfied: accelerate>=0.34 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (1.13.0)
Requirement already satisfied: safetensors in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.7.0)
Requirement already satisfied: sentencepiece in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.2.1)
Requirement already satisfied: datasets in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (4.8.5)
Requirement already satisfied: tqdm in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (4.67.3)
Requirement already satisfied: huggingface_hub[cli] in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (1.16.1)
Requirement already satisfied: numpy>=1.17 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (2.4.6)
Requirement already satisfied: packaging>=20.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (26.2)
Requirement already satisfied: pyyaml>=5.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (6.0.3)
Requirement already satisfied: regex>=2025.10.22 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (2026.5.9)
Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (0.22.2)
Requirement already satisfied: typer in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (0.25.1)
Requirement already satisfied: filelock>=3.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (3.29.0)
Requirement already satisfied: fsspec>=2023.5.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (2026.2.0)
Requirement already satisfied: hf-xet<2.0.0,>=1.4.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (1.5.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (0.28.1)
Requirement already satisfied: typing-extensions>=4.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (4.15.0)
Requirement already satisfied: anyio in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (4.13.0)
Requirement already satisfied: certifi in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (2026.5.20)
Requirement already satisfied: httpcore==1.* in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (1.0.9)
Requirement already satisfied: idna in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (3.16)
Requirement already satisfied: h11>=0.16 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub[cli]) (0.16.0)
Requirement already satisfied: psutil in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from peft>=0.12) (7.2.2)
Requirement already satisfied: torch>=1.13.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from peft>=0.12) (2.6.0+cu124)
Requirement already satisfied: pyarrow>=21.0.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (24.0.0)
Requirement already satisfied: dill<0.4.2,>=0.3.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (0.4.1)
Requirement already satisfied: pandas in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (3.0.3)
Requirement already satisfied: requests>=2.32.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (2.34.2)
Requirement already satisfied: xxhash in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (3.7.0)
Requirement already satisfied: multiprocess<0.70.20 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (0.70.19)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (3.13.5)
WARNING: huggingface-hub 1.16.1 does not provide the extra 'cli'
Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (2.6.2)
Requirement already satisfied: aiosignal>=1.4.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (1.4.0)
Requirement already satisfied: attrs>=17.3.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (26.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (1.8.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (6.7.1)
Requirement already satisfied: propcache>=0.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (0.5.2)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (1.24.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (3.4.7)
Requirement already satisfied: urllib3<3,>=1.26 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (2.7.0)
Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.6.1)
Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.1.6)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (9.1.0.70)
Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.5.8)
Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.2.1.3)
Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (10.3.5.147)
Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.6.1.9)
Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.3.1.170)
Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (0.6.2)
Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (2.21.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127)
Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.2.0)
Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (1.13.1)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.13.0->peft>=0.12) (1.3.0)
Requirement already satisfied: click>=8.2.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (8.4.1)
Requirement already satisfied: shellingham>=1.3.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (1.5.4)
Requirement already satisfied: rich>=13.8.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (15.0.0)
Requirement already satisfied: annotated-doc>=0.0.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (0.0.4)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from rich>=13.8.0->typer->transformers>=4.46) (4.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from rich>=13.8.0->typer->transformers>=4.46) (2.20.0)
Requirement already satisfied: mdurl~=0.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers>=4.46) (0.1.2)
Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch>=1.13.0->peft>=0.12) (3.0.3)
Requirement already satisfied: python-dateutil>=2.8.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from pandas->datasets) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0)
[info] skipping flash-attn (default). Set INSTALL_FLASH_ATTN=1 to try.
[info] downloading nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 -> /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
path=/home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
=== Environment ready ===
 env : nemo
 base model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
Next: conda activate nemo && bash run_train.sh
=== [Fri May 22 15:23:06 UTC 2026] Phase 1: smoke test (20 steps) ===
[stage 1] generating 100 examples per task -> /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
wrote 700 examples to /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
--- sample ---
### Task: cryptarithm
### Problem:
Cryptarithm-style mapping. Examples:
 14,51 -> 1451
 13,96 -> 1396
 11,68 -> 1168
Apply to: 42,37
### Reasoning:
<|cot_start|>Examples:
 14,51 -> 1451
 13,96 -> 1396
 11,68 -> 1168
Check both candidate forms:
 concat(14,51)=1451 rev_concat(14,51)=5114 target=1451 concat
 concat(13,96)=1396 rev_concat(13,96)=9613 target=1396 concat
 concat(11,68)=1168 rev_concat(11,68)=6811 target=1168 concat
Conclude operator: concat.
Apply to (42, 37): answer = 4237
Final: \boxed{4237}
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/smoke
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 20
 grad_accum: 1
 batch_size: 1
 seq_len: 1024
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
Traceback (most recent call last):
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 64, in <module>
 from mamba_ssm.ops.triton.layernorm_gated import rmsnorm_fn
ModuleNotFoundError: No module named 'mamba_ssm'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 315, in <module>
 train(cfg)
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 221, in train
 model = build_model(cfg)
 ^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 187, in build_model
 model = AutoModelForCausalLM.from_pretrained(cfg.base_model, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained
 model_class = get_class_from_dynamic_module(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 627, in get_class_from_dynamic_module
 return get_class_in_module(class_name, final_module, force_reload=force_download)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 309, in get_class_in_module
 module_spec.loader.exec_module(module)
 File "<frozen importlib._bootstrap_external>", line 940, in exec_module
 File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 66, in <module>
 raise ImportError("mamba-ssm is required by the Mamba model but cannot be imported")
ImportError: mamba-ssm is required by the Mamba model but cannot be imported
EXIT_CODE=1
=== RESUME2 Fri May 22 16:37:57 UTC 2026 ===
=== [Fri May 22 16:37:57 UTC 2026] RESUME: verify mamba-ssm ===
mamba_ssm OK
=== [Fri May 22 16:37:59 UTC 2026] Phase 1: smoke test (20 steps) ===
[stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/smoke
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 20
 grad_accum: 1
 batch_size: 1
 seq_len: 1024
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 131/6243 [00:00<00:04, 1302.67it/s] Loading weights: 4%|▍ | 262/6243 [00:00<00:07, 827.29it/s] Loading weights: 6%|▌ | 355/6243 [00:00<00:07, 751.81it/s] Loading weights: 7%|▋ | 435/6243 [00:00<00:07, 733.31it/s] Loading weights: 8%|▊ | 511/6243 [00:00<00:07, 718.08it/s] Loading weights: 9%|▉ | 585/6243 [00:00<00:07, 707.71it/s] Loading weights: 11%|█ | 657/6243 [00:00<00:08, 684.43it/s] Loading weights: 12%|█▏ | 726/6243 [00:00<00:08, 666.98it/s] Loading weights: 13%|█▎ | 793/6243 [00:01<00:08, 657.98it/s] Loading weights: 14%|█▍ | 862/6243 [00:01<00:08, 665.35it/s] Loading weights: 15%|█▍ | 930/6243 [00:01<00:07, 667.23it/s] Loading weights: 16%|█▌ | 997/6243 [00:01<00:07, 664.56it/s] Loading weights: 17%|█▋ | 1064/6243 [00:01<00:07, 654.62it/s] Loading weights: 18%|█▊ | 1132/6243 [00:01<00:07, 660.64it/s] Loading weights: 19%|█▉ | 1199/6243 [00:01<00:07, 654.64it/s] Loading weights: 20%|██ | 1265/6243 [00:01<00:07, 651.35it/s] Loading weights: 21%|██▏ | 1333/6243 [00:01<00:07, 657.42it/s] Loading weights: 22%|██▏ | 1402/6243 [00:02<00:07, 666.57it/s] Loading weights: 24%|██▎ | 1469/6243 [00:02<00:07, 666.18it/s] Loading weights: 25%|██▍ | 1536/6243 [00:02<00:07, 664.66it/s] Loading weights: 26%|██▌ | 1606/6243 [00:02<00:06, 673.98it/s] Loading weights: 27%|██▋ | 1674/6243 [00:02<00:06, 672.92it/s] Loading weights: 28%|██▊ | 1742/6243 [00:02<00:06, 666.69it/s] Loading weights: 29%|██▉ | 1809/6243 [00:02<00:06, 652.35it/s] Loading weights: 30%|███ | 1875/6243 [00:02<00:06, 649.76it/s] Loading weights: 31%|███ | 1941/6243 [00:02<00:06, 647.19it/s] Loading weights: 32%|███▏ | 2007/6243 [00:02<00:06, 649.32it/s] Loading weights: 33%|███▎ | 2072/6243 [00:03<00:06, 646.03it/s] Loading weights: 34%|███▍ | 2137/6243 [00:03<00:06, 645.73it/s] Loading weights: 35%|███▌ | 2205/6243 [00:03<00:06, 655.69it/s] Loading weights: 36%|███▋ | 2271/6243 [00:03<00:06, 646.41it/s] Loading weights: 37%|███▋ | 2336/6243 [00:03<00:06, 640.61it/s] Loading weights: 38%|███▊ | 2402/6243 [00:03<00:05, 643.24it/s] Loading weights: 40%|███▉ | 2468/6243 [00:03<00:05, 648.07it/s] Loading weights: 41%|████ | 2533/6243 [00:03<00:05, 646.83it/s] Loading weights: 42%|████▏ | 2598/6243 [00:03<00:05, 639.87it/s] Loading weights: 43%|████▎ | 2664/6243 [00:03<00:05, 643.40it/s] Loading weights: 44%|████▎ | 2729/6243 [00:04<00:05, 642.78it/s] Loading weights: 45%|████▍ | 2795/6243 [00:04<00:05, 646.45it/s] Loading weights: 46%|████▌ | 2861/6243 [00:04<00:05, 649.87it/s] Loading weights: 47%|████▋ | 2929/6243 [00:04<00:05, 656.59it/s] Loading weights: 48%|████▊ | 2996/6243 [00:04<00:04, 660.35it/s] Loading weights: 49%|████▉ | 3063/6243 [00:04<00:04, 660.89it/s] Loading weights: 50%|█████ | 3130/6243 [00:04<00:04, 646.32it/s] Loading weights: 51%|█████ | 3195/6243 [00:04<00:04, 641.54it/s] Loading weights: 52%|█████▏ | 3260/6243 [00:04<00:04, 637.94it/s] Loading weights: 53%|█████▎ | 3325/6243 [00:04<00:04, 638.77it/s] Loading weights: 54%|█████▍ | 3389/6243 [00:05<00:04, 638.20it/s] Loading weights: 55%|█████▌ | 3458/6243 [00:05<00:04, 651.83it/s] Loading weights: 57%|█████▋ | 3529/6243 [00:05<00:04, 665.64it/s] Loading weights: 58%|█████▊ | 3596/6243 [00:05<00:04, 654.56it/s] Loading weights: 59%|█████▊ | 3662/6243 [00:05<00:04, 644.17it/s] Loading weights: 60%|█████▉ | 3727/6243 [00:05<00:03, 636.88it/s] Loading weights: 61%|██████ | 3791/6243 [00:05<00:03, 635.43it/s] Loading weights: 62%|██████▏ | 3857/6243 [00:05<00:03, 641.79it/s] Loading weights: 63%|██████▎ | 3922/6243 [00:05<00:03, 635.76it/s] Loading weights: 64%|██████▍ | 3989/6243 [00:06<00:03, 645.20it/s] Loading weights: 65%|██████▍ | 4057/6243 [00:06<00:03, 654.55it/s] Loading weights: 66%|██████▌ | 4123/6243 [00:06<00:03, 655.23it/s] Loading weights: 67%|██████▋ | 4189/6243 [00:06<00:03, 649.20it/s] Loading weights: 68%|██████▊ | 4254/6243 [00:06<00:03, 640.62it/s] Loading weights: 69%|██████▉ | 4320/6243 [00:06<00:02, 646.03it/s] Loading weights: 70%|███████ | 4387/6243 [00:06<00:02, 650.70it/s] Loading weights: 71%|███████▏ | 4453/6243 [00:06<00:02, 647.50it/s] Loading weights: 72%|███████▏ | 4519/6243 [00:06<00:02, 649.33it/s] Loading weights: 73%|███████▎ | 4587/6243 [00:06<00:02, 655.67it/s] Loading weights: 75%|███████▍ | 4656/6243 [00:07<00:02, 663.12it/s] Loading weights: 76%|███████▌ | 4723/6243 [00:07<00:02, 659.60it/s] Loading weights: 77%|███████▋ | 4790/6243 [00:07<00:02, 660.04it/s] Loading weights: 78%|███████▊ | 4857/6243 [00:07<00:02, 659.62it/s] Loading weights: 79%|███████▉ | 4924/6243 [00:07<00:01, 660.82it/s] Loading weights: 80%|███████▉ | 4991/6243 [00:07<00:01, 650.71it/s] Loading weights: 81%|████████ | 5057/6243 [00:07<00:01, 644.39it/s] Loading weights: 82%|████████▏ | 5122/6243 [00:07<00:01, 643.45it/s] Loading weights: 83%|████████▎ | 5188/6243 [00:07<00:01, 647.95it/s] Loading weights: 84%|████████▍ | 5253/6243 [00:07<00:01, 634.58it/s] Loading weights: 85%|████████▌ | 5319/6243 [00:08<00:01, 640.39it/s] Loading weights: 86%|████████▌ | 5384/6243 [00:08<00:01, 641.59it/s] Loading weights: 87%|████████▋ | 5450/6243 [00:08<00:01, 645.76it/s] Loading weights: 88%|████████▊ | 5515/6243 [00:08<00:01, 636.50it/s] Loading weights: 89%|████████▉ | 5580/6243 [00:08<00:01, 638.48it/s] Loading weights: 90%|█████████ | 5645/6243 [00:08<00:00, 640.05it/s] Loading weights: 91%|█████████▏| 5712/6243 [00:08<00:00, 648.03it/s] Loading weights: 93%|█████████▎| 5779/6243 [00:08<00:00, 652.65it/s] Loading weights: 94%|█████████▎| 5845/6243 [00:08<00:00, 648.99it/s] Loading weights: 95%|█████████▍| 5910/6243 [00:08<00:00, 645.80it/s] Loading weights: 96%|█████████▌| 5977/6243 [00:09<00:00, 652.68it/s] Loading weights: 97%|█████████▋| 6043/6243 [00:09<00:00, 641.02it/s] Loading weights: 98%|█████████▊| 6108/6243 [00:09<00:00, 641.17it/s] Loading weights: 99%|█████████▉| 6173/6243 [00:09<00:00, 638.44it/s] Loading weights: 100%|█████████▉| 6237/6243 [00:09<00:00, 638.17it/s] Loading weights: 100%|██████████| 6243/6243 [00:09<00:00, 650.43it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
Traceback (most recent call last):
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 315, in <module>
 train(cfg)
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 254, in train
 out = model(input_ids=input_ids, attention_mask=attn_mask, use_cache=False)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1993, in forward
 return self.base_model(
 ^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 330, in forward
 return self.model.forward(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1702, in forward
 nemotron_h_outputs = self.backbone(
 ^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1489, in forward
 hidden_states = self._gradient_checkpointing_func(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner
 return disable_fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
 return fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint
 ret = function(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 777, in forward
 hidden_states = self.mixer(
 ^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 715, in forward
 return self.cuda_kernels_forward(hidden_states, cache_params, cache_position, attention_mask)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 429, in cuda_kernels_forward
 out = mamba_split_conv1d_scan_combined(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 997, in mamba_split_conv1d_scan_combined
 return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
 return super().apply(*args, **kwargs) # type: ignore[misc]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 503, in decorate_fwd
 return fwd(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 841, in forward
 causal_conv1d_fwd_function(rearrange(ensure_stride(xBC), "b s d -> b d s"),
TypeError: 'NoneType' object is not callable
EXIT_CODE=1
=== RESUME3 Fri May 22 16:59:38 UTC 2026 ===
=== [Fri May 22 16:59:38 UTC 2026] patch Nemotron model cache (force slow mamba path) ===
=== [Fri May 22 16:59:38 UTC 2026] RESUME: verify mamba-ssm ===
mamba_ssm OK
=== [Fri May 22 16:59:40 UTC 2026] Phase 1: smoke test (20 steps) ===
[stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/smoke
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 20
 grad_accum: 1
 batch_size: 1
 seq_len: 1024
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 144/6243 [00:00<00:04, 1432.41it/s] Loading weights: 5%|▍ | 288/6243 [00:00<00:06, 917.08it/s] Loading weights: 6%|▋ | 391/6243 [00:00<00:07, 834.73it/s] Loading weights: 8%|▊ | 480/6243 [00:00<00:07, 795.05it/s] Loading weights: 9%|▉ | 563/6243 [00:00<00:07, 775.27it/s] Loading weights: 10%|█ | 642/6243 [00:00<00:07, 757.28it/s] Loading weights: 12%|█▏ | 719/6243 [00:00<00:07, 745.99it/s] Loading weights: 13%|█▎ | 794/6243 [00:00<00:07, 738.15it/s] Loading weights: 14%|█▍ | 869/6243 [00:01<00:07, 740.64it/s] Loading weights: 15%|█▌ | 944/6243 [00:01<00:07, 738.49it/s] Loading weights: 16%|█▋ | 1018/6243 [00:01<00:07, 728.73it/s] Loading weights: 17%|█▋ | 1091/6243 [00:01<00:07, 723.85it/s] Loading weights: 19%|█▊ | 1165/6243 [00:01<00:06, 726.24it/s] Loading weights: 20%|█▉ | 1238/6243 [00:01<00:06, 723.96it/s] Loading weights: 21%|██ | 1311/6243 [00:01<00:06, 720.69it/s] Loading weights: 22%|██▏ | 1386/6243 [00:01<00:06, 727.76it/s] Loading weights: 23%|██▎ | 1459/6243 [00:01<00:06, 723.97it/s] Loading weights: 25%|██▍ | 1532/6243 [00:02<00:06, 715.86it/s] Loading weights: 26%|██▌ | 1604/6243 [00:02<00:06, 713.90it/s] Loading weights: 27%|██▋ | 1676/6243 [00:02<00:06, 711.96it/s] Loading weights: 28%|██▊ | 1748/6243 [00:02<00:06, 708.37it/s] Loading weights: 29%|██▉ | 1819/6243 [00:02<00:06, 703.90it/s] Loading weights: 30%|███ | 1890/6243 [00:02<00:06, 700.86it/s] Loading weights: 31%|███▏ | 1962/6243 [00:02<00:06, 705.05it/s] Loading weights: 33%|███▎ | 2033/6243 [00:02<00:06, 695.50it/s] Loading weights: 34%|███▎ | 2103/6243 [00:02<00:05, 690.86it/s] Loading weights: 35%|███▍ | 2176/6243 [00:02<00:05, 685.11it/s] Loading weights: 36%|███▌ | 2250/6243 [00:03<00:05, 700.26it/s] Loading weights: 37%|███▋ | 2321/6243 [00:03<00:05, 699.55it/s] Loading weights: 38%|███▊ | 2392/6243 [00:03<00:05, 689.88it/s] Loading weights: 39%|███▉ | 2463/6243 [00:03<00:05, 694.80it/s] Loading weights: 41%|████ | 2534/6243 [00:03<00:05, 696.87it/s] Loading weights: 42%|████▏ | 2605/6243 [00:03<00:05, 699.54it/s] Loading weights: 43%|████▎ | 2676/6243 [00:03<00:05, 702.48it/s] Loading weights: 44%|████▍ | 2748/6243 [00:03<00:04, 705.31it/s] Loading weights: 45%|████▌ | 2819/6243 [00:03<00:04, 700.79it/s] Loading weights: 46%|████▋ | 2890/6243 [00:03<00:04, 694.79it/s] Loading weights: 47%|████▋ | 2960/6243 [00:04<00:04, 687.62it/s] Loading weights: 49%|████▊ | 3030/6243 [00:04<00:04, 689.73it/s] Loading weights: 50%|████▉ | 3101/6243 [00:04<00:04, 694.36it/s] Loading weights: 51%|█████ | 3172/6243 [00:04<00:04, 697.74it/s] Loading weights: 52%|█████▏ | 3243/6243 [00:04<00:04, 701.25it/s] Loading weights: 53%|█████▎ | 3314/6243 [00:04<00:04, 703.79it/s] Loading weights: 54%|█████▍ | 3385/6243 [00:04<00:04, 703.68it/s] Loading weights: 55%|█████▌ | 3456/6243 [00:04<00:03, 703.80it/s] Loading weights: 57%|█████▋ | 3529/6243 [00:04<00:03, 709.63it/s] Loading weights: 58%|█████▊ | 3600/6243 [00:04<00:03, 703.69it/s] Loading weights: 59%|█████▉ | 3671/6243 [00:05<00:03, 704.62it/s] Loading weights: 60%|█████▉ | 3742/6243 [00:05<00:03, 702.95it/s] Loading weights: 61%|██████ | 3813/6243 [00:05<00:03, 701.61it/s] Loading weights: 62%|██████▏ | 3885/6243 [00:05<00:03, 704.78it/s] Loading weights: 63%|██████▎ | 3956/6243 [00:05<00:03, 698.17it/s] Loading weights: 64%|██████▍ | 4026/6243 [00:05<00:03, 691.23it/s] Loading weights: 66%|██████▌ | 4097/6243 [00:05<00:03, 694.15it/s] Loading weights: 67%|██████▋ | 4167/6243 [00:05<00:02, 695.46it/s] Loading weights: 68%|██████▊ | 4237/6243 [00:05<00:02, 694.12it/s] Loading weights: 69%|██████▉ | 4308/6243 [00:05<00:02, 696.86it/s] Loading weights: 70%|███████ | 4381/6243 [00:06<00:02, 705.34it/s] Loading weights: 71%|███████▏ | 4452/6243 [00:06<00:02, 705.24it/s] Loading weights: 72%|███████▏ | 4523/6243 [00:06<00:02, 704.04it/s] Loading weights: 74%|███████▎ | 4594/6243 [00:06<00:02, 703.46it/s] Loading weights: 75%|███████▍ | 4667/6243 [00:06<00:02, 709.56it/s] Loading weights: 76%|███████▌ | 4738/6243 [00:06<00:02, 709.03it/s] Loading weights: 77%|███████▋ | 4809/6243 [00:06<00:02, 703.41it/s] Loading weights: 78%|███████▊ | 4881/6243 [00:06<00:01, 705.28it/s] Loading weights: 79%|███████▉ | 4954/6243 [00:06<00:01, 709.75it/s] Loading weights: 80%|████████ | 5025/6243 [00:07<00:01, 702.69it/s] Loading weights: 82%|████████▏ | 5096/6243 [00:07<00:01, 700.93it/s] Loading weights: 83%|████████▎ | 5167/6243 [00:07<00:01, 696.72it/s] Loading weights: 84%|████████▍ | 5240/6243 [00:07<00:01, 705.31it/s] Loading weights: 85%|████████▌ | 5311/6243 [00:07<00:01, 703.84it/s] Loading weights: 86%|████████▌ | 5382/6243 [00:07<00:01, 703.30it/s] Loading weights: 87%|████████▋ | 5455/6243 [00:07<00:01, 711.16it/s] Loading weights: 89%|████████▊ | 5527/6243 [00:07<00:01, 704.75it/s] Loading weights: 90%|████████▉ | 5598/6243 [00:07<00:00, 704.61it/s] Loading weights: 91%|█████████ | 5669/6243 [00:07<00:00, 701.83it/s] Loading weights: 92%|█████████▏| 5741/6243 [00:08<00:00, 704.37it/s] Loading weights: 93%|█████████▎| 5812/6243 [00:08<00:00, 698.56it/s] Loading weights: 94%|█████████▍| 5882/6243 [00:08<00:00, 694.99it/s] Loading weights: 95%|█████████▌| 5952/6243 [00:08<00:00, 685.70it/s] Loading weights: 96%|█████████▋| 6023/6243 [00:08<00:00, 692.74it/s] Loading weights: 98%|█████████▊| 6093/6243 [00:08<00:00, 688.44it/s] Loading weights: 99%|█████████▊| 6162/6243 [00:08<00:00, 687.72it/s] Loading weights: 100%|█████████▉| 6232/6243 [00:08<00:00, 690.41it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 705.17it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
Traceback (most recent call last):
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 315, in <module>
 train(cfg)
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 254, in train
 out = model(input_ids=input_ids, attention_mask=attn_mask, use_cache=False)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1993, in forward
 return self.base_model(
 ^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 330, in forward
 return self.model.forward(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1702, in forward
 nemotron_h_outputs = self.backbone(
 ^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1489, in forward
 hidden_states = self._gradient_checkpointing_func(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner
 return disable_fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
 return fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint
 ret = function(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 777, in forward
 hidden_states = self.mixer(
 ^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 715, in forward
 return self.cuda_kernels_forward(hidden_states, cache_params, cache_position, attention_mask)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 429, in cuda_kernels_forward
 out = mamba_split_conv1d_scan_combined(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 997, in mamba_split_conv1d_scan_combined
 return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
 return super().apply(*args, **kwargs) # type: ignore[misc]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 503, in decorate_fwd
 return fwd(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 841, in forward
 causal_conv1d_fwd_function(rearrange(ensure_stride(xBC), "b s d -> b d s"),
TypeError: 'NoneType' object is not callable
EXIT_CODE=1
=== RESUME4 Fri May 22 17:02:26 UTC 2026 ===
=== [Fri May 22 17:02:27 UTC 2026] patch Nemotron model cache (force slow mamba path) ===
=== [Fri May 22 17:02:27 UTC 2026] RESUME: verify mamba-ssm ===
mamba_ssm OK
=== [Fri May 22 17:02:28 UTC 2026] Phase 1: smoke test (20 steps) ===
[stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/smoke
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 20
 grad_accum: 1
 batch_size: 1
 seq_len: 1024
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 144/6243 [00:00<00:04, 1428.17it/s] Loading weights: 5%|▍ | 287/6243 [00:00<00:06, 901.69it/s] Loading weights: 6%|▌ | 388/6243 [00:00<00:07, 819.10it/s] Loading weights: 8%|▊ | 476/6243 [00:00<00:07, 773.98it/s] Loading weights: 9%|▉ | 557/6243 [00:00<00:07, 758.22it/s] Loading weights: 10%|█ | 635/6243 [00:00<00:07, 737.69it/s] Loading weights: 11%|█▏ | 710/6243 [00:00<00:07, 727.06it/s] Loading weights: 13%|█▎ | 784/6243 [00:01<00:07, 721.37it/s] Loading weights: 14%|█▎ | 857/6243 [00:01<00:07, 723.10it/s] Loading weights: 15%|█▍ | 930/6243 [00:01<00:07, 724.56it/s] Loading weights: 16%|█▌ | 1003/6243 [00:01<00:07, 716.50it/s] Loading weights: 17%|█▋ | 1075/6243 [00:01<00:07, 715.37it/s] Loading weights: 18%|█▊ | 1150/6243 [00:01<00:07, 724.41it/s] Loading weights: 20%|█▉ | 1223/6243 [00:01<00:06, 720.61it/s] Loading weights: 21%|██ | 1296/6243 [00:01<00:06, 722.63it/s] Loading weights: 22%|██▏ | 1371/6243 [00:01<00:06, 729.82it/s] Loading weights: 23%|██▎ | 1445/6243 [00:01<00:06, 729.80it/s] Loading weights: 24%|██▍ | 1519/6243 [00:02<00:06, 729.36it/s] Loading weights: 26%|██▌ | 1592/6243 [00:02<00:06, 728.76it/s] Loading weights: 27%|██▋ | 1666/6243 [00:02<00:06, 730.62it/s] Loading weights: 28%|██▊ | 1740/6243 [00:02<00:06, 729.72it/s] Loading weights: 29%|██▉ | 1813/6243 [00:02<00:06, 728.65it/s] Loading weights: 30%|███ | 1886/6243 [00:02<00:06, 724.40it/s] Loading weights: 31%|███▏ | 1962/6243 [00:02<00:05, 733.56it/s] Loading weights: 33%|███▎ | 2036/6243 [00:02<00:05, 732.17it/s] Loading weights: 34%|███▍ | 2110/6243 [00:02<00:05, 726.57it/s] Loading weights: 35%|███▌ | 2186/6243 [00:02<00:05, 735.98it/s] Loading weights: 36%|███▌ | 2260/6243 [00:03<00:05, 734.70it/s] Loading weights: 37%|███▋ | 2334/6243 [00:03<00:05, 730.44it/s] Loading weights: 39%|███▊ | 2408/6243 [00:03<00:05, 723.32it/s] Loading weights: 40%|███▉ | 2481/6243 [00:03<00:05, 724.22it/s] Loading weights: 41%|████ | 2554/6243 [00:03<00:05, 717.07it/s] Loading weights: 42%|████▏ | 2626/6243 [00:03<00:05, 714.99it/s] Loading weights: 43%|████▎ | 2698/6243 [00:03<00:04, 713.02it/s] Loading weights: 44%|████▍ | 2770/6243 [00:03<00:04, 714.40it/s] Loading weights: 46%|████▌ | 2842/6243 [00:03<00:04, 710.49it/s] Loading weights: 47%|████▋ | 2914/6243 [00:03<00:04, 702.36it/s] Loading weights: 48%|████▊ | 2985/6243 [00:04<00:04, 704.28it/s] Loading weights: 49%|████▉ | 3058/6243 [00:04<00:04, 710.18it/s] Loading weights: 50%|█████ | 3130/6243 [00:04<00:04, 708.72it/s] Loading weights: 51%|█████▏ | 3201/6243 [00:04<00:04, 708.50it/s] Loading weights: 52%|█████▏ | 3273/6243 [00:04<00:04, 711.07it/s] Loading weights: 54%|█████▎ | 3345/6243 [00:04<00:04, 711.59it/s] Loading weights: 55%|█████▍ | 3417/6243 [00:04<00:03, 713.66it/s] Loading weights: 56%|█████▌ | 3489/6243 [00:04<00:03, 712.78it/s] Loading weights: 57%|█████▋ | 3563/6243 [00:04<00:03, 719.22it/s] Loading weights: 58%|█████▊ | 3635/6243 [00:04<00:03, 716.97it/s] Loading weights: 59%|█████▉ | 3707/6243 [00:05<00:03, 717.05it/s] Loading weights: 61%|██████ | 3779/6243 [00:05<00:03, 713.56it/s] Loading weights: 62%|██████▏ | 3852/6243 [00:05<00:03, 716.82it/s] Loading weights: 63%|██████▎ | 3924/6243 [00:05<00:03, 708.25it/s] Loading weights: 64%|██████▍ | 3995/6243 [00:05<00:03, 692.24it/s] Loading weights: 65%|██████▌ | 4068/6243 [00:05<00:03, 701.20it/s] Loading weights: 66%|██████▋ | 4142/6243 [00:05<00:02, 712.44it/s] Loading weights: 68%|██████▊ | 4215/6243 [00:05<00:02, 717.40it/s] Loading weights: 69%|██████▊ | 4289/6243 [00:05<00:02, 722.91it/s] Loading weights: 70%|██████▉ | 4364/6243 [00:05<00:02, 728.73it/s] Loading weights: 71%|███████ | 4437/6243 [00:06<00:02, 722.98it/s] Loading weights: 72%|███████▏ | 4510/6243 [00:06<00:02, 722.82it/s] Loading weights: 73%|███████▎ | 4583/6243 [00:06<00:02, 721.13it/s] Loading weights: 75%|███████▍ | 4658/6243 [00:06<00:02, 728.36it/s] Loading weights: 76%|███████▌ | 4731/6243 [00:06<00:02, 728.35it/s] Loading weights: 77%|███████▋ | 4804/6243 [00:06<00:01, 720.69it/s] Loading weights: 78%|███████▊ | 4878/6243 [00:06<00:01, 724.05it/s] Loading weights: 79%|███████▉ | 4952/6243 [00:06<00:01, 728.46it/s] Loading weights: 80%|████████ | 5025/6243 [00:06<00:01, 723.83it/s] Loading weights: 82%|████████▏ | 5098/6243 [00:06<00:01, 721.53it/s] Loading weights: 83%|████████▎ | 5174/6243 [00:07<00:01, 731.40it/s] Loading weights: 84%|████████▍ | 5248/6243 [00:07<00:01, 723.09it/s] Loading weights: 85%|████████▌ | 5321/6243 [00:07<00:01, 723.88it/s] Loading weights: 86%|████████▋ | 5394/6243 [00:07<00:01, 723.20it/s] Loading weights: 88%|████████▊ | 5467/6243 [00:07<00:01, 722.94it/s] Loading weights: 89%|████████▊ | 5540/6243 [00:07<00:00, 718.23it/s] Loading weights: 90%|████████▉ | 5613/6243 [00:07<00:00, 720.98it/s] Loading weights: 91%|█████████ | 5686/6243 [00:07<00:00, 720.31it/s] Loading weights: 92%|█████████▏| 5759/6243 [00:07<00:00, 718.51it/s] Loading weights: 93%|█████████▎| 5831/6243 [00:08<00:00, 716.27it/s] Loading weights: 95%|█████████▍| 5903/6243 [00:08<00:00, 711.46it/s] Loading weights: 96%|█████████▌| 5975/6243 [00:08<00:00, 711.18it/s] Loading weights: 97%|█████████▋| 6049/6243 [00:08<00:00, 719.06it/s] Loading weights: 98%|█████████▊| 6122/6243 [00:08<00:00, 720.79it/s] Loading weights: 99%|█████████▉| 6195/6243 [00:08<00:00, 720.80it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 719.15it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
Traceback (most recent call last):
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 315, in <module>
 train(cfg)
 File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 254, in train
 out = model(input_ids=input_ids, attention_mask=attn_mask, use_cache=False)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1993, in forward
 return self.base_model(
 ^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 330, in forward
 return self.model.forward(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1702, in forward
 nemotron_h_outputs = self.backbone(
 ^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1489, in forward
 hidden_states = self._gradient_checkpointing_func(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner
 return disable_fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn
 return fn(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint
 ret = function(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 777, in forward
 hidden_states = self.mixer(
 ^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
 return self._call_impl(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
 return forward_call(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 715, in forward
 return self.cuda_kernels_forward(hidden_states, cache_params, cache_position, attention_mask)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 429, in cuda_kernels_forward
 out = mamba_split_conv1d_scan_combined(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 997, in mamba_split_conv1d_scan_combined
 return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
 return super().apply(*args, **kwargs) # type: ignore[misc]
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 503, in decorate_fwd
 return fwd(*args, **kwargs)
 ^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 841, in forward
 causal_conv1d_fwd_function(rearrange(ensure_stride(xBC), "b s d -> b d s"),
TypeError: 'NoneType' object is not callable
EXIT_CODE=1
=== RESUME5 Fri May 22 17:06:20 UTC 2026 ===
=== [Fri May 22 17:06:20 UTC 2026] patch Nemotron model (force slow mamba path) ===
=== [Fri May 22 17:06:20 UTC 2026] RESUME: verify mamba-ssm ===
mamba_ssm OK
=== [Fri May 22 17:06:21 UTC 2026] Phase 1: smoke test (20 steps) ===
[stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/smoke
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 20
 grad_accum: 1
 batch_size: 1
 seq_len: 1024
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 135/6243 [00:00<00:04, 1333.95it/s] Loading weights: 4%|▍ | 269/6243 [00:00<00:06, 870.58it/s] Loading weights: 6%|▌ | 366/6243 [00:00<00:07, 786.30it/s] Loading weights: 7%|▋ | 450/6243 [00:00<00:07, 751.80it/s] Loading weights: 8%|▊ | 528/6243 [00:00<00:07, 729.23it/s] Loading weights: 10%|▉ | 603/6243 [00:00<00:07, 722.99it/s] Loading weights: 11%|█ | 676/6243 [00:00<00:07, 707.10it/s] Loading weights: 12%|█▏ | 748/6243 [00:00<00:07, 696.82it/s] Loading weights: 13%|█▎ | 818/6243 [00:01<00:07, 686.39it/s] Loading weights: 14%|█▍ | 888/6243 [00:01<00:07, 688.93it/s] Loading weights: 15%|█▌ | 959/6243 [00:01<00:07, 694.05it/s] Loading weights: 17%|█▋ | 1031/6243 [00:01<00:07, 700.06it/s] Loading weights: 18%|█▊ | 1103/6243 [00:01<00:07, 703.03it/s] Loading weights: 19%|█▉ | 1174/6243 [00:01<00:07, 702.73it/s] Loading weights: 20%|█▉ | 1245/6243 [00:01<00:07, 700.35it/s] Loading weights: 21%|██ | 1316/6243 [00:01<00:07, 697.63it/s] Loading weights: 22%|██▏ | 1389/6243 [00:01<00:06, 705.24it/s] Loading weights: 23%|██▎ | 1461/6243 [00:02<00:06, 706.80it/s] Loading weights: 25%|██▍ | 1533/6243 [00:02<00:06, 708.92it/s] Loading weights: 26%|██▌ | 1606/6243 [00:02<00:06, 712.34it/s] Loading weights: 27%|██▋ | 1678/6243 [00:02<00:06, 713.92it/s] Loading weights: 28%|██▊ | 1750/6243 [00:02<00:06, 712.27it/s] Loading weights: 29%|██▉ | 1822/6243 [00:02<00:06, 705.22it/s] Loading weights: 30%|███ | 1893/6243 [00:02<00:06, 697.07it/s] Loading weights: 31%|███▏ | 1965/6243 [00:02<00:06, 703.65it/s] Loading weights: 33%|███▎ | 2036/6243 [00:02<00:05, 704.83it/s] Loading weights: 34%|███▎ | 2107/6243 [00:02<00:05, 701.66it/s] Loading weights: 35%|███▍ | 2182/6243 [00:03<00:05, 710.83it/s] Loading weights: 36%|███▌ | 2254/6243 [00:03<00:05, 712.11it/s] Loading weights: 37%|███▋ | 2326/6243 [00:03<00:05, 705.82it/s] Loading weights: 38%|███▊ | 2397/6243 [00:03<00:05, 702.14it/s] Loading weights: 40%|███▉ | 2468/6243 [00:03<00:05, 704.42it/s] Loading weights: 41%|████ | 2539/6243 [00:03<00:05, 698.88it/s] Loading weights: 42%|████▏ | 2609/6243 [00:03<00:05, 696.62it/s] Loading weights: 43%|████▎ | 2680/6243 [00:03<00:05, 698.47it/s] Loading weights: 44%|████▍ | 2752/6243 [00:03<00:04, 702.73it/s] Loading weights: 45%|████▌ | 2823/6243 [00:03<00:04, 699.74it/s] Loading weights: 46%|████▋ | 2893/6243 [00:04<00:04, 697.22it/s] Loading weights: 47%|████▋ | 2964/6243 [00:04<00:04, 698.40it/s] Loading weights: 49%|████▊ | 3037/6243 [00:04<00:04, 705.12it/s] Loading weights: 50%|████▉ | 3108/6243 [00:04<00:04, 703.25it/s] Loading weights: 51%|█████ | 3180/6243 [00:04<00:04, 705.65it/s] Loading weights: 52%|█████▏ | 3251/6243 [00:04<00:04, 704.70it/s] Loading weights: 53%|█████▎ | 3324/6243 [00:04<00:04, 711.18it/s] Loading weights: 54%|█████▍ | 3396/6243 [00:04<00:04, 711.53it/s] Loading weights: 56%|█████▌ | 3468/6243 [00:04<00:03, 710.35it/s] Loading weights: 57%|█████▋ | 3542/6243 [00:04<00:03, 710.60it/s] Loading weights: 58%|█████▊ | 3615/6243 [00:05<00:03, 715.57it/s] Loading weights: 59%|█████▉ | 3687/6243 [00:05<00:03, 714.62it/s] Loading weights: 60%|██████ | 3759/6243 [00:05<00:03, 707.15it/s] Loading weights: 61%|██████▏ | 3831/6243 [00:05<00:03, 710.04it/s] Loading weights: 63%|██████▎ | 3903/6243 [00:05<00:03, 706.59it/s] Loading weights: 64%|██████▎ | 3974/6243 [00:05<00:03, 688.64it/s] Loading weights: 65%|██████▍ | 4045/6243 [00:05<00:03, 693.09it/s] Loading weights: 66%|██████▌ | 4118/6243 [00:05<00:03, 703.72it/s] Loading weights: 67%|██████▋ | 4191/6243 [00:05<00:02, 708.83it/s] Loading weights: 68%|██████▊ | 4262/6243 [00:05<00:02, 708.73it/s] Loading weights: 69%|██████▉ | 4334/6243 [00:06<00:02, 710.83it/s] Loading weights: 71%|███████ | 4406/6243 [00:06<00:02, 712.07it/s] Loading weights: 72%|███████▏ | 4478/6243 [00:06<00:02, 710.12it/s] Loading weights: 73%|███████▎ | 4550/6243 [00:06<00:02, 710.07it/s] Loading weights: 74%|███████▍ | 4622/6243 [00:06<00:02, 706.80it/s] Loading weights: 75%|███████▌ | 4697/6243 [00:06<00:02, 718.48it/s] Loading weights: 76%|███████▋ | 4769/6243 [00:06<00:02, 713.87it/s] Loading weights: 78%|███████▊ | 4841/6243 [00:06<00:01, 705.62it/s] Loading weights: 79%|███████▊ | 4914/6243 [00:06<00:01, 711.77it/s] Loading weights: 80%|███████▉ | 4986/6243 [00:07<00:01, 697.93it/s] Loading weights: 81%|████████ | 5057/6243 [00:07<00:01, 699.80it/s] Loading weights: 82%|████████▏ | 5128/6243 [00:07<00:01, 700.05it/s] Loading weights: 83%|████████▎ | 5200/6243 [00:07<00:01, 704.28it/s] Loading weights: 84%|████████▍ | 5272/6243 [00:07<00:01, 706.22it/s] Loading weights: 86%|████████▌ | 5344/6243 [00:07<00:01, 708.13it/s] Loading weights: 87%|████████▋ | 5416/6243 [00:07<00:01, 711.08it/s] Loading weights: 88%|████████▊ | 5488/6243 [00:07<00:01, 713.28it/s] Loading weights: 89%|████████▉ | 5560/6243 [00:07<00:00, 707.29it/s] Loading weights: 90%|█████████ | 5631/6243 [00:07<00:00, 706.84it/s] Loading weights: 91%|█████████▏| 5706/6243 [00:08<00:00, 706.96it/s] Loading weights: 93%|█████████▎| 5782/6243 [00:08<00:00, 720.03it/s] Loading weights: 94%|█████████▍| 5855/6243 [00:08<00:00, 718.50it/s] Loading weights: 95%|█████████▍| 5927/6243 [00:08<00:00, 718.54it/s] Loading weights: 96%|█████████▌| 6000/6243 [00:08<00:00, 720.71it/s] Loading weights: 97%|█████████▋| 6073/6243 [00:08<00:00, 720.27it/s] Loading weights: 98%|█████████▊| 6146/6243 [00:08<00:00, 717.90it/s] Loading weights: 100%|█████████▉| 6218/6243 [00:08<00:00, 717.99it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 704.83it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
step 10/20 loss=0.4303 ema=0.6710 lr=2.00e-05 mem=69.1/78.7 GiB
step 20/20 loss=0.4214 ema=0.7938 lr=4.00e-05 mem=69.0/80.9 GiB
saved checkpoint -> /home/lakesenberg/nemotron_lora_work/smoke/step-00020
DONE. final adapter -> /home/lakesenberg/nemotron_lora_work/smoke/final
=== DONE ===
Final adapter: /home/lakesenberg/nemotron_lora_work/smoke/final
Next: upload that folder to Kaggle as a Model dataset,
then point CFG.tinker_adapter_path at it in the AA-SVD notebook.
=== [Fri May 22 17:09:11 UTC 2026] Phase 2: full training (2000 steps) ===
[stage 1] generating 4000 examples per task -> /home/lakesenberg/nemotron_lora_work/cot.jsonl
wrote 28000 examples to /home/lakesenberg/nemotron_lora_work/cot.jsonl
--- sample ---
### Task: numeral
### Problem:
Convert the Roman numeral LXXVI to an integer.
### Reasoning:
<|cot_start|>Reading symbols one by one:
 L = 50 -> 0 + 50 = 50
 X = 10 -> 50 + 10 = 60
 X = 10 -> 60 + 10 = 70
 V = 5 -> 70 + 5 = 75
 I = 1 -> 75 + 1 = 76
Final: \boxed{76}
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/adapter
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 2000
 grad_accum: 4
 batch_size: 1
 seq_len: 7680
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 151/6243 [00:00<00:04, 1488.36it/s] Loading weights: 5%|▍ | 300/6243 [00:00<00:06, 915.17it/s] Loading weights: 6%|▋ | 404/6243 [00:00<00:07, 830.90it/s] Loading weights: 8%|▊ | 494/6243 [00:00<00:07, 788.13it/s] Loading weights: 9%|▉ | 576/6243 [00:00<00:07, 760.63it/s] Loading weights: 10%|█ | 654/6243 [00:00<00:07, 744.72it/s] Loading weights: 12%|█▏ | 730/6243 [00:00<00:07, 735.13it/s] Loading weights: 13%|█▎ | 804/6243 [00:01<00:07, 727.16it/s] Loading weights: 14%|█▍ | 877/6243 [00:01<00:07, 726.57it/s] Loading weights: 15%|█▌ | 950/6243 [00:01<00:07, 718.28it/s] Loading weights: 16%|█▋ | 1022/6243 [00:01<00:07, 716.55it/s] Loading weights: 18%|█▊ | 1096/6243 [00:01<00:07, 721.02it/s] Loading weights: 19%|█▊ | 1169/6243 [00:01<00:07, 714.84it/s] Loading weights: 20%|█▉ | 1241/6243 [00:01<00:07, 710.87it/s] Loading weights: 21%|██ | 1313/6243 [00:01<00:06, 706.08it/s] Loading weights: 22%|██▏ | 1386/6243 [00:01<00:06, 710.25it/s] Loading weights: 23%|██▎ | 1458/6243 [00:01<00:06, 711.21it/s] Loading weights: 25%|██▍ | 1530/6243 [00:02<00:06, 702.94it/s] Loading weights: 26%|██▌ | 1601/6243 [00:02<00:06, 704.05it/s] Loading weights: 27%|██▋ | 1673/6243 [00:02<00:06, 708.19it/s] Loading weights: 28%|██▊ | 1744/6243 [00:02<00:06, 705.31it/s] Loading weights: 29%|██▉ | 1815/6243 [00:02<00:06, 704.99it/s] Loading weights: 30%|███ | 1886/6243 [00:02<00:06, 702.80it/s] Loading weights: 31%|███▏ | 1959/6243 [00:02<00:06, 710.26it/s] Loading weights: 33%|███▎ | 2031/6243 [00:02<00:05, 712.04it/s] Loading weights: 34%|███▎ | 2103/6243 [00:02<00:05, 705.27it/s] Loading weights: 35%|███▍ | 2176/6243 [00:02<00:05, 704.59it/s] Loading weights: 36%|███▌ | 2250/6243 [00:03<00:05, 714.21it/s] Loading weights: 37%|███▋ | 2322/6243 [00:03<00:05, 707.79it/s] Loading weights: 38%|███▊ | 2393/6243 [00:03<00:05, 704.65it/s] Loading weights: 40%|███▉ | 2466/6243 [00:03<00:05, 711.27it/s] Loading weights: 41%|████ | 2538/6243 [00:03<00:05, 710.04it/s] Loading weights: 42%|████▏ | 2610/6243 [00:03<00:05, 711.03it/s] Loading weights: 43%|████▎ | 2682/6243 [00:03<00:05, 710.42it/s] Loading weights: 44%|████▍ | 2754/6243 [00:03<00:04, 712.74it/s] Loading weights: 45%|████▌ | 2826/6243 [00:03<00:04, 713.87it/s] Loading weights: 46%|████▋ | 2898/6243 [00:03<00:04, 712.60it/s] Loading weights: 48%|████▊ | 2970/6243 [00:04<00:04, 713.90it/s] Loading weights: 49%|████▉ | 3044/6243 [00:04<00:04, 721.44it/s] Loading weights: 50%|████▉ | 3117/6243 [00:04<00:04, 713.96it/s] Loading weights: 51%|█████ | 3189/6243 [00:04<00:04, 708.97it/s] Loading weights: 52%|█████▏ | 3260/6243 [00:04<00:04, 704.73it/s] Loading weights: 53%|█████▎ | 3333/6243 [00:04<00:04, 711.30it/s] Loading weights: 55%|█████▍ | 3405/6243 [00:04<00:03, 709.88it/s] Loading weights: 56%|█████▌ | 3477/6243 [00:04<00:03, 706.62it/s] Loading weights: 57%|█████▋ | 3551/6243 [00:04<00:03, 714.54it/s] Loading weights: 58%|█████▊ | 3623/6243 [00:04<00:03, 712.78it/s] Loading weights: 59%|█████▉ | 3695/6243 [00:05<00:03, 713.25it/s] Loading weights: 60%|██████ | 3767/6243 [00:05<00:03, 708.10it/s] Loading weights: 61%|██████▏ | 3839/6243 [00:05<00:03, 709.26it/s] Loading weights: 63%|██████▎ | 3910/6243 [00:05<00:03, 705.50it/s] Loading weights: 64%|██████▍ | 3981/6243 [00:05<00:03, 700.09it/s] Loading weights: 65%|██████▍ | 4054/6243 [00:05<00:03, 706.24it/s] Loading weights: 66%|██████▌ | 4128/6243 [00:05<00:02, 714.97it/s] Loading weights: 67%|██████▋ | 4200/6243 [00:05<00:02, 716.46it/s] Loading weights: 68%|██████▊ | 4273/6243 [00:05<00:02, 718.96it/s] Loading weights: 70%|██████▉ | 4348/6243 [00:06<00:02, 727.84it/s] Loading weights: 71%|███████ | 4421/6243 [00:06<00:02, 719.98it/s] Loading weights: 72%|███████▏ | 4494/6243 [00:06<00:02, 722.79it/s] Loading weights: 73%|███████▎ | 4568/6243 [00:06<00:02, 725.10it/s] Loading weights: 74%|███████▍ | 4643/6243 [00:06<00:02, 731.15it/s] Loading weights: 76%|███████▌ | 4717/6243 [00:06<00:02, 732.70it/s] Loading weights: 77%|███████▋ | 4791/6243 [00:06<00:01, 731.70it/s] Loading weights: 78%|███████▊ | 4865/6243 [00:06<00:01, 730.29it/s] Loading weights: 79%|███████▉ | 4939/6243 [00:06<00:01, 730.72it/s] Loading weights: 80%|████████ | 5013/6243 [00:06<00:01, 725.45it/s] Loading weights: 81%|████████▏ | 5086/6243 [00:07<00:01, 723.66it/s] Loading weights: 83%|████████▎ | 5160/6243 [00:07<00:01, 723.56it/s] Loading weights: 84%|████████▍ | 5233/6243 [00:07<00:01, 723.27it/s] Loading weights: 85%|████████▍ | 5306/6243 [00:07<00:01, 719.86it/s] Loading weights: 86%|████████▌ | 5378/6243 [00:07<00:01, 717.66it/s] Loading weights: 87%|████████▋ | 5452/6243 [00:07<00:01, 722.17it/s] Loading weights: 88%|████████▊ | 5525/6243 [00:07<00:01, 716.64it/s] Loading weights: 90%|████████▉ | 5597/6243 [00:07<00:00, 716.69it/s] Loading weights: 91%|█████████ | 5669/6243 [00:07<00:00, 715.01it/s] Loading weights: 92%|█████████▏| 5742/6243 [00:07<00:00, 716.79it/s] Loading weights: 93%|█████████▎| 5814/6243 [00:08<00:00, 711.37it/s] Loading weights: 94%|█████████▍| 5886/6243 [00:08<00:00, 708.09it/s] Loading weights: 95%|█████████▌| 5957/6243 [00:08<00:00, 704.84it/s] Loading weights: 97%|█████████▋| 6030/6243 [00:08<00:00, 709.22it/s] Loading weights: 98%|█████████▊| 6101/6243 [00:08<00:00, 704.84it/s] Loading weights: 99%|█████████▉| 6172/6243 [00:08<00:00, 703.55it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 549.50it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 714.08it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
=== STOP 7680 / RESTART 4096 Fri May 22 19:00:50 UTC 2026 ===
=== [Fri May 22 19:00:51 UTC 2026] SEQ_LEN=4096 restart ===
=== [Fri May 22 19:00:51 UTC 2026] Phase 2: full training (2000 steps, seq=4096) ===
[stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/cot.jsonl
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/adapter
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 2000
 grad_accum: 4
 batch_size: 1
 seq_len: 4096
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 150/6243 [00:00<00:04, 1489.04it/s] Loading weights: 5%|▍ | 299/6243 [00:00<00:06, 898.70it/s] Loading weights: 6%|▋ | 402/6243 [00:00<00:07, 819.89it/s] Loading weights: 8%|▊ | 491/6243 [00:00<00:07, 782.99it/s] Loading weights: 9%|▉ | 573/6243 [00:00<00:07, 763.22it/s] Loading weights: 10%|█ | 652/6243 [00:00<00:07, 743.82it/s] Loading weights: 12%|█▏ | 728/6243 [00:00<00:07, 726.32it/s] Loading weights: 13%|█▎ | 802/6243 [00:01<00:07, 716.46it/s] Loading weights: 14%|█▍ | 874/6243 [00:01<00:07, 716.70it/s] Loading weights: 15%|█▌ | 946/6243 [00:01<00:07, 710.50it/s] Loading weights: 16%|█▋ | 1018/6243 [00:01<00:07, 709.37it/s] Loading weights: 17%|█▋ | 1092/6243 [00:01<00:07, 711.73it/s] Loading weights: 19%|█▊ | 1164/6243 [00:01<00:07, 701.92it/s] Loading weights: 20%|█▉ | 1235/6243 [00:01<00:07, 700.78it/s] Loading weights: 21%|██ | 1306/6243 [00:01<00:07, 700.93it/s] Loading weights: 22%|██▏ | 1377/6243 [00:01<00:06, 700.80it/s] Loading weights: 23%|██▎ | 1451/6243 [00:01<00:06, 711.59it/s] Loading weights: 24%|██▍ | 1523/6243 [00:02<00:06, 712.89it/s] Loading weights: 26%|██▌ | 1595/6243 [00:02<00:06, 714.19it/s] Loading weights: 27%|██▋ | 1668/6243 [00:02<00:06, 717.40it/s] Loading weights: 28%|██▊ | 1740/6243 [00:02<00:06, 708.44it/s] Loading weights: 29%|██▉ | 1812/6243 [00:02<00:06, 710.10it/s] Loading weights: 30%|███ | 1884/6243 [00:02<00:06, 704.69it/s] Loading weights: 31%|███▏ | 1956/6243 [00:02<00:06, 706.93it/s] Loading weights: 32%|███▏ | 2028/6243 [00:02<00:05, 709.44it/s] Loading weights: 34%|███▎ | 2099/6243 [00:02<00:05, 708.85it/s] Loading weights: 35%|███▍ | 2170/6243 [00:02<00:05, 703.75it/s] Loading weights: 36%|███▌ | 2241/6243 [00:03<00:05, 704.29it/s] Loading weights: 37%|███▋ | 2313/6243 [00:03<00:05, 706.81it/s] Loading weights: 38%|███▊ | 2384/6243 [00:03<00:05, 705.17it/s] Loading weights: 39%|███▉ | 2456/6243 [00:03<00:05, 707.90it/s] Loading weights: 40%|████ | 2527/6243 [00:03<00:05, 706.78it/s] Loading weights: 42%|████▏ | 2598/6243 [00:03<00:05, 707.16it/s] Loading weights: 43%|████▎ | 2669/6243 [00:03<00:05, 706.53it/s] Loading weights: 44%|████▍ | 2740/6243 [00:03<00:04, 707.49it/s] Loading weights: 45%|████▌ | 2811/6243 [00:03<00:04, 702.75it/s] Loading weights: 46%|████▌ | 2882/6243 [00:03<00:04, 701.79it/s] Loading weights: 47%|████▋ | 2953/6243 [00:04<00:04, 703.32it/s] Loading weights: 48%|████▊ | 3027/6243 [00:04<00:04, 712.96it/s] Loading weights: 50%|████▉ | 3099/6243 [00:04<00:04, 711.50it/s] Loading weights: 51%|█████ | 3171/6243 [00:04<00:04, 706.39it/s] Loading weights: 52%|█████▏ | 3242/6243 [00:04<00:04, 706.12it/s] Loading weights: 53%|█████▎ | 3316/6243 [00:04<00:04, 714.61it/s] Loading weights: 54%|█████▍ | 3389/6243 [00:04<00:03, 717.23it/s] Loading weights: 55%|█████▌ | 3461/6243 [00:04<00:03, 716.15it/s] Loading weights: 57%|█████▋ | 3534/6243 [00:04<00:03, 711.36it/s] Loading weights: 58%|█████▊ | 3610/6243 [00:04<00:03, 724.95it/s] Loading weights: 59%|█████▉ | 3683/6243 [00:05<00:03, 723.48it/s] Loading weights: 60%|██████ | 3756/6243 [00:05<00:03, 720.96it/s] Loading weights: 61%|██████▏ | 3829/6243 [00:05<00:03, 721.70it/s] Loading weights: 63%|██████▎ | 3902/6243 [00:05<00:03, 720.39it/s] Loading weights: 64%|██████▎ | 3975/6243 [00:05<00:03, 711.84it/s] Loading weights: 65%|██████▍ | 4048/6243 [00:05<00:03, 716.41it/s] Loading weights: 66%|██████▌ | 4123/6243 [00:05<00:02, 723.84it/s] Loading weights: 67%|██████▋ | 4196/6243 [00:05<00:02, 721.04it/s] Loading weights: 68%|██████▊ | 4269/6243 [00:05<00:02, 719.17it/s] Loading weights: 70%|██████▉ | 4341/6243 [00:06<00:02, 709.27it/s] Loading weights: 71%|███████ | 4414/6243 [00:06<00:02, 712.65it/s] Loading weights: 72%|███████▏ | 4487/6243 [00:06<00:02, 716.36it/s] Loading weights: 73%|███████▎ | 4560/6243 [00:06<00:02, 718.82it/s] Loading weights: 74%|███████▍ | 4635/6243 [00:06<00:02, 726.60it/s] Loading weights: 75%|███████▌ | 4708/6243 [00:06<00:02, 721.62it/s] Loading weights: 77%|███████▋ | 4781/6243 [00:06<00:02, 721.58it/s] Loading weights: 78%|███████▊ | 4854/6243 [00:06<00:01, 719.91it/s] Loading weights: 79%|███████▉ | 4929/6243 [00:06<00:01, 726.03it/s] Loading weights: 80%|████████ | 5002/6243 [00:06<00:01, 722.60it/s] Loading weights: 81%|████████▏ | 5075/6243 [00:07<00:01, 724.61it/s] Loading weights: 82%|████████▏ | 5148/6243 [00:07<00:01, 723.64it/s] Loading weights: 84%|████████▎ | 5221/6243 [00:07<00:01, 721.29it/s] Loading weights: 85%|████████▍ | 5294/6243 [00:07<00:01, 719.45it/s] Loading weights: 86%|████████▌ | 5366/6243 [00:07<00:01, 704.68it/s] Loading weights: 87%|████████▋ | 5437/6243 [00:07<00:01, 697.71it/s] Loading weights: 88%|████████▊ | 5510/6243 [00:07<00:01, 705.82it/s] Loading weights: 89%|████████▉ | 5581/6243 [00:07<00:00, 704.24it/s] Loading weights: 91%|█████████ | 5652/6243 [00:07<00:00, 705.38it/s] Loading weights: 92%|█████████▏| 5725/6243 [00:07<00:00, 711.93it/s] Loading weights: 93%|█████████▎| 5797/6243 [00:08<00:00, 704.23it/s] Loading weights: 94%|█████████▍| 5868/6243 [00:08<00:00, 700.02it/s] Loading weights: 95%|█████████▌| 5939/6243 [00:08<00:00, 694.33it/s] Loading weights: 96%|█████████▋| 6011/6243 [00:08<00:00, 700.65it/s] Loading weights: 97%|█████████▋| 6082/6243 [00:08<00:00, 700.02it/s] Loading weights: 99%|█████████▊| 6153/6243 [00:08<00:00, 699.75it/s] Loading weights: 100%|█████████▉| 6224/6243 [00:08<00:00, 700.52it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 711.24it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
=== STOP 4096 / RESTART 2048 Fri May 22 19:44:26 UTC 2026 ===
=== [Fri May 22 19:44:26 UTC 2026] SEQ_LEN=2048 restart ===
=== [Fri May 22 19:44:26 UTC 2026] Phase 2: full training (2000 steps, seq=2048) ===
[stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/cot.jsonl
[stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
=== TrainConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 data_path: /home/lakesenberg/nemotron_lora_work/cot.jsonl
 out_dir: /home/lakesenberg/nemotron_lora_work/adapter
 lora_rank: 32
 lora_alpha: 32
 lora_dropout: 0.0
 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2')
 lr: 0.0001
 weight_decay: 0.0
 betas: (0.9, 0.95)
 max_grad_norm: 1.0
 warmup_steps: 50
 total_steps: 2000
 grad_accum: 4
 batch_size: 1
 seq_len: 2048
 save_every: 500
 log_every: 10
 seed: 0
 bf16: True
 use_flash_attn: False
 gradient_checkpointing: True
 resume_adapter: None
Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ...
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 2%|▏ | 142/6243 [00:00<00:04, 1409.92it/s] Loading weights: 5%|▍ | 283/6243 [00:00<00:06, 894.94it/s] Loading weights: 6%|▌ | 383/6243 [00:00<00:07, 820.20it/s] Loading weights: 8%|▊ | 471/6243 [00:00<00:07, 771.14it/s] Loading weights: 9%|▉ | 551/6243 [00:00<00:07, 754.74it/s] Loading weights: 10%|█ | 628/6243 [00:00<00:07, 747.75it/s] Loading weights: 11%|█▏ | 704/6243 [00:00<00:07, 738.57it/s] Loading weights: 12%|█▏ | 779/6243 [00:00<00:07, 731.94it/s] Loading weights: 14%|█▎ | 853/6243 [00:01<00:07, 731.54it/s] Loading weights: 15%|█▍ | 927/6243 [00:01<00:07, 731.86it/s] Loading weights: 16%|█▌ | 1001/6243 [00:01<00:07, 724.54it/s] Loading weights: 17%|█▋ | 1074/6243 [00:01<00:07, 724.27it/s] Loading weights: 18%|█▊ | 1147/6243 [00:01<00:07, 725.15it/s] Loading weights: 20%|█▉ | 1220/6243 [00:01<00:06, 719.99it/s] Loading weights: 21%|██ | 1293/6243 [00:01<00:06, 716.92it/s] Loading weights: 22%|██▏ | 1368/6243 [00:01<00:06, 722.25it/s] Loading weights: 23%|██▎ | 1441/6243 [00:01<00:06, 722.70it/s] Loading weights: 24%|██▍ | 1514/6243 [00:02<00:06, 716.55it/s] Loading weights: 25%|██▌ | 1586/6243 [00:02<00:06, 715.57it/s] Loading weights: 27%|██▋ | 1658/6243 [00:02<00:06, 716.43it/s] Loading weights: 28%|██▊ | 1730/6243 [00:02<00:06, 713.14it/s] Loading weights: 29%|██▉ | 1802/6243 [00:02<00:06, 711.93it/s] Loading weights: 30%|███ | 1874/6243 [00:02<00:06, 708.10it/s] Loading weights: 31%|███ | 1946/6243 [00:02<00:06, 710.74it/s] Loading weights: 32%|███▏ | 2018/6243 [00:02<00:05, 710.57it/s] Loading weights: 33%|███▎ | 2090/6243 [00:02<00:05, 705.27it/s] Loading weights: 35%|███▍ | 2162/6243 [00:02<00:05, 706.75it/s] Loading weights: 36%|███▌ | 2236/6243 [00:03<00:05, 715.43it/s] Loading weights: 37%|███▋ | 2309/6243 [00:03<00:05, 716.75it/s] Loading weights: 38%|███▊ | 2381/6243 [00:03<00:05, 711.68it/s] Loading weights: 39%|███▉ | 2454/6243 [00:03<00:05, 711.96it/s] Loading weights: 40%|████ | 2526/6243 [00:03<00:05, 712.95it/s] Loading weights: 42%|████▏ | 2598/6243 [00:03<00:05, 713.58it/s] Loading weights: 43%|████▎ | 2670/6243 [00:03<00:04, 715.33it/s] Loading weights: 44%|████▍ | 2742/6243 [00:03<00:04, 716.33it/s] Loading weights: 45%|████▌ | 2814/6243 [00:03<00:04, 708.85it/s] Loading weights: 46%|████▌ | 2885/6243 [00:03<00:04, 703.37it/s] Loading weights: 47%|████▋ | 2956/6243 [00:04<00:04, 698.83it/s] Loading weights: 49%|████▊ | 3028/6243 [00:04<00:04, 703.56it/s] Loading weights: 50%|████▉ | 3099/6243 [00:04<00:04, 697.48it/s] Loading weights: 51%|█████ | 3169/6243 [00:04<00:04, 693.32it/s] Loading weights: 52%|█████▏ | 3241/6243 [00:04<00:04, 699.84it/s] Loading weights: 53%|█████▎ | 3314/6243 [00:04<00:04, 705.90it/s] Loading weights: 54%|█████▍ | 3386/6243 [00:04<00:04, 709.22it/s] Loading weights: 55%|█████▌ | 3457/6243 [00:04<00:03, 707.19it/s] Loading weights: 57%|█████▋ | 3530/6243 [00:04<00:03, 708.23it/s] Loading weights: 58%|█████▊ | 3603/6243 [00:04<00:03, 713.99it/s] Loading weights: 59%|█████▉ | 3675/6243 [00:05<00:03, 711.44it/s] Loading weights: 60%|██████ | 3747/6243 [00:05<00:03, 704.68it/s] Loading weights: 61%|██████ | 3820/6243 [00:05<00:03, 709.52it/s] Loading weights: 62%|██████▏ | 3891/6243 [00:05<00:03, 707.53it/s] Loading weights: 63%|██████▎ | 3962/6243 [00:05<00:03, 697.40it/s] Loading weights: 65%|██████▍ | 4032/6243 [00:05<00:03, 695.04it/s] Loading weights: 66%|██████▌ | 4106/6243 [00:05<00:03, 706.90it/s] Loading weights: 67%|██████▋ | 4179/6243 [00:05<00:02, 710.70it/s] Loading weights: 68%|██████▊ | 4251/6243 [00:05<00:02, 712.38it/s] Loading weights: 69%|██████▉ | 4324/6243 [00:05<00:02, 716.78it/s] Loading weights: 70%|███████ | 4397/6243 [00:06<00:02, 720.66it/s] Loading weights: 72%|███████▏ | 4470/6243 [00:06<00:02, 720.10it/s] Loading weights: 73%|███████▎ | 4543/6243 [00:06<00:02, 720.30it/s] Loading weights: 74%|███████▍ | 4616/6243 [00:06<00:02, 721.88it/s] Loading weights: 75%|███████▌ | 4690/6243 [00:06<00:02, 726.68it/s] Loading weights: 76%|███████▋ | 4763/6243 [00:06<00:02, 726.28it/s] Loading weights: 77%|███████▋ | 4836/6243 [00:06<00:01, 719.20it/s] Loading weights: 79%|███████▊ | 4910/6243 [00:06<00:01, 722.75it/s] Loading weights: 80%|███████▉ | 4983/6243 [00:06<00:01, 718.49it/s] Loading weights: 81%|████████ | 5056/6243 [00:06<00:01, 719.72it/s] Loading weights: 82%|████████▏ | 5128/6243 [00:07<00:01, 717.53it/s] Loading weights: 83%|████████▎ | 5201/6243 [00:07<00:01, 718.37it/s] Loading weights: 84%|████████▍ | 5273/6243 [00:07<00:01, 716.85it/s] Loading weights: 86%|████████▌ | 5346/6243 [00:07<00:01, 718.44it/s] Loading weights: 87%|████████▋ | 5418/6243 [00:07<00:01, 716.81it/s] Loading weights: 88%|████████▊ | 5491/6243 [00:07<00:01, 718.47it/s] Loading weights: 89%|████████▉ | 5563/6243 [00:07<00:00, 717.67it/s] Loading weights: 90%|█████████ | 5635/6243 [00:07<00:00, 717.03it/s] Loading weights: 91%|█████████▏| 5707/6243 [00:07<00:00, 717.20it/s] Loading weights: 93%|█████████▎| 5781/6243 [00:08<00:00, 721.96it/s] Loading weights: 94%|█████████▍| 5854/6243 [00:08<00:00, 718.29it/s] Loading weights: 95%|█████████▍| 5926/6243 [00:08<00:00, 716.06it/s] Loading weights: 96%|█████████▌| 5999/6243 [00:08<00:00, 717.23it/s] Loading weights: 97%|█████████▋| 6071/6243 [00:08<00:00, 711.79it/s] Loading weights: 98%|█████████▊| 6143/6243 [00:08<00:00, 708.21it/s] Loading weights: 100%|█████████▉| 6215/6243 [00:08<00:00, 709.37it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 713.49it/s]
trainable params: 883,873,792 || all params: 32,461,811,136 || trainable%: 2.7228
[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
step 10/2000 loss=0.4620 ema=0.5800 lr=2.00e-05 mem=69.0/86.8 GiB
step 20/2000 loss=0.6800 ema=0.6877 lr=4.00e-05 mem=68.9/86.8 GiB
step 30/2000 loss=0.6103 ema=0.5642 lr=6.00e-05 mem=68.8/86.8 GiB
step 40/2000 loss=0.1662 ema=0.4054 lr=8.00e-05 mem=68.9/86.8 GiB
step 50/2000 loss=0.0279 ema=0.2234 lr=1.00e-04 mem=68.8/86.8 GiB
step 60/2000 loss=0.0005 ema=0.1067 lr=1.00e-04 mem=68.9/86.8 GiB
step 70/2000 loss=0.0146 ema=0.0605 lr=1.00e-04 mem=69.0/86.8 GiB
step 80/2000 loss=0.0240 ema=0.0324 lr=9.99e-05 mem=68.9/86.8 GiB
step 90/2000 loss=0.0007 ema=0.0192 lr=9.99e-05 mem=68.8/86.8 GiB
step 100/2000 loss=0.0049 ema=0.0158 lr=9.98e-05 mem=68.9/86.8 GiB
step 110/2000 loss=0.0445 ema=0.0155 lr=9.98e-05 mem=69.1/86.8 GiB
step 120/2000 loss=0.0108 ema=0.0163 lr=9.97e-05 mem=69.1/86.8 GiB
step 130/2000 loss=0.0096 ema=0.0134 lr=9.96e-05 mem=68.8/86.8 GiB
step 140/2000 loss=0.0099 ema=0.0130 lr=9.95e-05 mem=69.0/86.8 GiB
step 150/2000 loss=0.0003 ema=0.0135 lr=9.94e-05 mem=68.9/86.8 GiB
step 160/2000 loss=0.0023 ema=0.0084 lr=9.92e-05 mem=68.9/86.8 GiB
step 170/2000 loss=0.0071 ema=0.0081 lr=9.91e-05 mem=68.8/86.8 GiB
step 180/2000 loss=0.0004 ema=0.0071 lr=9.89e-05 mem=68.8/86.8 GiB
step 190/2000 loss=0.0080 ema=0.0055 lr=9.87e-05 mem=69.1/86.8 GiB
step 200/2000 loss=0.0001 ema=0.0046 lr=9.85e-05 mem=68.8/86.8 GiB
step 210/2000 loss=0.0128 ema=0.0086 lr=9.83e-05 mem=69.1/86.8 GiB
step 220/2000 loss=0.0001 ema=0.0086 lr=9.81e-05 mem=68.8/86.8 GiB
step 230/2000 loss=0.0049 ema=0.0056 lr=9.79e-05 mem=68.8/86.8 GiB
step 240/2000 loss=0.0008 ema=0.0105 lr=9.77e-05 mem=68.9/86.8 GiB
step 250/2000 loss=0.0066 ema=0.0074 lr=9.74e-05 mem=69.0/86.8 GiB
step 260/2000 loss=0.0011 ema=0.0055 lr=9.72e-05 mem=68.8/86.8 GiB
step 270/2000 loss=0.0009 ema=0.0055 lr=9.69e-05 mem=69.0/86.8 GiB
step 280/2000 loss=0.0005 ema=0.0046 lr=9.66e-05 mem=69.0/86.8 GiB
step 290/2000 loss=0.0018 ema=0.0043 lr=9.63e-05 mem=69.1/86.8 GiB
step 300/2000 loss=0.0001 ema=0.0036 lr=9.60e-05 mem=68.8/86.8 GiB
step 310/2000 loss=0.0023 ema=0.0054 lr=9.57e-05 mem=69.5/86.8 GiB
step 320/2000 loss=0.0036 ema=0.0047 lr=9.53e-05 mem=68.8/86.8 GiB
step 330/2000 loss=0.0046 ema=0.0035 lr=9.50e-05 mem=69.0/86.8 GiB
step 340/2000 loss=0.0000 ema=0.0034 lr=9.46e-05 mem=68.8/86.8 GiB
step 350/2000 loss=0.0043 ema=0.0049 lr=9.43e-05 mem=69.0/86.8 GiB
step 360/2000 loss=0.0071 ema=0.0078 lr=9.39e-05 mem=68.8/86.8 GiB
step 370/2000 loss=0.0000 ema=0.0046 lr=9.35e-05 mem=68.9/86.8 GiB
step 380/2000 loss=0.0024 ema=0.0034 lr=9.31e-05 mem=68.8/86.8 GiB
step 390/2000 loss=0.0004 ema=0.0027 lr=9.27e-05 mem=69.1/86.8 GiB
step 400/2000 loss=0.0024 ema=0.0020 lr=9.23e-05 mem=69.0/86.8 GiB
step 410/2000 loss=0.0012 ema=0.0020 lr=9.18e-05 mem=69.6/86.8 GiB
step 420/2000 loss=0.0029 ema=0.0027 lr=9.14e-05 mem=68.8/86.8 GiB
step 430/2000 loss=0.0002 ema=0.0030 lr=9.09e-05 mem=69.0/86.8 GiB
step 440/2000 loss=0.0000 ema=0.0020 lr=9.05e-05 mem=68.9/86.8 GiB
step 450/2000 loss=0.0000 ema=0.0041 lr=9.00e-05 mem=68.9/86.8 GiB
step 460/2000 loss=0.0039 ema=0.0045 lr=8.95e-05 mem=69.0/86.8 GiB
step 470/2000 loss=0.0001 ema=0.0053 lr=8.90e-05 mem=68.9/86.8 GiB
step 480/2000 loss=0.0004 ema=0.0041 lr=8.85e-05 mem=68.9/86.8 GiB
step 490/2000 loss=0.0005 ema=0.0134 lr=8.80e-05 mem=68.8/86.8 GiB
step 500/2000 loss=0.0038 ema=0.0072 lr=8.74e-05 mem=69.1/86.8 GiB
saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-00500
step 510/2000 loss=0.0114 ema=0.0061 lr=8.69e-05 mem=69.0/86.8 GiB
step 520/2000 loss=0.0041 ema=0.0037 lr=8.63e-05 mem=69.1/86.8 GiB
step 530/2000 loss=0.0031 ema=0.0021 lr=8.58e-05 mem=69.5/86.8 GiB
step 540/2000 loss=0.0002 ema=0.0017 lr=8.52e-05 mem=69.1/86.8 GiB
step 550/2000 loss=0.0071 ema=0.0023 lr=8.46e-05 mem=69.5/86.8 GiB
step 560/2000 loss=0.0042 ema=0.0025 lr=8.41e-05 mem=69.1/86.8 GiB
step 570/2000 loss=0.0047 ema=0.0021 lr=8.35e-05 mem=69.1/86.8 GiB
step 580/2000 loss=0.0015 ema=0.0019 lr=8.29e-05 mem=69.5/86.8 GiB
step 590/2000 loss=0.0033 ema=0.0018 lr=8.22e-05 mem=69.1/86.8 GiB
step 600/2000 loss=0.0103 ema=0.0033 lr=8.16e-05 mem=69.1/86.8 GiB
step 610/2000 loss=0.0000 ema=0.0021 lr=8.10e-05 mem=68.9/86.8 GiB
step 620/2000 loss=0.0027 ema=0.0019 lr=8.04e-05 mem=69.0/86.8 GiB
step 630/2000 loss=0.0000 ema=0.0032 lr=7.97e-05 mem=68.8/86.8 GiB
step 640/2000 loss=0.0004 ema=0.0027 lr=7.91e-05 mem=68.8/86.8 GiB
step 650/2000 loss=0.0001 ema=0.0017 lr=7.84e-05 mem=69.1/86.8 GiB
step 660/2000 loss=0.0000 ema=0.0079 lr=7.77e-05 mem=68.8/86.8 GiB
step 670/2000 loss=0.0019 ema=0.0051 lr=7.71e-05 mem=68.8/86.8 GiB
step 680/2000 loss=0.0001 ema=0.0028 lr=7.64e-05 mem=68.8/86.8 GiB
step 690/2000 loss=0.0000 ema=0.0015 lr=7.57e-05 mem=69.0/86.8 GiB
step 700/2000 loss=0.0000 ema=0.0016 lr=7.50e-05 mem=69.1/86.8 GiB
step 710/2000 loss=0.0028 ema=0.0017 lr=7.43e-05 mem=68.8/86.8 GiB
step 720/2000 loss=0.0036 ema=0.0015 lr=7.36e-05 mem=68.8/86.8 GiB
step 730/2000 loss=0.0015 ema=0.0015 lr=7.29e-05 mem=68.9/86.8 GiB
step 740/2000 loss=0.0055 ema=0.0020 lr=7.22e-05 mem=68.9/86.8 GiB
step 750/2000 loss=0.0002 ema=0.0034 lr=7.14e-05 mem=68.8/86.8 GiB
step 760/2000 loss=0.0033 ema=0.0022 lr=7.07e-05 mem=68.9/86.8 GiB
step 770/2000 loss=0.0000 ema=0.0020 lr=7.00e-05 mem=68.9/86.8 GiB
step 780/2000 loss=0.0000 ema=0.0013 lr=6.92e-05 mem=68.8/86.8 GiB
step 790/2000 loss=0.0019 ema=0.0012 lr=6.85e-05 mem=69.5/86.8 GiB
step 800/2000 loss=0.0030 ema=0.0019 lr=6.77e-05 mem=68.9/86.8 GiB
step 810/2000 loss=0.0000 ema=0.0017 lr=6.70e-05 mem=68.8/86.8 GiB
step 820/2000 loss=0.0000 ema=0.0012 lr=6.62e-05 mem=68.8/86.8 GiB
step 830/2000 loss=0.0000 ema=0.0014 lr=6.55e-05 mem=69.1/86.8 GiB
step 840/2000 loss=0.0015 ema=0.0021 lr=6.47e-05 mem=68.9/86.8 GiB
step 850/2000 loss=0.0000 ema=0.0015 lr=6.39e-05 mem=68.8/86.8 GiB
step 860/2000 loss=0.0000 ema=0.0011 lr=6.31e-05 mem=68.9/86.8 GiB
step 870/2000 loss=0.0046 ema=0.0017 lr=6.24e-05 mem=69.0/86.8 GiB
step 880/2000 loss=0.0027 ema=0.0018 lr=6.16e-05 mem=68.8/86.8 GiB
step 890/2000 loss=0.0048 ema=0.0012 lr=6.08e-05 mem=69.1/86.8 GiB
step 900/2000 loss=0.0018 ema=0.0012 lr=6.00e-05 mem=68.8/86.8 GiB
step 910/2000 loss=0.0010 ema=0.0031 lr=5.92e-05 mem=69.6/86.8 GiB
step 920/2000 loss=0.0019 ema=0.0023 lr=5.84e-05 mem=69.5/86.8 GiB
step 930/2000 loss=0.0026 ema=0.0034 lr=5.76e-05 mem=69.1/86.8 GiB
step 940/2000 loss=0.0023 ema=0.0020 lr=5.68e-05 mem=69.0/86.8 GiB
step 950/2000 loss=0.0018 ema=0.0016 lr=5.60e-05 mem=69.1/86.8 GiB
step 960/2000 loss=0.0072 ema=0.0020 lr=5.52e-05 mem=69.5/86.8 GiB
step 970/2000 loss=0.0001 ema=0.0010 lr=5.44e-05 mem=68.9/86.8 GiB
step 980/2000 loss=0.0000 ema=0.0016 lr=5.36e-05 mem=68.8/86.8 GiB
step 990/2000 loss=0.0003 ema=0.0013 lr=5.28e-05 mem=69.0/86.8 GiB
step 1000/2000 loss=0.0000 ema=0.0007 lr=5.20e-05 mem=69.0/86.8 GiB
saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-01000
step 1010/2000 loss=0.0001 ema=0.0012 lr=5.12e-05 mem=68.9/86.8 GiB
step 1020/2000 loss=0.0098 ema=0.0023 lr=5.04e-05 mem=69.0/86.8 GiB
step 1030/2000 loss=0.0000 ema=0.0015 lr=4.96e-05 mem=68.9/86.8 GiB
step 1040/2000 loss=0.0000 ema=0.0011 lr=4.88e-05 mem=68.8/86.8 GiB
step 1050/2000 loss=0.0013 ema=0.0007 lr=4.80e-05 mem=69.5/86.8 GiB
step 1060/2000 loss=0.0000 ema=0.0005 lr=4.72e-05 mem=68.9/86.8 GiB
step 1070/2000 loss=0.0000 ema=0.0009 lr=4.64e-05 mem=68.8/86.8 GiB
step 1080/2000 loss=0.0000 ema=0.0012 lr=4.56e-05 mem=68.8/86.8 GiB
step 1090/2000 loss=0.0022 ema=0.0014 lr=4.48e-05 mem=68.8/86.8 GiB
step 1100/2000 loss=0.0022 ema=0.0008 lr=4.40e-05 mem=69.0/86.8 GiB
step 1110/2000 loss=0.0035 ema=0.0014 lr=4.32e-05 mem=69.5/86.8 GiB
step 1120/2000 loss=0.0000 ema=0.0009 lr=4.24e-05 mem=68.9/86.8 GiB
step 1130/2000 loss=0.0000 ema=0.0012 lr=4.16e-05 mem=69.1/86.8 GiB
step 1140/2000 loss=0.0026 ema=0.0010 lr=4.08e-05 mem=69.1/87.9 GiB
step 1150/2000 loss=0.0000 ema=0.0009 lr=4.00e-05 mem=68.9/87.9 GiB
step 1160/2000 loss=0.0019 ema=0.0010 lr=3.92e-05 mem=69.5/87.9 GiB
step 1170/2000 loss=0.0000 ema=0.0013 lr=3.84e-05 mem=68.8/87.9 GiB
step 1180/2000 loss=0.0000 ema=0.0014 lr=3.76e-05 mem=68.9/87.9 GiB
step 1190/2000 loss=0.0032 ema=0.0019 lr=3.69e-05 mem=69.1/87.9 GiB
step 1200/2000 loss=0.0000 ema=0.0013 lr=3.61e-05 mem=68.9/87.9 GiB
step 1210/2000 loss=0.0000 ema=0.0007 lr=3.53e-05 mem=68.8/87.9 GiB
step 1220/2000 loss=0.0000 ema=0.0006 lr=3.45e-05 mem=68.9/87.9 GiB
step 1230/2000 loss=0.0000 ema=0.0004 lr=3.38e-05 mem=68.9/87.9 GiB
step 1240/2000 loss=0.0000 ema=0.0006 lr=3.30e-05 mem=68.9/87.9 GiB
step 1250/2000 loss=0.0002 ema=0.0008 lr=3.23e-05 mem=69.1/87.9 GiB
step 1260/2000 loss=0.0000 ema=0.0005 lr=3.15e-05 mem=68.8/87.9 GiB
step 1270/2000 loss=0.0000 ema=0.0041 lr=3.08e-05 mem=68.8/87.9 GiB
step 1280/2000 loss=0.0000 ema=0.0017 lr=3.00e-05 mem=69.1/87.9 GiB
step 1290/2000 loss=0.0003 ema=0.0013 lr=2.93e-05 mem=69.0/87.9 GiB
step 1300/2000 loss=0.0045 ema=0.0015 lr=2.86e-05 mem=69.1/87.9 GiB
step 1310/2000 loss=0.0000 ema=0.0013 lr=2.78e-05 mem=68.8/87.9 GiB
step 1320/2000 loss=0.0000 ema=0.0012 lr=2.71e-05 mem=69.1/87.9 GiB
step 1330/2000 loss=0.0005 ema=0.0009 lr=2.64e-05 mem=69.1/87.9 GiB
step 1340/2000 loss=0.0000 ema=0.0005 lr=2.57e-05 mem=68.9/87.9 GiB
step 1350/2000 loss=0.0011 ema=0.0005 lr=2.50e-05 mem=68.8/87.9 GiB
step 1360/2000 loss=0.0008 ema=0.0008 lr=2.43e-05 mem=69.1/87.9 GiB
step 1370/2000 loss=0.0015 ema=0.0007 lr=2.36e-05 mem=69.5/87.9 GiB
step 1380/2000 loss=0.0018 ema=0.0010 lr=2.29e-05 mem=69.5/87.9 GiB
step 1390/2000 loss=0.0009 ema=0.0005 lr=2.23e-05 mem=68.9/87.9 GiB
step 1400/2000 loss=0.0000 ema=0.0007 lr=2.16e-05 mem=68.9/87.9 GiB
step 1410/2000 loss=0.0000 ema=0.0009 lr=2.09e-05 mem=68.9/87.9 GiB
step 1420/2000 loss=0.0016 ema=0.0011 lr=2.03e-05 mem=69.5/87.9 GiB
step 1430/2000 loss=0.0000 ema=0.0007 lr=1.96e-05 mem=68.9/87.9 GiB
step 1440/2000 loss=0.0007 ema=0.0008 lr=1.90e-05 mem=69.5/87.9 GiB
step 1450/2000 loss=0.0026 ema=0.0009 lr=1.84e-05 mem=69.5/87.9 GiB
step 1460/2000 loss=0.0001 ema=0.0010 lr=1.78e-05 mem=68.9/87.9 GiB
step 1470/2000 loss=0.0025 ema=0.0012 lr=1.71e-05 mem=69.1/87.9 GiB
step 1480/2000 loss=0.0000 ema=0.0006 lr=1.65e-05 mem=68.9/87.9 GiB
step 1490/2000 loss=0.0034 ema=0.0011 lr=1.59e-05 mem=69.0/87.9 GiB
step 1500/2000 loss=0.0005 ema=0.0008 lr=1.54e-05 mem=69.0/87.9 GiB
saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-01500
step 1510/2000 loss=0.0000 ema=0.0006 lr=1.48e-05 mem=69.1/87.9 GiB
step 1520/2000 loss=0.0010 ema=0.0004 lr=1.42e-05 mem=69.5/87.9 GiB
step 1530/2000 loss=0.0000 ema=0.0004 lr=1.37e-05 mem=68.8/87.9 GiB
step 1540/2000 loss=0.0001 ema=0.0009 lr=1.31e-05 mem=69.1/87.9 GiB
step 1550/2000 loss=0.0012 ema=0.0008 lr=1.26e-05 mem=68.9/87.9 GiB
step 1560/2000 loss=0.0000 ema=0.0005 lr=1.20e-05 mem=68.8/87.9 GiB
step 1570/2000 loss=0.0014 ema=0.0004 lr=1.15e-05 mem=69.1/87.9 GiB
step 1580/2000 loss=0.0000 ema=0.0008 lr=1.10e-05 mem=68.8/87.9 GiB
step 1590/2000 loss=0.0000 ema=0.0007 lr=1.05e-05 mem=69.0/87.9 GiB
step 1600/2000 loss=0.0043 ema=0.0012 lr=1.00e-05 mem=69.5/87.9 GiB
step 1610/2000 loss=0.0000 ema=0.0007 lr=9.55e-06 mem=68.8/87.9 GiB
step 1620/2000 loss=0.0002 ema=0.0007 lr=9.08e-06 mem=69.0/87.9 GiB
step 1630/2000 loss=0.0000 ema=0.0006 lr=8.62e-06 mem=69.1/87.9 GiB
step 1640/2000 loss=0.0004 ema=0.0008 lr=8.18e-06 mem=69.0/87.9 GiB
step 1650/2000 loss=0.0013 ema=0.0006 lr=7.74e-06 mem=69.5/87.9 GiB
step 1660/2000 loss=0.0171 ema=0.0031 lr=7.32e-06 mem=69.1/87.9 GiB
step 1670/2000 loss=0.0015 ema=0.0014 lr=6.90e-06 mem=69.5/87.9 GiB
step 1680/2000 loss=0.0015 ema=0.0008 lr=6.50e-06 mem=68.8/87.9 GiB
step 1690/2000 loss=0.0001 ema=0.0009 lr=6.11e-06 mem=68.8/87.9 GiB
step 1700/2000 loss=0.0000 ema=0.0005 lr=5.73e-06 mem=69.0/87.9 GiB
step 1710/2000 loss=0.0000 ema=0.0005 lr=5.36e-06 mem=68.8/87.9 GiB
step 1720/2000 loss=0.0001 ema=0.0005 lr=5.00e-06 mem=69.1/87.9 GiB
step 1730/2000 loss=0.0000 ema=0.0005 lr=4.66e-06 mem=69.1/87.9 GiB
step 1740/2000 loss=0.0000 ema=0.0008 lr=4.32e-06 mem=68.8/87.9 GiB
step 1750/2000 loss=0.0000 ema=0.0005 lr=4.00e-06 mem=68.9/87.9 GiB
step 1760/2000 loss=0.0009 ema=0.0006 lr=3.69e-06 mem=69.6/87.9 GiB
step 1770/2000 loss=0.0021 ema=0.0007 lr=3.39e-06 mem=69.5/87.9 GiB
step 1780/2000 loss=0.0000 ema=0.0003 lr=3.11e-06 mem=68.8/87.9 GiB
step 1790/2000 loss=0.0000 ema=0.0006 lr=2.83e-06 mem=68.8/87.9 GiB
step 1800/2000 loss=0.0013 ema=0.0004 lr=2.57e-06 mem=68.9/87.9 GiB
step 1810/2000 loss=0.0000 ema=0.0005 lr=2.32e-06 mem=68.9/87.9 GiB
step 1820/2000 loss=0.0030 ema=0.0006 lr=2.09e-06 mem=69.1/87.9 GiB
step 1830/2000 loss=0.0016 ema=0.0008 lr=1.86e-06 mem=69.5/87.9 GiB
step 1840/2000 loss=0.0000 ema=0.0005 lr=1.65e-06 mem=68.8/87.9 GiB
step 1850/2000 loss=0.0020 ema=0.0006 lr=1.45e-06 mem=69.0/87.9 GiB
step 1860/2000 loss=0.0000 ema=0.0005 lr=1.27e-06 mem=68.9/87.9 GiB
step 1870/2000 loss=0.0000 ema=0.0005 lr=1.09e-06 mem=68.9/87.9 GiB
step 1880/2000 loss=0.0000 ema=0.0004 lr=9.31e-07 mem=68.9/87.9 GiB
step 1890/2000 loss=0.0000 ema=0.0003 lr=7.83e-07 mem=68.9/87.9 GiB
step 1900/2000 loss=0.0001 ema=0.0008 lr=6.47e-07 mem=69.1/87.9 GiB
step 1910/2000 loss=0.0000 ema=0.0008 lr=5.25e-07 mem=68.9/87.9 GiB
step 1920/2000 loss=0.0000 ema=0.0006 lr=4.15e-07 mem=68.9/87.9 GiB
step 1930/2000 loss=0.0008 ema=0.0004 lr=3.18e-07 mem=69.5/87.9 GiB
step 1940/2000 loss=0.0000 ema=0.0006 lr=2.33e-07 mem=68.8/87.9 GiB
step 1950/2000 loss=0.0000 ema=0.0008 lr=1.62e-07 mem=69.0/87.9 GiB
step 1960/2000 loss=0.0015 ema=0.0006 lr=1.04e-07 mem=69.5/87.9 GiB
step 1970/2000 loss=0.0015 ema=0.0011 lr=5.84e-08 mem=69.0/87.9 GiB
step 1980/2000 loss=0.0002 ema=0.0008 lr=2.60e-08 mem=69.0/87.9 GiB
step 1990/2000 loss=0.0000 ema=0.0006 lr=6.49e-09 mem=69.1/87.9 GiB
step 2000/2000 loss=0.0000 ema=0.0003 lr=0.00e+00 mem=68.8/87.9 GiB
saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-02000
DONE. final adapter -> /home/lakesenberg/nemotron_lora_work/adapter/final
=== DONE ===
Final adapter: /home/lakesenberg/nemotron_lora_work/adapter/final
Next: upload that folder to Kaggle as a Model dataset,
then point CFG.tinker_adapter_path at it in the AA-SVD notebook.
=== [Sat May 23 09:28:53 UTC 2026] Phase 3: smoke eval ===
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d
=== EvalConfig ===
 base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 adapter: /home/lakesenberg/nemotron_lora_work/adapter/final
 out_dir: /home/lakesenberg/nemotron_lora_work/eval_smoke
 n_per_task: 3
 max_new_tokens: 256
 batch_size: 2
 seed: 0
 tasks: ('numeral', 'unit', 'gravity', 'cipher', 'bit', 'equation', 'cryptarithm')
 use_flash_attn: False
 bf16: True
 repetition_threshold: 0.3
[load] base = /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16
 Loading weights: 0%| | 0/6243 [00:00<?, ?it/s] Loading weights: 3%|▎ | 161/6243 [00:00<00:03, 1609.42it/s] Loading weights: 5%|▌ | 322/6243 [00:00<00:06, 924.09it/s] Loading weights: 7%|▋ | 431/6243 [00:00<00:06, 835.47it/s] Loading weights: 8%|▊ | 523/6243 [00:00<00:07, 792.74it/s] Loading weights: 10%|▉ | 607/6243 [00:00<00:07, 768.50it/s] Loading weights: 11%|█ | 687/6243 [00:00<00:07, 744.99it/s] Loading weights: 12%|█▏ | 763/6243 [00:00<00:07, 729.00it/s] Loading weights: 13%|█▎ | 837/6243 [00:01<00:07, 722.50it/s] Loading weights: 15%|█▍ | 910/6243 [00:01<00:07, 716.61it/s] Loading weights: 16%|█▌ | 982/6243 [00:01<00:07, 706.95it/s] Loading weights: 17%|█▋ | 1053/6243 [00:01<00:07, 697.30it/s] Loading weights: 18%|█▊ | 1124/6243 [00:01<00:07, 699.92it/s] Loading weights: 19%|█▉ | 1195/6243 [00:01<00:07, 695.68it/s] Loading weights: 20%|██ | 1265/6243 [00:01<00:07, 693.35it/s] Loading weights: 21%|██▏ | 1335/6243 [00:01<00:07, 689.91it/s] Loading weights: 23%|██▎ | 1406/6243 [00:01<00:06, 695.05it/s] Loading weights: 24%|██▎ | 1477/6243 [00:01<00:06, 697.00it/s] Loading weights: 25%|██▍ | 1547/6243 [00:02<00:06, 697.78it/s] Loading weights: 26%|██▌ | 1618/6243 [00:02<00:06, 699.76it/s] Loading weights: 27%|██▋ | 1688/6243 [00:02<00:06, 699.47it/s] Loading weights: 28%|██▊ | 1759/6243 [00:02<00:06, 701.90it/s] Loading weights: 29%|██▉ | 1830/6243 [00:02<00:06, 698.04it/s] Loading weights: 30%|███ | 1901/6243 [00:02<00:06, 701.28it/s] Loading weights: 32%|███▏ | 1972/6243 [00:02<00:06, 703.27it/s] Loading weights: 33%|███▎ | 2043/6243 [00:02<00:06, 697.96it/s] Loading weights: 34%|███▍ | 2113/6243 [00:02<00:05, 694.97it/s] Loading weights: 35%|███▍ | 2183/6243 [00:02<00:05, 693.61it/s] Loading weights: 36%|███▌ | 2256/6243 [00:03<00:05, 701.91it/s] Loading weights: 37%|███▋ | 2327/6243 [00:03<00:05, 701.58it/s] Loading weights: 38%|███▊ | 2398/6243 [00:03<00:05, 697.61it/s] Loading weights: 40%|███▉ | 2468/6243 [00:03<00:05, 698.10it/s] Loading weights: 41%|████ | 2538/6243 [00:03<00:05, 695.04it/s] Loading weights: 42%|████▏ | 2608/6243 [00:03<00:05, 695.30it/s] Loading weights: 43%|████▎ | 2678/6243 [00:03<00:05, 694.05it/s] Loading weights: 44%|████▍ | 2748/6243 [00:03<00:05, 695.72it/s] Loading weights: 45%|████▌ | 2818/6243 [00:03<00:04, 693.97it/s] Loading weights: 46%|████▋ | 2888/6243 [00:04<00:04, 689.60it/s] Loading weights: 47%|████▋ | 2957/6243 [00:04<00:04, 687.27it/s] Loading weights: 49%|████▊ | 3028/6243 [00:04<00:04, 693.15it/s] Loading weights: 50%|████▉ | 3098/6243 [00:04<00:04, 694.11it/s] Loading weights: 51%|█████ | 3168/6243 [00:04<00:04, 694.72it/s] Loading weights: 52%|█████▏ | 3238/6243 [00:04<00:04, 695.46it/s] Loading weights: 53%|█████▎ | 3309/6243 [00:04<00:04, 698.78it/s] Loading weights: 54%|█████▍ | 3379/6243 [00:04<00:04, 696.67it/s] Loading weights: 55%|█████▌ | 3449/6243 [00:04<00:04, 697.34it/s] Loading weights: 56%|█████▋ | 3519/6243 [00:04<00:03, 697.38it/s] Loading weights: 58%|█████▊ | 3590/6243 [00:05<00:03, 699.63it/s] Loading weights: 59%|█████▊ | 3660/6243 [00:05<00:03, 697.46it/s] Loading weights: 60%|█████▉ | 3730/6243 [00:05<00:03, 695.11it/s] Loading weights: 61%|██████ | 3800/6243 [00:05<00:03, 689.91it/s] Loading weights: 62%|██████▏ | 3872/6243 [00:05<00:03, 697.57it/s] Loading weights: 63%|██████▎ | 3942/6243 [00:05<00:03, 691.58it/s] Loading weights: 64%|██████▍ | 4012/6243 [00:05<00:03, 688.18it/s] Loading weights: 65%|██████▌ | 4082/6243 [00:05<00:03, 688.40it/s] Loading weights: 67%|██████▋ | 4154/6243 [00:05<00:03, 695.44it/s] Loading weights: 68%|██████▊ | 4225/6243 [00:05<00:02, 698.71it/s] Loading weights: 69%|██████▉ | 4296/6243 [00:06<00:02, 700.31it/s] Loading weights: 70%|██████▉ | 4370/6243 [00:06<00:02, 709.50it/s] Loading weights: 71%|███████ | 4441/6243 [00:06<00:02, 706.51it/s] Loading weights: 72%|███████▏ | 4513/6243 [00:06<00:02, 708.42it/s] Loading weights: 73%|███████▎ | 4586/6243 [00:06<00:02, 712.98it/s] Loading weights: 75%|███████▍ | 4659/6243 [00:06<00:02, 716.88it/s] Loading weights: 76%|███████▌ | 4731/6243 [00:06<00:02, 715.51it/s] Loading weights: 77%|███████▋ | 4803/6243 [00:06<00:02, 710.45it/s] Loading weights: 78%|███████▊ | 4875/6243 [00:06<00:01, 711.47it/s] Loading weights: 79%|███████▉ | 4947/6243 [00:06<00:01, 710.40it/s] Loading weights: 80%|████████ | 5019/6243 [00:07<00:01, 700.61it/s] Loading weights: 82%|████████▏ | 5090/6243 [00:07<00:01, 696.97it/s] Loading weights: 83%|████████▎ | 5160/6243 [00:07<00:01, 683.82it/s] Loading weights: 84%|████████▍ | 5230/6243 [00:07<00:01, 687.03it/s] Loading weights: 85%|████████▍ | 5301/6243 [00:07<00:01, 693.28it/s] Loading weights: 86%|████████▌ | 5371/6243 [00:07<00:01, 689.89it/s] Loading weights: 87%|████████▋ | 5441/6243 [00:07<00:01, 688.40it/s] Loading weights: 88%|████████▊ | 5514/6243 [00:07<00:01, 700.35it/s] Loading weights: 89%|████████▉ | 5585/6243 [00:07<00:00, 701.41it/s] Loading weights: 91%|█████████ | 5656/6243 [00:07<00:00, 702.44it/s] Loading weights: 92%|█████████▏| 5728/6243 [00:08<00:00, 706.43it/s] Loading weights: 93%|█████████▎| 5799/6243 [00:08<00:00, 696.18it/s] Loading weights: 94%|█████████▍| 5869/6243 [00:08<00:00, 695.62it/s] Loading weights: 95%|█████████▌| 5939/6243 [00:08<00:00, 693.53it/s] Loading weights: 96%|█████████▋| 6011/6243 [00:08<00:00, 699.15it/s] Loading weights: 97%|█████████▋| 6081/6243 [00:08<00:00, 694.50it/s] Loading weights: 99%|█████████▊| 6151/6243 [00:08<00:00, 689.54it/s] Loading weights: 100%|█████████▉| 6222/6243 [00:08<00:00, 693.55it/s] Loading weights: 100%|██████████| 6243/6243 [00:08<00:00, 701.08it/s]
[load] adapter = /home/lakesenberg/nemotron_lora_work/adapter/final
Traceback (most recent call last):
 File "/home/lakesenberg/code/nemotron_lora_train/evaluate_adapter.py", line 365, in <module>
 summary = evaluate(cfg)
 ^^^^^^^^^^^^^
 File "/home/lakesenberg/code/nemotron_lora_train/evaluate_adapter.py", line 230, in evaluate
 model, tokenizer = load_model(cfg)
 ^^^^^^^^^^^^^^^
 File "/home/lakesenberg/code/nemotron_lora_train/evaluate_adapter.py", line 183, in load_model
 model = PeftModel.from_pretrained(model, cfg.adapter)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 582, in from_pretrained
 load_result = model.load_adapter(
 ^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1408, in load_adapter
 load_result = set_peft_model_state_dict(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/utils/save_and_load.py", line 644, in set_peft_model_state_dict
 state_dict = convert_peft_adapter_state_dict_for_transformers(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/utils/transformers_weight_conversion.py", line 509, in convert_peft_adapter_state_dict_for_transformers
 peft_weight_conversions = build_peft_weight_mapping(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/utils/transformers_weight_conversion.py", line 310, in build_peft_weight_mapping
 new_conversion = orig_conversion.__class__(
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: WeightConverter.__init__() got an unexpected keyword argument 'distributed_operation'
EXIT_CODE=1