=== [Fri May 22 15:03:42 UTC 2026] Phase 0: setup_env.sh === Retrieving notices: - \ | / - \ | done Channels: - conda-forge Platform: linux-64 Collecting package metadata (repodata.json): - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done Solving environment: | / done ==> WARNING: A newer version of conda exists. <== current version: 26.1.1 latest version: 26.5.0 Please update conda by running $ conda update -n base -c conda-forge conda ## Package Plan ## environment location: /home/lakesenberg/miniforge3/envs/nemo added / updated specs: - python=3.11 The following packages will be downloaded: package | build ---------------------------|----------------- ca-certificates-2026.5.20 | hbd8a1cb_0 127 KB conda-forge libexpat-2.8.1 | hecca717_0 75 KB conda-forge libgcc-15.2.0 | he0feb66_19 1017 KB conda-forge libgcc-ng-15.2.0 | h69a702a_19 27 KB conda-forge libgomp-15.2.0 | he0feb66_19 590 KB conda-forge libsqlite-3.53.1 | h0c1763c_0 933 KB conda-forge libuuid-2.42.1 | h5347b49_0 39 KB conda-forge ncurses-6.6 | hdb14827_0 897 KB conda-forge pip-26.1.1 | pyh8b19718_0 1.1 MB conda-forge python-3.11.15 |hd63d673_0_cpython 29.5 MB conda-forge ------------------------------------------------------------ Total: 34.3 MB The following NEW packages will be INSTALLED: _openmp_mutex conda-forge/linux-64::_openmp_mutex-4.5-20_gnu bzip2 conda-forge/linux-64::bzip2-1.0.8-hda65f42_9 ca-certificates conda-forge/noarch::ca-certificates-2026.5.20-hbd8a1cb_0 ld_impl_linux-64 conda-forge/linux-64::ld_impl_linux-64-2.45.1-default_hbd61a6d_102 libexpat conda-forge/linux-64::libexpat-2.8.1-hecca717_0 libffi conda-forge/linux-64::libffi-3.5.2-h3435931_0 libgcc conda-forge/linux-64::libgcc-15.2.0-he0feb66_19 libgcc-ng conda-forge/linux-64::libgcc-ng-15.2.0-h69a702a_19 libgomp conda-forge/linux-64::libgomp-15.2.0-he0feb66_19 liblzma conda-forge/linux-64::liblzma-5.8.3-hb03c661_0 libnsl conda-forge/linux-64::libnsl-2.0.1-hb9d3cd8_1 libsqlite conda-forge/linux-64::libsqlite-3.53.1-h0c1763c_0 libuuid conda-forge/linux-64::libuuid-2.42.1-h5347b49_0 libxcrypt conda-forge/linux-64::libxcrypt-4.4.36-hd590300_1 libzlib conda-forge/linux-64::libzlib-1.3.2-h25fd6f3_2 ncurses conda-forge/linux-64::ncurses-6.6-hdb14827_0 openssl conda-forge/linux-64::openssl-3.6.2-h35e630c_0 packaging conda-forge/noarch::packaging-26.2-pyhc364b38_0 pip conda-forge/noarch::pip-26.1.1-pyh8b19718_0 python conda-forge/linux-64::python-3.11.15-hd63d673_0_cpython readline conda-forge/linux-64::readline-8.3-h853b02a_0 setuptools conda-forge/noarch::setuptools-82.0.1-pyh332efcf_0 tk conda-forge/linux-64::tk-8.6.13-noxft_h366c992_103 tzdata conda-forge/noarch::tzdata-2025c-hc9c84f9_1 wheel conda-forge/noarch::wheel-0.47.0-pyhd8ed1ab_0 zstd conda-forge/linux-64::zstd-1.5.7-hb78ec9c_6 Downloading and Extracting Packages: ...working... python-3.11.15 | 29.5 MB | | 0% pip-26.1.1 | 1.1 MB | | 0%  libgcc-15.2.0 | 1017 KB | | 0%  libsqlite-3.53.1 | 933 KB | | 0%  ncurses-6.6 | 897 KB | | 0%  libgomp-15.2.0 | 590 KB | | 0%  ca-certificates-2026 | 127 KB | | 0%  libexpat-2.8.1 | 75 KB | | 0%  libuuid-2.42.1 | 39 KB | | 0%  libgcc-ng-15.2.0 | 27 KB | | 0%  pip-26.1.1 | 1.1 MB | ########## | 100%  ncurses-6.6 | 897 KB | ########## | 100%  python-3.11.15 | 29.5 MB | 1 | 1% libgcc-15.2.0 | 1017 KB | ########## | 100%  libgcc-15.2.0 | 1017 KB | ########## | 100%  libsqlite-3.53.1 | 933 KB | ########## | 100%  libsqlite-3.53.1 | 933 KB | ########## | 100%  libgomp-15.2.0 | 590 KB | 2 | 3%  libgomp-15.2.0 | 590 KB | ########## | 100%  ca-certificates-2026 | 127 KB | #2 | 13%  libuuid-2.42.1 | 39 KB | #### | 41%  libuuid-2.42.1 | 39 KB | ########## | 100%  ca-certificates-2026 | 127 KB | ########## | 100%  libgcc-ng-15.2.0 | 27 KB | #####9 | 59%  libexpat-2.8.1 | 75 KB | ##1 | 21%  libgcc-ng-15.2.0 | 27 KB | ########## | 100%  libexpat-2.8.1 | 75 KB | ########## | 100%  python-3.11.15 | 29.5 MB | ##2 | 22% libgcc-15.2.0 | 1017 KB | ########## | 100%  libsqlite-3.53.1 | 933 KB | ########## | 100%  python-3.11.15 | 29.5 MB | ####5 | 46% libgomp-15.2.0 | 590 KB | ########## | 100%  libgomp-15.2.0 | 590 KB | ########## | 100%  libuuid-2.42.1 | 39 KB | ########## | 100%  libuuid-2.42.1 | 39 KB | ########## | 100%  ca-certificates-2026 | 127 KB | ########## | 100%  ca-certificates-2026 | 127 KB | ########## | 100%  libgcc-ng-15.2.0 | 27 KB | ########## | 100%  libgcc-ng-15.2.0 | 27 KB | ########## | 100%  python-3.11.15 | 29.5 MB | #######3 | 73% libexpat-2.8.1 | 75 KB | ########## | 100%  libexpat-2.8.1 | 75 KB | ########## | 100%  pip-26.1.1 | 1.1 MB | ########## | 100%  pip-26.1.1 | 1.1 MB | ########## | 100%  python-3.11.15 | 29.5 MB | ########## | 100% python-3.11.15 | 29.5 MB | ########## | 100% ncurses-6.6 | 897 KB | ########## | 100%  ncurses-6.6 | 897 KB | ########## | 100%  python-3.11.15 | 29.5 MB | ########## | 100%          done Preparing transaction: \ | done Verifying transaction: - \ | / - \ | / - done Executing transaction: | / - \ | / - \ | / - \ | / - \ | / - done # # To activate this environment, use # # $ conda activate nemo # # To deactivate an active environment, use # # $ conda deactivate Requirement already satisfied: pip in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (26.1.1) Requirement already satisfied: wheel in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.47.0) Requirement already satisfied: packaging>=24.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from wheel) (26.2) Looking in indexes: https://download.pytorch.org/whl/cu124 Collecting torch>=2.4 Downloading torch-2.6.0%2Bcu124-cp311-cp311-linux_x86_64.whl.metadata (28 kB) Collecting filelock (from torch>=2.4) Using cached filelock-3.29.0-py3-none-any.whl.metadata (2.0 kB) Collecting typing-extensions>=4.10.0 (from torch>=2.4) Using cached typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB) Collecting networkx (from torch>=2.4) Using cached networkx-3.6.1-py3-none-any.whl.metadata (6.8 kB) Collecting jinja2 (from torch>=2.4) Using cached jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB) Collecting fsspec (from torch>=2.4) Downloading fsspec-2026.4.0-py3-none-any.whl.metadata (10 kB) Collecting nvidia-cuda-nvrtc-cu12==12.4.127 (from torch>=2.4) Using cached nvidia_cuda_nvrtc_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (24.6 MB) Collecting nvidia-cuda-runtime-cu12==12.4.127 (from torch>=2.4) Using cached nvidia_cuda_runtime_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (883 kB) Collecting nvidia-cuda-cupti-cu12==12.4.127 (from torch>=2.4) Using cached nvidia_cuda_cupti_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (13.8 MB) Collecting nvidia-cudnn-cu12==9.1.0.70 (from torch>=2.4) Downloading nvidia_cudnn_cu12-9.1.0.70-py3-none-manylinux2014_x86_64.whl (664.8 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/664.8 MB ? eta -:--:--  ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.9/664.8 MB 338.4 MB/s eta 0:00:02  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.3/664.8 MB 322.8 MB/s eta 0:00:02  ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━ 184.0/664.8 MB 312.2 MB/s eta 0:00:02  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 229.4/664.8 MB 290.5 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━ 284.7/664.8 MB 287.7 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━ 341.3/664.8 MB 271.6 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 399.2/664.8 MB 270.1 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━ 456.9/664.8 MB 271.6 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 503.6/664.8 MB 277.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━ 551.3/664.8 MB 263.8 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 605.6/664.8 MB 262.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 664.5/664.8 MB 263.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 664.8/664.8 MB 139.6 MB/s 0:00:03 [?25hCollecting nvidia-cublas-cu12==12.4.5.8 (from torch>=2.4) Using cached nvidia_cublas_cu12-12.4.5.8-py3-none-manylinux2014_x86_64.whl (363.4 MB) Collecting nvidia-cufft-cu12==11.2.1.3 (from torch>=2.4) Using cached nvidia_cufft_cu12-11.2.1.3-py3-none-manylinux2014_x86_64.whl (211.5 MB) Collecting nvidia-curand-cu12==10.3.5.147 (from torch>=2.4) Using cached nvidia_curand_cu12-10.3.5.147-py3-none-manylinux2014_x86_64.whl (56.3 MB) Collecting nvidia-cusolver-cu12==11.6.1.9 (from torch>=2.4) Using cached nvidia_cusolver_cu12-11.6.1.9-py3-none-manylinux2014_x86_64.whl (127.9 MB) Collecting nvidia-cusparse-cu12==12.3.1.170 (from torch>=2.4) Using cached nvidia_cusparse_cu12-12.3.1.170-py3-none-manylinux2014_x86_64.whl (207.5 MB) Collecting nvidia-cusparselt-cu12==0.6.2 (from torch>=2.4) Using cached nvidia_cusparselt_cu12-0.6.2-py3-none-manylinux2014_x86_64.whl.metadata (6.8 kB) Collecting nvidia-nccl-cu12==2.21.5 (from torch>=2.4) Using cached nvidia_nccl_cu12-2.21.5-py3-none-manylinux2014_x86_64.whl (188.7 MB) Collecting nvidia-nvtx-cu12==12.4.127 (from torch>=2.4) Using cached nvidia_nvtx_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (99 kB) Collecting nvidia-nvjitlink-cu12==12.4.127 (from torch>=2.4) Using cached nvidia_nvjitlink_cu12-12.4.127-py3-none-manylinux2014_x86_64.whl (21.1 MB) Collecting triton==3.2.0 (from torch>=2.4) Downloading triton-3.2.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (1.4 kB) Collecting sympy==1.13.1 (from torch>=2.4) Using cached sympy-1.13.1-py3-none-any.whl.metadata (12 kB) Collecting mpmath<1.4,>=1.1.0 (from sympy==1.13.1->torch>=2.4) Using cached mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB) Collecting MarkupSafe>=2.0 (from jinja2->torch>=2.4) Downloading markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.7 kB) Downloading torch-2.6.0%2Bcu124-cp311-cp311-linux_x86_64.whl (768.5 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/768.5 MB ? eta -:--:--  ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.1/768.5 MB 98.5 MB/s eta 0:00:08  ━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.4/768.5 MB 113.7 MB/s eta 0:00:07  ━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.8/768.5 MB 126.8 MB/s eta 0:00:06  ━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.9/768.5 MB 132.4 MB/s eta 0:00:06  ━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.2/768.5 MB 140.0 MB/s eta 0:00:05  ━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.0/768.5 MB 133.9 MB/s eta 0:00:05  ━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 188.2/768.5 MB 134.1 MB/s eta 0:00:05  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━ 221.0/768.5 MB 137.8 MB/s eta 0:00:04  ━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━ 250.6/768.5 MB 138.8 MB/s eta 0:00:04  ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━ 280.8/768.5 MB 143.9 MB/s eta 0:00:04  ━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━ 304.9/768.5 MB 143.2 MB/s eta 0:00:04  ━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━ 336.3/768.5 MB 143.8 MB/s eta 0:00:04  ━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 363.9/768.5 MB 142.7 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━ 396.1/768.5 MB 140.9 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 425.5/768.5 MB 146.3 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 450.1/768.5 MB 144.5 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━ 478.7/768.5 MB 142.8 MB/s eta 0:00:03  ━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 514.1/768.5 MB 145.6 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━ 548.4/768.5 MB 149.2 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━ 576.2/768.5 MB 149.4 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━ 607.4/768.5 MB 149.6 MB/s eta 0:00:02  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 635.4/768.5 MB 150.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 663.5/768.5 MB 146.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━ 694.2/768.5 MB 149.1 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━ 724.6/768.5 MB 151.7 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 767.3/768.5 MB 157.6 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸ 768.3/768.5 MB 157.5 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 768.5/768.5 MB 93.1 MB/s 0:00:06 [?25hUsing cached nvidia_cusparselt_cu12-0.6.2-py3-none-manylinux2014_x86_64.whl (150.1 MB) Using cached sympy-1.13.1-py3-none-any.whl (6.2 MB) Downloading triton-3.2.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (166.7 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/166.7 MB ? eta -:--:--  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.0/166.7 MB 159.9 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━ 66.1/166.7 MB 165.3 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━ 87.6/166.7 MB 162.4 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 95.7/166.7 MB 119.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 128.5/166.7 MB 127.8 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 158.6/166.7 MB 131.4 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 166.7/166.7 MB 118.9 MB/s 0:00:01 [?25hUsing cached mpmath-1.3.0-py3-none-any.whl (536 kB) Using cached typing_extensions-4.15.0-py3-none-any.whl (44 kB) Using cached filelock-3.29.0-py3-none-any.whl (39 kB) Downloading fsspec-2026.4.0-py3-none-any.whl (203 kB) Using cached jinja2-3.1.6-py3-none-any.whl (134 kB) Downloading markupsafe-3.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (22 kB) Using cached networkx-3.6.1-py3-none-any.whl (2.1 MB) Installing collected packages: triton, nvidia-cusparselt-cu12, mpmath, typing-extensions, sympy, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, networkx, MarkupSafe, fsspec, filelock, nvidia-cusparse-cu12, nvidia-cudnn-cu12, jinja2, nvidia-cusolver-cu12, torch [?25l  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  0/23 [triton]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  1/23 [nvidia-cusparselt-cu12]  ━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  2/23 [mpmath]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  4/23 [sympy]  ━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  6/23 [nvidia-nvjitlink-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/23 [nvidia-nccl-cu12]  ━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━  8/23 [nvidia-curand-cu12]  ━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━  8/23 [nvidia-curand-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━  9/23 [nvidia-cufft-cu12]  ━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 11/23 [nvidia-cuda-nvrtc-cu12]  ━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━ 11/23 [nvidia-cuda-nvrtc-cu12]  ━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━ 12/23 [nvidia-cuda-cupti-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━ 13/23 [nvidia-cublas-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━ 14/23 [networkx]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━ 16/23 [fsspec]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 18/23 [nvidia-cusparse-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 19/23 [nvidia-cudnn-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 21/23 [nvidia-cusolver-cu12]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 22/23 [torch]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 23/23 [torch] [?25h Successfully installed MarkupSafe-3.0.3 filelock-3.29.0 fsspec-2026.4.0 jinja2-3.1.6 mpmath-1.3.0 networkx-3.6.1 nvidia-cublas-cu12-12.4.5.8 nvidia-cuda-cupti-cu12-12.4.127 nvidia-cuda-nvrtc-cu12-12.4.127 nvidia-cuda-runtime-cu12-12.4.127 nvidia-cudnn-cu12-9.1.0.70 nvidia-cufft-cu12-11.2.1.3 nvidia-curand-cu12-10.3.5.147 nvidia-cusolver-cu12-11.6.1.9 nvidia-cusparse-cu12-12.3.1.170 nvidia-cusparselt-cu12-0.6.2 nvidia-nccl-cu12-2.21.5 nvidia-nvjitlink-cu12-12.4.127 nvidia-nvtx-cu12-12.4.127 sympy-1.13.1 torch-2.6.0+cu124 triton-3.2.0 typing-extensions-4.15.0 Collecting transformers>=4.46 Downloading transformers-5.9.0-py3-none-any.whl.metadata (33 kB) Collecting peft>=0.12 Downloading peft-0.19.1-py3-none-any.whl.metadata (15 kB) Collecting accelerate>=0.34 Using cached accelerate-1.13.0-py3-none-any.whl.metadata (19 kB) Collecting safetensors Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB) Collecting sentencepiece Downloading sentencepiece-0.2.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (10 kB) Collecting datasets Downloading datasets-4.8.5-py3-none-any.whl.metadata (19 kB) Collecting tqdm Using cached tqdm-4.67.3-py3-none-any.whl.metadata (57 kB) Collecting huggingface_hub[cli] Downloading huggingface_hub-1.16.1-py3-none-any.whl.metadata (14 kB) Collecting numpy>=1.17 (from transformers>=4.46) Downloading numpy-2.4.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (6.6 kB) Requirement already satisfied: packaging>=20.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (26.2) Collecting pyyaml>=5.1 (from transformers>=4.46) Downloading pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (2.4 kB) Collecting regex>=2025.10.22 (from transformers>=4.46) Downloading regex-2026.5.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB) Collecting tokenizers<=0.23.0,>=0.22.0 (from transformers>=4.46) Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB) Collecting typer (from transformers>=4.46) Downloading typer-0.25.1-py3-none-any.whl.metadata (15 kB) Requirement already satisfied: filelock>=3.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (3.29.0) Requirement already satisfied: fsspec>=2023.5.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (2026.4.0) Collecting hf-xet<2.0.0,>=1.4.3 (from huggingface_hub[cli]) Downloading hf_xet-1.5.0-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (4.9 kB) Collecting httpx<1,>=0.23.0 (from huggingface_hub[cli]) Using cached httpx-0.28.1-py3-none-any.whl.metadata (7.1 kB) Requirement already satisfied: typing-extensions>=4.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (4.15.0) Collecting anyio (from httpx<1,>=0.23.0->huggingface_hub[cli]) Downloading anyio-4.13.0-py3-none-any.whl.metadata (4.5 kB) Collecting certifi (from httpx<1,>=0.23.0->huggingface_hub[cli]) Downloading certifi-2026.5.20-py3-none-any.whl.metadata (2.5 kB) Collecting httpcore==1.* (from httpx<1,>=0.23.0->huggingface_hub[cli]) Using cached httpcore-1.0.9-py3-none-any.whl.metadata (21 kB) Collecting idna (from httpx<1,>=0.23.0->huggingface_hub[cli]) Downloading idna-3.16-py3-none-any.whl.metadata (6.4 kB) Collecting h11>=0.16 (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub[cli]) Using cached h11-0.16.0-py3-none-any.whl.metadata (8.3 kB) Collecting psutil (from peft>=0.12) Using cached psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (22 kB) Requirement already satisfied: torch>=1.13.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from peft>=0.12) (2.6.0+cu124) Collecting pyarrow>=21.0.0 (from datasets) Downloading pyarrow-24.0.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (3.0 kB) Collecting dill<0.4.2,>=0.3.0 (from datasets) Downloading dill-0.4.1-py3-none-any.whl.metadata (10 kB) Collecting pandas (from datasets) Downloading pandas-3.0.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (79 kB) Collecting requests>=2.32.2 (from datasets) Downloading requests-2.34.2-py3-none-any.whl.metadata (4.8 kB) Collecting xxhash (from datasets) Downloading xxhash-3.7.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (13 kB) Collecting multiprocess<0.70.20 (from datasets) Downloading multiprocess-0.70.19-py311-none-any.whl.metadata (7.5 kB) Collecting fsspec>=2023.5.0 (from huggingface_hub[cli]) Downloading fsspec-2026.2.0-py3-none-any.whl.metadata (10 kB) Collecting aiohttp!=4.0.0a0,!=4.0.0a1 (from fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Downloading aiohttp-3.13.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (8.1 kB) WARNING: huggingface-hub 1.16.1 does not provide the extra 'cli' Collecting aiohappyeyeballs>=2.5.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Downloading aiohappyeyeballs-2.6.2-py3-none-any.whl.metadata (5.9 kB) Collecting aiosignal>=1.4.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Using cached aiosignal-1.4.0-py3-none-any.whl.metadata (3.7 kB) Collecting attrs>=17.3.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Using cached attrs-26.1.0-py3-none-any.whl.metadata (8.8 kB) Collecting frozenlist>=1.1.1 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Downloading frozenlist-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl.metadata (20 kB) Collecting multidict<7.0,>=4.5 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Downloading multidict-6.7.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (5.3 kB) Collecting propcache>=0.2.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Downloading propcache-0.5.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (16 kB) Collecting yarl<2.0,>=1.17.0 (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) Downloading yarl-1.24.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (94 kB) Collecting charset_normalizer<4,>=2 (from requests>=2.32.2->datasets) Downloading charset_normalizer-3.4.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl.metadata (40 kB) Collecting urllib3<3,>=1.26 (from requests>=2.32.2->datasets) Downloading urllib3-2.7.0-py3-none-any.whl.metadata (6.9 kB) Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.6.1) Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.1.6) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (9.1.0.70) Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.5.8) Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.2.1.3) Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (10.3.5.147) Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.6.1.9) Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.3.1.170) Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (0.6.2) Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (2.21.5) Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.2.0) Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (1.13.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.13.0->peft>=0.12) (1.3.0) Collecting click>=8.2.1 (from typer->transformers>=4.46) Downloading click-8.4.1-py3-none-any.whl.metadata (2.6 kB) Collecting shellingham>=1.3.0 (from typer->transformers>=4.46) Using cached shellingham-1.5.4-py2.py3-none-any.whl.metadata (3.5 kB) Collecting rich>=13.8.0 (from typer->transformers>=4.46) Downloading rich-15.0.0-py3-none-any.whl.metadata (18 kB) Collecting annotated-doc>=0.0.2 (from typer->transformers>=4.46) Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB) Collecting markdown-it-py>=2.2.0 (from rich>=13.8.0->typer->transformers>=4.46) Downloading markdown_it_py-4.2.0-py3-none-any.whl.metadata (7.4 kB) Collecting pygments<3.0.0,>=2.13.0 (from rich>=13.8.0->typer->transformers>=4.46) Downloading pygments-2.20.0-py3-none-any.whl.metadata (2.5 kB) Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers>=4.46) Using cached mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB) Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch>=1.13.0->peft>=0.12) (3.0.3) Collecting python-dateutil>=2.8.2 (from pandas->datasets) Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB) Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas->datasets) Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB) Downloading transformers-5.9.0-py3-none-any.whl (10.8 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/10.8 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.8/10.8 MB 247.0 MB/s 0:00:00 [?25hDownloading huggingface_hub-1.16.1-py3-none-any.whl (668 kB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/668.2 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 668.2/668.2 kB 48.4 MB/s 0:00:00 [?25hDownloading hf_xet-1.5.0-cp37-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (4.5 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/4.5 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 140.5 MB/s 0:00:00 [?25hUsing cached httpx-0.28.1-py3-none-any.whl (73 kB) Using cached httpcore-1.0.9-py3-none-any.whl (78 kB) Using cached tokenizers-0.22.2-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB) Downloading peft-0.19.1-py3-none-any.whl (680 kB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/680.7 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 680.7/680.7 kB 27.1 MB/s 0:00:00 [?25hUsing cached accelerate-1.13.0-py3-none-any.whl (383 kB) Using cached safetensors-0.7.0-cp38-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (507 kB) Downloading sentencepiece-0.2.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (1.4 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.4 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 113.1 MB/s 0:00:00 [?25hDownloading datasets-4.8.5-py3-none-any.whl (528 kB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/529.0 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 529.0/529.0 kB 24.6 MB/s 0:00:00 [?25hDownloading dill-0.4.1-py3-none-any.whl (120 kB) Downloading fsspec-2026.2.0-py3-none-any.whl (202 kB) Downloading multiprocess-0.70.19-py311-none-any.whl (144 kB) Using cached tqdm-4.67.3-py3-none-any.whl (78 kB) Downloading aiohttp-3.13.5-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (1.8 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.8 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 91.1 MB/s 0:00:00 [?25hDownloading multidict-6.7.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (246 kB) Downloading yarl-1.24.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (106 kB) Downloading aiohappyeyeballs-2.6.2-py3-none-any.whl (15 kB) Using cached aiosignal-1.4.0-py3-none-any.whl (7.5 kB) Using cached attrs-26.1.0-py3-none-any.whl (67 kB) Downloading frozenlist-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (231 kB) Using cached h11-0.16.0-py3-none-any.whl (37 kB) Downloading idna-3.16-py3-none-any.whl (74 kB) Downloading numpy-2.4.6-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.9 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/16.9 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 356.4 MB/s 0:00:00 [?25hDownloading propcache-0.5.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (58 kB) Downloading pyarrow-24.0.0-cp311-cp311-manylinux_2_28_x86_64.whl (48.8 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/48.8 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 38.0/48.8 MB 189.2 MB/s eta 0:00:01  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.8/48.8 MB 156.9 MB/s 0:00:00 [?25hDownloading pyyaml-6.0.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (806 kB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/806.6 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 806.6/806.6 kB 15.8 MB/s 0:00:00 [?25hDownloading regex-2026.5.9-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (799 kB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/799.8 kB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 799.8/799.8 kB 3.9 MB/s 0:00:00 [?25hDownloading requests-2.34.2-py3-none-any.whl (73 kB) Downloading charset_normalizer-3.4.7-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (214 kB) Downloading urllib3-2.7.0-py3-none-any.whl (131 kB) Downloading certifi-2026.5.20-py3-none-any.whl (134 kB) Downloading typer-0.25.1-py3-none-any.whl (58 kB) Downloading annotated_doc-0.0.4-py3-none-any.whl (5.3 kB) Downloading click-8.4.1-py3-none-any.whl (116 kB) Downloading rich-15.0.0-py3-none-any.whl (310 kB) Downloading pygments-2.20.0-py3-none-any.whl (1.2 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/1.2 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 84.7 MB/s 0:00:00 [?25hDownloading markdown_it_py-4.2.0-py3-none-any.whl (91 kB) Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB) Using cached shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB) Downloading anyio-4.13.0-py3-none-any.whl (114 kB) Downloading pandas-3.0.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (11.3 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/11.3 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.3/11.3 MB 140.0 MB/s 0:00:00 [?25hUsing cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB) Using cached six-1.17.0-py2.py3-none-any.whl (11 kB) Using cached psutil-7.2.2-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl (155 kB) Downloading xxhash-3.7.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (193 kB) Installing collected packages: xxhash, urllib3, tqdm, six, shellingham, sentencepiece, safetensors, regex, pyyaml, pygments, pyarrow, psutil, propcache, numpy, multidict, mdurl, idna, hf-xet, h11, fsspec, frozenlist, dill, click, charset_normalizer, certifi, attrs, annotated-doc, aiohappyeyeballs, yarl, requests, python-dateutil, multiprocess, markdown-it-py, httpcore, anyio, aiosignal, rich, pandas, httpx, aiohttp, typer, huggingface_hub, tokenizers, datasets, accelerate, transformers, peft [?25l  ━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  7/47 [regex]  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  9/47 [pygments]  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  9/47 [pygments]  ━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  9/47 [pygments]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10/47 [pyarrow]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13/47 [numpy]  ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Attempting uninstall: fsspec ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Found existing installation: fsspec 2026.4.0 ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Uninstalling fsspec-2026.4.0: ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  Successfully uninstalled fsspec-2026.4.0 ━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━━━━━━━ 17/47 [hf-xet]  ━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━━━━━━━ 21/47 [dill]  ━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━━━━ 28/47 [yarl]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━ 32/47 [markdown-it-py]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━ 36/47 [rich]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━ 37/47 [pandas]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━━━━━ 39/47 [aiohttp]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 41/47 [huggingface_hub]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━ 41/47 [huggingface_hub]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━ 43/47 [datasets]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━━ 44/47 [accelerate]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺━ 45/47 [transformers]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 46/47 [peft]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╺ 46/47 [peft]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47/47 [peft] [?25h Successfully installed accelerate-1.13.0 aiohappyeyeballs-2.6.2 aiohttp-3.13.5 aiosignal-1.4.0 annotated-doc-0.0.4 anyio-4.13.0 attrs-26.1.0 certifi-2026.5.20 charset_normalizer-3.4.7 click-8.4.1 datasets-4.8.5 dill-0.4.1 frozenlist-1.8.0 fsspec-2026.2.0 h11-0.16.0 hf-xet-1.5.0 httpcore-1.0.9 httpx-0.28.1 huggingface_hub-1.16.1 idna-3.16 markdown-it-py-4.2.0 mdurl-0.1.2 multidict-6.7.1 multiprocess-0.70.19 numpy-2.4.6 pandas-3.0.3 peft-0.19.1 propcache-0.5.2 psutil-7.2.2 pyarrow-24.0.0 pygments-2.20.0 python-dateutil-2.9.0.post0 pyyaml-6.0.3 regex-2026.5.9 requests-2.34.2 rich-15.0.0 safetensors-0.7.0 sentencepiece-0.2.1 shellingham-1.5.4 six-1.17.0 tokenizers-0.22.2 tqdm-4.67.3 transformers-5.9.0 typer-0.25.1 urllib3-2.7.0 xxhash-3.7.0 yarl-1.24.2 Collecting flash-attn Downloading flash_attn-2.8.3.tar.gz (8.4 MB) [?25l ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/8.4 MB ? eta -:--:--  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.4/8.4 MB 112.2 MB/s 0:00:00 [?25h Preparing metadata (pyproject.toml): started Preparing metadata (pyproject.toml): finished with status 'done' Requirement already satisfied: torch in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from flash-attn) (2.6.0+cu124) Collecting einops (from flash-attn) Using cached einops-0.8.2-py3-none-any.whl.metadata (13 kB) Requirement already satisfied: filelock in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.29.0) Requirement already satisfied: typing-extensions>=4.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (4.15.0) Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.6.1) Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.1.6) Requirement already satisfied: fsspec in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (2026.2.0) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (9.1.0.70) Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.5.8) Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (11.2.1.3) Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (10.3.5.147) Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (11.6.1.9) Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.3.1.170) Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (0.6.2) Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (2.21.5) Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127) Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (12.4.127) Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (3.2.0) Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch->flash-attn) (1.13.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch->flash-attn) (1.3.0) Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch->flash-attn) (3.0.3) Using cached einops-0.8.2-py3-none-any.whl (65 kB) Building wheels for collected packages: flash-attn Building wheel for flash-attn (pyproject.toml): started Building wheel for flash-attn (pyproject.toml): finished with status 'done' Created wheel for flash-attn: filename=flash_attn-2.8.3-cp311-cp311-linux_x86_64.whl size=256022485 sha256=0abc62d04f28f140f4f76ab7cfd1d8ce24a69c6ab0cbace8d4ab99640b68dc0a Stored in directory: /home/lakesenberg/.cache/pip/wheels/42/31/1f/4b22dd7295b3cb064b8fa9038f6d58fb15c9571555b2d7c39c Successfully built flash-attn Installing collected packages: einops, flash-attn [?25l  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━━━ 1/2 [flash-attn]  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2/2 [flash-attn] [?25h Successfully installed einops-0.8.2 flash-attn-2.8.3 [info] downloading nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 -> /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 Warning: `huggingface-cli` is deprecated and no longer works. Use `hf` instead. Hint: `hf` is already installed! Use it directly. Hint: Examples: hf auth login hf download unsloth/gemma-4-31B-it-GGUF hf upload my-cool-model . . hf models ls --search "gemma" hf repos ls --format json hf jobs run python:3.12 python -c 'print("Hello!")' hf --help EXIT_CODE=1 === RESTART Fri May 22 15:20:38 UTC 2026 === === [Fri May 22 15:20:38 UTC 2026] RESTART: Phase 0 setup + model download === Requirement already satisfied: pip in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (26.1.1) Requirement already satisfied: wheel in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.47.0) Requirement already satisfied: packaging>=24.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from wheel) (26.2) Looking in indexes: https://download.pytorch.org/whl/cu124 Requirement already satisfied: torch>=2.4 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (2.6.0+cu124) Requirement already satisfied: filelock in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.29.0) Requirement already satisfied: typing-extensions>=4.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (4.15.0) Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.6.1) Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.1.6) Requirement already satisfied: fsspec in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (2026.2.0) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (9.1.0.70) Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.5.8) Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (11.2.1.3) Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (10.3.5.147) Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (11.6.1.9) Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.3.1.170) Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (0.6.2) Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (2.21.5) Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127) Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (12.4.127) Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (3.2.0) Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=2.4) (1.13.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch>=2.4) (1.3.0) Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch>=2.4) (3.0.3) Requirement already satisfied: transformers>=4.46 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (5.9.0) Requirement already satisfied: peft>=0.12 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.19.1) Requirement already satisfied: accelerate>=0.34 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (1.13.0) Requirement already satisfied: safetensors in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.7.0) Requirement already satisfied: sentencepiece in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (0.2.1) Requirement already satisfied: datasets in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (4.8.5) Requirement already satisfied: tqdm in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (4.67.3) Requirement already satisfied: huggingface_hub[cli] in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (1.16.1) Requirement already satisfied: numpy>=1.17 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (2.4.6) Requirement already satisfied: packaging>=20.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (26.2) Requirement already satisfied: pyyaml>=5.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (6.0.3) Requirement already satisfied: regex>=2025.10.22 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (2026.5.9) Requirement already satisfied: tokenizers<=0.23.0,>=0.22.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (0.22.2) Requirement already satisfied: typer in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from transformers>=4.46) (0.25.1) Requirement already satisfied: filelock>=3.10.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (3.29.0) Requirement already satisfied: fsspec>=2023.5.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (2026.2.0) Requirement already satisfied: hf-xet<2.0.0,>=1.4.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (1.5.0) Requirement already satisfied: httpx<1,>=0.23.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (0.28.1) Requirement already satisfied: typing-extensions>=4.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from huggingface_hub[cli]) (4.15.0) Requirement already satisfied: anyio in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (4.13.0) Requirement already satisfied: certifi in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (2026.5.20) Requirement already satisfied: httpcore==1.* in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (1.0.9) Requirement already satisfied: idna in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpx<1,>=0.23.0->huggingface_hub[cli]) (3.16) Requirement already satisfied: h11>=0.16 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->huggingface_hub[cli]) (0.16.0) Requirement already satisfied: psutil in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from peft>=0.12) (7.2.2) Requirement already satisfied: torch>=1.13.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from peft>=0.12) (2.6.0+cu124) Requirement already satisfied: pyarrow>=21.0.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (24.0.0) Requirement already satisfied: dill<0.4.2,>=0.3.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (0.4.1) Requirement already satisfied: pandas in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (3.0.3) Requirement already satisfied: requests>=2.32.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (2.34.2) Requirement already satisfied: xxhash in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (3.7.0) Requirement already satisfied: multiprocess<0.70.20 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from datasets) (0.70.19) Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (3.13.5) WARNING: huggingface-hub 1.16.1 does not provide the extra 'cli' Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (2.6.2) Requirement already satisfied: aiosignal>=1.4.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (1.4.0) Requirement already satisfied: attrs>=17.3.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (26.1.0) Requirement already satisfied: frozenlist>=1.1.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (1.8.0) Requirement already satisfied: multidict<7.0,>=4.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (6.7.1) Requirement already satisfied: propcache>=0.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (0.5.2) Requirement already satisfied: yarl<2.0,>=1.17.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]<=2026.2.0,>=2023.1.0->datasets) (1.24.2) Requirement already satisfied: charset_normalizer<4,>=2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (3.4.7) Requirement already satisfied: urllib3<3,>=1.26 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from requests>=2.32.2->datasets) (2.7.0) Requirement already satisfied: networkx in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.6.1) Requirement already satisfied: jinja2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.1.6) Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-cuda-runtime-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-cuda-cupti-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-cudnn-cu12==9.1.0.70 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (9.1.0.70) Requirement already satisfied: nvidia-cublas-cu12==12.4.5.8 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.5.8) Requirement already satisfied: nvidia-cufft-cu12==11.2.1.3 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.2.1.3) Requirement already satisfied: nvidia-curand-cu12==10.3.5.147 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (10.3.5.147) Requirement already satisfied: nvidia-cusolver-cu12==11.6.1.9 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (11.6.1.9) Requirement already satisfied: nvidia-cusparse-cu12==12.3.1.170 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.3.1.170) Requirement already satisfied: nvidia-cusparselt-cu12==0.6.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (0.6.2) Requirement already satisfied: nvidia-nccl-cu12==2.21.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (2.21.5) Requirement already satisfied: nvidia-nvtx-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: nvidia-nvjitlink-cu12==12.4.127 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (12.4.127) Requirement already satisfied: triton==3.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (3.2.0) Requirement already satisfied: sympy==1.13.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from torch>=1.13.0->peft>=0.12) (1.13.1) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from sympy==1.13.1->torch>=1.13.0->peft>=0.12) (1.3.0) Requirement already satisfied: click>=8.2.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (8.4.1) Requirement already satisfied: shellingham>=1.3.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (1.5.4) Requirement already satisfied: rich>=13.8.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (15.0.0) Requirement already satisfied: annotated-doc>=0.0.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from typer->transformers>=4.46) (0.0.4) Requirement already satisfied: markdown-it-py>=2.2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from rich>=13.8.0->typer->transformers>=4.46) (4.2.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from rich>=13.8.0->typer->transformers>=4.46) (2.20.0) Requirement already satisfied: mdurl~=0.1 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich>=13.8.0->typer->transformers>=4.46) (0.1.2) Requirement already satisfied: MarkupSafe>=2.0 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from jinja2->torch>=1.13.0->peft>=0.12) (3.0.3) Requirement already satisfied: python-dateutil>=2.8.2 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from pandas->datasets) (2.9.0.post0) Requirement already satisfied: six>=1.5 in /home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.17.0) [info] skipping flash-attn (default). Set INSTALL_FLASH_ATTN=1 to try. [info] downloading nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 -> /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 path=/home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 === Environment ready === env : nemo base model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 Next: conda activate nemo && bash run_train.sh === [Fri May 22 15:23:06 UTC 2026] Phase 1: smoke test (20 steps) === [stage 1] generating 100 examples per task -> /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl wrote 700 examples to /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl --- sample --- ### Task: cryptarithm ### Problem: Cryptarithm-style mapping. Examples: 14,51 -> 1451 13,96 -> 1396 11,68 -> 1168 Apply to: 42,37 ### Reasoning: <|cot_start|>Examples: 14,51 -> 1451 13,96 -> 1396 11,68 -> 1168 Check both candidate forms: concat(14,51)=1451 rev_concat(14,51)=5114 target=1451 concat concat(13,96)=1396 rev_concat(13,96)=9613 target=1396 concat concat(11,68)=1168 rev_concat(11,68)=6811 target=1168 concat Conclude operator: concat. Apply to (42, 37): answer = 4237 Final: \boxed{4237} [stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 [transformers] `torch_dtype` is deprecated! Use `dtype` instead! === TrainConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl out_dir: /home/lakesenberg/nemotron_lora_work/smoke lora_rank: 32 lora_alpha: 32 lora_dropout: 0.0 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2') lr: 0.0001 weight_decay: 0.0 betas: (0.9, 0.95) max_grad_norm: 1.0 warmup_steps: 50 total_steps: 20 grad_accum: 1 batch_size: 1 seq_len: 1024 save_every: 500 log_every: 10 seed: 0 bf16: True use_flash_attn: False gradient_checkpointing: True resume_adapter: None Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ... Traceback (most recent call last): File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 64, in from mamba_ssm.ops.triton.layernorm_gated import rmsnorm_fn ModuleNotFoundError: No module named 'mamba_ssm' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 315, in train(cfg) File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 221, in train model = build_model(cfg) ^^^^^^^^^^^^^^^^ File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 187, in build_model model = AutoModelForCausalLM.from_pretrained(cfg.base_model, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 379, in from_pretrained model_class = get_class_from_dynamic_module( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 627, in get_class_from_dynamic_module return get_class_in_module(class_name, final_module, force_reload=force_download) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 309, in get_class_in_module module_spec.loader.exec_module(module) File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 66, in raise ImportError("mamba-ssm is required by the Mamba model but cannot be imported") ImportError: mamba-ssm is required by the Mamba model but cannot be imported EXIT_CODE=1 === RESUME2 Fri May 22 16:37:57 UTC 2026 === === [Fri May 22 16:37:57 UTC 2026] RESUME: verify mamba-ssm === mamba_ssm OK === [Fri May 22 16:37:59 UTC 2026] Phase 1: smoke test (20 steps) === [stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl [stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 [transformers] `torch_dtype` is deprecated! Use `dtype` instead! === TrainConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl out_dir: /home/lakesenberg/nemotron_lora_work/smoke lora_rank: 32 lora_alpha: 32 lora_dropout: 0.0 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2') lr: 0.0001 weight_decay: 0.0 betas: (0.9, 0.95) max_grad_norm: 1.0 warmup_steps: 50 total_steps: 20 grad_accum: 1 batch_size: 1 seq_len: 1024 save_every: 500 log_every: 10 seed: 0 bf16: True use_flash_attn: False gradient_checkpointing: True resume_adapter: None Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ... Loading weights: 0%| | 0/6243 [00:00 train(cfg) File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 254, in train out = model(input_ids=input_ids, attention_mask=attn_mask, use_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1993, in forward return self.base_model( ^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 330, in forward return self.model.forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1702, in forward nemotron_h_outputs = self.backbone( ^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1489, in forward hidden_states = self._gradient_checkpointing_func( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner return disable_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint ret = function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 777, in forward hidden_states = self.mixer( ^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 715, in forward return self.cuda_kernels_forward(hidden_states, cache_params, cache_position, attention_mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 429, in cuda_kernels_forward out = mamba_split_conv1d_scan_combined( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 997, in mamba_split_conv1d_scan_combined return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 503, in decorate_fwd return fwd(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 841, in forward causal_conv1d_fwd_function(rearrange(ensure_stride(xBC), "b s d -> b d s"), TypeError: 'NoneType' object is not callable EXIT_CODE=1 === RESUME3 Fri May 22 16:59:38 UTC 2026 === === [Fri May 22 16:59:38 UTC 2026] patch Nemotron model cache (force slow mamba path) === === [Fri May 22 16:59:38 UTC 2026] RESUME: verify mamba-ssm === mamba_ssm OK === [Fri May 22 16:59:40 UTC 2026] Phase 1: smoke test (20 steps) === [stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl [stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 [transformers] `torch_dtype` is deprecated! Use `dtype` instead! === TrainConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl out_dir: /home/lakesenberg/nemotron_lora_work/smoke lora_rank: 32 lora_alpha: 32 lora_dropout: 0.0 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2') lr: 0.0001 weight_decay: 0.0 betas: (0.9, 0.95) max_grad_norm: 1.0 warmup_steps: 50 total_steps: 20 grad_accum: 1 batch_size: 1 seq_len: 1024 save_every: 500 log_every: 10 seed: 0 bf16: True use_flash_attn: False gradient_checkpointing: True resume_adapter: None Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ... Loading weights: 0%| | 0/6243 [00:00 train(cfg) File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 254, in train out = model(input_ids=input_ids, attention_mask=attn_mask, use_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1993, in forward return self.base_model( ^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 330, in forward return self.model.forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1702, in forward nemotron_h_outputs = self.backbone( ^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1489, in forward hidden_states = self._gradient_checkpointing_func( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner return disable_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint ret = function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 777, in forward hidden_states = self.mixer( ^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 715, in forward return self.cuda_kernels_forward(hidden_states, cache_params, cache_position, attention_mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 429, in cuda_kernels_forward out = mamba_split_conv1d_scan_combined( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 997, in mamba_split_conv1d_scan_combined return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 503, in decorate_fwd return fwd(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 841, in forward causal_conv1d_fwd_function(rearrange(ensure_stride(xBC), "b s d -> b d s"), TypeError: 'NoneType' object is not callable EXIT_CODE=1 === RESUME4 Fri May 22 17:02:26 UTC 2026 === === [Fri May 22 17:02:27 UTC 2026] patch Nemotron model cache (force slow mamba path) === === [Fri May 22 17:02:27 UTC 2026] RESUME: verify mamba-ssm === mamba_ssm OK === [Fri May 22 17:02:28 UTC 2026] Phase 1: smoke test (20 steps) === [stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl [stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 [transformers] `torch_dtype` is deprecated! Use `dtype` instead! === TrainConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl out_dir: /home/lakesenberg/nemotron_lora_work/smoke lora_rank: 32 lora_alpha: 32 lora_dropout: 0.0 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2') lr: 0.0001 weight_decay: 0.0 betas: (0.9, 0.95) max_grad_norm: 1.0 warmup_steps: 50 total_steps: 20 grad_accum: 1 batch_size: 1 seq_len: 1024 save_every: 500 log_every: 10 seed: 0 bf16: True use_flash_attn: False gradient_checkpointing: True resume_adapter: None Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ... Loading weights: 0%| | 0/6243 [00:00 train(cfg) File "/home/lakesenberg/code/nemotron_lora_train/train_nemotron_lora_h200.py", line 254, in train out = model(input_ids=input_ids, attention_mask=attn_mask, use_cache=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1993, in forward return self.base_model( ^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/tuners/tuners_utils.py", line 330, in forward return self.model.forward(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1702, in forward nemotron_h_outputs = self.backbone( ^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 1489, in forward hidden_states = self._gradient_checkpointing_func( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_compile.py", line 32, in inner return disable_fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 745, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/utils/checkpoint.py", line 496, in checkpoint ret = function(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 777, in forward hidden_states = self.mixer( ^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 715, in forward return self.cuda_kernels_forward(hidden_states, cache_params, cache_position, attention_mask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/.cache/huggingface/modules/transformers_modules/nemotron_hyphen_3_hyphen_nano_hyphen_30b_hyphen_a3b_hyphen_bf16/f06245fae7870619/modeling_nemotron_h.py", line 429, in cuda_kernels_forward out = mamba_split_conv1d_scan_combined( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 997, in mamba_split_conv1d_scan_combined return MambaSplitConv1dScanCombinedFn.apply(zxbcdt, conv1d_weight, conv1d_bias, dt_bias, A, D, chunk_size, initial_states, seq_idx, dt_limit, return_final_states, activation, rmsnorm_weight, rmsnorm_eps, outproj_weight, outproj_bias, headdim, ngroups, norm_before_gate) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply return super().apply(*args, **kwargs) # type: ignore[misc] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/torch/amp/autocast_mode.py", line 503, in decorate_fwd return fwd(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/mamba_ssm/ops/triton/ssd_combined.py", line 841, in forward causal_conv1d_fwd_function(rearrange(ensure_stride(xBC), "b s d -> b d s"), TypeError: 'NoneType' object is not callable EXIT_CODE=1 === RESUME5 Fri May 22 17:06:20 UTC 2026 === === [Fri May 22 17:06:20 UTC 2026] patch Nemotron model (force slow mamba path) === === [Fri May 22 17:06:20 UTC 2026] RESUME: verify mamba-ssm === mamba_ssm OK === [Fri May 22 17:06:21 UTC 2026] Phase 1: smoke test (20 steps) === [stage 1] data file exists, skipping: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl [stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 [transformers] `torch_dtype` is deprecated! Use `dtype` instead! The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d === TrainConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 data_path: /home/lakesenberg/nemotron_lora_work/smoke_cot.jsonl out_dir: /home/lakesenberg/nemotron_lora_work/smoke lora_rank: 32 lora_alpha: 32 lora_dropout: 0.0 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2') lr: 0.0001 weight_decay: 0.0 betas: (0.9, 0.95) max_grad_norm: 1.0 warmup_steps: 50 total_steps: 20 grad_accum: 1 batch_size: 1 seq_len: 1024 save_every: 500 log_every: 10 seed: 0 bf16: True use_flash_attn: False gradient_checkpointing: True resume_adapter: None Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ... Loading weights: 0%| | 0/6243 [00:00 /home/lakesenberg/nemotron_lora_work/smoke/step-00020 DONE. final adapter -> /home/lakesenberg/nemotron_lora_work/smoke/final === DONE === Final adapter: /home/lakesenberg/nemotron_lora_work/smoke/final Next: upload that folder to Kaggle as a Model dataset, then point CFG.tinker_adapter_path at it in the AA-SVD notebook. === [Fri May 22 17:09:11 UTC 2026] Phase 2: full training (2000 steps) === [stage 1] generating 4000 examples per task -> /home/lakesenberg/nemotron_lora_work/cot.jsonl wrote 28000 examples to /home/lakesenberg/nemotron_lora_work/cot.jsonl --- sample --- ### Task: numeral ### Problem: Convert the Roman numeral LXXVI to an integer. ### Reasoning: <|cot_start|>Reading symbols one by one: L = 50 -> 0 + 50 = 50 X = 10 -> 50 + 10 = 60 X = 10 -> 60 + 10 = 70 V = 5 -> 70 + 5 = 75 I = 1 -> 75 + 1 = 76 Final: \boxed{76} [stage 2] training LoRA on /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 [transformers] `torch_dtype` is deprecated! Use `dtype` instead! The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d === TrainConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 data_path: /home/lakesenberg/nemotron_lora_work/cot.jsonl out_dir: /home/lakesenberg/nemotron_lora_work/adapter lora_rank: 32 lora_alpha: 32 lora_dropout: 0.0 target_modules: ('q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj', 'x_proj', 'in_proj', 'out_proj', 'dt_proj', 'experts.w1', 'experts.w2') lr: 0.0001 weight_decay: 0.0 betas: (0.9, 0.95) max_grad_norm: 1.0 warmup_steps: 50 total_steps: 2000 grad_accum: 4 batch_size: 1 seq_len: 7680 save_every: 500 log_every: 10 seed: 0 bf16: True use_flash_attn: False gradient_checkpointing: True resume_adapter: None Loading /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 ... Loading weights: 0%| | 0/6243 [00:00 /home/lakesenberg/nemotron_lora_work/adapter/step-00500 step 510/2000 loss=0.0114 ema=0.0061 lr=8.69e-05 mem=69.0/86.8 GiB step 520/2000 loss=0.0041 ema=0.0037 lr=8.63e-05 mem=69.1/86.8 GiB step 530/2000 loss=0.0031 ema=0.0021 lr=8.58e-05 mem=69.5/86.8 GiB step 540/2000 loss=0.0002 ema=0.0017 lr=8.52e-05 mem=69.1/86.8 GiB step 550/2000 loss=0.0071 ema=0.0023 lr=8.46e-05 mem=69.5/86.8 GiB step 560/2000 loss=0.0042 ema=0.0025 lr=8.41e-05 mem=69.1/86.8 GiB step 570/2000 loss=0.0047 ema=0.0021 lr=8.35e-05 mem=69.1/86.8 GiB step 580/2000 loss=0.0015 ema=0.0019 lr=8.29e-05 mem=69.5/86.8 GiB step 590/2000 loss=0.0033 ema=0.0018 lr=8.22e-05 mem=69.1/86.8 GiB step 600/2000 loss=0.0103 ema=0.0033 lr=8.16e-05 mem=69.1/86.8 GiB step 610/2000 loss=0.0000 ema=0.0021 lr=8.10e-05 mem=68.9/86.8 GiB step 620/2000 loss=0.0027 ema=0.0019 lr=8.04e-05 mem=69.0/86.8 GiB step 630/2000 loss=0.0000 ema=0.0032 lr=7.97e-05 mem=68.8/86.8 GiB step 640/2000 loss=0.0004 ema=0.0027 lr=7.91e-05 mem=68.8/86.8 GiB step 650/2000 loss=0.0001 ema=0.0017 lr=7.84e-05 mem=69.1/86.8 GiB step 660/2000 loss=0.0000 ema=0.0079 lr=7.77e-05 mem=68.8/86.8 GiB step 670/2000 loss=0.0019 ema=0.0051 lr=7.71e-05 mem=68.8/86.8 GiB step 680/2000 loss=0.0001 ema=0.0028 lr=7.64e-05 mem=68.8/86.8 GiB step 690/2000 loss=0.0000 ema=0.0015 lr=7.57e-05 mem=69.0/86.8 GiB step 700/2000 loss=0.0000 ema=0.0016 lr=7.50e-05 mem=69.1/86.8 GiB step 710/2000 loss=0.0028 ema=0.0017 lr=7.43e-05 mem=68.8/86.8 GiB step 720/2000 loss=0.0036 ema=0.0015 lr=7.36e-05 mem=68.8/86.8 GiB step 730/2000 loss=0.0015 ema=0.0015 lr=7.29e-05 mem=68.9/86.8 GiB step 740/2000 loss=0.0055 ema=0.0020 lr=7.22e-05 mem=68.9/86.8 GiB step 750/2000 loss=0.0002 ema=0.0034 lr=7.14e-05 mem=68.8/86.8 GiB step 760/2000 loss=0.0033 ema=0.0022 lr=7.07e-05 mem=68.9/86.8 GiB step 770/2000 loss=0.0000 ema=0.0020 lr=7.00e-05 mem=68.9/86.8 GiB step 780/2000 loss=0.0000 ema=0.0013 lr=6.92e-05 mem=68.8/86.8 GiB step 790/2000 loss=0.0019 ema=0.0012 lr=6.85e-05 mem=69.5/86.8 GiB step 800/2000 loss=0.0030 ema=0.0019 lr=6.77e-05 mem=68.9/86.8 GiB step 810/2000 loss=0.0000 ema=0.0017 lr=6.70e-05 mem=68.8/86.8 GiB step 820/2000 loss=0.0000 ema=0.0012 lr=6.62e-05 mem=68.8/86.8 GiB step 830/2000 loss=0.0000 ema=0.0014 lr=6.55e-05 mem=69.1/86.8 GiB step 840/2000 loss=0.0015 ema=0.0021 lr=6.47e-05 mem=68.9/86.8 GiB step 850/2000 loss=0.0000 ema=0.0015 lr=6.39e-05 mem=68.8/86.8 GiB step 860/2000 loss=0.0000 ema=0.0011 lr=6.31e-05 mem=68.9/86.8 GiB step 870/2000 loss=0.0046 ema=0.0017 lr=6.24e-05 mem=69.0/86.8 GiB step 880/2000 loss=0.0027 ema=0.0018 lr=6.16e-05 mem=68.8/86.8 GiB step 890/2000 loss=0.0048 ema=0.0012 lr=6.08e-05 mem=69.1/86.8 GiB step 900/2000 loss=0.0018 ema=0.0012 lr=6.00e-05 mem=68.8/86.8 GiB step 910/2000 loss=0.0010 ema=0.0031 lr=5.92e-05 mem=69.6/86.8 GiB step 920/2000 loss=0.0019 ema=0.0023 lr=5.84e-05 mem=69.5/86.8 GiB step 930/2000 loss=0.0026 ema=0.0034 lr=5.76e-05 mem=69.1/86.8 GiB step 940/2000 loss=0.0023 ema=0.0020 lr=5.68e-05 mem=69.0/86.8 GiB step 950/2000 loss=0.0018 ema=0.0016 lr=5.60e-05 mem=69.1/86.8 GiB step 960/2000 loss=0.0072 ema=0.0020 lr=5.52e-05 mem=69.5/86.8 GiB step 970/2000 loss=0.0001 ema=0.0010 lr=5.44e-05 mem=68.9/86.8 GiB step 980/2000 loss=0.0000 ema=0.0016 lr=5.36e-05 mem=68.8/86.8 GiB step 990/2000 loss=0.0003 ema=0.0013 lr=5.28e-05 mem=69.0/86.8 GiB step 1000/2000 loss=0.0000 ema=0.0007 lr=5.20e-05 mem=69.0/86.8 GiB saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-01000 step 1010/2000 loss=0.0001 ema=0.0012 lr=5.12e-05 mem=68.9/86.8 GiB step 1020/2000 loss=0.0098 ema=0.0023 lr=5.04e-05 mem=69.0/86.8 GiB step 1030/2000 loss=0.0000 ema=0.0015 lr=4.96e-05 mem=68.9/86.8 GiB step 1040/2000 loss=0.0000 ema=0.0011 lr=4.88e-05 mem=68.8/86.8 GiB step 1050/2000 loss=0.0013 ema=0.0007 lr=4.80e-05 mem=69.5/86.8 GiB step 1060/2000 loss=0.0000 ema=0.0005 lr=4.72e-05 mem=68.9/86.8 GiB step 1070/2000 loss=0.0000 ema=0.0009 lr=4.64e-05 mem=68.8/86.8 GiB step 1080/2000 loss=0.0000 ema=0.0012 lr=4.56e-05 mem=68.8/86.8 GiB step 1090/2000 loss=0.0022 ema=0.0014 lr=4.48e-05 mem=68.8/86.8 GiB step 1100/2000 loss=0.0022 ema=0.0008 lr=4.40e-05 mem=69.0/86.8 GiB step 1110/2000 loss=0.0035 ema=0.0014 lr=4.32e-05 mem=69.5/86.8 GiB step 1120/2000 loss=0.0000 ema=0.0009 lr=4.24e-05 mem=68.9/86.8 GiB step 1130/2000 loss=0.0000 ema=0.0012 lr=4.16e-05 mem=69.1/86.8 GiB step 1140/2000 loss=0.0026 ema=0.0010 lr=4.08e-05 mem=69.1/87.9 GiB step 1150/2000 loss=0.0000 ema=0.0009 lr=4.00e-05 mem=68.9/87.9 GiB step 1160/2000 loss=0.0019 ema=0.0010 lr=3.92e-05 mem=69.5/87.9 GiB step 1170/2000 loss=0.0000 ema=0.0013 lr=3.84e-05 mem=68.8/87.9 GiB step 1180/2000 loss=0.0000 ema=0.0014 lr=3.76e-05 mem=68.9/87.9 GiB step 1190/2000 loss=0.0032 ema=0.0019 lr=3.69e-05 mem=69.1/87.9 GiB step 1200/2000 loss=0.0000 ema=0.0013 lr=3.61e-05 mem=68.9/87.9 GiB step 1210/2000 loss=0.0000 ema=0.0007 lr=3.53e-05 mem=68.8/87.9 GiB step 1220/2000 loss=0.0000 ema=0.0006 lr=3.45e-05 mem=68.9/87.9 GiB step 1230/2000 loss=0.0000 ema=0.0004 lr=3.38e-05 mem=68.9/87.9 GiB step 1240/2000 loss=0.0000 ema=0.0006 lr=3.30e-05 mem=68.9/87.9 GiB step 1250/2000 loss=0.0002 ema=0.0008 lr=3.23e-05 mem=69.1/87.9 GiB step 1260/2000 loss=0.0000 ema=0.0005 lr=3.15e-05 mem=68.8/87.9 GiB step 1270/2000 loss=0.0000 ema=0.0041 lr=3.08e-05 mem=68.8/87.9 GiB step 1280/2000 loss=0.0000 ema=0.0017 lr=3.00e-05 mem=69.1/87.9 GiB step 1290/2000 loss=0.0003 ema=0.0013 lr=2.93e-05 mem=69.0/87.9 GiB step 1300/2000 loss=0.0045 ema=0.0015 lr=2.86e-05 mem=69.1/87.9 GiB step 1310/2000 loss=0.0000 ema=0.0013 lr=2.78e-05 mem=68.8/87.9 GiB step 1320/2000 loss=0.0000 ema=0.0012 lr=2.71e-05 mem=69.1/87.9 GiB step 1330/2000 loss=0.0005 ema=0.0009 lr=2.64e-05 mem=69.1/87.9 GiB step 1340/2000 loss=0.0000 ema=0.0005 lr=2.57e-05 mem=68.9/87.9 GiB step 1350/2000 loss=0.0011 ema=0.0005 lr=2.50e-05 mem=68.8/87.9 GiB step 1360/2000 loss=0.0008 ema=0.0008 lr=2.43e-05 mem=69.1/87.9 GiB step 1370/2000 loss=0.0015 ema=0.0007 lr=2.36e-05 mem=69.5/87.9 GiB step 1380/2000 loss=0.0018 ema=0.0010 lr=2.29e-05 mem=69.5/87.9 GiB step 1390/2000 loss=0.0009 ema=0.0005 lr=2.23e-05 mem=68.9/87.9 GiB step 1400/2000 loss=0.0000 ema=0.0007 lr=2.16e-05 mem=68.9/87.9 GiB step 1410/2000 loss=0.0000 ema=0.0009 lr=2.09e-05 mem=68.9/87.9 GiB step 1420/2000 loss=0.0016 ema=0.0011 lr=2.03e-05 mem=69.5/87.9 GiB step 1430/2000 loss=0.0000 ema=0.0007 lr=1.96e-05 mem=68.9/87.9 GiB step 1440/2000 loss=0.0007 ema=0.0008 lr=1.90e-05 mem=69.5/87.9 GiB step 1450/2000 loss=0.0026 ema=0.0009 lr=1.84e-05 mem=69.5/87.9 GiB step 1460/2000 loss=0.0001 ema=0.0010 lr=1.78e-05 mem=68.9/87.9 GiB step 1470/2000 loss=0.0025 ema=0.0012 lr=1.71e-05 mem=69.1/87.9 GiB step 1480/2000 loss=0.0000 ema=0.0006 lr=1.65e-05 mem=68.9/87.9 GiB step 1490/2000 loss=0.0034 ema=0.0011 lr=1.59e-05 mem=69.0/87.9 GiB step 1500/2000 loss=0.0005 ema=0.0008 lr=1.54e-05 mem=69.0/87.9 GiB saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-01500 step 1510/2000 loss=0.0000 ema=0.0006 lr=1.48e-05 mem=69.1/87.9 GiB step 1520/2000 loss=0.0010 ema=0.0004 lr=1.42e-05 mem=69.5/87.9 GiB step 1530/2000 loss=0.0000 ema=0.0004 lr=1.37e-05 mem=68.8/87.9 GiB step 1540/2000 loss=0.0001 ema=0.0009 lr=1.31e-05 mem=69.1/87.9 GiB step 1550/2000 loss=0.0012 ema=0.0008 lr=1.26e-05 mem=68.9/87.9 GiB step 1560/2000 loss=0.0000 ema=0.0005 lr=1.20e-05 mem=68.8/87.9 GiB step 1570/2000 loss=0.0014 ema=0.0004 lr=1.15e-05 mem=69.1/87.9 GiB step 1580/2000 loss=0.0000 ema=0.0008 lr=1.10e-05 mem=68.8/87.9 GiB step 1590/2000 loss=0.0000 ema=0.0007 lr=1.05e-05 mem=69.0/87.9 GiB step 1600/2000 loss=0.0043 ema=0.0012 lr=1.00e-05 mem=69.5/87.9 GiB step 1610/2000 loss=0.0000 ema=0.0007 lr=9.55e-06 mem=68.8/87.9 GiB step 1620/2000 loss=0.0002 ema=0.0007 lr=9.08e-06 mem=69.0/87.9 GiB step 1630/2000 loss=0.0000 ema=0.0006 lr=8.62e-06 mem=69.1/87.9 GiB step 1640/2000 loss=0.0004 ema=0.0008 lr=8.18e-06 mem=69.0/87.9 GiB step 1650/2000 loss=0.0013 ema=0.0006 lr=7.74e-06 mem=69.5/87.9 GiB step 1660/2000 loss=0.0171 ema=0.0031 lr=7.32e-06 mem=69.1/87.9 GiB step 1670/2000 loss=0.0015 ema=0.0014 lr=6.90e-06 mem=69.5/87.9 GiB step 1680/2000 loss=0.0015 ema=0.0008 lr=6.50e-06 mem=68.8/87.9 GiB step 1690/2000 loss=0.0001 ema=0.0009 lr=6.11e-06 mem=68.8/87.9 GiB step 1700/2000 loss=0.0000 ema=0.0005 lr=5.73e-06 mem=69.0/87.9 GiB step 1710/2000 loss=0.0000 ema=0.0005 lr=5.36e-06 mem=68.8/87.9 GiB step 1720/2000 loss=0.0001 ema=0.0005 lr=5.00e-06 mem=69.1/87.9 GiB step 1730/2000 loss=0.0000 ema=0.0005 lr=4.66e-06 mem=69.1/87.9 GiB step 1740/2000 loss=0.0000 ema=0.0008 lr=4.32e-06 mem=68.8/87.9 GiB step 1750/2000 loss=0.0000 ema=0.0005 lr=4.00e-06 mem=68.9/87.9 GiB step 1760/2000 loss=0.0009 ema=0.0006 lr=3.69e-06 mem=69.6/87.9 GiB step 1770/2000 loss=0.0021 ema=0.0007 lr=3.39e-06 mem=69.5/87.9 GiB step 1780/2000 loss=0.0000 ema=0.0003 lr=3.11e-06 mem=68.8/87.9 GiB step 1790/2000 loss=0.0000 ema=0.0006 lr=2.83e-06 mem=68.8/87.9 GiB step 1800/2000 loss=0.0013 ema=0.0004 lr=2.57e-06 mem=68.9/87.9 GiB step 1810/2000 loss=0.0000 ema=0.0005 lr=2.32e-06 mem=68.9/87.9 GiB step 1820/2000 loss=0.0030 ema=0.0006 lr=2.09e-06 mem=69.1/87.9 GiB step 1830/2000 loss=0.0016 ema=0.0008 lr=1.86e-06 mem=69.5/87.9 GiB step 1840/2000 loss=0.0000 ema=0.0005 lr=1.65e-06 mem=68.8/87.9 GiB step 1850/2000 loss=0.0020 ema=0.0006 lr=1.45e-06 mem=69.0/87.9 GiB step 1860/2000 loss=0.0000 ema=0.0005 lr=1.27e-06 mem=68.9/87.9 GiB step 1870/2000 loss=0.0000 ema=0.0005 lr=1.09e-06 mem=68.9/87.9 GiB step 1880/2000 loss=0.0000 ema=0.0004 lr=9.31e-07 mem=68.9/87.9 GiB step 1890/2000 loss=0.0000 ema=0.0003 lr=7.83e-07 mem=68.9/87.9 GiB step 1900/2000 loss=0.0001 ema=0.0008 lr=6.47e-07 mem=69.1/87.9 GiB step 1910/2000 loss=0.0000 ema=0.0008 lr=5.25e-07 mem=68.9/87.9 GiB step 1920/2000 loss=0.0000 ema=0.0006 lr=4.15e-07 mem=68.9/87.9 GiB step 1930/2000 loss=0.0008 ema=0.0004 lr=3.18e-07 mem=69.5/87.9 GiB step 1940/2000 loss=0.0000 ema=0.0006 lr=2.33e-07 mem=68.8/87.9 GiB step 1950/2000 loss=0.0000 ema=0.0008 lr=1.62e-07 mem=69.0/87.9 GiB step 1960/2000 loss=0.0015 ema=0.0006 lr=1.04e-07 mem=69.5/87.9 GiB step 1970/2000 loss=0.0015 ema=0.0011 lr=5.84e-08 mem=69.0/87.9 GiB step 1980/2000 loss=0.0002 ema=0.0008 lr=2.60e-08 mem=69.0/87.9 GiB step 1990/2000 loss=0.0000 ema=0.0006 lr=6.49e-09 mem=69.1/87.9 GiB step 2000/2000 loss=0.0000 ema=0.0003 lr=0.00e+00 mem=68.8/87.9 GiB saved checkpoint -> /home/lakesenberg/nemotron_lora_work/adapter/step-02000 DONE. final adapter -> /home/lakesenberg/nemotron_lora_work/adapter/final === DONE === Final adapter: /home/lakesenberg/nemotron_lora_work/adapter/final Next: upload that folder to Kaggle as a Model dataset, then point CFG.tinker_adapter_path at it in the AA-SVD notebook. === [Sat May 23 09:28:53 UTC 2026] Phase 3: smoke eval === [transformers] `torch_dtype` is deprecated! Use `dtype` instead! The fast path is not available because on of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d === EvalConfig === base_model: /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 adapter: /home/lakesenberg/nemotron_lora_work/adapter/final out_dir: /home/lakesenberg/nemotron_lora_work/eval_smoke n_per_task: 3 max_new_tokens: 256 batch_size: 2 seed: 0 tasks: ('numeral', 'unit', 'gravity', 'cipher', 'bit', 'equation', 'cryptarithm') use_flash_attn: False bf16: True repetition_threshold: 0.3 [load] base = /home/lakesenberg/models/nemotron-3-nano-30b-a3b-bf16 Loading weights: 0%| | 0/6243 [00:00 summary = evaluate(cfg) ^^^^^^^^^^^^^ File "/home/lakesenberg/code/nemotron_lora_train/evaluate_adapter.py", line 230, in evaluate model, tokenizer = load_model(cfg) ^^^^^^^^^^^^^^^ File "/home/lakesenberg/code/nemotron_lora_train/evaluate_adapter.py", line 183, in load_model model = PeftModel.from_pretrained(model, cfg.adapter) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 582, in from_pretrained load_result = model.load_adapter( ^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/peft_model.py", line 1408, in load_adapter load_result = set_peft_model_state_dict( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/utils/save_and_load.py", line 644, in set_peft_model_state_dict state_dict = convert_peft_adapter_state_dict_for_transformers( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/utils/transformers_weight_conversion.py", line 509, in convert_peft_adapter_state_dict_for_transformers peft_weight_conversions = build_peft_weight_mapping( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/lakesenberg/miniforge3/envs/nemo/lib/python3.11/site-packages/peft/utils/transformers_weight_conversion.py", line 310, in build_peft_weight_mapping new_conversion = orig_conversion.__class__( ^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: WeightConverter.__init__() got an unexpected keyword argument 'distributed_operation' EXIT_CODE=1