Xseg-Baseline / 9.log
YuqianFu's picture
Upload folder using huggingface_hub
944cdc2 verified
Solving dependencies
Installing conda packages
Empty environment created at prefix: /scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd
error libmamba Could not lock non-existing path '/scratch/yuqian_fu/micromamba/pkgs'
Transaction
Prefix: /scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd
No specs added or removed.
Package Version Build Channel Size
─────────────────────────────────────────────────────────────────────────────────────────────
Install:
─────────────────────────────────────────────────────────────────────────────────────────────
+ _libgcc_mutex 0.1 conda_forge conda-forge
+ _openmp_mutex 4.5 2_gnu conda-forge
+ bzip2 1.0.8 h4bc722e_7 conda-forge
+ ca-certificates 2024.8.30 hbcca054_0 conda-forge
+ ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
+ libexpat 2.6.2 h59595ed_0 conda-forge
+ libffi 3.4.2 h7f98852_5 conda-forge
+ libgcc 14.1.0 h77fa898_1 conda-forge
+ libgcc-ng 14.1.0 h69a702a_1 conda-forge
+ libgomp 14.1.0 h77fa898_1 conda-forge
+ libnsl 2.0.1 hd590300_0 conda-forge
+ libsqlite 3.46.1 hadc24fc_0 conda-forge
+ libuuid 2.38.1 h0b41bf4_0 conda-forge
+ libxcrypt 4.4.36 hd590300_1 conda-forge
+ libzlib 1.3.1 h4ab18f5_1 conda-forge
+ ncurses 6.5 he02047a_1 conda-forge
+ openssl 3.3.2 hb9d3cd8_0 conda-forge
+ pip 24.2 pyh8b19718_1 conda-forge
+ python 3.11.9 hb806964_0_cpython /work/conda/cache/conda-forge
+ readline 8.2 h8228510_1 conda-forge
+ setuptools 73.0.1 pyhd8ed1ab_0 conda-forge
+ tk 8.6.13 noxft_h4845f30_101 /work/conda/cache/conda-forge
+ tzdata 2024a h8827d51_1 conda-forge
+ wheel 0.44.0 pyhd8ed1ab_0 conda-forge
+ xz 5.2.6 h166bdaf_0 conda-forge
Summary:
Install: 25 packages
Total download: 0 B
─────────────────────────────────────────────────────────────────────────────────────────────
Transaction starting
Linking _libgcc_mutex-0.1-conda_forge
Linking ld_impl_linux-64-2.40-hf3520f5_7
Linking ca-certificates-2024.8.30-hbcca054_0
Linking libgomp-14.1.0-h77fa898_1
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-14.1.0-h77fa898_1
Linking openssl-3.3.2-hb9d3cd8_0
Linking libgcc-ng-14.1.0-h69a702a_1
Linking libffi-3.4.2-h7f98852_5
Linking libzlib-1.3.1-h4ab18f5_1
Linking libxcrypt-4.4.36-hd590300_1
Linking bzip2-1.0.8-h4bc722e_7
Linking ncurses-6.5-he02047a_1
Linking xz-5.2.6-h166bdaf_0
Linking libuuid-2.38.1-h0b41bf4_0
Linking libnsl-2.0.1-hd590300_0
Linking libexpat-2.6.2-h59595ed_0
Linking tk-8.6.13-noxft_h4845f30_101
Linking libsqlite-3.46.1-hadc24fc_0
Linking readline-8.2-h8228510_1
Linking tzdata-2024a-h8827d51_1
Linking python-3.11.9-hb806964_0_cpython
Linking pip-24.2-pyh8b19718_1
Linking setuptools-73.0.1-pyhd8ed1ab_0
Linking wheel-0.44.0-pyhd8ed1ab_0
Transaction finished
To activate this environment, use:
mamba activate auto-gfmiyxa3evbd
Or to execute a single command in this environment, use:
mamba run -n auto-gfmiyxa3evbd mycommand
Installing pip packages
Collecting absl-py==1.4.0
Downloading absl_py-1.4.0-py3-none-any.whl.metadata (2.3 kB)
Collecting appdirs==1.4.4
Downloading appdirs-1.4.4-py2.py3-none-any.whl.metadata (9.0 kB)
Collecting cachetools==5.3.1
Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.5.7
Downloading certifi-2023.5.7-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.1.0
Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (30 kB)
Collecting click==8.1.4
Downloading click-8.1.4-py3-none-any.whl.metadata (3.0 kB)
Collecting cmake==3.26.4
Downloading cmake-3.26.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (6.7 kB)
Collecting contourpy==1.1.0
Downloading contourpy-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.7 kB)
Collecting cycler==0.11.0
Downloading cycler-0.11.0-py3-none-any.whl.metadata (785 bytes)
Collecting debugpy==1.5.1
Downloading debugpy-1.5.1-py2.py3-none-any.whl.metadata (1.2 kB)
Collecting decord==0.6.0
Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl.metadata (422 bytes)
Collecting docker-pycreds==0.4.0
Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting filelock==3.12.2
Downloading filelock-3.12.2-py3-none-any.whl.metadata (2.7 kB)
Collecting fonttools==4.40.0
Downloading fonttools-4.40.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (148 kB)
Collecting gitdb==4.0.10
Downloading gitdb-4.0.10-py3-none-any.whl.metadata (1.1 kB)
Collecting gitpython==3.1.31
Downloading GitPython-3.1.31-py3-none-any.whl.metadata (1.3 kB)
Collecting google-auth-oauthlib==1.0.0
Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting google-auth==2.22.0
Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting grpcio==1.56.2
Downloading grpcio-1.56.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting idna==3.4
Downloading idna-3.4-py3-none-any.whl.metadata (9.8 kB)
Collecting imageio==2.31.5
Downloading imageio-2.31.5-py3-none-any.whl.metadata (4.6 kB)
Collecting jinja2==3.1.2
Downloading Jinja2-3.1.2-py3-none-any.whl.metadata (3.5 kB)
Collecting joblib==1.2.0
Downloading joblib-1.2.0-py3-none-any.whl.metadata (5.3 kB)
Collecting jupyter-core==5.3.1
Downloading jupyter_core-5.3.1-py3-none-any.whl.metadata (3.4 kB)
Collecting kiwisolver==1.4.4
Downloading kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.3 kB)
Collecting kornia==0.6.12
Downloading kornia-0.6.12-py2.py3-none-any.whl.metadata (12 kB)
Collecting lazy-loader==0.3
Downloading lazy_loader-0.3-py3-none-any.whl.metadata (4.3 kB)
Collecting lit==16.0.6
Downloading lit-16.0.6.tar.gz (153 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting lzstring
Downloading lzstring-1.0.4.tar.gz (4.3 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting markdown==3.4.3
Downloading Markdown-3.4.3-py3-none-any.whl.metadata (6.7 kB)
Collecting markupsafe==2.1.3
Downloading MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting matplotlib==3.7.1
Downloading matplotlib-3.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Collecting mpmath==1.3.0
Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)
Collecting natsort==8.4.0
Downloading natsort-8.4.0-py3-none-any.whl.metadata (21 kB)
Collecting networkx==3.1
Downloading networkx-3.1-py3-none-any.whl.metadata (5.3 kB)
Collecting numpy==1.25.0
Downloading numpy-1.25.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Collecting nvidia-cublas-cu11==11.10.3.66
Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cuda-cupti-cu11==11.7.101
Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cuda-nvrtc-cu11==11.7.99
Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-cuda-runtime-cu11==11.7.99
Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cudnn-cu11==8.5.0.96
Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cufft-cu11==10.9.0.58
Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting nvidia-curand-cu11==10.2.10.91
Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusolver-cu11==11.4.0.1
Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-cusparse-cu11==11.7.4.91
Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl.metadata (1.6 kB)
Collecting nvidia-nccl-cu11==2.14.3
Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl.metadata (1.8 kB)
Collecting nvidia-nvtx-cu11==11.7.91
Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl.metadata (1.7 kB)
Collecting oauthlib==3.2.2
Downloading oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB)
Collecting opencv-python==4.7.0.72
Downloading opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Collecting pandas==2.0.3
Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (18 kB)
Collecting pathtools==0.1.2
Downloading pathtools-0.1.2.tar.gz (11 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting pillow==9.5.0
Downloading Pillow-9.5.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (9.5 kB)
Collecting pip==23.1.2
Downloading pip-23.1.2-py3-none-any.whl.metadata (4.1 kB)
Collecting protobuf==4.23.4
Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting psutil==5.9.0
Downloading psutil-5.9.0.tar.gz (478 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting pyasn1-modules==0.3.0
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl.metadata (3.6 kB)
Collecting pyasn1==0.5.0
Downloading pyasn1-0.5.0-py2.py3-none-any.whl.metadata (8.5 kB)
Collecting pycocotools==2.0.6
Downloading pycocotools-2.0.6.tar.gz (24 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status 'done'
Collecting pyparsing==3.1.0
Downloading pyparsing-3.1.0-py3-none-any.whl.metadata (4.9 kB)
Collecting pytz==2023.3
Downloading pytz-2023.3-py2.py3-none-any.whl.metadata (22 kB)
Collecting pyyaml==6.0
Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
Collecting pyzmq==25.1.0
Downloading pyzmq-25.1.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (4.8 kB)
Collecting requests-oauthlib==1.3.1
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl.metadata (10 kB)
Collecting requests==2.31.0
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting rsa==4.9
Downloading rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Collecting scikit-image==0.22.0
Downloading scikit_image-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (13 kB)
Collecting scikit-learn==1.2.2
Downloading scikit_learn-1.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (11 kB)
Collecting scipy==1.11.0
Downloading scipy-1.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (59 kB)
Collecting sentry-sdk==1.27.1
Downloading sentry_sdk-1.27.1-py2.py3-none-any.whl.metadata (8.8 kB)
Collecting setproctitle==1.3.2
Downloading setproctitle-1.3.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.8 kB)
Collecting setuptools==67.8.0
Downloading setuptools-67.8.0-py3-none-any.whl.metadata (6.2 kB)
Collecting smmap==5.0.0
Downloading smmap-5.0.0-py3-none-any.whl.metadata (4.2 kB)
Collecting sympy==1.12
Downloading sympy-1.12-py3-none-any.whl.metadata (12 kB)
Collecting tensorboard-data-server==0.7.1
Downloading tensorboard_data_server-0.7.1-py3-none-manylinux2014_x86_64.whl.metadata (1.1 kB)
Collecting tensorboard==2.13.0
Downloading tensorboard-2.13.0-py3-none-any.whl.metadata (1.8 kB)
Collecting tensorboardx==2.6.1
Downloading tensorboardX-2.6.1-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting threadpoolctl==3.1.0
Downloading threadpoolctl-3.1.0-py3-none-any.whl.metadata (9.2 kB)
Collecting tifffile==2023.9.26
Downloading tifffile-2023.9.26-py3-none-any.whl.metadata (30 kB)
Collecting torch==2.0.1
Downloading torch-2.0.1-cp311-cp311-manylinux1_x86_64.whl.metadata (24 kB)
Collecting torchvision==0.15.2
Downloading torchvision-0.15.2-cp311-cp311-manylinux1_x86_64.whl.metadata (11 kB)
Collecting tornado==6.1
Downloading tornado-6.1.tar.gz (497 kB)
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status 'done'
Collecting tqdm==4.65.0
Downloading tqdm-4.65.0-py3-none-any.whl.metadata (56 kB)
Collecting triton==2.0.0
Downloading triton-2.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.0 kB)
Collecting tzdata==2023.3
Downloading tzdata-2023.3-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting urllib3==1.26.16
Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
Collecting wandb==0.15.5
Downloading wandb-0.15.5-py3-none-any.whl.metadata (8.2 kB)
Collecting werkzeug==2.3.6
Downloading Werkzeug-2.3.6-py3-none-any.whl.metadata (4.1 kB)
Collecting wheel==0.38.4
Downloading wheel-0.38.4-py3-none-any.whl.metadata (2.1 kB)
Collecting six>=1.4.0 (from docker-pycreds==0.4.0)
Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting platformdirs>=2.5 (from jupyter-core==5.3.1)
Downloading platformdirs-4.2.2-py3-none-any.whl.metadata (11 kB)
Collecting traitlets>=5.3 (from jupyter-core==5.3.1)
Downloading traitlets-5.14.3-py3-none-any.whl.metadata (10 kB)
Collecting packaging (from kornia==0.6.12)
Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting python-dateutil>=2.7 (from matplotlib==3.7.1)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting typing-extensions (from torch==2.0.1)
Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Collecting future>=0.14.0 (from lzstring)
Downloading future-1.0.0-py3-none-any.whl.metadata (4.0 kB)
WARNING: The candidate selected for download or install is a yanked version: 'scipy' candidate (version 1.11.0 at https://files.pythonhosted.org/packages/27/10/985c0902929a424c8e0483f16ee54f92fc5e4fa07fbd83a36f3aa469b3fe/scipy-1.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/scipy/) (requires-python:<3.13,>=3.9))
Reason for being yanked: License Violation
Downloading absl_py-1.4.0-py3-none-any.whl (126 kB)
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Downloading certifi-2023.5.7-py3-none-any.whl (156 kB)
Downloading charset_normalizer-3.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (197 kB)
Downloading click-8.1.4-py3-none-any.whl (98 kB)
Downloading cmake-3.26.4-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (24.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.0/24.0 MB 201.3 MB/s eta 0:00:00
Downloading contourpy-1.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (300 kB)
Downloading cycler-0.11.0-py3-none-any.whl (6.4 kB)
Downloading debugpy-1.5.1-py2.py3-none-any.whl (4.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 223.4 MB/s eta 0:00:00
Downloading decord-0.6.0-py3-none-manylinux2010_x86_64.whl (13.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.6/13.6 MB 298.9 MB/s eta 0:00:00
Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Downloading filelock-3.12.2-py3-none-any.whl (10 kB)
Downloading fonttools-4.40.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.3/4.3 MB 11.5 MB/s eta 0:00:00
Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)
Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)
Downloading google_auth_oauthlib-1.0.0-py2.py3-none-any.whl (18 kB)
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
Downloading grpcio-1.56.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.2/5.2 MB 44.5 MB/s eta 0:00:00
Downloading idna-3.4-py3-none-any.whl (61 kB)
Downloading imageio-2.31.5-py3-none-any.whl (313 kB)
Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
Downloading joblib-1.2.0-py3-none-any.whl (297 kB)
Downloading jupyter_core-5.3.1-py3-none-any.whl (93 kB)
Downloading kiwisolver-1.4.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.4/1.4 MB 172.0 MB/s eta 0:00:00
Downloading kornia-0.6.12-py2.py3-none-any.whl (653 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 653.4/653.4 kB 62.5 MB/s eta 0:00:00
Downloading lazy_loader-0.3-py3-none-any.whl (9.1 kB)
Downloading Markdown-3.4.3-py3-none-any.whl (93 kB)
Downloading MarkupSafe-2.1.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Downloading matplotlib-3.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.6/11.6 MB 104.5 MB/s eta 0:00:00
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 100.7 MB/s eta 0:00:00
Downloading natsort-8.4.0-py3-none-any.whl (38 kB)
Downloading networkx-3.1-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 199.2 MB/s eta 0:00:00
Downloading numpy-1.25.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 17.6/17.6 MB 106.4 MB/s eta 0:00:00
Downloading nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 317.1/317.1 MB 160.2 MB/s eta 0:00:00
Downloading nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 11.8/11.8 MB 70.9 MB/s eta 0:00:00
Downloading nvidia_cuda_nvrtc_cu11-11.7.99-2-py3-none-manylinux1_x86_64.whl (21.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.0/21.0 MB 269.9 MB/s eta 0:00:00
Downloading nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 849.3/849.3 kB 156.8 MB/s eta 0:00:00
Downloading nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 557.1/557.1 MB 117.8 MB/s eta 0:00:00
Downloading nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux2014_x86_64.whl (168.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.4/168.4 MB 152.6 MB/s eta 0:00:00
Downloading nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.6/54.6 MB 278.2 MB/s eta 0:00:00
Downloading nvidia_cusolver_cu11-11.4.0.1-2-py3-none-manylinux1_x86_64.whl (102.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.6/102.6 MB 144.2 MB/s eta 0:00:00
Downloading nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 173.2/173.2 MB 167.0 MB/s eta 0:00:00
Downloading nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.1/177.1 MB 278.1 MB/s eta 0:00:00
Downloading nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
Downloading opencv_python-4.7.0.72-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.8/61.8 MB 137.4 MB/s eta 0:00:00
Downloading pandas-2.0.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 299.1 MB/s eta 0:00:00
Downloading Pillow-9.5.0-cp311-cp311-manylinux_2_28_x86_64.whl (3.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 188.0 MB/s eta 0:00:00
Downloading pip-23.1.2-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 184.7 MB/s eta 0:00:00
Downloading protobuf-4.23.4-cp37-abi3-manylinux2014_x86_64.whl (304 kB)
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
Downloading pyparsing-3.1.0-py3-none-any.whl (102 kB)
Downloading pytz-2023.3-py2.py3-none-any.whl (502 kB)
Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.9/757.9 kB 152.1 MB/s eta 0:00:00
Downloading pyzmq-25.1.0-cp311-cp311-manylinux_2_28_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 180.9 MB/s eta 0:00:00
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Downloading scikit_image-0.22.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (14.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.7/14.7 MB 282.4 MB/s eta 0:00:00
Downloading scikit_learn-1.2.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (9.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 9.6/9.6 MB 96.8 MB/s eta 0:00:00
Downloading scipy-1.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.2 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 36.2/36.2 MB 24.4 MB/s eta 0:00:00
Downloading sentry_sdk-1.27.1-py2.py3-none-any.whl (211 kB)
Downloading setproctitle-1.3.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (31 kB)
Downloading setuptools-67.8.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 152.8 MB/s eta 0:00:00
Downloading smmap-5.0.0-py3-none-any.whl (24 kB)
Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 126.3 MB/s eta 0:00:00
Downloading tensorboard_data_server-0.7.1-py3-none-manylinux2014_x86_64.whl (6.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 282.9 MB/s eta 0:00:00
Downloading tensorboard-2.13.0-py3-none-any.whl (5.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 211.9 MB/s eta 0:00:00
Downloading tensorboardX-2.6.1-py2.py3-none-any.whl (101 kB)
Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Downloading tifffile-2023.9.26-py3-none-any.whl (222 kB)
Downloading torch-2.0.1-cp311-cp311-manylinux1_x86_64.whl (619.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 619.9/619.9 MB 60.2 MB/s eta 0:00:00
Downloading torchvision-0.15.2-cp311-cp311-manylinux1_x86_64.whl (6.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.0/6.0 MB 65.2 MB/s eta 0:00:00
Downloading tqdm-4.65.0-py3-none-any.whl (77 kB)
Downloading triton-2.0.0-1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.3/63.3 MB 104.8 MB/s eta 0:00:00
Downloading tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
Downloading wandb-0.15.5-py3-none-any.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 265.9 MB/s eta 0:00:00
Downloading Werkzeug-2.3.6-py3-none-any.whl (242 kB)
Downloading wheel-0.38.4-py3-none-any.whl (36 kB)
Downloading future-1.0.0-py3-none-any.whl (491 kB)
Downloading packaging-24.1-py3-none-any.whl (53 kB)
Downloading platformdirs-4.2.2-py3-none-any.whl (18 kB)
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading traitlets-5.14.3-py3-none-any.whl (85 kB)
Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Building wheels for collected packages: lit, pathtools, psutil, pycocotools, tornado, lzstring
Building wheel for lit (pyproject.toml): started
Building wheel for lit (pyproject.toml): finished with status 'done'
Created wheel for lit: filename=lit-16.0.6-py3-none-any.whl size=93586 sha256=2fc0f4da9619296241c3540c311e37bbc1817c9a3733bf3a6a0fef272f60939a
Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/ab/84/e4/5af8c76af9e5bee472e825f1451c18bb3b261d80a7b3ec7f8a
Building wheel for pathtools (setup.py): started
Building wheel for pathtools (setup.py): finished with status 'done'
Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8791 sha256=c159257a6daf4d9c9a17d19386e0faaa5dc7753be7c0ef36e1434f17b0957fe7
Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/ea/b7/8b/84e94095ea418b9442f5abeba4ca7b0ad52d3fe7b69d6238a6
Building wheel for psutil (setup.py): started
Building wheel for psutil (setup.py): finished with status 'done'
Created wheel for psutil: filename=psutil-5.9.0-cp311-cp311-linux_x86_64.whl size=230265 sha256=b88ee6fba7b6d6be83ff34993629c84da6c6611417aa3214429079a872287eed
Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/b7/3f/a5/6da60cf045cc7d875f3f6746c27e9afa159642fc5acaa5ae08
Building wheel for pycocotools (pyproject.toml): started
Building wheel for pycocotools (pyproject.toml): finished with status 'done'
Created wheel for pycocotools: filename=pycocotools-2.0.6-cp311-cp311-linux_x86_64.whl size=104816 sha256=72aac5487d6718737633f38298519705c39e716d4efff67132801dcd2dd1159a
Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/ad/ca/ea/fb115e04c841c3f71fd369b7d9805a43a5193f4f9251bed0ec
Building wheel for tornado (setup.py): started
Building wheel for tornado (setup.py): finished with status 'done'
Created wheel for tornado: filename=tornado-6.1-cp311-cp311-linux_x86_64.whl size=416969 sha256=3d6b1c8473d83655df9c18572a71fc93230c8907cd00a1a497dcb690e7e4ed6c
Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/f2/59/06/a9c85c7b17ec0fc9b1e2ae0c59e3d39255d5c0a38492e33fea
Building wheel for lzstring (setup.py): started
Building wheel for lzstring (setup.py): finished with status 'done'
Created wheel for lzstring: filename=lzstring-1.0.4-py2.py3-none-any.whl size=4568 sha256=6055ffda849cd05b517aeff466d1110f2cc7133b4267a764ba2685fed07b971f
Stored in directory: /scratch/yuqian_fu/.cache/pip/wheels/a2/68/ae/ab2520f1ce001476a22df0405a1066359d6c4f14aad5f6aae1
Successfully built lit pathtools psutil pycocotools tornado lzstring
Installing collected packages: pytz, pathtools, mpmath, lit, cmake, appdirs, wheel, urllib3, tzdata, typing-extensions, traitlets, tqdm, tornado, threadpoolctl, tensorboard-data-server, sympy, smmap, six, setuptools, setproctitle, pyzmq, pyyaml, pyparsing, pyasn1, psutil, protobuf, platformdirs, pip, pillow, packaging, oauthlib, nvidia-nccl-cu11, nvidia-cufft-cu11, nvidia-cuda-nvrtc-cu11, numpy, networkx, natsort, markupsafe, markdown, lazy-loader, kiwisolver, joblib, idna, grpcio, future, fonttools, filelock, debugpy, cycler, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tifffile, tensorboardx, sentry-sdk, scipy, rsa, requests, python-dateutil, pyasn1-modules, opencv-python, nvidia-nvtx-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, lzstring, jupyter-core, jinja2, imageio, gitdb, docker-pycreds, decord, contourpy, scikit-learn, scikit-image, requests-oauthlib, pandas, nvidia-cusolver-cu11, nvidia-cudnn-cu11, matplotlib, google-auth, gitpython, wandb, pycocotools, google-auth-oauthlib, tensorboard, triton, torch, torchvision, kornia
Attempting uninstall: wheel
Found existing installation: wheel 0.44.0
Uninstalling wheel-0.44.0:
Successfully uninstalled wheel-0.44.0
Attempting uninstall: setuptools
Found existing installation: setuptools 73.0.1
Uninstalling setuptools-73.0.1:
Successfully uninstalled setuptools-73.0.1
Attempting uninstall: pip
Found existing installation: pip 24.2
Uninstalling pip-24.2:
Successfully uninstalled pip-24.2
Successfully installed absl-py-1.4.0 appdirs-1.4.4 cachetools-5.3.1 certifi-2023.5.7 charset-normalizer-3.1.0 click-8.1.4 cmake-3.26.4 contourpy-1.1.0 cycler-0.11.0 debugpy-1.5.1 decord-0.6.0 docker-pycreds-0.4.0 filelock-3.12.2 fonttools-4.40.0 future-1.0.0 gitdb-4.0.10 gitpython-3.1.31 google-auth-2.22.0 google-auth-oauthlib-1.0.0 grpcio-1.56.2 idna-3.4 imageio-2.31.5 jinja2-3.1.2 joblib-1.2.0 jupyter-core-5.3.1 kiwisolver-1.4.4 kornia-0.6.12 lazy-loader-0.3 lit-16.0.6 lzstring-1.0.4 markdown-3.4.3 markupsafe-2.1.3 matplotlib-3.7.1 mpmath-1.3.0 natsort-8.4.0 networkx-3.1 numpy-1.25.0 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 oauthlib-3.2.2 opencv-python-4.7.0.72 packaging-24.1 pandas-2.0.3 pathtools-0.1.2 pillow-9.5.0 pip-23.1.2 platformdirs-4.2.2 protobuf-4.23.4 psutil-5.9.0 pyasn1-0.5.0 pyasn1-modules-0.3.0 pycocotools-2.0.6 pyparsing-3.1.0 python-dateutil-2.9.0.post0 pytz-2023.3 pyyaml-6.0 pyzmq-25.1.0 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 scikit-image-0.22.0 scikit-learn-1.2.2 scipy-1.11.0 sentry-sdk-1.27.1 setproctitle-1.3.2 setuptools-67.8.0 six-1.16.0 smmap-5.0.0 sympy-1.12 tensorboard-2.13.0 tensorboard-data-server-0.7.1 tensorboardx-2.6.1 threadpoolctl-3.1.0 tifffile-2023.9.26 torch-2.0.1 torchvision-0.15.2 tornado-6.1 tqdm-4.65.0 traitlets-5.14.3 triton-2.0.0 typing-extensions-4.12.2 tzdata-2023.3 urllib3-1.26.16 wandb-0.15.5 werkzeug-2.3.6 wheel-0.38.4
[notice] A new release of pip is available: 23.1.2 -> 24.2
[notice] To update, run: pip install --upgrade pip
2024-09-04 01:49:17,020 INFO Namespace(n_epoch=250, lr_schedule=[50], lr=0.0002, out_dir='/data/work-gcp-europe-west4-a/yuqian_fu/Ego/checkpoints/egoexo_v2_480x480', train_dir=['/data/work-gcp-europe-west4-a/yuqian_fu/Ego/data_segswap'], prob_dir=[0.5, 0.5], batch_pos=32, batch_neg=15, feat_pth='../evalBrueghel/Moco_resnet50_feat_1Scale_640p.pkl', warp_mask=False, warmUpIter=1000, resume_pth=None, resume_epoch=0, mode='small', pos_weight=0.1, feat_weight=1, dropout=0.1, activation='relu', prob_style=0.5, layer_type=['I', 'C', 'I', 'C', 'I', 'N'], drop_feat=0.1, tps_grid=[4, 6], eta_corr=8, iter_epoch=1000, iter_epoch_val=100, weight_decay=0, reverse=False)
2024-09-04 01:49:17,021 INFO Load MocoV2 pre-trained ResNet-50 feature...
LOADING: train_egoexo_pairs.json
LOADING: val_egoexo_pairs.json
0%| | 0/1000 [00:00<?, ?it/s] 0%| | 0/1000 [00:07<?, ?it/s]
Traceback (most recent call last):
File "/home/yuqian_fu/Projects/ego-exo4d-relation/correspondence/SegSwap/train/Main.py", line 195, in <module>
backbone, netEncoder, optimizer, history = Train.trainEpoch(trainLoader, backbone, netEncoder, optimizer, history,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuqian_fu/Projects/ego-exo4d-relation/correspondence/SegSwap/train/Train.py", line 80, in trainEpoch
O1, O2, O3 = netEncoder(X, Y, FMTX, RS, RT)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuqian_fu/Projects/ego-exo4d-relation/correspondence/SegSwap/model/transformer.py", line 342, in forward
outx, outy, out_cls = self.net(x, y, fmask, x_mask, y_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuqian_fu/Projects/ego-exo4d-relation/correspondence/SegSwap/model/transformer.py", line 291, in forward
featx, featy, x_mask, y_mask = self.encoder_blocks[i](featx, featy, featmask, x_mask, y_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuqian_fu/Projects/ego-exo4d-relation/correspondence/SegSwap/model/transformer.py", line 205, in forward
featx, featy, x_mask, y_mask = self.layer1(featx, featy, featmask, x_mask, y_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yuqian_fu/Projects/ego-exo4d-relation/correspondence/SegSwap/model/transformer.py", line 105, in forward
output = self.inner_encoder_layer(output, src_mask=src_mask, src_key_padding_mask=src_key_padding_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/transformer.py", line 591, in forward
x = self.norm1(x + self._sa_block(x, src_mask, src_key_padding_mask, is_causal=is_causal))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/transformer.py", line 599, in _sa_block
x = self.self_attn(x, x, x,
^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/modules/activation.py", line 1205, in forward
attn_output, attn_output_weights = F.multi_head_attention_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/scratch/yuqian_fu/micromamba/envs/auto-gfmiyxa3evbd/lib/python3.11/site-packages/torch/nn/functional.py", line 5373, in multi_head_attention_forward
attn_output = scaled_dot_product_attention(q, k, v, attn_mask, dropout_p, is_causal)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 792.00 MiB (GPU 0; 21.95 GiB total capacity; 20.03 GiB already allocated; 790.12 MiB free; 20.94 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
sh: 3: --resume_path: not found
srun: error: gcpl4-eu-3: task 0: Exited with exit code 127