INSTALL_PORTABLE.md

RNASeek Portable Environment Notes

This repo now has a project-local conda prefix at portable_runtime/env plus local package artifacts/caches under portable_runtime/.

What Was Installed Here

• Conda prefix: portable_runtime/env
• Conda package cache: portable_runtime/conda_pkgs
• Pip cache: portable_runtime/pip_cache
• Local wheelhouse: portable_runtime/wheelhouse
• Prebuilt local flash-attn wheel: portable_runtime/wheelhouse/flash_attn-2.6.3-cp311-cp311-linux_x86_64.whl
• Relocatable conda-pack archive: portable_runtime/rnaseek-conda-env-linux-64.tar.gz
• Split pip requirement file: portable_runtime/requirements-pip-no-flash-attn.txt
• Portable conda spec: portable_runtime/environment-portable.yml
• Resolved conda env export: portable_runtime/environment-resolved.yml
• Explicit conda package URLs: portable_runtime/conda-explicit-linux-64.txt
• Pip freeze: portable_runtime/pip-freeze.txt
• Checksums for large artifacts: portable_runtime/SHA256SUMS
• Runtime environment exports: portable_runtime/env-vars.sh

The main project imports require PyTorch, Transformers, Datasets, TRL, PEFT, FastAPI, ViennaRNA, scikit-learn/scipy/numpy/pandas, plotting/notebook packages, and bio/audio utility packages. Those are installed in portable_runtime/env.

Approximate artifact sizes from this build:

13G   portable_runtime/env
6.1G  portable_runtime/rnaseek-conda-env-linux-64.tar.gz
177M  portable_runtime/wheelhouse
5.8G  portable_runtime/conda_pkgs
3.3G  portable_runtime/pip_cache

Reusing The Packed Environment

The preferred transfer artifact is portable_runtime/rnaseek-conda-env-linux-64.tar.gz. On another Linux x86_64 machine with a sufficiently new NVIDIA driver:

cd rnaseek
sha256sum -c portable_runtime/SHA256SUMS
mkdir -p portable_runtime/env
tar -xzf portable_runtime/rnaseek-conda-env-linux-64.tar.gz -C portable_runtime/env
portable_runtime/env/bin/conda-unpack
conda activate "$PWD/portable_runtime/env"
source portable_runtime/env-vars.sh
python -c "import torch, flash_attn; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

Directly copying portable_runtime/env may work only when the repo is restored to the same absolute path. Use the tarball above when the destination path differs.

Recreating The Prefix With Minimal Compilation

From a fresh checkout on another Linux x86_64 CUDA machine:

cd rnaseek
mkdir -p portable_runtime/conda_pkgs portable_runtime/pip_cache portable_runtime/wheelhouse
XDG_CACHE_HOME="$PWD/portable_runtime/xdg_cache" \
CONDA_PKGS_DIRS="$PWD/portable_runtime/conda_pkgs" \
PIP_CACHE_DIR="$PWD/portable_runtime/pip_cache" \
conda env create -p "$PWD/portable_runtime/env" -f portable_runtime/environment-portable.yml
conda activate "$PWD/portable_runtime/env"
source portable_runtime/env-vars.sh

Install flash-attn from the wheelhouse if a compatible wheel is present:

python -m pip install --no-index --find-links "$PWD/portable_runtime/wheelhouse" flash-attn==2.6.3

The included wheel is for Linux x86_64, CPython 3.11, PyTorch 2.10/CUDA 12.x. If no compatible wheel exists, build it once and keep the wheel:

MAX_JOBS=2 python -m pip wheel flash-attn==2.6.3 --no-build-isolation --no-deps -w "$PWD/portable_runtime/wheelhouse"
python -m pip install --no-index --find-links "$PWD/portable_runtime/wheelhouse" flash-attn==2.6.3

Flash-Attn Build Notes

flash-attn==2.6.3 cannot be installed during conda env create because pip build isolation cannot import the just-installed torch. The working sequence is:

1. Install conda PyTorch/CUDA packages.
2. Install all pip packages except flash-attn.
3. Build/install flash-attn with --no-build-isolation --no-deps.

The local prefix needed these CUDA development packages for source builds:

cuda-cudart-dev
cuda-crt-dev_linux-64=12.9.86
cuda-nvcc-dev_linux-64=12.9.86
libcublas-dev=12.9.2.10
libcusparse-dev=12.5.10.65
libcusolver-dev=11.7.5.82
ffmpeg

The conda CUDA layout also required this symlink for nvcc:

ln -s ../../nvvm portable_runtime/env/targets/x86_64-linux/nvvm

Smoke Test

After activation:

source portable_runtime/env-vars.sh
python - <<'PY'
import torch, transformers, datasets, peft, trl, RNA
import numpy, pandas, scipy, sklearn
print("torch", torch.__version__, "cuda", torch.version.cuda, "available", torch.cuda.is_available())
print("transformers", transformers.__version__)
print("RNA", RNA.__version__ if hasattr(RNA, "__version__") else "import-ok")
PY

The smoke test passed locally for the major imports, including torch, transformers, datasets, peft, trl, RNA, flash_attn, bitsandbytes, torchcodec, cutadapt, multiqc, pysam, anndata, pydeseq2, and audio packages. In this sandboxed run PyTorch reported CUDA 12.8 but torch.cuda.is_available() was False because NVML could not be initialized from the sandbox; rerun the smoke test on the target GPU host.

Most training scripts hard-code local checkpoint/data paths. Review path constants near the top of each script before running training on another machine.