scripts/readme.md

Fix hnswlib Issue

What’s going on (1 minute)

• hnswlib is a compiled Python package (.so file).
• If it’s compiled on a CPU with newer instruction support, then running it later on an older CPU can crash (SIGILL).
• The fix is to compile it in a CPU-generic way (-march=x86-64 -mtune=generic) and then reuse that same compiled wheel everywhere.

That’s why we build a “portable wheel” once and then just install it repeatedly.

---

Where/when to run each script

Script A: build_portable_wheels.sh

Run it once in an environment that:

1. has a compiler toolchain available (g++, make, etc.), and
2. can download the hnswlib source (from your allowed pip indexes), and
3. uses the same Python/venv you’ll run with (your /venv/bin/python).

On Pegasus, the safest way is:

✅ Run it inside a SLURM interactive job on a compute node (same way you ran debugpy), inside the same container.

Example flow:

1. Request an interactive job (like you already do with scripts/slurm_pty.sh ...)

2. Inside that job, inside the container:

   cd /home/abuali/projects/KBExtraction
   bash scripts/build_portable_wheels.sh
   

This produces a wheel file in ./.wheelhouse/.

Why compute node? Because that environment is closest to where your job will run and has the right architecture/permissions.

Is it one-time? ➡️ Yes, as long as you don’t change Python version / hnswlib version / compiler environment. If you upgrade Python (e.g., 3.10 → 3.11) or want a different hnswlib version, rebuild once again.

---

Script B: install_from_wheelhouse.sh

Run it at the start of every job (or at least every time you start a fresh container/venv).

Because:

• a job might land on a node where your venv is “clean” or missing that package
• reinstalling from your local wheelhouse is fast and deterministic

So you typically do:

• add this in your job startup (debug.sh, train.sh, etc.):

  bash scripts/install_from_wheelhouse.sh
  

Then run your actual program.

---

A very practical “do this” recipe

One-time setup (first time only)

Inside an interactive SLURM job + container:

cd /home/abuali/projects/KBExtraction
bash scripts/build_portable_wheels.sh

You should now have something like: /home/abuali/projects/KBExtraction/.wheelhouse/hnswlib-...whl

Every time you run on any node

At the beginning of your debug.sh (or right before starting python):

bash scripts/install_from_wheelhouse.sh

That’s it.

---

What “where building works” really meant

Sometimes building fails because of one of these:

• missing g++ / python3-dev headers
• pip can’t fetch build requirements (common on HPC with restricted indexes)
• build isolation tries to download stuff you can’t access

So “where building works” = “a node/container where the compile + pip install succeeds”.

Since you already successfully built once: you’ve found such a place.

---

Tiny improvement I strongly recommend

Make .wheelhouse live on a shared filesystem you always see from any node (your project dir probably is). If you ever run from a different working directory, set:

export WHEELHOUSE=/home/abuali/projects/KBExtraction/.wheelhouse

---

If you paste the output of:

ls -lh .wheelhouse
python -c "import sys; print(sys.version)"

I can tell you if your current wheelhouse is correctly set up and version-safe.