docs/INSTALL.md

## h2oGPT Installation Help

The following sections describe how to get a working Python environment on a Linux system.

### Install for A100+

E.g. for Ubuntu 20.04, install driver if you haven't already done so:

```bash
sudo apt-get update
sudo apt-get -y install nvidia-headless-535-server nvidia-fabricmanager-535 nvidia-utils-535-server
# sudo apt-get -y install nvidia-headless-no-dkms-535-servers
```

Note that if you run the preceding commands, you don't need to use the NVIDIA developer downloads in the following sections.

### Install CUDA Toolkit

If happy with above drivers, then just get run local file for [CUDA 11.8](https://developer.nvidia.com/cuda-11-8-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=runfile_local):
```bash
wget wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.run
```
only choose to install toolkit and do not replace existing `/usr/local/cuda` link if you already have one.

If instead, you want full deb CUDA [install cuda coolkit](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_local).  Pick deb local, e.g. for Ubuntu:
```bash
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-12-1-local_12.1.0-530.30.02-1_amd64.deb
sudo cp /var/cuda-repo-ubuntu2004-12-1-local/cuda-*-keyring.gpg /usr/share/keyrings/
sudo apt-get update
sudo apt-get -y install cuda
```

Then set the system up to use the freshly installed CUDA location:
```bash
echo "export LD_LIBRARY_PATH=\$LD_LIBRARY_PATH:/usr/local/cuda/lib64/" >> ~/.bashrc
echo "export CUDA_HOME=/usr/local/cuda" >> ~/.bashrc
echo "export PATH=\$PATH:/usr/local/cuda/bin/" >> ~/.bashrc
source ~/.bashrc
```

Then reboot the machine, to get everything sync'ed up on restart.
```bash
sudo reboot
```

### Compile bitsandbytes

For fast 4-bit and 8-bit training, you need to use [bitsandbytes](https://github.com/TimDettmers/bitsandbytes/tree/main#readme). Note that [compiling bitsandbytes](https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md) is only required if you have a different CUDA version from the ones built into the [bitsandbytes PyPI package](https://pypi.org/project/bitsandbytes/),
which includes CUDA 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6, 11.7, 11.8, 12.0, and 12.1. In the following example, bitsandbytes is compiled for CUDA 12.1:
```bash
git clone http://github.com/TimDettmers/bitsandbytes.git
cd bitsandbytes
git checkout 7c651012fce87881bb4e194a26af25790cadea4f
CUDA_VERSION=121 make cuda12x
CUDA_VERSION=121 python setup.py install
cd ..
```

### Install NVIDIA GPU Manager on systems with multiple A100 or H100 GPUs

To install NVIDIA GPU Manager, run the following:

```bash
sudo apt-key del 7fa2af80
distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-keyring_1.0-1_all.deb
sudo dpkg -i cuda-keyring_1.0-1_all.deb
sudo apt-get update
sudo apt-get install -y datacenter-gpu-manager
# if use 535 drivers, then use 535 below
sudo apt-get install -y libnvidia-nscq-535
sudo systemctl --now enable nvidia-dcgm
dcgmi discovery -l
```
For more information, see the official [GPU Manager user guide](https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html).

### Install and run NVIDIA Fabric Manager on systems with multiple A100 or H100 GPUs

To install the CUDA drivers for NVIDIA Fabric Manager, run the following:

```bash
sudo apt-get install -y cuda-drivers-fabricmanager
```

Once you've installed Fabric Manager and rebooted your system, run the following to start the NVIDIA Fabric Manager service:

```bash
sudo systemctl --now enable nvidia-dcgm
dcgmi discovery -l
sudo systemctl start nvidia-fabricmanager
sudo systemctl status nvidia-fabricmanager
```

For more information, see the official [Fabric Manager user guide](https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html).

### Optional: Use TensorBoard to inspect training

You can use [TensorBoard](https://www.tensorflow.org/tensorboard/get_started) to inspect the training process. To launch TensorBoard and instruct it to read event files from the `runs/` directory, use the following command:

```bash
tensorboard --logdir=runs/
```

For more information, see [TensorBoard usage](https://github.com/tensorflow/tensorboard/blob/master/README.md#usage).

### Flash Attention

**Update:** Flash attention specifics are no longer needed. For more information, see https://github.com/h2oai/h2ogpt/issues/128.

To use flash attention with LLaMa, need cuda 11.7 so flash attention module compiles against torch.

E.g. for Ubuntu, one goes to [cuda toolkit](https://developer.nvidia.com/cuda-11-7-0-download-archive?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=20.04&target_type=runfile_local), then:
```bash
wget https://developer.download.nvidia.com/compute/cuda/11.7.0/local_installers/cuda_11.7.0_515.43.04_linux.run
sudo bash ./cuda_11.7.0_515.43.04_linux.run
```
Then No for symlink change, say continue (not abort), accept license, keep only toolkit selected, select install.

If cuda 11.7 is not your base installation, then when doing pip install -r requirements.txt do instead:
```bash
CUDA_HOME=/usr/local/cuda-11.8 pip install -r reqs_optional/requirements_optional_flashattention.txt
```