Update README.md
Browse files
README.md
CHANGED
|
@@ -22,6 +22,8 @@ conda create -n custom python=3.10 -y
|
|
| 22 |
conda activate custom
|
| 23 |
```
|
| 24 |
|
|
|
|
|
|
|
| 25 |
|
| 26 |
2. Export the environment with `conda-pack` to a `.tar.gz` file.
|
| 27 |
Run `docker-compose up`. This will pull the base competition image, copy the files into the container and create a new environment save it.
|
|
@@ -40,13 +42,30 @@ conda-pack -n custom -o $OUTPUT
|
|
| 40 |
```
|
| 41 |
|
| 42 |
3. Upload the enviroment. Since the environment file is large. It's better to upload it seperately using fast HF_TRANSFER.
|
| 43 |
-
Make sure to install `pip install huggingface_hub[
|
| 44 |
|
| 45 |
```bash
|
| 46 |
bash upload_env_file
|
| 47 |
```
|
| 48 |
|
| 49 |
-
Make sure to change the path to your env file in the script.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
|
| 52 |
-
5. During runtime, `script.py`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
conda activate custom
|
| 23 |
```
|
| 24 |
|
| 25 |
+
Then you can install packages with pip, conda, etc.
|
| 26 |
+
|
| 27 |
|
| 28 |
2. Export the environment with `conda-pack` to a `.tar.gz` file.
|
| 29 |
Run `docker-compose up`. This will pull the base competition image, copy the files into the container and create a new environment save it.
|
|
|
|
| 42 |
```
|
| 43 |
|
| 44 |
3. Upload the enviroment. Since the environment file is large. It's better to upload it seperately using fast HF_TRANSFER.
|
| 45 |
+
Make sure to install `pip install huggingface_hub[hf_transfer]`.
|
| 46 |
|
| 47 |
```bash
|
| 48 |
bash upload_env_file
|
| 49 |
```
|
| 50 |
|
| 51 |
+
Make sure to change the path to your env file in the script. This is what the script does:
|
| 52 |
+
|
| 53 |
+
```bash
|
| 54 |
+
ENV_FILE=/disk1/kirill/custom-env/custom.tar.gz
|
| 55 |
+
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli upload safe-challenge/safe-example-submission-custom-env $ENV_FILE custom.tar.gz
|
| 56 |
+
```
|
| 57 |
|
| 58 |
|
| 59 |
+
5. During runtime, `script.py` unpacks the environment to run your model in your custom environment. Rename your actual script to `_script.py`.
|
| 60 |
+
`script.py` just calls `bash run.sh` to do all the steps:
|
| 61 |
+
|
| 62 |
+
```bash
|
| 63 |
+
echo "uncompressing environment"
|
| 64 |
+
mkdir -p custom
|
| 65 |
+
tar -xzf custom.tar.gz -C custom
|
| 66 |
+
source custom/bin/activate
|
| 67 |
+
echo "unpacking"
|
| 68 |
+
conda-unpack
|
| 69 |
+
echo "running"
|
| 70 |
+
python _script.py
|
| 71 |
+
```
|