config-tb2/proxy-config-modification.md

# Harbor Docker Proxy Configuration Modification

## Overview

This document records the modification made to harbor's Docker environment to support dynamic proxy URL selection based on task name.

## Modification Date

2026-03-12

## Problem Statement

Some tasks (e.g., `hf-model-inference`, `count-dataset-tokens`) need to use a different proxy server for better network access to certain resources (like Hugging Face).

## Solution

Modified `/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker.py` to dynamically set proxy URLs based on the task name.

## Proxy Configuration

| Task Name | Proxy URL |
|-----------|-----------|
| `hf-model-inference` | `http://oversea-squid2.ko.txyun:11080` |
| `count-dataset-tokens` | `http://oversea-squid2.ko.txyun:11080` |
| `make-doom-for-mips` | `http://oversea-squid2.ko.txyun:11080` |
| `rstan-to-pystan` | `http://oversea-squid2.ko.txyun:11080` |
| `gcode-to-text` | `http://oversea-squid2.ko.txyun:11080` |
| `mcmc-sampling-stan` | `http://oversea-squid2.ko.txyun:11080` |
| `make-mips-interpreter` | `http://oversea-squid2.ko.txyun:11080` |
| `sam-cell-seg` | `http://oversea-squid2.ko.txyun:11080` |
| `reshard-c4-data` | `http://oversea-squid2.ko.txyun:11080` |
| `caffe-cifar-10` | `http://oversea-squid2.ko.txyun:11080` |
| `install-windows-3.11` | `http://oversea-squid2.ko.txyun:11080` |
| `pytorch-model-recovery` | `http://oversea-squid2.ko.txyun:11080` |
| `compile-compcert` | `http://oversea-squid2.ko.txyun:11080` |
| All other tasks | `http://oversea-squid5.sgp.txyun:11080` |

## Code Changes

### 1. `docker.py` - Added class-level constants (line 50-65)

```python
# Proxy configuration based on task name
_SPECIAL_PROXY_TASKS = [
    "hf-model-inference",
    "count-dataset-tokens",
    "make-doom-for-mips",
    "rstan-to-pystan",
    "gcode-to-text",
    "mcmc-sampling-stan",
    "make-mips-interpreter",
    "sam-cell-seg",
    "reshard-c4-data",
    "caffe-cifar-10",
    "install-windows-3.11",
    "pytorch-model-recovery",
    "compile-compcert",
]
_DEFAULT_PROXY = "http://oversea-squid5.sgp.txyun:11080"
_SPECIAL_PROXY = "http://oversea-squid2.ko.txyun:11080"
```

### 2. `docker.py` - Added `_get_proxy_url` method and initialization (after line 88)

```python
self._proxy_url = self._get_proxy_url()

def _get_proxy_url(self) -> str:
    """Get the proxy URL based on the task name."""
    if self.environment_name.lower() in [t.lower() for t in self._SPECIAL_PROXY_TASKS]:
        return self._SPECIAL_PROXY
    return self._DEFAULT_PROXY
```

### 3. `docker.py` - Modified `_run_docker_compose_command` method (line 188-193)

Before:
```python
process = await asyncio.create_subprocess_exec(
    *full_command,
    env=self._env_vars.to_env_dict(include_os_env=True),
    ...
)
```

After:
```python
# Get env dict and inject proxy settings based on task name
env = self._env_vars.to_env_dict(include_os_env=True)
env["HTTP_PROXY"] = self._proxy_url
env["HTTPS_PROXY"] = self._proxy_url
env["http_proxy"] = self._proxy_url
env["https_proxy"] = self._proxy_url

process = await asyncio.create_subprocess_exec(
    *full_command,
    env=env,
    ...
)
```

### 4. `docker-compose-build.yaml` - Changed hardcoded proxy to environment variables

Before:
```yaml
build:
  args:
    - http_proxy=http://oversea-squid5.sgp.txyun:11080
    - https_proxy=http://oversea-squid5.sgp.txyun:11080
    - HTTP_PROXY=http://oversea-squid5.sgp.txyun:11080
    - HTTPS_PROXY=http://oversea-squid5.sgp.txyun:11080
environment:
  - http_proxy=http://oversea-squid5.sgp.txyun:11080
  - https_proxy=http://oversea-squid5.sgp.txyun:11080
  - HTTP_PROXY=http://oversea-squid5.sgp.txyun:11080
  - HTTPS_PROXY=http://oversea-squid5.sgp.txyun:11080
```

After:
```yaml
build:
  args:
    - http_proxy=${http_proxy}
    - https_proxy=${https_proxy}
    - HTTP_PROXY=${HTTP_PROXY}
    - HTTPS_PROXY=${HTTPS_PROXY}
environment:
  - http_proxy=${http_proxy}
  - https_proxy=${https_proxy}
  - HTTP_PROXY=${HTTP_PROXY}
  - HTTPS_PROXY=${HTTPS_PROXY}
```

### 5. `docker-compose-prebuilt.yaml` - Changed hardcoded proxy to environment variables

Before:
```yaml
environment:
  - http_proxy=http://oversea-squid5.sgp.txyun:11080
  - https_proxy=http://oversea-squid5.sgp.txyun:11080
  - HTTP_PROXY=http://oversea-squid5.sgp.txyun:11080
  - HTTPS_PROXY=http://oversea-squid5.sgp.txyun:11080
```

After:
```yaml
environment:
  - http_proxy=${http_proxy}
  - https_proxy=${https_proxy}
  - HTTP_PROXY=${HTTP_PROXY}
  - HTTPS_PROXY=${HTTPS_PROXY}
```

## How It Works

1. When `DockerEnvironment` is initialized, `environment_name` contains the task name
2. The `_get_proxy_url()` method checks if the task name is in the special proxy list
3. The appropriate proxy URL is stored in `self._proxy_url`
4. When running docker compose commands, the proxy environment variables are injected with the correct URL
5. Docker compose YAML files use `${http_proxy}` etc. to read the proxy from environment variables
6. This affects both:
   - **Build args**: Proxy used during `docker build` (for apt-get, pip install, etc.)
   - **Environment**: Proxy used at container runtime

## Adding New Tasks

To add a new task that should use the special proxy, simply add the task name to the `_SPECIAL_PROXY_TASKS` list:

```python
_SPECIAL_PROXY_TASKS = ["hf-model-inference", "count-dataset-tokens", "new-task-name"]
```

## Modified Files

| File | Path |
|------|------|
| docker.py | `/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker.py` |
| docker-compose-build.yaml | `/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker-compose-build.yaml` |
| docker-compose-prebuilt.yaml | `/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker-compose-prebuilt.yaml` |

## Notes

- Task name matching is case-insensitive
- This modification affects all docker compose operations (build, up, exec, etc.)
- The proxy is set via environment variables passed to the subprocess
- Docker compose YAML files reference these environment variables for both build args and runtime environment