config-tb2/proxy-config-modification.md

Harbor Docker Proxy Configuration Modification

Overview

This document records the modification made to harbor's Docker environment to support dynamic proxy URL selection based on task name.

Modification Date

2026-03-12

Problem Statement

Some tasks (e.g., hf-model-inference, count-dataset-tokens) need to use a different proxy server for better network access to certain resources (like Hugging Face).

Solution

Modified /root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker.py to dynamically set proxy URLs based on the task name.

Proxy Configuration

Task Name	Proxy URL
hf-model-inference	http://oversea-squid2.ko.txyun:11080
count-dataset-tokens	http://oversea-squid2.ko.txyun:11080
make-doom-for-mips	http://oversea-squid2.ko.txyun:11080
rstan-to-pystan	http://oversea-squid2.ko.txyun:11080
gcode-to-text	http://oversea-squid2.ko.txyun:11080
mcmc-sampling-stan	http://oversea-squid2.ko.txyun:11080
make-mips-interpreter	http://oversea-squid2.ko.txyun:11080
sam-cell-seg	http://oversea-squid2.ko.txyun:11080
reshard-c4-data	http://oversea-squid2.ko.txyun:11080
caffe-cifar-10	http://oversea-squid2.ko.txyun:11080
install-windows-3.11	http://oversea-squid2.ko.txyun:11080
pytorch-model-recovery	http://oversea-squid2.ko.txyun:11080
compile-compcert	http://oversea-squid2.ko.txyun:11080
All other tasks	http://oversea-squid5.sgp.txyun:11080

Code Changes

1. docker.py - Added class-level constants (line 50-65)

# Proxy configuration based on task name
_SPECIAL_PROXY_TASKS = [
    "hf-model-inference",
    "count-dataset-tokens",
    "make-doom-for-mips",
    "rstan-to-pystan",
    "gcode-to-text",
    "mcmc-sampling-stan",
    "make-mips-interpreter",
    "sam-cell-seg",
    "reshard-c4-data",
    "caffe-cifar-10",
    "install-windows-3.11",
    "pytorch-model-recovery",
    "compile-compcert",
]
_DEFAULT_PROXY = "http://oversea-squid5.sgp.txyun:11080"
_SPECIAL_PROXY = "http://oversea-squid2.ko.txyun:11080"

2. docker.py - Added _get_proxy_url method and initialization (after line 88)

self._proxy_url = self._get_proxy_url()

def _get_proxy_url(self) -> str:
    """Get the proxy URL based on the task name."""
    if self.environment_name.lower() in [t.lower() for t in self._SPECIAL_PROXY_TASKS]:
        return self._SPECIAL_PROXY
    return self._DEFAULT_PROXY

3. docker.py - Modified _run_docker_compose_command method (line 188-193)

Before:

process = await asyncio.create_subprocess_exec(
    *full_command,
    env=self._env_vars.to_env_dict(include_os_env=True),
    ...
)

After:

# Get env dict and inject proxy settings based on task name
env = self._env_vars.to_env_dict(include_os_env=True)
env["HTTP_PROXY"] = self._proxy_url
env["HTTPS_PROXY"] = self._proxy_url
env["http_proxy"] = self._proxy_url
env["https_proxy"] = self._proxy_url

process = await asyncio.create_subprocess_exec(
    *full_command,
    env=env,
    ...
)

4. docker-compose-build.yaml - Changed hardcoded proxy to environment variables

Before:

build:
  args:
    - http_proxy=http://oversea-squid5.sgp.txyun:11080
    - https_proxy=http://oversea-squid5.sgp.txyun:11080
    - HTTP_PROXY=http://oversea-squid5.sgp.txyun:11080
    - HTTPS_PROXY=http://oversea-squid5.sgp.txyun:11080
environment:
  - http_proxy=http://oversea-squid5.sgp.txyun:11080
  - https_proxy=http://oversea-squid5.sgp.txyun:11080
  - HTTP_PROXY=http://oversea-squid5.sgp.txyun:11080
  - HTTPS_PROXY=http://oversea-squid5.sgp.txyun:11080

After:

build:
  args:
    - http_proxy=${http_proxy}
    - https_proxy=${https_proxy}
    - HTTP_PROXY=${HTTP_PROXY}
    - HTTPS_PROXY=${HTTPS_PROXY}
environment:
  - http_proxy=${http_proxy}
  - https_proxy=${https_proxy}
  - HTTP_PROXY=${HTTP_PROXY}
  - HTTPS_PROXY=${HTTPS_PROXY}

5. docker-compose-prebuilt.yaml - Changed hardcoded proxy to environment variables

Before:

environment:
  - http_proxy=http://oversea-squid5.sgp.txyun:11080
  - https_proxy=http://oversea-squid5.sgp.txyun:11080
  - HTTP_PROXY=http://oversea-squid5.sgp.txyun:11080
  - HTTPS_PROXY=http://oversea-squid5.sgp.txyun:11080

After:

environment:
  - http_proxy=${http_proxy}
  - https_proxy=${https_proxy}
  - HTTP_PROXY=${HTTP_PROXY}
  - HTTPS_PROXY=${HTTPS_PROXY}

How It Works

1. When DockerEnvironment is initialized, environment_name contains the task name
2. The _get_proxy_url() method checks if the task name is in the special proxy list
3. The appropriate proxy URL is stored in self._proxy_url
4. When running docker compose commands, the proxy environment variables are injected with the correct URL
5. Docker compose YAML files use ${http_proxy} etc. to read the proxy from environment variables
6. This affects both:
   • Build args: Proxy used during docker build (for apt-get, pip install, etc.)
   • Environment: Proxy used at container runtime

Adding New Tasks

To add a new task that should use the special proxy, simply add the task name to the _SPECIAL_PROXY_TASKS list:

_SPECIAL_PROXY_TASKS = ["hf-model-inference", "count-dataset-tokens", "new-task-name"]

Modified Files

File	Path
docker.py	/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker.py
docker-compose-build.yaml	/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker-compose-build.yaml
docker-compose-prebuilt.yaml	/root/.local/share/uv/tools/harbor/lib/python3.12/site-packages/harbor/environments/docker/docker-compose-prebuilt.yaml

Notes

• Task name matching is case-insensitive
• This modification affects all docker compose operations (build, up, exec, etc.)
• The proxy is set via environment variables passed to the subprocess
• Docker compose YAML files reference these environment variables for both build args and runtime environment