--- title: README emoji: ๐ŸŒ colorFrom: gray colorTo: blue sdk: static pinned: false --- # Tracking Hugging Face Jobs with track.py Oct 21, 2026 ยท @Clifford Anderson ## What it does `track.py` starts a Hugging Face Job from your computer, or attaches to one already running, and records it in Weights & Biases (W&B), MLflow, or both. It works with any job script, ours or anyone else's (Daniel van Strien's [uv-scripts](https://huggingface.co/uv-scripts), for example), without changing it. Each run records the script's URL and arguments, the job's image and hardware, and its outcome and duration. It also keeps the full job log and the CPU, GPU, memory and network use that Hugging Face reports every 15 seconds. It can't see inside a script beyond what the script prints. Tracking runs on your computer, so your W&B and MLflow logins never enter the job. An MLflow server only needs to be reachable from your computer. ```mermaid flowchart LR PC["Your computer
runs track.py
W&B and MLflow logins never leave it"] HF["Hugging Face Jobs
runs the script unchanged
any script, any author"] TR["W&B and/or MLflow
settings, full log, outcome
CPU, GPU, memory, network"] PC -- starts the job --> HF HF -- log and resource use --> PC PC -- a run per job --> TR ``` Every arrow starts or ends on your computer: the job never talks to W&B or MLflow. ## Set up once Install [uv](https://docs.astral.sh/uv/), open a new terminal, and log in to Hugging Face. `uvx` runs the `hf` command without installing it. Bash or zsh (macOS, Linux): ```bash curl -LsSf https://astral.sh/uv/install.sh | sh uvx hf auth login ``` PowerShell (Windows): ```powershell powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" uvx hf auth login ``` ## Connect W&B and MLflow Log in to W&B once with `uvx wandb login`, pasting your key from [wandb.ai/authorize](https://wandb.ai/authorize). Everything else comes from environment variables on your computer: | Variable | Sets | Default | | --- | --- | --- | | `WANDB_PROJECT` | W&B project the runs go to | `hf-jobs` | | `WANDB_ENTITY` | W&B team to log to | your own account | | `MLFLOW_TRACKING_URI` | MLflow server URL | none; required for `--mlflow` | | `MLFLOW_TRACKING_USERNAME`, `MLFLOW_TRACKING_PASSWORD` | MLflow login, if the server has one | none | | `MLFLOW_TRACKING_TOKEN` | MLflow token, if the server uses one instead | none | | `MLFLOW_EXPERIMENT_NAME` | MLflow experiment the runs go to | `hf-jobs` | For MLflow on Databricks, set `MLFLOW_TRACKING_URI` to `databricks` and add `DATABRICKS_HOST` and `DATABRICKS_TOKEN`. The experiment name must then be a workspace path, such as `/Users/you@yale.edu/open-theology`. Bash or zsh (add the `export` lines to `~/.bashrc` or `~/.zshrc` to keep them): ```bash uvx wandb login export WANDB_PROJECT=open-theology export MLFLOW_TRACKING_URI=https://YOUR-MLFLOW-SERVER export MLFLOW_TRACKING_USERNAME=you MLFLOW_TRACKING_PASSWORD=your-password export MLFLOW_EXPERIMENT_NAME=open-theology ``` PowerShell (`$env:` settings last for the current window; to keep one, run `[Environment]::SetEnvironmentVariable("WANDB_PROJECT", "open-theology", "User")` and open a new window): ```powershell uvx wandb login $env:WANDB_PROJECT = "open-theology" $env:MLFLOW_TRACKING_URI = "https://YOUR-MLFLOW-SERVER" $env:MLFLOW_TRACKING_USERNAME = "you"; $env:MLFLOW_TRACKING_PASSWORD = "your-password" $env:MLFLOW_EXPERIMENT_NAME = "open-theology" ``` ## Run a tracked job Put `uv run $TRACK --wandb --` in front of the `hf jobs` command you would normally run. Use `--mlflow` instead for MLflow, or both flags to log to both. The job's own options (`--flavor`, `-s HF_TOKEN`, `-v`, `--name`) stay in the `hf jobs` part, and its log streams to your terminal as usual. This example runs Daniel's GLM-OCR script on our page images. The script's header already picks the GPU, the Docker image and the Hugging Face token. Bash or zsh: ```bash TRACK=https://huggingface.co/datasets/divinity-library/scripts/resolve/main/track.py OCR=https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py uv run $TRACK --wandb -- hf jobs uv run --name open-theology-ocr $OCR \ divinity-library/open-theology-pages divinity-library/open-theology-ocr ``` PowerShell (the backtick continues the line and must be its last character): ```powershell $TRACK = "https://huggingface.co/datasets/divinity-library/scripts/resolve/main/track.py" $OCR = "https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py" uv run $TRACK --wandb -- hf jobs uv run --name open-theology-ocr $OCR ` divinity-library/open-theology-pages divinity-library/open-theology-ocr ``` The run takes the job's name, here `open-theology-ocr`, and keeps the job's URL in its settings. When the job ends, the run is marked finished or failed to match it. ## Resume or stop `track.py` has to keep running until the job ends. Ctrl-C, closing the terminal or a sleeping laptop stops the tracking, not the job. To pick it up again, run `track.py` with the `NAMESPACE/JOB_ID` it printed at the start; it continues the same W&B or MLflow run. To stop the job itself, cancel it. The commands are the same in Bash, zsh and PowerShell: ```bash uv run $TRACK --wandb NAMESPACE/JOB_ID uvx hf jobs cancel NAMESPACE/JOB_ID ``` The first command also follows a job started without `track.py`. Its log is recorded in full, but resource use only from when you attach. ## Good practice - Jobs run, and are billed, under your own Hugging Face account unless you add `--namespace divinity-library` to the `hf jobs` command. That needs write access to the organization. - Pin other people's scripts to a commit, as in `.../uv-scripts/ocr/raw//glm-ocr.py`, so the run records exactly what ran. A URL with `main` points at whatever the file says later. - Keep W&B and MLflow logins out of the job: don't pass them with `-s`. `track.py` never needs them there.