training-template/SECURITY_REVIEW.md

# Security Review: Daimonion RunPod Training Infrastructure

**Reviewer:** Automated security audit  
**Date:** 2026-06-30  
**Scope:** `/home/HumboldtJoker/Daimonion/runpod-template/` + existing `cloud_sft/` infrastructure  
**Context:** Full-parameter SFT of Qwen3.6-35B-A3B on rented RunPod GPUs for Liberation Labs

---

## Executive Summary

No hardcoded secrets found. HF_TOKEN is handled correctly via environment variable. The primary risks are **supply-chain** (unpinned pip dependencies on rented infra), **trust_remote_code execution** (required for Qwen but is arbitrary code execution from HuggingFace Hub), and **data-at-rest exposure** on RunPod's unencrypted persistent volumes. The training loop itself is network-isolated after setup, which is good.

---

## Findings by Category

### 1. HF_TOKEN Handling — ACCEPTABLE WITH NOTES

**Status:** No tokens hardcoded anywhere. Both `setup.sh` (new template) and `pod_setup.sh` (existing) pass `HF_TOKEN` via environment variable, which is the correct approach.

The new `setup.sh` properly checks for `HF_TOKEN` presence early (line 96) and aborts with a clear error message if missing. It passes the token explicitly to `snapshot_download()` (line 192) and `load_dataset()` (line 224). The training script `train_daimon.py` also passes `os.environ.get("HF_TOKEN")` to `load_dataset()` (line 189).

**Token exposure on setup.sh line 104:**
```bash
echo "HF_TOKEN is set (${#HF_TOKEN} chars)"
```
This logs the token LENGTH, not the token itself. Safe.

**Risks:**
- RunPod's environment variable UI stores `HF_TOKEN` in their backend. It is visible in the pod config panel. RunPod staff with database access could theoretically read it.
- `snapshot_download()` and `load_dataset()` may cache the token to `~/.cache/huggingface/token` on the pod filesystem. This persists on the persistent volume after pod termination.
- Shell history: if a user runs `export HF_TOKEN=hf_xxx...` interactively, it lands in `~/.bash_history`. The usage comment in `setup.sh` line 17 even shows `export HF_TOKEN="hf_your_token_here"` as the example, which users will copy-paste with their real token.

**Recommendations:**
- Use a **read-only, fine-grained HF token** scoped only to the model and dataset repos needed. Create at https://huggingface.co/settings/tokens with minimal permissions.
- Add to `pod_setup.sh`: `unset HISTFILE` at the top to prevent bash history capture.
- Add a cleanup step: `rm -f ~/.cache/huggingface/token` after data download completes.
- Consider downloading the dataset to the persistent volume once, then revoking the token before training starts.

### 2. Training Data Privacy — GOOD

**Status:** `report_to="none"` is set in all training configs (both `cloud_sft/` scripts and new `train_daimon_config.yaml`). No telemetry goes to Weights & Biases, TensorBoard cloud, or any external service.

**Data flow:**
1. Training data downloaded from `HumboldtJoker/daimon-sft-data` (or `HumboldtJoker/sonnet-voice-sft-data`) on HuggingFace Hub.
2. Converted to Arrow format at `/workspace/sonnet-data/` on the pod.
3. Read locally during training. No upload.

**Risks:**
- The 226K Sonnet distillation examples are on HuggingFace Hub. If the dataset repo is public, anyone can access them. Verify the repo visibility is set to **private**.
- Training data sits unencrypted on the RunPod persistent volume. RunPod's infrastructure team has physical/logical access to the underlying storage.
- The bare `except:` on `pod_setup.sh` lines 100-101 swallows authentication errors silently. If `HF_TOKEN` is missing or wrong, the download fails but the script continues, producing confusing errors later.

**Recommendations:**
- Confirm `HumboldtJoker/daimon-sft-data` and `HumboldtJoker/sonnet-voice-sft-data` are set to **private** on HuggingFace Hub.
- Replace bare `except:` with `except Exception as e: print(f"Download failed: {e}"); sys.exit(1)` — fail loudly on auth errors.
- After training completes and checkpoints are downloaded, delete the training data from the persistent volume.

### 3. Checkpoint Security — MEDIUM RISK

**Status:** Checkpoints are written to `/workspace/daimon-sft/` (or `/workspace/sonnet-voice-sft/`) on RunPod's persistent volume. `save_total_limit=5` caps at 5 checkpoints (~70 GB each = ~350 GB).

**Risks:**
- **No encryption at rest.** RunPod persistent volumes use standard ext4 on their NVMe/SSD pool. RunPod staff with datacenter access can read the raw blocks. This is true of all cloud providers — RunPod is not worse than AWS EBS in this regard, but also not better.
- **Checkpoints survive pod termination** if on a persistent volume (which they should be — `/workspace/` maps to the volume). This is by design for resumability, but means trained weights persist on RunPod infrastructure until explicitly deleted.
- **No upload mechanism.** The scripts don't push checkpoints anywhere. You must manually `scp` or `rsync` them off the pod. Good for security (no accidental upload), but risky for data loss if the volume is accidentally deleted.

**Recommendations:**
- After training: immediately copy final checkpoint off the pod, then delete the volume.
- Consider encrypting the persistent volume with LUKS (add to setup script): `cryptsetup luksFormat /dev/...` — but this adds significant complexity and RunPod may not support it on all volume types.
- At minimum, after copying checkpoints off: `rm -rf /workspace/daimon-sft/` before terminating the pod.

### 4. Network Isolation — GOOD

**Status:** All network access happens during the setup phase:
1. `pip install` — downloads from PyPI
2. `hf_hub_download` / `from_pretrained` — downloads model + data from HuggingFace Hub
3. SSH key injection for remote access

**During training: zero outbound network calls.** The training loop is purely local computation. `report_to="none"` prevents any telemetry.

**Recommendations:**
- For maximum isolation, after setup completes, you could block outbound traffic: `iptables -A OUTPUT -p tcp --dport 443 -j DROP` (but this breaks SSH access for monitoring). Not recommended unless paranoia level is very high.
- The current approach of "download everything during setup, then train offline" is the right architecture.

### 5. Pod Termination Data Survival — ACCEPTABLE

**Status:** The config correctly targets `/workspace/` for all outputs, which maps to the RunPod persistent volume. `save_steps=1000` means at most 1000 steps of work can be lost on unexpected termination.

**The `train_daimon_config.yaml` explicitly notes:** "Checkpoints go to persistent volume. After pod termination, resume from last." This is correct.

**Risks:**
- Spot/preemptible pods can be terminated with 30 seconds notice. A checkpoint save for a 35B model takes minutes (due to `stage3_gather_16bit_weights_on_model_save`). An in-progress save during termination could produce a corrupted checkpoint.
- `save_total_limit=5` means old checkpoints are auto-deleted. If the latest checkpoint is corrupted, the previous 4 are available as fallback.

**Recommendations:**
- Use on-demand pods, not spot instances, for this training run. The cost difference is small relative to the value of the trained model.
- Consider `save_total_limit=3` to save disk space, since 5 checkpoints at 70 GB each is 350 GB.

### 6. Credential Hygiene — CLEAN

**No API keys, tokens, or secrets hardcoded anywhere in the codebase.** Verified across all files in `cloud_sft/`, `runpod-template/`, `configs/`, `full_sft/`, and `merge/`.

The `.gitignore` includes `.env`, which is correct.

**One note:** The SSH public key `ssh-ed25519 AAAAC3Nza...` in `pod_setup.sh` line 301 is a public key (not a secret) and is safe to commit. However, it does identify Thomas's key fingerprint. Anyone reading the repo knows which key grants pod access. Low risk since pods are ephemeral.

### 7. Supply Chain — HIGH RISK (Primary Concern)

**Status:** All `pip install` commands use **unpinned versions** with no integrity verification.

**New template `setup.sh` lines 113-127:**
```bash
pip install --upgrade -q torch torchvision --index-url https://download.pytorch.org/whl/cu124
pip install --upgrade -q transformers trl datasets accelerate deepspeed peft safetensors mpi4py pyyaml
pip install --upgrade -q flash-attn --no-build-isolation
```
Note: The setup script comments say "Pin versions for reproducibility" on line 110 but then does NOT actually pin any versions. The comment is misleading.

**Existing `pod_setup.sh` line 64:**
```bash
pip install --upgrade -q transformers trl datasets accelerate deepspeed mpi4py safetensors peft
```

**Existing `preflight_check.sh` lines 49, 69, 74:**
```bash
pip install $pkg -q        # for torch, transformers, trl, etc.
pip install deepspeed -q
pip install mpi4py -q
```

**Existing `train_sonnet_h100.py` line 69:**
```python
os.system("pip install trl -q")   # Runtime install — worst pattern
```

**Risks:**
- A compromised PyPI package (or typosquatted name) gets full root access on the pod, plus access to the HF_TOKEN, training data, and model weights.
- Running `pip install --upgrade` on rented infra means you cannot reproduce the exact environment later.
- The `train_sonnet_h100.py` runtime `os.system("pip install trl -q")` is the worst variant — it installs an arbitrary version of `trl` mid-execution.

**Recommendations (STRONGLY RECOMMENDED):**
Add a `requirements.txt` to the template with pinned versions and hashes:
```
transformers==4.46.3
trl==1.7.0
datasets==3.2.0
accelerate==1.2.0
deepspeed==0.16.0
mpi4py==4.0.1
safetensors==0.4.5
peft==0.14.0
```
Then install with: `pip install -r requirements.txt --require-hashes` (if hash pinning) or at minimum `pip install -r requirements.txt` (version pinning only).

Remove the `os.system("pip install trl -q")` from `train_sonnet_h100.py` — it should never install packages at runtime.

---

## Additional Findings

### 8. `trust_remote_code=True` — ACCEPTED RISK

Every model loading call passes `trust_remote_code=True`. This is **required** for Qwen3.6 models (they include custom modeling code). However, it means the HuggingFace Hub model repo executes arbitrary Python on your pod. If `Qwen/Qwen3.6-35B-A3B` were compromised or you typo the model name, this is full RCE.

**Mitigation:** Pin the model revision. In training scripts, use:
```python
AutoModelForCausalLM.from_pretrained(MODEL_ID, revision="<specific_commit_hash>", trust_remote_code=True)
```
This ensures you get the exact code you audited, not whatever the latest push is.

### 9. Windows Path Artifacts — LOW RISK / HYGIENE

Two files with Windows paths are in the repo:
- `Daimonion/C:\Users\Thomas\.claude\hooks\vera_stop_capture.log` (24KB)
- `cloud_sft/C:\Users\Thomas\.claude\hooks\vera_stop_capture.log` (13KB)

These contain Claude hook output logs. They expose the Windows username "Thomas" and operational details, though no secrets. Should be added to `.gitignore` and removed from tracking.

### 10. Large Training Data in Git — HYGIENE

`cloud_sft/train.jsonl` (814 MB) and `cloud_sft/valid.jsonl` (42 MB) plus gzipped copies appear to be committed to git. These should be in `.gitignore` or managed via Git LFS. The `.gitignore` does not exclude `*.jsonl` or `*.jsonl.gz`.

### 11. New Template Config Review (`train_daimon_config.yaml`)

The new template config is clean:
- Correctly targets `/workspace/` for outputs (persistent volume)
- `report_to: "none"` — no telemetry
- References HuggingFace dataset by repo name (`HumboldtJoker/daimon-sft-data`) — will need `HF_TOKEN` if private
- `max_seq_length: 8192` — addresses the previous silent truncation bug
- Optimizer correctly set to `adamw_torch` (not fused, which crashes with DeepSpeed)
- No secrets, no hardcoded paths except model ID

### 12. DeepSpeed Config Review (`runpod-template/ds_config_zero3.json`)

Clean. Notable that **param offload to CPU is removed** (only optimizer offload remains) compared to the `cloud_sft/` configs which offload both. The comments explain this is because CPU param offload makes MoE expert gathering catastrophically slow. This is a good architectural decision and has no security implications.

### 13. New Training Script Review (`runpod-template/train_daimon.py`)

Overall well-structured and security-conscious. Specific findings:

**GOOD:**
- `report_to="none"` hardcoded on line 348, not configurable from YAML. This prevents accidental telemetry even if someone edits the config. Correct decision.
- Logging goes to local files only (`/workspace/daimon-sft/logs/`), no remote logging.
- `TOKENIZERS_PARALLELISM=false` (line 41) prevents data races. Good hygiene.
- Environment variable overrides for paths (lines 256-258) allow runtime flexibility without modifying code. Clean pattern.
- Training summary saved as JSON (lines 399-409) does NOT include the HF token or any secrets. Clean.
- Resume-from-checkpoint support (line 365, 385) is critical for pod interruption recovery. Well implemented.

**CONCERNS:**
- `trust_remote_code=True` on lines 288 and 308 (same accepted risk as finding #8).
- Bare `except Exception:` on line 107 in `pre_split_long_sequences()` silently falls back to character-count estimation. Not a security issue, but could mask data corruption.
- `load_dataset()` on line 189 passes `token=os.environ.get("HF_TOKEN")` which is correct, but if `HF_TOKEN` is unset, returns `None` and the call proceeds without auth. If the dataset is private, this would fail with a confusing 404 instead of an auth error. The setup script checks for `HF_TOKEN` early so this is unlikely in practice, but the training script can be run independently.

### 14. New Setup Script Review (`runpod-template/setup.sh`)

**GOOD:**
- `set -e` on line 22 means the script aborts on any error. Correct.
- Comprehensive pre-flight checks: GPU count, VRAM, RAM, disk, HF_TOKEN.
- Hard aborts at unsafe thresholds (RAM < 200GB, disk < 200GB, GPU < 2).
- SSH key is a public key (same one as existing scripts). Not a secret.

**CONCERNS:**
- **Lines 110-127: "Pin versions for reproducibility" comment is a lie.** The comment says "Pin versions" but the actual pip commands use `--upgrade` with no version constraints. This is the primary supply-chain risk. Anyone reading the comment would assume versions are pinned. Fix the comment or actually pin versions.
- **Line 114: PyTorch installed from pytorch.org index.** `--index-url https://download.pytorch.org/whl/cu124` is a trusted source (official PyTorch wheel index). Acceptable. But combined with `--upgrade`, it could pull a new PyTorch version that breaks compatibility.
- **Line 130: `flash-attn --no-build-isolation`** compiles from source. The `--no-build-isolation` flag means build dependencies come from the existing environment, not a clean virtualenv. Low risk since it's a well-known package, but worth noting.
- **Lines 240-253: Bare `except Exception as e2`** catches download failures but prints "DATA MUST BE UPLOADED MANUALLY" and continues. If auth fails, this message is misleading — the real issue is the token, not manual upload.
- **Line 277: Inner bare `except: pass`** in JSONL parsing silently drops malformed lines. Same pattern as the existing scripts. Not a security issue but could mask data corruption.
- **Line 291: Outer bare `except:`** at the final verification step swallows all errors. If data verification fails, the setup reports success anyway. Should abort or at least warn loudly.

---

### 15. Launch Script Review (`runpod-template/launch.sh`)

Clean and well-structured. Specific notes:

- Pre-flight checks are thorough (GPU count, disk, model, config, DeepSpeed config, existing checkpoints).
- `HF_TOKEN` warning on line 63 is a soft warning, not a hard abort. This is fine since the model may already be cached locally.
- NCCL environment variables (lines 106-122) are standard multi-GPU training config. `NCCL_IB_DISABLE=1` disables InfiniBand (correct for RunPod pods which use NVLink within a node). No security implications.
- `NCCL_TIMEOUT=3600000` (60 minutes) is generous but appropriate for MoE layer gathering.
- Training output piped through `tee` to both console and log file. Log file is on persistent volume. No sensitive data is logged (only loss values, step counts, GPU metrics).
- **No security issues found in this file.**

### 16. Test Script Review (`runpod-template/test_template.py`)

Clean validation script. Notes:

- Writes a temporary test file to `/workspace/.daimon_write_test` (line 206) and immediately removes it (line 209). Clean pattern.
- Creates temporary directory `/tmp/daimon_test` for SFTConfig validation and cleans it up (lines 290-292). Clean.
- `trust_remote_code=True` on lines 127, 135, 167, 247. Same accepted risk.
- The smoke test (test 6) does NOT load the full model on a single GPU (would OOM). It only verifies imports and config. Well-designed — avoids false OOM signals.
- **No security issues found in this file.**

---

## Risk Summary

| # | Finding | Severity | Status |
|---|---------|----------|--------|
| 1 | HF_TOKEN via env var (correct) but persists in cache/history | Low | Add cleanup steps |
| 2 | Training data privacy (no exfiltration) | Clean | Verify HF repo is private |
| 3 | Checkpoints unencrypted on persistent volume | Medium | Delete after extraction |
| 4 | Network isolation during training | Clean | Good architecture |
| 5 | Pod termination resilience | Clean | Use on-demand, not spot |
| 6 | No hardcoded credentials | Clean | No action needed |
| 7 | **Unpinned pip dependencies (all scripts)** | **HIGH** | **Pin versions immediately** |
| 8 | `trust_remote_code=True` (all model loads) | Medium | Pin model revision hash |
| 9 | Windows path log artifacts | Low | Add to .gitignore |
| 10 | Large data files in git | Low | Use LFS or .gitignore |
| 11 | `setup.sh` misleading "pin versions" comment | Medium | Fix comment or pin versions |
| 12 | Silent error swallowing in data verification | Low | Make failures loud |
| 13 | `train_daimon.py` clean architecture | Clean | No action needed |
| 14 | `ds_config_zero3.json` well-commented | Clean | No action needed |

---

## Recommended Actions Before First RunPod Launch

1. **Create `requirements.txt` with pinned versions** in this template directory.
2. **Pin the Qwen model revision** to a specific commit hash in `train_daimon_config.yaml`.
3. **Verify HuggingFace dataset repos are private** (`daimon-sft-data`, `sonnet-voice-sft-data`).
4. **Create a fine-grained, read-only HF token** scoped to only the needed repos.
5. **Add cleanup steps** to the setup script: clear HF token cache and bash history after downloads.
6. **After training completes:** copy checkpoints off the pod, then delete the volume.

---

## What Should NOT Worry Thomas

- **RunPod seeing the training process:** They can see GPU utilization and process names, but cannot see into the training data or model weights without accessing the volume storage directly. This is equivalent to any cloud provider.
- **Network exfiltration:** The training loop makes zero outbound calls. `report_to="none"` is correctly set everywhere.
- **The SSH key in the repo:** It's a public key. The private key is what matters, and that stays on Thomas's machine.
- **DeepSpeed / HuggingFace telemetry:** No telemetry endpoints are configured. The scripts are clean.

The main thing that should concern a privacy-first organization running on rented infrastructure is the **supply-chain risk from unpinned pip packages**. Everything else is either clean or has straightforward mitigations.