training-template/SECURITY_REVIEW.md

Security Review: Daimonion RunPod Training Infrastructure

Reviewer: Automated security audit
Date: 2026-06-30
Scope: /home/HumboldtJoker/Daimonion/runpod-template/ + existing cloud_sft/ infrastructure
Context: Full-parameter SFT of Qwen3.6-35B-A3B on rented RunPod GPUs for Liberation Labs

---

Executive Summary

No hardcoded secrets found. HF_TOKEN is handled correctly via environment variable. The primary risks are supply-chain (unpinned pip dependencies on rented infra), trust_remote_code execution (required for Qwen but is arbitrary code execution from HuggingFace Hub), and data-at-rest exposure on RunPod's unencrypted persistent volumes. The training loop itself is network-isolated after setup, which is good.

---

Findings by Category

1. HF_TOKEN Handling — ACCEPTABLE WITH NOTES

Status: No tokens hardcoded anywhere. Both setup.sh (new template) and pod_setup.sh (existing) pass HF_TOKEN via environment variable, which is the correct approach.

The new setup.sh properly checks for HF_TOKEN presence early (line 96) and aborts with a clear error message if missing. It passes the token explicitly to snapshot_download() (line 192) and load_dataset() (line 224). The training script train_daimon.py also passes os.environ.get("HF_TOKEN") to load_dataset() (line 189).

Token exposure on setup.sh line 104:

echo "HF_TOKEN is set (${#HF_TOKEN} chars)"

This logs the token LENGTH, not the token itself. Safe.

Risks:

• RunPod's environment variable UI stores HF_TOKEN in their backend. It is visible in the pod config panel. RunPod staff with database access could theoretically read it.
• snapshot_download() and load_dataset() may cache the token to ~/.cache/huggingface/token on the pod filesystem. This persists on the persistent volume after pod termination.
• Shell history: if a user runs export HF_TOKEN=hf_xxx... interactively, it lands in ~/.bash_history. The usage comment in setup.sh line 17 even shows export HF_TOKEN="hf_your_token_here" as the example, which users will copy-paste with their real token.

Recommendations:

• Use a read-only, fine-grained HF token scoped only to the model and dataset repos needed. Create at https://huggingface.co/settings/tokens with minimal permissions.
• Add to pod_setup.sh: unset HISTFILE at the top to prevent bash history capture.
• Add a cleanup step: rm -f ~/.cache/huggingface/token after data download completes.
• Consider downloading the dataset to the persistent volume once, then revoking the token before training starts.

2. Training Data Privacy — GOOD

Status: report_to="none" is set in all training configs (both cloud_sft/ scripts and new train_daimon_config.yaml). No telemetry goes to Weights & Biases, TensorBoard cloud, or any external service.

Data flow:

1. Training data downloaded from HumboldtJoker/daimon-sft-data (or HumboldtJoker/sonnet-voice-sft-data) on HuggingFace Hub.
2. Converted to Arrow format at /workspace/sonnet-data/ on the pod.
3. Read locally during training. No upload.

Risks:

• The 226K Sonnet distillation examples are on HuggingFace Hub. If the dataset repo is public, anyone can access them. Verify the repo visibility is set to private.
• Training data sits unencrypted on the RunPod persistent volume. RunPod's infrastructure team has physical/logical access to the underlying storage.
• The bare except: on pod_setup.sh lines 100-101 swallows authentication errors silently. If HF_TOKEN is missing or wrong, the download fails but the script continues, producing confusing errors later.

Recommendations:

• Confirm HumboldtJoker/daimon-sft-data and HumboldtJoker/sonnet-voice-sft-data are set to private on HuggingFace Hub.
• Replace bare except: with except Exception as e: print(f"Download failed: {e}"); sys.exit(1) — fail loudly on auth errors.
• After training completes and checkpoints are downloaded, delete the training data from the persistent volume.

3. Checkpoint Security — MEDIUM RISK

Status: Checkpoints are written to /workspace/daimon-sft/ (or /workspace/sonnet-voice-sft/) on RunPod's persistent volume. save_total_limit=5 caps at 5 checkpoints (~70 GB each = ~350 GB).

Risks:

• No encryption at rest. RunPod persistent volumes use standard ext4 on their NVMe/SSD pool. RunPod staff with datacenter access can read the raw blocks. This is true of all cloud providers — RunPod is not worse than AWS EBS in this regard, but also not better.
• Checkpoints survive pod termination if on a persistent volume (which they should be — /workspace/ maps to the volume). This is by design for resumability, but means trained weights persist on RunPod infrastructure until explicitly deleted.
• No upload mechanism. The scripts don't push checkpoints anywhere. You must manually scp or rsync them off the pod. Good for security (no accidental upload), but risky for data loss if the volume is accidentally deleted.

Recommendations:

• After training: immediately copy final checkpoint off the pod, then delete the volume.
• Consider encrypting the persistent volume with LUKS (add to setup script): cryptsetup luksFormat /dev/... — but this adds significant complexity and RunPod may not support it on all volume types.
• At minimum, after copying checkpoints off: rm -rf /workspace/daimon-sft/ before terminating the pod.

4. Network Isolation — GOOD

Status: All network access happens during the setup phase:

1. pip install — downloads from PyPI
2. hf_hub_download / from_pretrained — downloads model + data from HuggingFace Hub
3. SSH key injection for remote access

During training: zero outbound network calls. The training loop is purely local computation. report_to="none" prevents any telemetry.

Recommendations:

• For maximum isolation, after setup completes, you could block outbound traffic: iptables -A OUTPUT -p tcp --dport 443 -j DROP (but this breaks SSH access for monitoring). Not recommended unless paranoia level is very high.
• The current approach of "download everything during setup, then train offline" is the right architecture.

5. Pod Termination Data Survival — ACCEPTABLE

Status: The config correctly targets /workspace/ for all outputs, which maps to the RunPod persistent volume. save_steps=1000 means at most 1000 steps of work can be lost on unexpected termination.

The train_daimon_config.yaml explicitly notes: "Checkpoints go to persistent volume. After pod termination, resume from last." This is correct.

Risks:

• Spot/preemptible pods can be terminated with 30 seconds notice. A checkpoint save for a 35B model takes minutes (due to stage3_gather_16bit_weights_on_model_save). An in-progress save during termination could produce a corrupted checkpoint.
• save_total_limit=5 means old checkpoints are auto-deleted. If the latest checkpoint is corrupted, the previous 4 are available as fallback.

Recommendations:

• Use on-demand pods, not spot instances, for this training run. The cost difference is small relative to the value of the trained model.
• Consider save_total_limit=3 to save disk space, since 5 checkpoints at 70 GB each is 350 GB.

6. Credential Hygiene — CLEAN

No API keys, tokens, or secrets hardcoded anywhere in the codebase. Verified across all files in cloud_sft/, runpod-template/, configs/, full_sft/, and merge/.

The .gitignore includes .env, which is correct.

One note: The SSH public key ssh-ed25519 AAAAC3Nza... in pod_setup.sh line 301 is a public key (not a secret) and is safe to commit. However, it does identify Thomas's key fingerprint. Anyone reading the repo knows which key grants pod access. Low risk since pods are ephemeral.

7. Supply Chain — HIGH RISK (Primary Concern)

Status: All pip install commands use unpinned versions with no integrity verification.

New template setup.sh lines 113-127:

pip install --upgrade -q torch torchvision --index-url https://download.pytorch.org/whl/cu124
pip install --upgrade -q transformers trl datasets accelerate deepspeed peft safetensors mpi4py pyyaml
pip install --upgrade -q flash-attn --no-build-isolation

Note: The setup script comments say "Pin versions for reproducibility" on line 110 but then does NOT actually pin any versions. The comment is misleading.

Existing pod_setup.sh line 64:

pip install --upgrade -q transformers trl datasets accelerate deepspeed mpi4py safetensors peft

Existing preflight_check.sh lines 49, 69, 74:

pip install $pkg -q        # for torch, transformers, trl, etc.
pip install deepspeed -q
pip install mpi4py -q

Existing train_sonnet_h100.py line 69:

os.system("pip install trl -q")   # Runtime install — worst pattern

Risks:

• A compromised PyPI package (or typosquatted name) gets full root access on the pod, plus access to the HF_TOKEN, training data, and model weights.
• Running pip install --upgrade on rented infra means you cannot reproduce the exact environment later.
• The train_sonnet_h100.py runtime os.system("pip install trl -q") is the worst variant — it installs an arbitrary version of trl mid-execution.

Recommendations (STRONGLY RECOMMENDED): Add a requirements.txt to the template with pinned versions and hashes:

transformers==4.46.3
trl==1.7.0
datasets==3.2.0
accelerate==1.2.0
deepspeed==0.16.0
mpi4py==4.0.1
safetensors==0.4.5
peft==0.14.0

Then install with: pip install -r requirements.txt --require-hashes (if hash pinning) or at minimum pip install -r requirements.txt (version pinning only).

Remove the os.system("pip install trl -q") from train_sonnet_h100.py — it should never install packages at runtime.

---

Additional Findings

8. trust_remote_code=True — ACCEPTED RISK

Every model loading call passes trust_remote_code=True. This is required for Qwen3.6 models (they include custom modeling code). However, it means the HuggingFace Hub model repo executes arbitrary Python on your pod. If Qwen/Qwen3.6-35B-A3B were compromised or you typo the model name, this is full RCE.

Mitigation: Pin the model revision. In training scripts, use:

AutoModelForCausalLM.from_pretrained(MODEL_ID, revision="<specific_commit_hash>", trust_remote_code=True)

This ensures you get the exact code you audited, not whatever the latest push is.

9. Windows Path Artifacts — LOW RISK / HYGIENE

Two files with Windows paths are in the repo:

• Daimonion/C:\Users\Thomas\.claude\hooks\vera_stop_capture.log (24KB)
• cloud_sft/C:\Users\Thomas\.claude\hooks\vera_stop_capture.log (13KB)

These contain Claude hook output logs. They expose the Windows username "Thomas" and operational details, though no secrets. Should be added to .gitignore and removed from tracking.

10. Large Training Data in Git — HYGIENE

cloud_sft/train.jsonl (814 MB) and cloud_sft/valid.jsonl (42 MB) plus gzipped copies appear to be committed to git. These should be in .gitignore or managed via Git LFS. The .gitignore does not exclude *.jsonl or *.jsonl.gz.

11. New Template Config Review (train_daimon_config.yaml)

The new template config is clean:

• Correctly targets /workspace/ for outputs (persistent volume)
• report_to: "none" — no telemetry
• References HuggingFace dataset by repo name (HumboldtJoker/daimon-sft-data) — will need HF_TOKEN if private
• max_seq_length: 8192 — addresses the previous silent truncation bug
• Optimizer correctly set to adamw_torch (not fused, which crashes with DeepSpeed)
• No secrets, no hardcoded paths except model ID

12. DeepSpeed Config Review (runpod-template/ds_config_zero3.json)

Clean. Notable that param offload to CPU is removed (only optimizer offload remains) compared to the cloud_sft/ configs which offload both. The comments explain this is because CPU param offload makes MoE expert gathering catastrophically slow. This is a good architectural decision and has no security implications.

13. New Training Script Review (runpod-template/train_daimon.py)

Overall well-structured and security-conscious. Specific findings:

GOOD:

• report_to="none" hardcoded on line 348, not configurable from YAML. This prevents accidental telemetry even if someone edits the config. Correct decision.
• Logging goes to local files only (/workspace/daimon-sft/logs/), no remote logging.
• TOKENIZERS_PARALLELISM=false (line 41) prevents data races. Good hygiene.
• Environment variable overrides for paths (lines 256-258) allow runtime flexibility without modifying code. Clean pattern.
• Training summary saved as JSON (lines 399-409) does NOT include the HF token or any secrets. Clean.
• Resume-from-checkpoint support (line 365, 385) is critical for pod interruption recovery. Well implemented.

CONCERNS:

• trust_remote_code=True on lines 288 and 308 (same accepted risk as finding #8).
• Bare except Exception: on line 107 in pre_split_long_sequences() silently falls back to character-count estimation. Not a security issue, but could mask data corruption.
• load_dataset() on line 189 passes token=os.environ.get("HF_TOKEN") which is correct, but if HF_TOKEN is unset, returns None and the call proceeds without auth. If the dataset is private, this would fail with a confusing 404 instead of an auth error. The setup script checks for HF_TOKEN early so this is unlikely in practice, but the training script can be run independently.

14. New Setup Script Review (runpod-template/setup.sh)

GOOD:

• set -e on line 22 means the script aborts on any error. Correct.
• Comprehensive pre-flight checks: GPU count, VRAM, RAM, disk, HF_TOKEN.
• Hard aborts at unsafe thresholds (RAM < 200GB, disk < 200GB, GPU < 2).
• SSH key is a public key (same one as existing scripts). Not a secret.

CONCERNS:

• Lines 110-127: "Pin versions for reproducibility" comment is a lie. The comment says "Pin versions" but the actual pip commands use --upgrade with no version constraints. This is the primary supply-chain risk. Anyone reading the comment would assume versions are pinned. Fix the comment or actually pin versions.
• Line 114: PyTorch installed from pytorch.org index. --index-url https://download.pytorch.org/whl/cu124 is a trusted source (official PyTorch wheel index). Acceptable. But combined with --upgrade, it could pull a new PyTorch version that breaks compatibility.
• Line 130: flash-attn --no-build-isolation compiles from source. The --no-build-isolation flag means build dependencies come from the existing environment, not a clean virtualenv. Low risk since it's a well-known package, but worth noting.
• Lines 240-253: Bare except Exception as e2 catches download failures but prints "DATA MUST BE UPLOADED MANUALLY" and continues. If auth fails, this message is misleading — the real issue is the token, not manual upload.
• Line 277: Inner bare except: pass in JSONL parsing silently drops malformed lines. Same pattern as the existing scripts. Not a security issue but could mask data corruption.
• Line 291: Outer bare except: at the final verification step swallows all errors. If data verification fails, the setup reports success anyway. Should abort or at least warn loudly.

---

15. Launch Script Review (runpod-template/launch.sh)

Clean and well-structured. Specific notes:

• Pre-flight checks are thorough (GPU count, disk, model, config, DeepSpeed config, existing checkpoints).
• HF_TOKEN warning on line 63 is a soft warning, not a hard abort. This is fine since the model may already be cached locally.
• NCCL environment variables (lines 106-122) are standard multi-GPU training config. NCCL_IB_DISABLE=1 disables InfiniBand (correct for RunPod pods which use NVLink within a node). No security implications.
• NCCL_TIMEOUT=3600000 (60 minutes) is generous but appropriate for MoE layer gathering.
• Training output piped through tee to both console and log file. Log file is on persistent volume. No sensitive data is logged (only loss values, step counts, GPU metrics).
• No security issues found in this file.

16. Test Script Review (runpod-template/test_template.py)

Clean validation script. Notes:

• Writes a temporary test file to /workspace/.daimon_write_test (line 206) and immediately removes it (line 209). Clean pattern.
• Creates temporary directory /tmp/daimon_test for SFTConfig validation and cleans it up (lines 290-292). Clean.
• trust_remote_code=True on lines 127, 135, 167, 247. Same accepted risk.
• The smoke test (test 6) does NOT load the full model on a single GPU (would OOM). It only verifies imports and config. Well-designed — avoids false OOM signals.
• No security issues found in this file.

---

Risk Summary

#	Finding	Severity	Status
1	HF_TOKEN via env var (correct) but persists in cache/history	Low	Add cleanup steps
2	Training data privacy (no exfiltration)	Clean	Verify HF repo is private
3	Checkpoints unencrypted on persistent volume	Medium	Delete after extraction
4	Network isolation during training	Clean	Good architecture
5	Pod termination resilience	Clean	Use on-demand, not spot
6	No hardcoded credentials	Clean	No action needed
7	Unpinned pip dependencies (all scripts)	HIGH	Pin versions immediately
8	trust_remote_code=True (all model loads)	Medium	Pin model revision hash
9	Windows path log artifacts	Low	Add to .gitignore
10	Large data files in git	Low	Use LFS or .gitignore
11	setup.sh misleading "pin versions" comment	Medium	Fix comment or pin versions
12	Silent error swallowing in data verification	Low	Make failures loud
13	train_daimon.py clean architecture	Clean	No action needed
14	ds_config_zero3.json well-commented	Clean	No action needed

---

Recommended Actions Before First RunPod Launch

1. Create requirements.txt with pinned versions in this template directory.
2. Pin the Qwen model revision to a specific commit hash in train_daimon_config.yaml.
3. Verify HuggingFace dataset repos are private (daimon-sft-data, sonnet-voice-sft-data).
4. Create a fine-grained, read-only HF token scoped to only the needed repos.
5. Add cleanup steps to the setup script: clear HF token cache and bash history after downloads.
6. After training completes: copy checkpoints off the pod, then delete the volume.

---

What Should NOT Worry Thomas

• RunPod seeing the training process: They can see GPU utilization and process names, but cannot see into the training data or model weights without accessing the volume storage directly. This is equivalent to any cloud provider.
• Network exfiltration: The training loop makes zero outbound calls. report_to="none" is correctly set everywhere.
• The SSH key in the repo: It's a public key. The private key is what matters, and that stays on Thomas's machine.
• DeepSpeed / HuggingFace telemetry: No telemetry endpoints are configured. The scripts are clean.

The main thing that should concern a privacy-first organization running on rented infrastructure is the supply-chain risk from unpinned pip packages. Everything else is either clean or has straightforward mitigations.