# Own classifier: code and pod runbook

**Spec:** [docs/market-psychology-own-classifier-training.md](../../../docs/market-psychology-own-classifier-training.md).

**What it is:** Qwen3.5-9B-Base with LoRA (r32) and one linear head per field. The design is adapted from
the read-only reference `~/yt_live_analysis/training`. Nothing there is imported or changed.

## Modules

| Module | Needs torch | Purpose |
|---|---|---|
| `taxonomy.py` | no | Fields and options from `taxonomy.json`; hard rules, rule fill, gating |
| `formats.py` | no | Segments to token ids, with segment-wise truncation (never headers or FACTS) |
| `data.py` | no | Subsets, steps of 32 similar-length posts, token-budget micro-batches, per-step null counts |
| `calibrate.py` | no | Softmax with temperature on stored logits; NLL grid fit on dev; author-half cross-fit |
| `metrics.py` | no | Accuracy, macro-F1, log-loss, ECE, shares, bootstrap |
| `robustness.py` | no | Injection, context-attack and date-shift copies of benchmark rows |
| `candidates.py` | no | Round-2 candidate ranking from the pool scores (thin-class probability, entropy, Korean quota, author cap) |
| `ledger.py` | no | Local spend ledger and stage guards ($35 no new stage, $45 stop) |
| `model.py` | yes | Text-only Qwen3.5 load, LoRA, heads, per-step masked loss, checkpoints |
| `train.py` | yes | Training loop: cumulative `--max-hours`, `--stop-after`, `--force-longest`, `--kernels fast\|torch` (fast asserted), best by dev log-loss; a non-finite step stops before the update |
| `predict.py` | yes | Logits for a split or JSONL; posts/s and tokens/s; `--repeat` for determinism |
| `check_env.py` | yes | GPU, versions, special tokens, 9B load, fast kernel path |
| `selftest.py` | yes | Tiny 4-layer model (full, DeltaNet, DeltaNet, full) with the real head sizes: per-tensor LoRA gradient parity against the PyTorch path, padding invariance, split step, save/load |

**Related tools** in `evaluation/market_psych_v2/`:
- `classifier_export.py` builds `data/market_psych/development/classifier-data-20261017/`.
- `classifier_eval.py` scores the downloaded predictions locally, including the comparison with d3.

## Local checks (no GPU)

```
python3 -m unittest discover -s evaluation/market_psych_v2/classifier -t evaluation/market_psych_v2
```

**CPU dry run of the torch code (2026-10-17, repeated after the code review's fixes):** it ran in a throwaway Python 3.12 venv with CPU torch 2.14
plus the pinned transformers 5.20.1 and peft 0.21.2, against the real Qwen3.5-9B-Base config and tokenizer.
- **Selftest:** padding invariance relative error 1e-6 (two DeltaNet layers), split-step gradient cosine 1.0,
  save/load exact.
  - **Parity** ran over 52 LoRA tensors. It correctly fails on CPU because no fast kernels are active.
  - **Default mode:** `train.py` and `predict.py` likewise refuse to start without kernels, unless given
    `--kernels torch`.
- **Training:** a tiny 2-layer model on a synthetic export:
  - ran train, then a planned stop (exit 3) and a resume to the end, with dev loss falling;
  - refused a new run over an existing one, and a resume with changed settings.
- **After training:**
  - Predict with an identical repeat. Re-predicting dev from the saved best checkpoint matches the
    training-time predictions exactly.
  - Calibration fit, and robustness build and predict.

## Pod session

All `sync.sh` commands run on this machine; the others run on the pod inside `tmux`.

1. **The user** registers `~/.ssh/dom_runpod.pub` in RunPod, then creates the pod:
   - 1× H100 SXM 80 GB;
   - a PyTorch devel image with torch 2.8 and CUDA 12.8, recording the exact tag;
   - 50 GB container disk, plus a 100 GB volume at `/workspace`;
   - TCP 22 exposed.

   The user then shares the direct SSH host, port and creation time.

   **The first pod (2026-10-17)** had a RunPod network volume at `/workspace` instead: a FUSE mount, measured at about
   125 MB/s writing and 210 MB/s reading, that refuses chown and setting file times. So:
   - `sync.sh up` compares by checksum (`-rl --checksum`);
   - the base checkpoint (`/root/models`) and the HF cache (`/root/hf`) live on the 50 GB container disk, next to
     the venv, so a pod restart means rerunning `bootstrap.sh`;
   - runs stay on `/workspace`, which outlives the pod;
   - watch logs on the container disk (`tee /root/logs/...`): a `tail -F` of a file on the volume saw no updates;
   - stages 2–5 ran as one tmux chain that stops at the first failure, after one ledger check for the whole chain.

   Results: `data/market_psych/development/classifier-v1-20261017/README.md`.
2. **Start the ledger:**
   ```
   python3 -m classifier.ledger start --ledger ~/twitter_models/classifier-v1/ledger.json --price <$/h> --started-at <ISO>
   ```
   Before each stage, run `ledger check --stage <name> --hours <estimate>` and go only on exit 0.
3. **Ship:** `classifier/runpod/sync.sh up HOST PORT`, then on the pod run
   `bash /workspace/code/classifier/runpod/bootstrap.sh`.
   - It must print `BOOTSTRAP OK`.
   - `check_env.json` and `selftest.json` are copied back by `sync.sh down`.
   - If parity fails, follow the spec's pre-declared fallback; do not train on the fast path.
4. **Train and evaluate on the pod**, each from `/workspace/code`:
   - `bash classifier/runpod/run.sh smoke`. Read the tokens/s and the forced-step memory, then re-plan with
     the ledger.
   - `run.sh curve`, then `run.sh full`.
   - `run.sh evaluate /workspace/runs/curve-8k` and `run.sh evaluate /workspace/runs/full`.
   - `run.sh pool /workspace/runs/full`, if the guards allow.
5. **Bring the results back:** `sync.sh down HOST PORT` copies them to `~/twitter_models/classifier-v1/`.
   Verify the files, then **tell the user to terminate the pod**.
6. **Score locally** with `classifier_eval.py`: `metrics`, `d3`, `teacher`, `curve` and `robustness`. Then run
   `python -m classifier.candidates` on the full run for the round-2 list.
   - **Determinism:** compare `RUN/predictions/dev.jsonl` (training time) with `dev-reload.jsonl` (the
     saved checkpoint).
   - **Afterwards:** write the run README, and commit only when asked.

Stopping and terminating the pod is the user's job (user decision). The ledger and the alerts only advise.