Correct the model card (pooling is at the LABELS suffix's last token); add code/classifier from project commit 4ced7554f
3d7c238 verified | # Own classifier: code and pod runbook | |
| **Spec:** [docs/market-psychology-own-classifier-training.md](../../../docs/market-psychology-own-classifier-training.md). | |
| **What it is:** Qwen3.5-9B-Base with LoRA (r32) and one linear head per field. The design is adapted from | |
| the read-only reference `~/yt_live_analysis/training`. Nothing there is imported or changed. | |
| ## Modules | |
| | Module | Needs torch | Purpose | | |
| |---|---|---| | |
| | `taxonomy.py` | no | Fields and options from `taxonomy.json`; hard rules, rule fill, gating | | |
| | `formats.py` | no | Segments to token ids, with segment-wise truncation (never headers or FACTS) | | |
| | `data.py` | no | Subsets, steps of 32 similar-length posts, token-budget micro-batches, per-step null counts | | |
| | `calibrate.py` | no | Softmax with temperature on stored logits; NLL grid fit on dev; author-half cross-fit | | |
| | `metrics.py` | no | Accuracy, macro-F1, log-loss, ECE, shares, bootstrap | | |
| | `robustness.py` | no | Injection, context-attack and date-shift copies of benchmark rows | | |
| | `candidates.py` | no | Round-2 candidate ranking from the pool scores (thin-class probability, entropy, Korean quota, author cap) | | |
| | `ledger.py` | no | Local spend ledger and stage guards ($35 no new stage, $45 stop) | | |
| | `model.py` | yes | Text-only Qwen3.5 load, LoRA, heads, per-step masked loss, checkpoints | | |
| | `train.py` | yes | Training loop: cumulative `--max-hours`, `--stop-after`, `--force-longest`, `--kernels fast\|torch` (fast asserted), best by dev log-loss; a non-finite step stops before the update | | |
| | `predict.py` | yes | Logits for a split or JSONL; posts/s and tokens/s; `--repeat` for determinism | | |
| | `check_env.py` | yes | GPU, versions, special tokens, 9B load, fast kernel path | | |
| | `selftest.py` | yes | Tiny 4-layer model (full, DeltaNet, DeltaNet, full) with the real head sizes: per-tensor LoRA gradient parity against the PyTorch path, padding invariance, split step, save/load | | |
| **Related tools** in `evaluation/market_psych_v2/`: | |
| - `classifier_export.py` builds `data/market_psych/development/classifier-data-20261017/`. | |
| - `classifier_eval.py` scores the downloaded predictions locally, including the comparison with d3. | |
| ## Local checks (no GPU) | |
| ``` | |
| python3 -m unittest discover -s evaluation/market_psych_v2/classifier -t evaluation/market_psych_v2 | |
| ``` | |
| **CPU dry run of the torch code (2026-10-17, repeated after the code review's fixes):** it ran in a throwaway Python 3.12 venv with CPU torch 2.14 | |
| plus the pinned transformers 5.20.1 and peft 0.21.2, against the real Qwen3.5-9B-Base config and tokenizer. | |
| - **Selftest:** padding invariance relative error 1e-6 (two DeltaNet layers), split-step gradient cosine 1.0, | |
| save/load exact. | |
| - **Parity** ran over 52 LoRA tensors. It correctly fails on CPU because no fast kernels are active. | |
| - **Default mode:** `train.py` and `predict.py` likewise refuse to start without kernels, unless given | |
| `--kernels torch`. | |
| - **Training:** a tiny 2-layer model on a synthetic export: | |
| - ran train, then a planned stop (exit 3) and a resume to the end, with dev loss falling; | |
| - refused a new run over an existing one, and a resume with changed settings. | |
| - **After training:** | |
| - Predict with an identical repeat. Re-predicting dev from the saved best checkpoint matches the | |
| training-time predictions exactly. | |
| - Calibration fit, and robustness build and predict. | |
| ## Pod session | |
| All `sync.sh` commands run on this machine; the others run on the pod inside `tmux`. | |
| 1. **The user** registers `~/.ssh/dom_runpod.pub` in RunPod, then creates the pod: | |
| - 1× H100 SXM 80 GB; | |
| - a PyTorch devel image with torch 2.8 and CUDA 12.8, recording the exact tag; | |
| - 50 GB container disk, plus a 100 GB volume at `/workspace`; | |
| - TCP 22 exposed. | |
| The user then shares the direct SSH host, port and creation time. | |
| **The first pod (2026-10-17)** had a RunPod network volume at `/workspace` instead: a FUSE mount, measured at about | |
| 125 MB/s writing and 210 MB/s reading, that refuses chown and setting file times. So: | |
| - `sync.sh up` compares by checksum (`-rl --checksum`); | |
| - the base checkpoint (`/root/models`) and the HF cache (`/root/hf`) live on the 50 GB container disk, next to | |
| the venv, so a pod restart means rerunning `bootstrap.sh`; | |
| - runs stay on `/workspace`, which outlives the pod; | |
| - watch logs on the container disk (`tee /root/logs/...`): a `tail -F` of a file on the volume saw no updates; | |
| - stages 2–5 ran as one tmux chain that stops at the first failure, after one ledger check for the whole chain. | |
| Results: `data/market_psych/development/classifier-v1-20261017/README.md`. | |
| 2. **Start the ledger:** | |
| ``` | |
| python3 -m classifier.ledger start --ledger ~/twitter_models/classifier-v1/ledger.json --price <$/h> --started-at <ISO> | |
| ``` | |
| Before each stage, run `ledger check --stage <name> --hours <estimate>` and go only on exit 0. | |
| 3. **Ship:** `classifier/runpod/sync.sh up HOST PORT`, then on the pod run | |
| `bash /workspace/code/classifier/runpod/bootstrap.sh`. | |
| - It must print `BOOTSTRAP OK`. | |
| - `check_env.json` and `selftest.json` are copied back by `sync.sh down`. | |
| - If parity fails, follow the spec's pre-declared fallback; do not train on the fast path. | |
| 4. **Train and evaluate on the pod**, each from `/workspace/code`: | |
| - `bash classifier/runpod/run.sh smoke`. Read the tokens/s and the forced-step memory, then re-plan with | |
| the ledger. | |
| - `run.sh curve`, then `run.sh full`. | |
| - `run.sh evaluate /workspace/runs/curve-8k` and `run.sh evaluate /workspace/runs/full`. | |
| - `run.sh pool /workspace/runs/full`, if the guards allow. | |
| 5. **Bring the results back:** `sync.sh down HOST PORT` copies them to `~/twitter_models/classifier-v1/`. | |
| Verify the files, then **tell the user to terminate the pod**. | |
| 6. **Score locally** with `classifier_eval.py`: `metrics`, `d3`, `teacher`, `curve` and `robustness`. Then run | |
| `python -m classifier.candidates` on the full run for the round-2 list. | |
| - **Determinism:** compare `RUN/predictions/dev.jsonl` (training time) with `dev-reload.jsonl` (the | |
| saved checkpoint). | |
| - **Afterwards:** write the run README, and commit only when asked. | |
| Stopping and terminating the pod is the user's job (user decision). The ledger and the alerts only advise. | |