code/classifier/README.md

Own classifier: code and pod runbook

Spec: docs/market-psychology-own-classifier-training.md.

What it is: Qwen3.5-9B-Base with LoRA (r32) and one linear head per field. The design is adapted from the read-only reference ~/yt_live_analysis/training. Nothing there is imported or changed.

Modules

Module	Needs torch	Purpose
taxonomy.py	no	Fields and options from taxonomy.json; hard rules, rule fill, gating
formats.py	no	Segments to token ids, with segment-wise truncation (never headers or FACTS)
data.py	no	Subsets, steps of 32 similar-length posts, token-budget micro-batches, per-step null counts
calibrate.py	no	Softmax with temperature on stored logits; NLL grid fit on dev; author-half cross-fit
metrics.py	no	Accuracy, macro-F1, log-loss, ECE, shares, bootstrap
robustness.py	no	Injection, context-attack and date-shift copies of benchmark rows
candidates.py	no	Round-2 candidate ranking from the pool scores (thin-class probability, entropy, Korean quota, author cap)
ledger.py	no	Local spend ledger and stage guards ($35 no new stage, $45 stop)
model.py	yes	Text-only Qwen3.5 load, LoRA, heads, per-step masked loss, checkpoints
train.py	yes	Training loop: cumulative --max-hours, --stop-after, --force-longest, --kernels fast|torch (fast asserted), best by dev log-loss; a non-finite step stops before the update
predict.py	yes	Logits for a split or JSONL; posts/s and tokens/s; --repeat for determinism
check_env.py	yes	GPU, versions, special tokens, 9B load, fast kernel path
selftest.py	yes	Tiny 4-layer model (full, DeltaNet, DeltaNet, full) with the real head sizes: per-tensor LoRA gradient parity against the PyTorch path, padding invariance, split step, save/load

Related tools in evaluation/market_psych_v2/:

• classifier_export.py builds data/market_psych/development/classifier-data-20261017/.
• classifier_eval.py scores the downloaded predictions locally, including the comparison with d3.

Local checks (no GPU)

python3 -m unittest discover -s evaluation/market_psych_v2/classifier -t evaluation/market_psych_v2

CPU dry run of the torch code (2026-10-17, repeated after the code review's fixes): it ran in a throwaway Python 3.12 venv with CPU torch 2.14 plus the pinned transformers 5.20.1 and peft 0.21.2, against the real Qwen3.5-9B-Base config and tokenizer.

• Selftest: padding invariance relative error 1e-6 (two DeltaNet layers), split-step gradient cosine 1.0, save/load exact.
  • Parity ran over 52 LoRA tensors. It correctly fails on CPU because no fast kernels are active.
  • Default mode: train.py and predict.py likewise refuse to start without kernels, unless given --kernels torch.
• Training: a tiny 2-layer model on a synthetic export:
  • ran train, then a planned stop (exit 3) and a resume to the end, with dev loss falling;
  • refused a new run over an existing one, and a resume with changed settings.
• After training:
  • Predict with an identical repeat. Re-predicting dev from the saved best checkpoint matches the training-time predictions exactly.
  • Calibration fit, and robustness build and predict.

Pod session

All sync.sh commands run on this machine; the others run on the pod inside tmux.

1. The user registers ~/.ssh/dom_runpod.pub in RunPod, then creates the pod:

   • 1× H100 SXM 80 GB;
   • a PyTorch devel image with torch 2.8 and CUDA 12.8, recording the exact tag;
   • 50 GB container disk, plus a 100 GB volume at /workspace;
   • TCP 22 exposed.

   The user then shares the direct SSH host, port and creation time.

   The first pod (2026-10-17) had a RunPod network volume at /workspace instead: a FUSE mount, measured at about 125 MB/s writing and 210 MB/s reading, that refuses chown and setting file times. So:

   • sync.sh up compares by checksum (-rl --checksum);
   • the base checkpoint (/root/models) and the HF cache (/root/hf) live on the 50 GB container disk, next to the venv, so a pod restart means rerunning bootstrap.sh;
   • runs stay on /workspace, which outlives the pod;
   • watch logs on the container disk (tee /root/logs/...): a tail -F of a file on the volume saw no updates;
   • stages 2–5 ran as one tmux chain that stops at the first failure, after one ledger check for the whole chain.

   Results: data/market_psych/development/classifier-v1-20261017/README.md.

2. Start the ledger:

   python3 -m classifier.ledger start --ledger ~/twitter_models/classifier-v1/ledger.json --price <$/h> --started-at <ISO>
   

   Before each stage, run ledger check --stage <name> --hours <estimate> and go only on exit 0.

3. Ship: classifier/runpod/sync.sh up HOST PORT, then on the pod run bash /workspace/code/classifier/runpod/bootstrap.sh.

   • It must print BOOTSTRAP OK.
   • check_env.json and selftest.json are copied back by sync.sh down.
   • If parity fails, follow the spec's pre-declared fallback; do not train on the fast path.

4. Train and evaluate on the pod, each from /workspace/code:

   • bash classifier/runpod/run.sh smoke. Read the tokens/s and the forced-step memory, then re-plan with the ledger.
   • run.sh curve, then run.sh full.
   • run.sh evaluate /workspace/runs/curve-8k and run.sh evaluate /workspace/runs/full.
   • run.sh pool /workspace/runs/full, if the guards allow.

5. Bring the results back: sync.sh down HOST PORT copies them to ~/twitter_models/classifier-v1/. Verify the files, then tell the user to terminate the pod.

6. Score locally with classifier_eval.py: metrics, d3, teacher, curve and robustness. Then run python -m classifier.candidates on the full run for the round-2 list.

   • Determinism: compare RUN/predictions/dev.jsonl (training time) with dev-reload.jsonl (the saved checkpoint).
   • Afterwards: write the run README, and commit only when asked.

Stopping and terminating the pod is the user's job (user decision). The ledger and the alerts only advise.