| ================================================================= |
| 🚀 Phase 2: SFT Fine-tuning on H100 |
| Data: /vol/data/phase2_sft_chunks.pt |
| Resume: /vol/checkpoints/novamind_pretrain_final.pt |
| Out: /vol/checkpoints |
| Fixes: 500x identity seeds | 2x EOS weight | pre-tokenized |
| ================================================================= |
| CMD: python /workspace/train.py --phase sft --data_file /vol/data/phase2_sft_chunks.pt --resume /vol/checkpoints/novamind_pretrain_final.pt --out_dir /vol/checkpoints --batch_size 16 --grad_accum_steps 4 --lr 1e-4 --warmup_steps 500 --epochs 2 --max_seq_len 2048 --save_every 500 --milestone_every 2000 --val_every 500 --val_batches 50 --log_every 10 --num_workers 4 --compile --no_wandb |
|
|
| Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. |
|
|
| ═════════════════════════════════════════════════════════════════ |
| 🚀 NovaMind-256M Training |
| Phase : SFT — Phase 2 — Instruction SFT (~300M tokens: Alpaca + DailyDialog) |
| Device: cuda Precision: torch.bfloat16 |
| GPU 0 : NVIDIA H100 80GB HBM3 (85.0 GB) |
| Peak LR: 1.00e-04 → min 1.00e-05 | Warmup: 500 steps |
| Batch: 16 seqs × 2048 tokens × 4 accum = 131,072 tok/update |
| Think-token loss weight: 0.5 |
| Checkpoint every: 500 steps | Milestone every: 2000 steps |
| ═════════════════════════════════════════════════════════════════ |
|
|
| 🔤 Loading tokenizer... |
| ✅ NousResearch/Llama-2-7b-hf — vocab: 32,000 |
|
|
| 📂 Loading sft dataset... |
| ⚡ Loading pre-tokenized SFT data from disk: /vol/data/phase2_sft_chunks.pt |
| ✅ 23,335 chunks (0.05B tokens) |
| Train: 22,169 chunks | Val: 1,166 chunks |
| DataLoader: 1,386 batches × 16 |
|
|
| 🏗️ Building model... |
| ⚡ torch.compile enabled... |
| ────────────────────────────────────────────── |
| NovaMind-256M Parameter Breakdown |
| ────────────────────────────────────────────── |
| Token Embedding (shared w/ lm_head) 32.77M |
| Attention QKV projections (per layer) 1.57M |
| Attention O projection (per layer) 1.05M |
| SwiGLU FFN (per layer) 7.08M |
| RMSNorm × 2 (per layer) 0.0020M |
| × {n} layers total 232.83M |
| Final RMSNorm 0.0010M |
| LM Head (weight-tied to embedding) → 0.00M |
| ────────────────────────────────────────────── |
| TOTAL (formula estimate) 265.60M |
| TOTAL (actual PyTorch count) 265.60M |
| ────────────────────────────────────────────── |
|
|
| Steps/epoch: 347 | Total max steps: 694 |
| Estimated training time on H100: ~0.1 hrs |
|
|
|
|
| 📂 Loading checkpoint: /vol/checkpoints/novamind_pretrain_final.pt |
| ✅ Resumed: phase=pretrain | step=17,829 | tokens=3.277B | last_loss=2.0787129402160645 |
|
|
| ⏩ Transitioning from phase 'pretrain' to 'sft'. Resetting step counter. |
|
|
| 📊 Logger ready: |
| JSON-lines → /vol/checkpoints/log_sft_20260501_083611.jsonl |
| CSV → /vol/checkpoints/log_sft_20260501_083611.csv |
| W&B → disabled (install wandb for cloud tracking) |
|
|
| ═════════════════════════════════════════════════════════════════ |
| 🚦 TRAINING STARTED — Phase: SFT |
| Resume from step: 0 | Target: 694 steps |
| ═════════════════════════════════════════════════════════════════ |
|
|
| 📅 Epoch 1/2 |
| step 10 [████████████████████] loss=1.9853 (think=1.0418 resp=3.3813) | lr=2.00e-06 gnorm=3.851 | 288,483 tok/s | mem=15.8GB | elapsed=0.5m ETA=32m |
| step 20 [████████████████████] loss=1.9734 (think=0.1326 resp=3.0440) | lr=4.00e-06 gnorm=2.967 | 300,708 tok/s | mem=15.8GB | elapsed=0.8m ETA=25m |
| step 30 [████████████████████] loss=1.6090 (think=0.0144 resp=2.4704) | lr=6.00e-06 gnorm=1.723 | 300,809 tok/s | mem=15.8GB | elapsed=1.0m ETA=23m |
| step 40 [████████████████████] loss=1.3728 (think=0.0043 resp=2.6637) | lr=8.00e-06 gnorm=0.987 | 291,490 tok/s | mem=15.8GB | elapsed=1.3m ETA=22m |
| step 50 [████████████████████] loss=1.5911 (think=0.0277 resp=2.0765) | lr=1.00e-05 gnorm=0.846 | 300,464 tok/s | mem=15.8GB | elapsed=1.6m ETA=21m |
| step 60 [████████████████████] loss=1.6621 (think=0.0030 resp=3.3728) | lr=1.20e-05 gnorm=0.855 | 300,868 tok/s | mem=15.8GB | elapsed=1.9m ETA=20m |
| step 70 [████████████████████] loss=1.2767 (think=0.0035 resp=2.7312) | lr=1.40e-05 gnorm=0.870 | 300,807 tok/s | mem=15.8GB | elapsed=2.2m ETA=19m |
| step 80 [████████████████████] loss=1.6785 (think=0.0032 resp=2.4103) | lr=1.60e-05 gnorm=0.912 | 300,788 tok/s | mem=15.8GB | elapsed=2.5m ETA=19m |
| step 90 [████████████████████] loss=1.1626 (think=0.0567 resp=1.7926) | lr=1.80e-05 gnorm=0.724 | 294,560 tok/s | mem=15.8GB | elapsed=2.7m ETA=18m |
| step 100 [████████████████████] loss=1.4615 (think=0.0068 resp=2.5660) | lr=2.00e-05 gnorm=1.506 | 300,820 tok/s | mem=15.8GB | elapsed=3.0m ETA=18m |
| step 110 [████████████████████] loss=1.2086 (think=0.0131 resp=2.4370) | lr=2.20e-05 gnorm=0.761 | 300,719 tok/s | mem=15.8GB | elapsed=3.3m ETA=18m |
| step 120 [████████████████████] loss=1.4017 (think=0.0066 resp=2.0912) | lr=2.40e-05 gnorm=0.999 | 294,308 tok/s | mem=15.8GB | elapsed=3.6m ETA=17m |
| step 130 [████████████████████] loss=1.3652 (think=0.0009 resp=2.4654) | lr=2.60e-05 gnorm=1.067 | 300,828 tok/s | mem=15.8GB | elapsed=3.9m ETA=17m |
| step 140 [████████████████████] loss=1.2592 (think=0.0079 resp=1.8993) | lr=2.80e-05 gnorm=1.056 | 294,823 tok/s | mem=15.8GB | elapsed=4.2m ETA=16m |
| step 150 [████████████████████] loss=1.5117 (think=0.0071 resp=2.6241) | lr=3.00e-05 gnorm=0.904 | 300,638 tok/s | mem=15.8GB | elapsed=4.5m ETA=16m |
| step 160 [████████████████████] loss=1.3731 (think=0.0051 resp=1.9573) | lr=3.20e-05 gnorm=0.886 | 300,765 tok/s | mem=15.8GB | elapsed=4.7m ETA=16m |
| step 170 [████████████████████] loss=1.2590 (think=0.0015 resp=2.3390) | lr=3.40e-05 gnorm=0.818 | 300,694 tok/s | mem=15.8GB | elapsed=5.0m ETA=16m |
| step 180 [████████████████████] loss=1.1489 (think=0.0068 resp=1.9164) | lr=3.60e-05 gnorm=0.846 | 300,702 tok/s | mem=15.8GB | elapsed=5.3m ETA=15m |
| step 190 [████████████████████] loss=1.2557 (think=0.0050 resp=2.0132) | lr=3.80e-05 gnorm=0.722 | 300,482 tok/s | mem=15.8GB | elapsed=5.6m ETA=15m |
| step 200 [████████████████████] loss=1.2009 (think=0.0088 resp=2.4839) | lr=4.00e-05 gnorm=0.735 | 300,978 tok/s | mem=15.8GB | elapsed=5.9m ETA=14m |
| step 210 [████████████████████] loss=1.1573 (think=0.0229 resp=2.3217) | lr=4.20e-05 gnorm=0.682 | 300,876 tok/s | mem=15.8GB | elapsed=6.2m ETA=14m |
| step 220 [████████████████████] loss=1.3889 (think=0.0084 resp=2.0943) | lr=4.40e-05 gnorm=0.742 | 292,315 tok/s | mem=15.8GB | elapsed=6.4m ETA=14m |
| step 230 [████████████████████] loss=1.0675 (think=0.0070 resp=1.4795) | lr=4.60e-05 gnorm=1.336 | 301,162 tok/s | mem=15.8GB | elapsed=6.7m ETA=14m |
| step 240 [████████████████████] loss=1.1577 (think=0.0113 resp=2.5419) | lr=4.80e-05 gnorm=1.044 | 300,878 tok/s | mem=15.8GB | elapsed=7.0m ETA=13m |
| step 250 [████████████████████] loss=1.2763 (think=0.0112 resp=2.2808) | lr=5.00e-05 gnorm=0.803 | 301,165 tok/s | mem=15.8GB | elapsed=7.3m ETA=13m |
| step 260 [████████████████████] loss=1.1743 (think=0.0072 resp=1.7669) | lr=5.20e-05 gnorm=0.970 | 300,995 tok/s | mem=15.8GB | elapsed=7.6m ETA=13m |
| step 270 [████████████████████] loss=1.1426 (think=0.0266 resp=1.7937) | lr=5.40e-05 gnorm=0.715 | 300,758 tok/s | mem=15.8GB | elapsed=7.8m ETA=12m |
| step 280 [████████████████████] loss=1.1629 (think=0.0180 resp=1.5782) | lr=5.60e-05 gnorm=0.930 | 300,286 tok/s | mem=15.8GB | elapsed=8.1m ETA=12m |
| step 290 [████████████████████] loss=1.1471 (think=0.0026 resp=2.0983) | lr=5.80e-05 gnorm=0.686 | 300,618 tok/s | mem=15.8GB | elapsed=8.4m ETA=12m |
| step 300 [████████████████████] loss=1.1391 (think=0.0262 resp=1.6352) | lr=6.00e-05 gnorm=0.847 | 300,490 tok/s | mem=15.8GB | elapsed=8.7m ETA=11m |
| step 310 [████████████████████] loss=1.2983 (think=0.0040 resp=2.3457) | lr=6.20e-05 gnorm=0.793 | 300,434 tok/s | mem=15.8GB | elapsed=9.0m ETA=11m |
| step 320 [████████████████████] loss=1.0637 (think=0.0040 resp=2.0149) | lr=6.40e-05 gnorm=0.755 | 300,815 tok/s | mem=15.8GB | elapsed=9.3m ETA=11m |
| step 330 [████████████████████] loss=1.1190 (think=0.0087 resp=1.6660) | lr=6.60e-05 gnorm=0.654 | 300,781 tok/s | mem=15.8GB | elapsed=9.6m ETA=10m |
| step 340 [████████████████████] loss=1.2121 (think=0.0165 resp=1.7479) | lr=6.80e-05 gnorm=0.746 | 301,057 tok/s | mem=15.8GB | elapsed=9.8m ETA=10m |
|
|
| ✅ Epoch 1/2 done | steps=346 | tokens=3.322B |
|
|
| 📅 Epoch 2/2 |
| step 350 [████████████████████] loss=1.1528 (think=0.0011 resp=1.7449) | lr=7.00e-05 gnorm=1.119 | 300,954 tok/s | mem=15.8GB | elapsed=10.2m ETA=10m |
| step 360 [████████████████████] loss=1.3285 (think=0.0071 resp=1.4439) | lr=7.20e-05 gnorm=0.579 | 300,932 tok/s | mem=15.8GB | elapsed=10.5m ETA=10m |
| step 370 [████████████████████] loss=1.2535 (think=0.0070 resp=1.2664) | lr=7.40e-05 gnorm=0.750 | 300,872 tok/s | mem=15.8GB | elapsed=10.8m ETA=10m |
| step 380 [████████████████████] loss=1.2360 (think=0.0010 resp=2.2591) | lr=7.60e-05 gnorm=0.757 | 292,942 tok/s | mem=15.8GB | elapsed=11.1m ETA=9m |
| step 390 [████████████████████] loss=1.0679 (think=0.0002 resp=2.2035) | lr=7.80e-05 gnorm=0.771 | 300,608 tok/s | mem=15.8GB | elapsed=11.4m ETA=9m |
| step 400 [████████████████████] loss=1.0097 (think=0.0099 resp=2.0128) | lr=8.00e-05 gnorm=0.718 | 300,775 tok/s | mem=15.8GB | elapsed=11.7m ETA=9m |
| step 410 [████████████████████] loss=1.0403 (think=0.0119 resp=1.4458) | lr=8.20e-05 gnorm=0.607 | 300,443 tok/s | mem=15.8GB | elapsed=11.9m ETA=8m |
| step 420 [████████████████████] loss=1.0789 (think=0.0047 resp=2.6394) | lr=8.40e-05 gnorm=0.665 | 300,605 tok/s | mem=15.8GB | elapsed=12.2m ETA=8m |
| step 430 [████████████████████] loss=1.0208 (think=0.0052 resp=2.1282) | lr=8.60e-05 gnorm=0.800 | 292,255 tok/s | mem=15.8GB | elapsed=12.5m ETA=8m |
| step 440 [████████████████████] loss=1.2623 (think=0.0106 resp=1.7723) | lr=8.80e-05 gnorm=0.975 | 300,682 tok/s | mem=15.8GB | elapsed=12.8m ETA=7m |
| step 450 [████████████████████] loss=1.2081 (think=0.0118 resp=2.2160) | lr=9.00e-05 gnorm=0.950 | 300,552 tok/s | mem=15.8GB | elapsed=13.1m ETA=7m |
| step 460 [████████████████████] loss=1.2113 (think=0.0076 resp=1.9569) | lr=9.20e-05 gnorm=0.846 | 300,548 tok/s | mem=15.8GB | elapsed=13.4m ETA=7m |
| step 470 [████████████████████] loss=1.2671 (think=0.0033 resp=1.7317) | lr=9.40e-05 gnorm=0.920 | 291,310 tok/s | mem=15.8GB | elapsed=13.7m ETA=6m |
| step 480 [████████████████████] loss=0.9992 (think=0.0169 resp=1.3905) | lr=9.60e-05 gnorm=0.924 | 294,298 tok/s | mem=15.8GB | elapsed=13.9m ETA=6m |
| step 490 [████████████████████] loss=0.9905 (think=0.0038 resp=2.2109) | lr=9.80e-05 gnorm=0.687 | 300,415 tok/s | mem=15.8GB | elapsed=14.2m ETA=6m |
| step 500 [████████████████████] loss=1.2137 (think=0.0296 resp=1.4885) | lr=1.00e-04 gnorm=0.584 | 300,886 tok/s | mem=15.8GB | elapsed=14.5m ETA=6m |
|
|
| ───────────────────────────────────────────────────────────────── |
| 📊 VALIDATION step=500 val_loss=1.093405 |
| ───────────────────────────────────────────────────────────────── |
| Q: What is the capital of France? |
| A: British Empire |
|
|
| Q: What is 15 multiplied by 17? |
| A: 15 x 3,480 = 2900.006x^2 |
|
|
| Q: Tell me a short joke. |
| A: Sorry about it, I don't know how to answer it in the first place! |
| But it is so funny that you can make up your mind of what happened if someone would just laugh at me and run away with my money! |
|
|
| Q: Explain what a neural network is in simple terms. |
| A: A neural network (or more accurately, the " Network of Brain Systems") is a subsystem of the human brain that provides information about neurons and their states by using algorithms to process it. The... |
|
|
| ───────────────────────────────────────────────────────────────── |
|
|
|
|
| ───────────────────────────────────────────────────────────────── |
| 💾 CHECKPOINT step=500 | loss=1.2052 | think=0.0000 | resp=1.2540 |
| tokens=3.342B | lr=1.00e-04 | size=3188 MB |
| → /vol/checkpoints/novamind_roll_latest.pt |
| ───────────────────────────────────────────────────────────────── |
|
|
| step 510 [████████████████████] loss=1.1125 (think=0.0297 resp=1.3397) | lr=9.94e-05 gnorm=0.663 | 300,398 tok/s | mem=15.8GB | elapsed=15.1m ETA=6m |
| step 520 [████████████████████] loss=1.0995 (think=0.0017 resp=1.9251) | lr=9.77e-05 gnorm=0.752 | 300,777 tok/s | mem=15.8GB | elapsed=15.4m ETA=5m |
| step 530 [████████████████████] loss=1.0558 (think=0.0090 resp=2.1797) | lr=9.48e-05 gnorm=0.767 | 300,701 tok/s | mem=15.8GB | elapsed=15.7m ETA=5m |
| step 540 [████████████████████] loss=0.9992 (think=0.0046 resp=2.1945) | lr=9.09e-05 gnorm=0.674 | 300,979 tok/s | mem=15.8GB | elapsed=16.0m ETA=5m |
| step 550 [████████████████████] loss=0.9163 (think=0.0026 resp=1.3548) | lr=8.60e-05 gnorm=0.649 | 301,041 tok/s | mem=15.8GB | elapsed=16.3m ETA=4m |
| step 560 [████████████████████] loss=1.1109 (think=0.0000 resp=2.4489) | lr=8.04e-05 gnorm=0.583 | 300,546 tok/s | mem=15.8GB | elapsed=16.6m ETA=4m |
| step 570 [████████████████████] loss=1.0719 (think=0.0093 resp=1.6699) | lr=7.41e-05 gnorm=0.558 | 300,623 tok/s | mem=15.8GB | elapsed=16.8m ETA=4m |
| step 580 [████████████████████] loss=0.9902 (think=0.0061 resp=2.3208) | lr=6.72e-05 gnorm=0.538 | 300,541 tok/s | mem=15.8GB | elapsed=17.1m ETA=3m |
| step 590 [████████████████████] loss=1.0740 (think=0.0029 resp=1.8986) | lr=6.01e-05 gnorm=0.540 | 300,462 tok/s | mem=15.8GB | elapsed=17.4m ETA=3m |
| step 600 [████████████████████] loss=1.1636 (think=0.0005 resp=2.4235) | lr=5.28e-05 gnorm=0.590 | 300,656 tok/s | mem=15.8GB | elapsed=17.7m ETA=3m |
| step 610 [████████████████████] loss=1.0070 (think=0.0103 resp=2.2433) | lr=4.56e-05 gnorm=0.700 | 300,705 tok/s | mem=15.8GB | elapsed=18.0m ETA=2m |
| step 620 [████████████████████] loss=1.1744 (think=0.0183 resp=2.0976) | lr=3.86e-05 gnorm=0.744 | 300,681 tok/s | mem=15.8GB | elapsed=18.3m ETA=2m |
| step 630 [████████████████████] loss=1.0927 (think=0.0053 resp=1.9465) | lr=3.21e-05 gnorm=0.491 | 300,820 tok/s | mem=15.8GB | elapsed=18.6m ETA=2m |
| step 640 [████████████████████] loss=0.8904 (think=0.0059 resp=1.4536) | lr=2.61e-05 gnorm=0.528 | 300,465 tok/s | mem=15.8GB | elapsed=18.8m ETA=2m |
| step 650 [████████████████████] loss=1.0574 (think=0.0039 resp=1.7072) | lr=2.09e-05 gnorm=0.514 | 300,387 tok/s | mem=15.8GB | elapsed=19.1m ETA=1m |
| step 660 [████████████████████] loss=1.0562 (think=0.0040 resp=1.8388) | lr=1.67e-05 gnorm=0.479 | 300,503 tok/s | mem=15.8GB | elapsed=19.4m ETA=1m |
| step 670 [████████████████████] loss=1.0360 (think=0.0057 resp=1.2987) | lr=1.34e-05 gnorm=0.799 | 300,473 tok/s | mem=15.8GB | elapsed=19.7m ETA=1m |
| step 680 [████████████████████] loss=0.9230 (think=0.0045 resp=1.9609) | lr=1.12e-05 gnorm=0.702 | 300,664 tok/s | mem=15.8GB | elapsed=20.0m ETA=0m |
| step 690 [████████████████████] loss=1.0236 (think=0.0034 resp=1.8634) | lr=1.01e-05 gnorm=0.506 | 300,496 tok/s | mem=15.8GB | elapsed=20.2m ETA=0m |
|
|
| ✅ Epoch 2/2 done | steps=692 | tokens=3.368B |
|
|
|
|
| ───────────────────────────────────────────────────────────────── |
| 💾 CHECKPOINT step=692 | loss=0.7103 | think=0.0176 | resp=0.7418 |
| tokens=3.368B | lr=1.00e-05 | size=3188 MB |
| → /vol/checkpoints/novamind_roll_latest.pt |
| → MILESTONE novamind_milestone_0000692.pt |
| → PHASE FINAL novamind_sft_final.pt |
| ───────────────────────────────────────────────────────────────── |
|
|
|
|
| ═════════════════════════════════════════════════════════════════ |
| ✅ Phase SFT COMPLETE |
| Steps : 692 / 694 |
| Tokens seen : 3.368B |
| Total time : 20.6 min |
| Final loss : 0.710349 |
| Checkpoints : /vol/checkpoints/ |
| ═════════════════════════════════════════════════════════════════ |
|
|
|
|
| 📊 Training complete. Logs saved: |
| /vol/checkpoints/log_sft_20260501_083611.jsonl |
| /vol/checkpoints/log_sft_20260501_083611.csv |
| ▶ Next: python train.py --phase cot_sft --resume /vol/checkpoints/novamind_sft_final.pt |
|
|
|
|
| ✅ Phase 2 SFT complete! |
| Final checkpoint: /vol/checkpoints/novamind_sft_final.pt |
|
|
| Download with: |
| modal volume get novamind-vol /checkpoints/novamind_sft_final.pt ./checkpoints/novamind_sft_final.pt |
| Stopping app - local entrypoint completed. |
