adityag-india/phase2_mini150_first5_mlp

Phase 2 mini150: first five Qwen MLP layers

This is a new small-data experiment named phase2_mini150_first5_mlp_seed1234. It trains only MLP modules in Qwen3.5-0.8B layers 0–4. It selects exactly 100 train, 20 validation, and 30 test records (150 distinct records total) from the saved index lists of the historical first-five run. The last-five mini150 package contains the identical index file and runner; its layer selection and run name are the only configuration differences.

The recorded first-five settings are retained: English Medical O1 Reasoning SFT data, reasoning plus response target, 1,536-token maximum, 10 epochs, learning rate 1e-5, cosine schedule with 3% warmup, batch size 2, gradient accumulation 8, BF16, zero weight decay/dropout, no gradient checkpointing, and no early stopping. Evaluation is configured every 200 optimizer steps and at each epoch end. With 100 training examples there are approximately 7 optimizer steps per epoch, so the 200-step trigger will not occur; baseline and epoch-end validation still run.

Run on RunPod

Use a PyTorch CUDA pod with a BF16-capable GPU and persistent /workspace storage. A 48 GB GPU is a conservative choice for batch size 2 and 1,536 tokens. Upload both zip files, then:

mkdir -p /workspace/phase2-mini150
unzip phase2_mini150_first5_mlp.zip -d /workspace/phase2-mini150
unzip phase2_mini150_last5_mlp.zip -d /workspace/phase2-mini150
cd /workspace/phase2-mini150/phase2_mini150_first5_mlp
python -m pip install -r requirements-runpod.txt
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.is_bf16_supported())"
python train_mini150.py --smoke-test
python train_mini150.py

Both CUDA checks must print True. The image must already provide a CUDA-enabled PyTorch build; if it does not, install a compatible CUDA build for that RunPod image before training. Access to Hugging Face is needed to download the base model and dataset.

The smoke test checks data loading, trainable parameter selection, and one forward/backward pass. Full training writes to mini150_outputs/phase2_mini150_first5_mlp_seed1234/. The run records the fetched model/dataset revisions and a hash of the 150 selected records. It saves the best validation checkpoint, resumable epoch checkpoints, test metrics, a run summary, and a full best_model/ export with tokenizer. Nothing is uploaded to Hugging Face automatically.

For the other condition, from /workspace/phase2-mini150/phase2_mini150_last5_mlp run the same install/check/smoke/train commands. After both finish, compare them with:

python /workspace/phase2-mini150/phase2_mini150_first5_mlp/compare_mini150.py \
  /workspace/phase2-mini150/phase2_mini150_first5_mlp/mini150_outputs/phase2_mini150_first5_mlp_seed1234 \
  /workspace/phase2-mini150/phase2_mini150_last5_mlp/mini150_outputs/phase2_mini150_last5_mlp_seed1234

The comparison checks the model revision, dataset revision, selected-record hash, split, and all training settings before reporting test-loss differences. Keep both output folders on persistent storage. To resume after an epoch checkpoint:

python train_mini150.py --resume mini150_outputs/phase2_mini150_first5_mlp_seed1234/epoch_3.pt

The historical first-five training source and exact prompt formatting were not in the supplied archive. This runner uses explicit ChatML with assistant-only loss for both new experiments. Compare the two new mini150 runs with each other, not directly with historical losses. A 30-example test set provides a quick controlled check, not a precise estimate of broader medical performance or clinical correctness.