# Full Fine-Tune Training and Exploration Playbook This playbook records the full fine-tuning path for `google/gemma-4-E4B-it` on `voidful/agent-sft`, targeting improvement on `claw-eval-zh --language tw`. This run is intentionally not LoRA or QLoRA: - No `adapter` field in the Axolotl config. - No `lora_model_dir`. - No `load_in_4bit` or `load_in_8bit`. - FSDP2 updates and saves full model weights. The earlier private full-weight repository was a LoRA-trained model merged into base weights. That artifact is not considered a valid full fine-tune for this run and must be replaced by a true full-FT checkpoint. ## Objective Train: ```text google/gemma-4-E4B-it ``` on: ```text voidful/agent-sft ``` and evaluate with: ```bash claw-eval-zh --language tw ``` Final judge model: ```text google/gemma-4-31B-it ``` Target Hugging Face repository: ```text voidful/gemma-4-e4b-it-agent-sft-tw ``` The repository was initially uploaded as private. It was later made public on 2026-06-23 CST per follow-up request. ## Slurm Environment Observed working training environment: - Account: `gov109183` - User: `voidful2nlp` - Main partition: `dev` - `dev` walltime: 4 hours - GPU nodes: H200, 8 GPUs per node - Node memory: about 1.9 TB with `--mem=0` - Useful full-FT allocation: `--gres=gpu:8 --cpus-per-task=64 --mem=0` - `8gpus` partition was available but had a much later predicted start time for this workload. - `slinky` and `taide` rejected this account/partition combination. Useful status commands: ```bash squeue -u "$USER" -h -o "%.18i|%.45j|%.2t|%.10M|%.6D|%.14b|%.12m|%R" | sort -n scontrol show job sprio -j -o "%i|%Y|%A|%F|%J|%P|%Q|%N" sinfo -o "%P|%a|%l|%D|%t|%G|%m|%N" sacct -j --format=JobID,JobName%35,State,ExitCode,Elapsed,MaxRSS,ReqMem,AllocTRES -P ``` ## Data Local prepared JSONL files: ```text /work/voidful2nlp/gemma-agent-sft/data/agent_sft/train.jsonl /work/voidful2nlp/gemma-agent-sft/data/agent_sft/valid.jsonl ``` Axolotl dataset fields: ```yaml type: chat_template field_messages: messages field_tools: tools chat_template: tokenizer_default sample_packing: true eval_sample_packing: true sequence_len: 8192 ``` ## Full-FT Config Smoke config: ```text /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_smoke_gemma4e4b_agent_sft_seq8192_steps5.yml ``` Main run config: ```text /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100.yml ``` Important settings: ```yaml base_model: google/gemma-4-E4B-it sequence_len: 8192 micro_batch_size: 1 gradient_accumulation_steps: 1 optimizer: adamw_torch_fused learning_rate: 2.0e-06 lr_scheduler: cosine bf16: true tf32: true gradient_checkpointing: true gradient_checkpointing_kwargs: use_reentrant: false flash_attention: false dataloader_num_workers: 0 max_steps: 400 eval_steps: 100 save_steps: 100 save_only_model: true save_safetensors: true fsdp_version: 2 fsdp_config: offload_params: false state_dict_type: FULL_STATE_DICT final_state_dict_type: FULL_STATE_DICT auto_wrap_policy: TRANSFORMER_BASED_WRAP transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer reshard_after_forward: true ``` `Gemma4TextDecoderLayer` was verified from the installed Transformers Gemma4 implementation before launching FSDP2 training. ## Smoke Tests First 8GPU full-FT smoke: ```text job 138371 ``` Result: ```text FAILED ``` Failure: ```text OSError: [Errno 12] Cannot allocate memory ``` The failure happened after the first eval when PyTorch tried to fork dataloader workers from already-large training processes. Fix: ```yaml dataloader_num_workers: 0 ``` Second 8GPU smoke: ```text job 138399 ``` Result: ```text COMPLETED ``` Observed smoke metrics: - Initial eval loss: 1.95 - Final eval loss after 5 steps: 1.878 - Training GPU memory max active: about 44 GiB per GPU - Training throughput after warmup: about 2.5k tokens/sec/GPU - Saved full-weight file: `model.safetensors` - Saved model size: about 17.2 GB The saved smoke output did not contain adapter weights. ## Training Submit Commands Smoke: ```bash CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_smoke_gemma4e4b_agent_sft_seq8192_steps5.yml \ sbatch --partition=dev --time=04:00:00 --gres=gpu:8 --cpus-per-task=64 --mem=0 \ --job-name=gemma4-fullft-smoke-nw0 \ /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch ``` Main wave: ```bash CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100.yml \ sbatch --partition=dev --time=03:00:00 --gres=gpu:8 --cpus-per-task=64 --mem=0 \ --job-name=gemma4-fullft-w001-lr2e6-s301-3h \ /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch ``` Main wave job: ```text 138433 ``` At submission time Slurm predicted: ```text StartTime=2026-06-23T00:02:57 SchedNodeList=25a-hgpn129 Reason=Priority ``` The job later started on: ```text job 138433 node 25a-hgpn171 world_size 8 started 2026-06-23 00:37:26 CST ``` Observed 8GPU metrics after warmup: - Training memory max active: about 44 GiB per GPU - Device reserved: about 58 GiB per GPU - Training throughput: about 2.5k tokens/sec/GPU - Step time after warmup: about 3.3 seconds - `checkpoint-100` and `checkpoint-200` were full checkpoints with `model.safetensors` about 17.2 GB. Because 8GPU full-node scheduling drifted repeatedly, a 2GPU equivalent-token backup run was launched while keeping the 8GPU job pending. To keep the total number of packed samples comparable to 8GPU x 400 steps, the 2GPU run uses 1600 steps and saves/evals every 400 steps. 2GPU equivalent-token config: ```text /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400.yml ``` 2GPU submit command: ```bash CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400.yml \ sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \ --job-name=gemma4-fullft-w001c-2g-eqtok \ /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch ``` 2GPU run: ```text job 138540 node 25a-hgpn174 world_size 2 ``` Observed early 2GPU metrics: - Initial eval loss: 1.947 - Training memory max active: about 60 GiB per GPU - Training throughput: about 2.5k tokens/sec/GPU - Step time after warmup: about 3.26 seconds This remains true full fine-tuning: no adapter, no LoRA, and no 4-bit or 8-bit base-model loading. ### Parallel LR Sweep To use partial-GPU slots while the main/equivalent runs were active, two more 2GPU true full-FT runs were queued: ```text /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400.yml /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave003_2g_eqtokens_gemma4e4b_agent_sft_lr3e-6_seed303_seq8192_steps1600_save400.yml ``` Submit commands: ```bash CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400.yml \ sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \ --job-name=gemma4-fullft-w002-2g-lr1e6 \ /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave003_2g_eqtokens_gemma4e4b_agent_sft_lr3e-6_seed303_seq8192_steps1600_save400.yml \ sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \ --job-name=gemma4-fullft-w003-2g-lr3e6 \ /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch ``` The first LR1e-6 submission, job `138577`, failed because Slurm placed it on `25a-hgpn062`, where the second allocated H200 reported `N/A/ERR` in `nvidia-smi`; PyTorch saw only one CUDA device and rank 1 failed. It was resubmitted as job `138582` with: ```bash --exclude=25a-hgpn062 ``` Active sweep jobs: ```text 138578 LR3e-6 node 25a-hgpn003 138582 LR1e-6 node 25a-hgpn003 ``` Both runs use FSDP2 full fine-tuning and do not include adapters. ### Main 8GPU Completion and Continuation Main 8GPU job `138433` completed successfully: ```text state: COMPLETED elapsed: 00:43:59 node: 25a-hgpn171 exit: 0:0 ``` Completed full checkpoints: ```text checkpoint-100/model.safetensors 17,224,656,900 bytes checkpoint-200/model.safetensors 17,224,656,900 bytes checkpoint-300/model.safetensors 17,224,656,900 bytes checkpoint-400/model.safetensors 17,224,656,900 bytes final model.safetensors 17,224,656,900 bytes ``` Final logged training/eval metrics for `138433`: ```text eval_loss: 0.986 eval_ppl: 2.681 train_loss: 1.118 ``` Because `save_only_model: true` skips optimizer and scheduler state, the next full-FT extension is not a trainer resume. Instead, `checkpoint-400` is used as the local `base_model` for another full fine-tune: ```text /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave004_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100.yml ``` This continuation remains full fine-tuning: no adapter, no LoRA, no QLoRA, and no 4-bit or 8-bit base loading. The 8GPU continuation was submitted as: ```text 138663 gemma4-fullft-w004-cont400-lr1e6 ``` This job failed before model training. The root cause was not GPU memory or FSDP, but local checkpoint metadata: Axolotl treated Gemma 4 as multimodal and called `AutoProcessor.from_pretrained(checkpoint-400)`, while the checkpoint subdirectory had model/tokenizer files but no `processor_config.json`. The fix was to copy the parent run's `processor_config.json` into the full checkpoint directory: ```bash install -m 0644 \ /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/processor_config.json \ /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/checkpoint-400/processor_config.json ``` After the metadata fix, `AutoProcessor.from_pretrained(checkpoint-400)` loaded successfully as `Gemma4Processor`. A corrected continuation config was added: ```text /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100.yml ``` It was submitted as: ```text 138685 gemma4-fullft-w004b-cont400-lr1e6 ``` This remains a full-weight continuation: `base_model` and `tokenizer_config` point to the local full checkpoint, `load_in_4bit=false`, `load_in_8bit=false`, and `adapter=None`. The cluster also enforced `QOSMaxSubmitJobPerUserLimit`; to free a job slot, the lowest-priority `checkpoint-100` eval retry job `138651` was cancelled and should be resubmitted later if the score curve needs it. ### Live Evaluation Queue Submitted evaluation jobs: ```text 138626 2GPU LR2e-6 checkpoint-400 running, node 25a-hgpn166 138635 8GPU LR2e-6 checkpoint-200 pending 138652 8GPU LR2e-6 checkpoint-300 pending 138660 2GPU LR1e-6 checkpoint-400 pending 138661 2GPU LR3e-6 checkpoint-400 pending 138662 8GPU LR2e-6 checkpoint-400 pending ``` All eval jobs use: ```text judge model: google/gemma-4-31B-it language: tw target GPUs: 0 judge GPUs: 1,2 ADAPTER: empty ``` ## Evaluation Evaluate full checkpoints directly as model paths. Leave `ADAPTER` empty. Template: ```bash MODEL=/work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/checkpoint-100 \ ADAPTER= \ SUITE=all LANGUAGE=tw \ OUTPUT_DIR=/work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_ckpt100_core_judge31b_3g \ sbatch --partition=dev --time=04:00:00 --gres=gpu:3 --cpus-per-task=40 --mem=800G \ --job-name=gemma4-fullft-w001c100-eval \ /home/voidful2nlp/gemma-agent-sft/slurm/eval_claw.sbatch \ --judge-model google/gemma-4-31B-it --core --no-parallel-judge \ --target-gpus 0 --judge-gpus 1,2 ``` Summarize results: ```bash python /home/voidful2nlp/gemma-agent-sft/scripts/summarize_claw_results.py \ /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_ckpt*_core_judge31b_3g ``` Helper script: ```bash /home/voidful2nlp/gemma-agent-sft/scripts/submit_fullft_wave001_evals.sh ``` The helper submits only checkpoints that already exist and always sets `ADAPTER=` to avoid adapter-based evaluation. `RUN_LABEL` and `JOB_PREFIX` can be set to keep output directories and Slurm job names distinct across runs. ### Eval Serving Fix The first eval attempt for 8GPU `checkpoint-100`, job `138611`, failed before benchmark execution: ```text OSError: Can't load feature extractor for .../checkpoint-100 ``` The full-FT checkpoint directories contain model/tokenizer/config files, but Axolotl did not copy `processor_config.json` into each checkpoint. The local text-only OpenAI-compatible server uses tokenizer chat templates and generation, not processor preprocessing, so `serve_transformers_openai.py` was changed to fall back to tokenizer-only serving when `AutoProcessor.from_pretrained(...)` raises `OSError`. During long-context file-analysis evals, one target server produced CUDA OOM and HTTP 503 responses after a single retry at 512 new tokens. The serving script now tries progressively smaller generation budgets (`1024`, `512`, `256`, `128`) before returning 503. This affects future eval jobs; already running eval jobs keep the script version they started with. The failed eval was resubmitted as: ```text 138651 8GPU LR2e-6 checkpoint-100 retry ``` ### Concurrent Eval Workspace Collision When multiple eval jobs served different runs whose checkpoint directory basename was the same, for example `checkpoint-400`, the eval driver originally passed the same model name to `claw-eval-zh`: ```text checkpoint-400 ``` That made `claw-eval-zh` create the same OpenClaw agent id: ```text bench-checkpoint-400 ``` If two such evals landed on the same node, their disposable workspaces could interfere. The observed symptom was: ```text OpenClaw command failed ... ENOENT: no such file or directory, uv_cwd ``` This affected LR sweep eval jobs `138660` and `138661`; both were cancelled and resubmitted with unique output labels as `138669` and `138670`. Fix in `/home/voidful2nlp/gemma-agent-sft/scripts/run_claw_eval.py`: ```text default model_name = - ``` `eval_claw.sbatch` was also updated to print Slurm GPU visibility in future eval logs. ### 8GPU Checkpoint-400 Original Eval The first complete 20-task eval for the 8GPU LR2e-6 `checkpoint-400` finished as job `138662`: ```text output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_ckpt_ckpt400_core_judge31b_3g/0001_checkpoint-400.json score: 8.79 / 20.0 mean: 0.439315 pass@1: 35.0% ``` This run is retained for audit, but it had target-server CUDA OOM/HTTP 503 events before the multi-step retry fix landed. A patched-server rerun was submitted: ```text 138694 gemma4-fullft-w001-8g-eval-oomfix-c400 output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_oomfix_ckpt400_core_judge31b_3g ``` The same 8GPU LR2e-6 run also completed `checkpoint-200` and `checkpoint-300` evals: ```text 138635 checkpoint-200 mean: 0.448565 score: 8.97 / 20.0 138652 checkpoint-300 mean: 0.439090 score: 8.78 / 20.0 ``` At this point the best completed clean full-FT score is `checkpoint-200` (`0.448565`). `checkpoint-400` is close but has serving OOM/503 artifacts, so the patched-server rerun `138694` is required before making a final comparison. ### LR Sweep Early Result The first completed 2GPU equivalent-token LR sweep eval was the LR3e-6 `checkpoint-400` job `138670`: ```text output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave003_2g_lr3e6_seed303_eqtokens_retryuniq_ckpt400_core_judge31b_3g/0002_checkpoint-400-138670.json score: 5.70 / 20.0 mean: 0.285135 pass@1: 25.0% ``` That result is well below the 8GPU LR2e-6 baseline, so LR3e-6 is deprioritized. With the next free submit slot, the LR2e-6 equivalent-token `checkpoint-1200` was queued using the patched serving script: ```text 138695 gemma4-fullft-w001c-2g-eval-oomfix-c1200 output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt1200_core_judge31b_3g ``` The eval submit helper now supports `EXCLUDE=...` so future evals can avoid known-bad nodes such as `25a-hgpn062`. After `checkpoint-200` and `checkpoint-300` evals completed, two more patched-server evals were submitted: ```text 138698 gemma4-fullft-w001-8g-eval-oomfix-c100 output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_oomfix_ckpt100_core_judge31b_3g 138699 gemma4-fullft-w002-2g-eval-oomfix-c800 output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave002_2g_lr1e6_seed302_eqtokens_oomfix_ckpt800_core_judge31b_3g ``` ## Selection Rule For this true full-FT run: 1. Evaluate checkpoints 100, 200, 300, and 400. 2. Select the highest completed `claw-eval-zh --language tw` score. 3. If a checkpoint improves over the current true full-FT best, continue from that checkpoint with a smaller neighborhood search. 4. Stop only when new full-FT continuation runs no longer improve the completed score. The previous LoRA score, 9.498095238095239, is useful context but is not a valid full-FT artifact. ## 2026-06-23 02:08 CST Full-FT Status The run policy was corrected to true full fine-tuning only. LoRA, QLoRA, adapter training, and merged-adapter outputs are excluded from final selection. Current clean completed full-FT best: ```text run: fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100 checkpoint: checkpoint-200 eval_output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_ckpt_ckpt200_core_judge31b_3g/0002_checkpoint-200.json passk.average_score: 0.448566 score_out_of_20: 8.97132 ``` Additional completed checkpoint comparison: ```text wave001 checkpoint-300: passk.average_score 0.439090 wave001 checkpoint-400: passk.average_score 0.439315 wave002 2GPU LR1e-6 checkpoint-400: passk.average_score 0.431448 wave003 2GPU LR3e-6 checkpoint-400: passk.average_score 0.285135 ``` The wave002 checkpoint-400 result did not beat the current clean full-FT best, and the LR3e-6 branch is deprioritized. The continuation run below is active and has entered actual training: ```text job: 138685 name: gemma4-fullft-w004b-cont400-lr1e6 node: 25a-hgpn164 gpus: 8x H200 source_weights: wave001 checkpoint-400 learning_rate: 1e-6 initial_eval_loss: 0.9885 initial_eval_ppl: 2.687 train_throughput_observed: about 2.5k tokens/sec/GPU after warmup ``` Submitted additional patched-server eval to keep the Slurm submit window full: ```text job: 138702 name: gemma4-fullft-w001c-eval-oomfix-c800 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-800 output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt800_core_judge31b_3g exclude: 25a-hgpn062 ``` ## 2026-06-23 02:17 CST Queue Adjustment The old non-patched `w001c checkpoint-400` eval was cancelled after 47 minutes because it was slow, low scoring at 8/20, and less useful than the patched evals: ```text job: 138626 state: CANCELLED reason: free 3 GPUs for higher-value patched checkpoint evals partial_score: completed 8/20, passk.average_score 0.125 ``` This allowed the queued `w002 checkpoint-800` eval to start: ```text job: 138699 name: gemma4-fullft-w002-2g-eval-oomfix-c800 node: 25a-hgpn166 ``` The `w001c` 2GPU equivalent-token run completed cleanly: ```text job: 138540 state: COMPLETED checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-1600 final_eval_loss: 0.8534 final_eval_ppl: 2.348 model_safetensors_size: 17224656900 bytes ``` New full-FT checkpoints submitted for patched eval: ```text job: 138705 name: gemma4-fullft-w004b-eval-oomfix-c100 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-100 job: 138706 name: gemma4-fullft-w001c-eval-oomfix-c1600 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-1600 ``` Two partial patched evals were cancelled once their running averages dropped below the clean best and better queued candidates were available: ```text job: 138695 name: gemma4-fullft-w001c-2g-eval-oomfix-c1200 state_at_cancel: completed 13/20, passk.average_score 0.389744 reason: low partial score, running on known-problem node 25a-hgpn062, c800/c1600 queued job: 138694 name: gemma4-fullft-w001-8g-eval-oomfix-c400 state_at_cancel: completed 13/20, passk.average_score 0.403205 reason: low partial score, older full checkpoint-400 eval already available, c100 looked stronger ``` The freed submit capacity was used for: ```text job: 138716 name: gemma4-fullft-w002-eval-oomfix-c1200 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1200 ``` The `w004b` continuation produced `checkpoint-200`, and it was queued for patched eval: ```text job: 138721 name: gemma4-fullft-w004b-eval-oomfix-c200 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200 ``` The `w004b` continuation also produced `checkpoint-300`, and it was queued: ```text job: 138729 name: gemma4-fullft-w004b-eval-oomfix-c300 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-300 ``` The `w002 checkpoint-800` eval was cancelled after it showed no progress toward the current best: ```text job: 138699 name: gemma4-fullft-w002-2g-eval-oomfix-c800 state_at_cancel: completed 4/20, passk.average_score 0.0 reason: free 3 GPUs for w002 checkpoint-1200 and later candidates ``` The `w002` LR1e-6 run produced `checkpoint-1600`, and it was queued for eval: ```text job: 138734 name: gemma4-fullft-w002-eval-oomfix-c1600 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1600 ``` The `w001c checkpoint-800` eval was cancelled after its partial average dropped well below the clean best: ```text job: 138702 name: gemma4-fullft-w001c-eval-oomfix-c800 state_at_cancel: completed 13/20, passk.average_score 0.389744 reason: free 3 GPUs for w002 checkpoint-1600 ``` The `w004b` continuation completed through `checkpoint-400`; `checkpoint-400` was queued for eval: ```text job: 138737 name: gemma4-fullft-w004b-eval-oomfix-c400 model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400 ``` The `w004b checkpoint-100` eval was cancelled after its partial score fell below the clean best: ```text job: 138705 name: gemma4-fullft-w004b-eval-oomfix-c100 state_at_cancel: completed 13/20, passk.average_score 0.403205 reason: free 3 GPUs for w004b checkpoint-400 ``` ## 2026-06-23 03:11 CST Full-FT Continuation Queue The `w001c checkpoint-1600` eval completed and did not beat the clean best: ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt1600_core_judge31b_3g/0005_checkpoint-1600-138706.json tasks: 20 passk.average_score: 0.4068333333333333 decision: reject; below clean best 0.44856547619047615 ``` Two low-LR continuation jobs were queued to keep using available Slurm capacity. Both are true full fine-tunes: no adapter config, no LoRA, no quantization, FSDP FULL_STATE_DICT, full `model.safetensors` checkpoints. ```text job: 138755 name: gemma4-fullft-w005-w002c1600-lr5e7 config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400.yml base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1600 resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062 status_at_submit: pending priority ``` ```text job: 138756 name: gemma4-fullft-w006-w004bc400-lr5e7 config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400.yml base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400 resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062 status_at_submit: pending priority ``` The continuation checkpoints will be evaluated only if their source branch or their own intermediate training losses justify continuing the search. ## 2026-06-23 03:19 CST New Full-FT Best `w004b checkpoint-200` completed evaluation and became the current best true full-FT checkpoint: ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt200_core_judge31b_3g/0042_checkpoint-200-138721.json tasks: 20 passk.average_score: 0.45305833333333334 previous_clean_best: 0.44856547619047615 delta: +0.00449285714285719 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200 ``` Because the new best came from `checkpoint-200`, a matching low-LR continuation was submitted from that checkpoint: ```text job: 138769 name: gemma4-fullft-w007-w004bc200-lr5e7 config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave007_cont_from_w004bckpt200_gemma4e4b_agent_sft_lr5e-7_seed307_seq8192_steps1600_save400.yml base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200 resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062 status_at_submit: pending priority ``` The `w002 checkpoint-1200` eval was cancelled to free GPUs for current-best continuation work: ```text job: 138716 name: gemma4-fullft-w002-eval-oomfix-c1200 state_at_cancel: completed 6/20, passk.average_score 0.166667 reason: lower-priority checkpoint from the same branch as checkpoint-1600; checkpoint-1600 was already under eval and had a continuation job running ``` Follow-up checkpoint evals completed: ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave002_2g_lr1e6_seed302_eqtokens_oomfix_ckpt1600_core_judge31b_3g/0006_checkpoint-1600-138734.json tasks: 20 passk.average_score: 0.4489583333333333 decision: reject for upload; above original clean best but below current best 0.45305833333333334 note: continuation job 138755 was already running from this checkpoint ``` ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt300_core_judge31b_3g/0043_checkpoint-300-138729.json tasks: 20 passk.average_score: 0.4428404761904762 decision: reject; below current best ``` `w007` started running after resources freed: ```text job: 138769 name: gemma4-fullft-w007-w004bc200-lr5e7 node: 25a-hgpn166 status: running ``` `w004b checkpoint-400` completed after a late recovery and became the new best: ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt400_core_judge31b_3g/0007_checkpoint-400-138737.json tasks: 20 passk.average_score: 0.4541071428571429 previous_best: 0.45305833333333334 delta: +0.001048809523809572 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400 note: continuation job 138756 was already running from this checkpoint ``` An additional conservative continuation from the current best was submitted: ```text job: 138788 name: gemma4-fullft-w008-w004bc400-lr2e7 config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400.yml base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400 resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062 status_at_submit: pending priority reason: compare LR 2e-7 against active LR 5e-7 continuation from the same current-best checkpoint ``` The `w007` continuation from the previous-best `checkpoint-200` was cancelled: ```text job: 138769 name: gemma4-fullft-w007-w004bc200-lr5e7 state_at_cancel: tokenization still in progress, about 21k/242k prompts reason: checkpoint-400 became the new best; prioritize current-best c400 continuations ``` `w005 checkpoint-400` was produced and queued for TW eval: ```text job: 138803 name: gemma4-fullft-w005-eval-c400 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-400 model_artifact: full 17G model.safetensors output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt400_core_judge31b_3g judge: google/gemma-4-31B-it language: tw ``` `w006 checkpoint-400` was produced and queued for TW eval: ```text job: 138806 name: gemma4-fullft-w006-eval-c400 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400/checkpoint-400 model_artifact: full 17G model.safetensors output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave006_cont_w004bckpt400_lr5e7_seed306_ckpt400_core_judge31b_3g judge: google/gemma-4-31B-it language: tw ``` `w008 checkpoint-400` was produced and queued for TW eval after confirming the full 17G checkpoint had finished writing: ```text job: 138814 name: gemma4-fullft-w008-eval-c400 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400/checkpoint-400 model_artifact: full 17G model.safetensors output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt400_core_judge31b_3g judge: google/gemma-4-31B-it language: tw ``` `w005 checkpoint-800` was produced and queued for TW eval: ```text job: 138818 name: gemma4-fullft-w005-eval-c800 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-800 model_artifact: full 17G model.safetensors output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt800_core_judge31b_3g judge: google/gemma-4-31B-it language: tw ``` The low-priority `w005 checkpoint-400` eval was cancelled after a poor partial: ```text job: 138803 name: gemma4-fullft-w005-eval-c400 state_at_cancel: completed 5/20, passk.average_score 0.2 reason: w005 is not the current-best branch; free 3 GPUs for current-best c400/c800 evals ``` `w006 checkpoint-800` was produced and queued for TW eval: ```text job: 138822 name: gemma4-fullft-w006-eval-c800 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400/checkpoint-800 model_artifact: full 17G model.safetensors output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave006_cont_w004bckpt400_lr5e7_seed306_ckpt800_core_judge31b_3g judge: google/gemma-4-31B-it language: tw ``` The `w006 checkpoint-400` eval was cancelled after its partial score fell below the current best: ```text job: 138806 name: gemma4-fullft-w006-eval-c400 state_at_cancel: completed 13/20, passk.average_score 0.38974358974358975 reason: free 3 GPUs for w008 checkpoint-400 and later checkpoint-800 evals ``` `w008 checkpoint-800` was produced and queued for TW eval: ```text job: 138827 name: gemma4-fullft-w008-eval-c800 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400/checkpoint-800 model_artifact: full 17G model.safetensors output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt800_core_judge31b_3g judge: google/gemma-4-31B-it language: tw ``` At submission time, `w005 checkpoint-800` was the strongest partial among active new evals: ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt800_core_judge31b_3g/0002_checkpoint-800-138818.json partial: completed 8/20, passk.average_score 0.5078125 ``` ## 2026-06-23 08:36 CST Final Full-FT Search Later continuation and eval waves were run under the same true full-FT policy: no LoRA, no QLoRA, no adapter field, no 4-bit or 8-bit base loading, and full 17,224,656,900 byte `model.safetensors` checkpoints. The strongest late candidates were: ```text wave005 checkpoint-1600: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt1600_core_judge31b_3g/0002_checkpoint-1600-138849.json tasks: 20 passk.average_score: 0.4599238095238095 wave008 checkpoint-800: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt800_core_judge31b_3g/0001_checkpoint-800-138827.json tasks: 20 passk.average_score: 0.45570714285714287 wave009b checkpoint-100: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt100_core_judge31b_3g/0045_checkpoint-100-138874.json tasks: 20 passk.average_score: 0.4680583333333333 wave009b checkpoint-200: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt200_core_judge31b_3g/0013_checkpoint-200-138890.json tasks: 20 passk.average_score: 0.4695104761904762 wave009b checkpoint-300: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt300_core_judge31b_3g/0002_checkpoint-300-138898.json tasks: 20 passk.average_score: 0.4662333333333334 wave009b checkpoint-400: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt400_core_judge31b_3g/0046_checkpoint-400-138903.json tasks: 20 passk.average_score: 0.45391523809523815 ``` `wave010` continued from the strongest `wave005 checkpoint-1600` branch with learning rate `5e-8`: ```text config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100.yml job: 138862 base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-1600 resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062 ``` Completed `wave010` evals: ```text checkpoint-100: 0.4479333333333333 checkpoint-200: 0.4415904761904762 checkpoint-300: 0.44665 checkpoint-400: 0.4789416666666667 ``` `wave010 checkpoint-400` became the best true full-FT checkpoint: ```text eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave010_cont_w005ckpt1600_lr5e8_seed310_ckpt400_core_judge31b_3g/0002_checkpoint-400-138904.json tasks: 20 passk.average_score: 0.4789416666666667 score_out_of_20: 9.578833333333334 checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400 model_safetensors_size: 17224656900 bytes ``` To test whether this best could still be improved, `wave011` continued from `wave010 checkpoint-400` with a smaller learning rate, `2e-8`: ```text config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave011_cont_from_w010ckpt400_gemma4e4b_agent_sft_lr2e-8_seed312_seq8192_steps400_save100.yml job: 138918 base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400 resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062 state: COMPLETED elapsed: 00:51:45 ``` `wave011` checkpoint evals all completed below `wave010 checkpoint-400`: ```text checkpoint-100: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt100_core_judge31b_3g/0047_checkpoint-100-138931.json tasks: 20 passk.average_score: 0.4315904761904762 checkpoint-200: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt200_core_judge31b_3g/0008_checkpoint-200-138936.json tasks: 20 passk.average_score: 0.45445833333333335 checkpoint-300: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt300_core_judge31b_3g/0014_checkpoint-300-138944.json tasks: 20 passk.average_score: 0.4371654761904762 checkpoint-400: eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt400_core_judge31b_3g/0048_checkpoint-400-138950.json tasks: 20 passk.average_score: 0.45445714285714284 ``` Stop decision: ```text current_best: wave010 checkpoint-400, 0.4789416666666667 next_smaller_lr_probe: wave011 lr2e-8 from current best best_wave011_score: 0.45445833333333335 decision: stop full-FT exploration; no later true full-FT continuation improved the completed TW eval score ``` ## Upload Plan Upload the selected true full-FT checkpoint to: ```text voidful/gemma-4-e4b-it-agent-sft-tw ``` Initial upload used `private=True`; current repository visibility is public. The final Hugging Face repository should include: - Full model weights and tokenizer files. - `README.md` model card. - `PLAYBOOK.md` copied from this playbook and updated with final scores. - `training_config.yml`. - Evaluation JSON files for the selected checkpoint. - A score summary table. Before upload, verify: ```bash find -maxdepth 2 -name 'adapter_*' -o -name '*lora*' ``` The command should not return adapter artifacts for the full-FT model repo. Staging helper: ```bash /home/voidful2nlp/gemma-agent-sft/scripts/stage_fullft_hf_upload.sh \ \ ``` Upload command template: ```bash hf upload voidful/gemma-4-e4b-it-agent-sft-tw \ --repo-type model --private --delete '*' \ --commit-message "Upload true full fine-tuned Gemma 4 E4B agent SFT TW" ``` ## Final Results ```text selected_checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400 selected_config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100.yml selected_eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave010_cont_w005ckpt1600_lr5e8_seed310_ckpt400_core_judge31b_3g/0002_checkpoint-400-138904.json claw_eval_tw_score: 0.4789416666666667 score_out_of_20: 9.578833333333334 hf_repo: voidful/gemma-4-e4b-it-agent-sft-tw hf_visibility: public hf_commit: dbef13b501b5277b057fa4fe23d9b5307452e108 hf_commit_note: initial private full artifact upload ```