Hugging Face's logo

voidful
/
gemma-4-e4b-it-agent-sft-tw

Safetensors
gemma4
gemma-4
full-finetune
agent-sft
traditional-chinese
claw-eval-zh
Model card Files
xet
 Community
gemma-4-e4b-it-agent-sft-tw / PLAYBOOK.md
voidful's picture
voidful
Update playbook for public release
5355734 verified
preview code
|
Raw
History Blame Contribute Delete
39.9 kB

Full Fine-Tune Training and Exploration Playbook

This playbook records the full fine-tuning path for google/gemma-4-E4B-it on voidful/agent-sft, targeting improvement on claw-eval-zh --language tw.

This run is intentionally not LoRA or QLoRA:

  • No adapter field in the Axolotl config.
  • No lora_model_dir.
  • No load_in_4bit or load_in_8bit.
  • FSDP2 updates and saves full model weights.

The earlier private full-weight repository was a LoRA-trained model merged into base weights. That artifact is not considered a valid full fine-tune for this run and must be replaced by a true full-FT checkpoint.

Objective

Train:

google/gemma-4-E4B-it

on:

voidful/agent-sft

and evaluate with:

claw-eval-zh --language tw

Final judge model:

google/gemma-4-31B-it

Target Hugging Face repository:

voidful/gemma-4-e4b-it-agent-sft-tw

The repository was initially uploaded as private. It was later made public on 2026-06-23 CST per follow-up request.

Slurm Environment

Observed working training environment:

  • Account: gov109183
  • User: voidful2nlp
  • Main partition: dev
  • dev walltime: 4 hours
  • GPU nodes: H200, 8 GPUs per node
  • Node memory: about 1.9 TB with --mem=0
  • Useful full-FT allocation: --gres=gpu:8 --cpus-per-task=64 --mem=0
  • 8gpus partition was available but had a much later predicted start time for this workload.
  • slinky and taide rejected this account/partition combination.

Useful status commands:

squeue -u "$USER" -h -o "%.18i|%.45j|%.2t|%.10M|%.6D|%.14b|%.12m|%R" | sort -n
scontrol show job <job_id>
sprio -j <job_id> -o "%i|%Y|%A|%F|%J|%P|%Q|%N"
sinfo -o "%P|%a|%l|%D|%t|%G|%m|%N"
sacct -j <job_id> --format=JobID,JobName%35,State,ExitCode,Elapsed,MaxRSS,ReqMem,AllocTRES -P

Data

Local prepared JSONL files:

/work/voidful2nlp/gemma-agent-sft/data/agent_sft/train.jsonl
/work/voidful2nlp/gemma-agent-sft/data/agent_sft/valid.jsonl

Axolotl dataset fields:

type: chat_template
field_messages: messages
field_tools: tools
chat_template: tokenizer_default
sample_packing: true
eval_sample_packing: true
sequence_len: 8192

Full-FT Config

Smoke config:

/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_smoke_gemma4e4b_agent_sft_seq8192_steps5.yml

Main run config:

/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100.yml

Important settings:

base_model: google/gemma-4-E4B-it
sequence_len: 8192
micro_batch_size: 1
gradient_accumulation_steps: 1
optimizer: adamw_torch_fused
learning_rate: 2.0e-06
lr_scheduler: cosine
bf16: true
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
flash_attention: false
dataloader_num_workers: 0
max_steps: 400
eval_steps: 100
save_steps: 100
save_only_model: true
save_safetensors: true
fsdp_version: 2
fsdp_config:
  offload_params: false
  state_dict_type: FULL_STATE_DICT
  final_state_dict_type: FULL_STATE_DICT
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  reshard_after_forward: true

Gemma4TextDecoderLayer was verified from the installed Transformers Gemma4 implementation before launching FSDP2 training.

Smoke Tests

First 8GPU full-FT smoke:

job 138371

Result:

FAILED

Failure:

OSError: [Errno 12] Cannot allocate memory

The failure happened after the first eval when PyTorch tried to fork dataloader workers from already-large training processes.

Fix:

dataloader_num_workers: 0

Second 8GPU smoke:

job 138399

Result:

COMPLETED

Observed smoke metrics:

  • Initial eval loss: 1.95
  • Final eval loss after 5 steps: 1.878
  • Training GPU memory max active: about 44 GiB per GPU
  • Training throughput after warmup: about 2.5k tokens/sec/GPU
  • Saved full-weight file: model.safetensors
  • Saved model size: about 17.2 GB

The saved smoke output did not contain adapter weights.

Training Submit Commands

Smoke:

CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_smoke_gemma4e4b_agent_sft_seq8192_steps5.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:8 --cpus-per-task=64 --mem=0 \
  --job-name=gemma4-fullft-smoke-nw0 \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch

Main wave:

CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100.yml \
sbatch --partition=dev --time=03:00:00 --gres=gpu:8 --cpus-per-task=64 --mem=0 \
  --job-name=gemma4-fullft-w001-lr2e6-s301-3h \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch

Main wave job:

138433

At submission time Slurm predicted:

StartTime=2026-06-23T00:02:57
SchedNodeList=25a-hgpn129
Reason=Priority

The job later started on:

job 138433
node 25a-hgpn171
world_size 8
started 2026-06-23 00:37:26 CST

Observed 8GPU metrics after warmup:

  • Training memory max active: about 44 GiB per GPU
  • Device reserved: about 58 GiB per GPU
  • Training throughput: about 2.5k tokens/sec/GPU
  • Step time after warmup: about 3.3 seconds
  • checkpoint-100 and checkpoint-200 were full checkpoints with model.safetensors about 17.2 GB.

Because 8GPU full-node scheduling drifted repeatedly, a 2GPU equivalent-token backup run was launched while keeping the 8GPU job pending. To keep the total number of packed samples comparable to 8GPU x 400 steps, the 2GPU run uses 1600 steps and saves/evals every 400 steps.

2GPU equivalent-token config:

/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400.yml

2GPU submit command:

CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \
  --job-name=gemma4-fullft-w001c-2g-eqtok \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch

2GPU run:

job 138540
node 25a-hgpn174
world_size 2

Observed early 2GPU metrics:

  • Initial eval loss: 1.947
  • Training memory max active: about 60 GiB per GPU
  • Training throughput: about 2.5k tokens/sec/GPU
  • Step time after warmup: about 3.26 seconds

This remains true full fine-tuning: no adapter, no LoRA, and no 4-bit or 8-bit base-model loading.

Parallel LR Sweep

To use partial-GPU slots while the main/equivalent runs were active, two more 2GPU true full-FT runs were queued:

/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400.yml
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave003_2g_eqtokens_gemma4e4b_agent_sft_lr3e-6_seed303_seq8192_steps1600_save400.yml

Submit commands:

CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \
  --job-name=gemma4-fullft-w002-2g-lr1e6 \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch

CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave003_2g_eqtokens_gemma4e4b_agent_sft_lr3e-6_seed303_seq8192_steps1600_save400.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \
  --job-name=gemma4-fullft-w003-2g-lr3e6 \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch

The first LR1e-6 submission, job 138577, failed because Slurm placed it on 25a-hgpn062, where the second allocated H200 reported N/A/ERR in nvidia-smi; PyTorch saw only one CUDA device and rank 1 failed. It was resubmitted as job 138582 with:

--exclude=25a-hgpn062

Active sweep jobs:

138578  LR3e-6  node 25a-hgpn003
138582  LR1e-6  node 25a-hgpn003

Both runs use FSDP2 full fine-tuning and do not include adapters.

Main 8GPU Completion and Continuation

Main 8GPU job 138433 completed successfully:

state: COMPLETED
elapsed: 00:43:59
node: 25a-hgpn171
exit: 0:0

Completed full checkpoints:

checkpoint-100/model.safetensors  17,224,656,900 bytes
checkpoint-200/model.safetensors  17,224,656,900 bytes
checkpoint-300/model.safetensors  17,224,656,900 bytes
checkpoint-400/model.safetensors  17,224,656,900 bytes
final model.safetensors           17,224,656,900 bytes

Final logged training/eval metrics for 138433:

eval_loss: 0.986
eval_ppl: 2.681
train_loss: 1.118

Because save_only_model: true skips optimizer and scheduler state, the next full-FT extension is not a trainer resume. Instead, checkpoint-400 is used as the local base_model for another full fine-tune:

/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave004_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100.yml

This continuation remains full fine-tuning: no adapter, no LoRA, no QLoRA, and no 4-bit or 8-bit base loading.

The 8GPU continuation was submitted as:

138663  gemma4-fullft-w004-cont400-lr1e6

This job failed before model training. The root cause was not GPU memory or FSDP, but local checkpoint metadata: Axolotl treated Gemma 4 as multimodal and called AutoProcessor.from_pretrained(checkpoint-400), while the checkpoint subdirectory had model/tokenizer files but no processor_config.json.

The fix was to copy the parent run's processor_config.json into the full checkpoint directory:

install -m 0644 \
  /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/processor_config.json \
  /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/checkpoint-400/processor_config.json

After the metadata fix, AutoProcessor.from_pretrained(checkpoint-400) loaded successfully as Gemma4Processor. A corrected continuation config was added:

/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100.yml

It was submitted as:

138685  gemma4-fullft-w004b-cont400-lr1e6

This remains a full-weight continuation: base_model and tokenizer_config point to the local full checkpoint, load_in_4bit=false, load_in_8bit=false, and adapter=None.

The cluster also enforced QOSMaxSubmitJobPerUserLimit; to free a job slot, the lowest-priority checkpoint-100 eval retry job 138651 was cancelled and should be resubmitted later if the score curve needs it.

Live Evaluation Queue

Submitted evaluation jobs:

138626  2GPU LR2e-6 checkpoint-400  running, node 25a-hgpn166
138635  8GPU LR2e-6 checkpoint-200  pending
138652  8GPU LR2e-6 checkpoint-300  pending
138660  2GPU LR1e-6 checkpoint-400  pending
138661  2GPU LR3e-6 checkpoint-400  pending
138662  8GPU LR2e-6 checkpoint-400  pending

All eval jobs use:

judge model: google/gemma-4-31B-it
language: tw
target GPUs: 0
judge GPUs: 1,2
ADAPTER: empty

Evaluation

Evaluate full checkpoints directly as model paths. Leave ADAPTER empty.

Template:

MODEL=/work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/checkpoint-100 \
ADAPTER= \
SUITE=all LANGUAGE=tw \
OUTPUT_DIR=/work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_ckpt100_core_judge31b_3g \
sbatch --partition=dev --time=04:00:00 --gres=gpu:3 --cpus-per-task=40 --mem=800G \
  --job-name=gemma4-fullft-w001c100-eval \
  /home/voidful2nlp/gemma-agent-sft/slurm/eval_claw.sbatch \
  --judge-model google/gemma-4-31B-it --core --no-parallel-judge \
  --target-gpus 0 --judge-gpus 1,2

Summarize results:

python /home/voidful2nlp/gemma-agent-sft/scripts/summarize_claw_results.py \
  /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_ckpt*_core_judge31b_3g

Helper script:

/home/voidful2nlp/gemma-agent-sft/scripts/submit_fullft_wave001_evals.sh

The helper submits only checkpoints that already exist and always sets ADAPTER= to avoid adapter-based evaluation. RUN_LABEL and JOB_PREFIX can be set to keep output directories and Slurm job names distinct across runs.

Eval Serving Fix

The first eval attempt for 8GPU checkpoint-100, job 138611, failed before benchmark execution:

OSError: Can't load feature extractor for .../checkpoint-100

The full-FT checkpoint directories contain model/tokenizer/config files, but Axolotl did not copy processor_config.json into each checkpoint. The local text-only OpenAI-compatible server uses tokenizer chat templates and generation, not processor preprocessing, so serve_transformers_openai.py was changed to fall back to tokenizer-only serving when AutoProcessor.from_pretrained(...) raises OSError.

During long-context file-analysis evals, one target server produced CUDA OOM and HTTP 503 responses after a single retry at 512 new tokens. The serving script now tries progressively smaller generation budgets (1024, 512, 256, 128) before returning 503. This affects future eval jobs; already running eval jobs keep the script version they started with.

The failed eval was resubmitted as:

138651  8GPU LR2e-6 checkpoint-100 retry

Concurrent Eval Workspace Collision

When multiple eval jobs served different runs whose checkpoint directory basename was the same, for example checkpoint-400, the eval driver originally passed the same model name to claw-eval-zh:

checkpoint-400

That made claw-eval-zh create the same OpenClaw agent id:

bench-checkpoint-400

If two such evals landed on the same node, their disposable workspaces could interfere. The observed symptom was:

OpenClaw command failed ... ENOENT: no such file or directory, uv_cwd

This affected LR sweep eval jobs 138660 and 138661; both were cancelled and resubmitted with unique output labels as 138669 and 138670.

Fix in /home/voidful2nlp/gemma-agent-sft/scripts/run_claw_eval.py:

default model_name = <checkpoint-basename>-<SLURM_JOB_ID>

eval_claw.sbatch was also updated to print Slurm GPU visibility in future eval logs.

8GPU Checkpoint-400 Original Eval

The first complete 20-task eval for the 8GPU LR2e-6 checkpoint-400 finished as job 138662:

output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_ckpt_ckpt400_core_judge31b_3g/0001_checkpoint-400.json
score: 8.79 / 20.0
mean: 0.439315
pass@1: 35.0%

This run is retained for audit, but it had target-server CUDA OOM/HTTP 503 events before the multi-step retry fix landed. A patched-server rerun was submitted:

138694  gemma4-fullft-w001-8g-eval-oomfix-c400
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_oomfix_ckpt400_core_judge31b_3g

The same 8GPU LR2e-6 run also completed checkpoint-200 and checkpoint-300 evals:

138635  checkpoint-200  mean: 0.448565  score: 8.97 / 20.0
138652  checkpoint-300  mean: 0.439090  score: 8.78 / 20.0

At this point the best completed clean full-FT score is checkpoint-200 (0.448565). checkpoint-400 is close but has serving OOM/503 artifacts, so the patched-server rerun 138694 is required before making a final comparison.

LR Sweep Early Result

The first completed 2GPU equivalent-token LR sweep eval was the LR3e-6 checkpoint-400 job 138670:

output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave003_2g_lr3e6_seed303_eqtokens_retryuniq_ckpt400_core_judge31b_3g/0002_checkpoint-400-138670.json
score: 5.70 / 20.0
mean: 0.285135
pass@1: 25.0%

That result is well below the 8GPU LR2e-6 baseline, so LR3e-6 is deprioritized. With the next free submit slot, the LR2e-6 equivalent-token checkpoint-1200 was queued using the patched serving script:

138695  gemma4-fullft-w001c-2g-eval-oomfix-c1200
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt1200_core_judge31b_3g

The eval submit helper now supports EXCLUDE=... so future evals can avoid known-bad nodes such as 25a-hgpn062. After checkpoint-200 and checkpoint-300 evals completed, two more patched-server evals were submitted:

138698  gemma4-fullft-w001-8g-eval-oomfix-c100
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_oomfix_ckpt100_core_judge31b_3g

138699  gemma4-fullft-w002-2g-eval-oomfix-c800
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave002_2g_lr1e6_seed302_eqtokens_oomfix_ckpt800_core_judge31b_3g

Selection Rule

For this true full-FT run:

  1. Evaluate checkpoints 100, 200, 300, and 400.
  2. Select the highest completed claw-eval-zh --language tw score.
  3. If a checkpoint improves over the current true full-FT best, continue from that checkpoint with a smaller neighborhood search.
  4. Stop only when new full-FT continuation runs no longer improve the completed score.

The previous LoRA score, 9.498095238095239, is useful context but is not a valid full-FT artifact.

2026-06-23 02:08 CST Full-FT Status

The run policy was corrected to true full fine-tuning only. LoRA, QLoRA, adapter training, and merged-adapter outputs are excluded from final selection.

Current clean completed full-FT best:

run: fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100
checkpoint: checkpoint-200
eval_output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_ckpt_ckpt200_core_judge31b_3g/0002_checkpoint-200.json
passk.average_score: 0.448566
score_out_of_20: 8.97132

Additional completed checkpoint comparison:

wave001 checkpoint-300: passk.average_score 0.439090
wave001 checkpoint-400: passk.average_score 0.439315
wave002 2GPU LR1e-6 checkpoint-400: passk.average_score 0.431448
wave003 2GPU LR3e-6 checkpoint-400: passk.average_score 0.285135

The wave002 checkpoint-400 result did not beat the current clean full-FT best, and the LR3e-6 branch is deprioritized.

The continuation run below is active and has entered actual training:

job: 138685
name: gemma4-fullft-w004b-cont400-lr1e6
node: 25a-hgpn164
gpus: 8x H200
source_weights: wave001 checkpoint-400
learning_rate: 1e-6
initial_eval_loss: 0.9885
initial_eval_ppl: 2.687
train_throughput_observed: about 2.5k tokens/sec/GPU after warmup

Submitted additional patched-server eval to keep the Slurm submit window full:

job: 138702
name: gemma4-fullft-w001c-eval-oomfix-c800
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-800
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt800_core_judge31b_3g
exclude: 25a-hgpn062

2026-06-23 02:17 CST Queue Adjustment

The old non-patched w001c checkpoint-400 eval was cancelled after 47 minutes because it was slow, low scoring at 8/20, and less useful than the patched evals:

job: 138626
state: CANCELLED
reason: free 3 GPUs for higher-value patched checkpoint evals
partial_score: completed 8/20, passk.average_score 0.125

This allowed the queued w002 checkpoint-800 eval to start:

job: 138699
name: gemma4-fullft-w002-2g-eval-oomfix-c800
node: 25a-hgpn166

The w001c 2GPU equivalent-token run completed cleanly:

job: 138540
state: COMPLETED
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-1600
final_eval_loss: 0.8534
final_eval_ppl: 2.348
model_safetensors_size: 17224656900 bytes

New full-FT checkpoints submitted for patched eval:

job: 138705
name: gemma4-fullft-w004b-eval-oomfix-c100
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-100

job: 138706
name: gemma4-fullft-w001c-eval-oomfix-c1600
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-1600

Two partial patched evals were cancelled once their running averages dropped below the clean best and better queued candidates were available:

job: 138695
name: gemma4-fullft-w001c-2g-eval-oomfix-c1200
state_at_cancel: completed 13/20, passk.average_score 0.389744
reason: low partial score, running on known-problem node 25a-hgpn062, c800/c1600 queued

job: 138694
name: gemma4-fullft-w001-8g-eval-oomfix-c400
state_at_cancel: completed 13/20, passk.average_score 0.403205
reason: low partial score, older full checkpoint-400 eval already available, c100 looked stronger

The freed submit capacity was used for:

job: 138716
name: gemma4-fullft-w002-eval-oomfix-c1200
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1200

The w004b continuation produced checkpoint-200, and it was queued for patched eval:

job: 138721
name: gemma4-fullft-w004b-eval-oomfix-c200
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200

The w004b continuation also produced checkpoint-300, and it was queued:

job: 138729
name: gemma4-fullft-w004b-eval-oomfix-c300
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-300

The w002 checkpoint-800 eval was cancelled after it showed no progress toward the current best:

job: 138699
name: gemma4-fullft-w002-2g-eval-oomfix-c800
state_at_cancel: completed 4/20, passk.average_score 0.0
reason: free 3 GPUs for w002 checkpoint-1200 and later candidates

The w002 LR1e-6 run produced checkpoint-1600, and it was queued for eval:

job: 138734
name: gemma4-fullft-w002-eval-oomfix-c1600
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1600

The w001c checkpoint-800 eval was cancelled after its partial average dropped well below the clean best:

job: 138702
name: gemma4-fullft-w001c-eval-oomfix-c800
state_at_cancel: completed 13/20, passk.average_score 0.389744
reason: free 3 GPUs for w002 checkpoint-1600

The w004b continuation completed through checkpoint-400; checkpoint-400 was queued for eval:

job: 138737
name: gemma4-fullft-w004b-eval-oomfix-c400
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400

The w004b checkpoint-100 eval was cancelled after its partial score fell below the clean best:

job: 138705
name: gemma4-fullft-w004b-eval-oomfix-c100
state_at_cancel: completed 13/20, passk.average_score 0.403205
reason: free 3 GPUs for w004b checkpoint-400

2026-06-23 03:11 CST Full-FT Continuation Queue

The w001c checkpoint-1600 eval completed and did not beat the clean best:

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt1600_core_judge31b_3g/0005_checkpoint-1600-138706.json
tasks: 20
passk.average_score: 0.4068333333333333
decision: reject; below clean best 0.44856547619047615

Two low-LR continuation jobs were queued to keep using available Slurm capacity. Both are true full fine-tunes: no adapter config, no LoRA, no quantization, FSDP FULL_STATE_DICT, full model.safetensors checkpoints.

job: 138755
name: gemma4-fullft-w005-w002c1600-lr5e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1600
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority

job: 138756
name: gemma4-fullft-w006-w004bc400-lr5e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority

The continuation checkpoints will be evaluated only if their source branch or their own intermediate training losses justify continuing the search.

2026-06-23 03:19 CST New Full-FT Best

w004b checkpoint-200 completed evaluation and became the current best true full-FT checkpoint:

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt200_core_judge31b_3g/0042_checkpoint-200-138721.json
tasks: 20
passk.average_score: 0.45305833333333334
previous_clean_best: 0.44856547619047615
delta: +0.00449285714285719
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200

Because the new best came from checkpoint-200, a matching low-LR continuation was submitted from that checkpoint:

job: 138769
name: gemma4-fullft-w007-w004bc200-lr5e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave007_cont_from_w004bckpt200_gemma4e4b_agent_sft_lr5e-7_seed307_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority

The w002 checkpoint-1200 eval was cancelled to free GPUs for current-best continuation work:

job: 138716
name: gemma4-fullft-w002-eval-oomfix-c1200
state_at_cancel: completed 6/20, passk.average_score 0.166667
reason: lower-priority checkpoint from the same branch as checkpoint-1600; checkpoint-1600 was already under eval and had a continuation job running

Follow-up checkpoint evals completed:

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave002_2g_lr1e6_seed302_eqtokens_oomfix_ckpt1600_core_judge31b_3g/0006_checkpoint-1600-138734.json
tasks: 20
passk.average_score: 0.4489583333333333
decision: reject for upload; above original clean best but below current best 0.45305833333333334
note: continuation job 138755 was already running from this checkpoint

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt300_core_judge31b_3g/0043_checkpoint-300-138729.json
tasks: 20
passk.average_score: 0.4428404761904762
decision: reject; below current best

w007 started running after resources freed:

job: 138769
name: gemma4-fullft-w007-w004bc200-lr5e7
node: 25a-hgpn166
status: running

w004b checkpoint-400 completed after a late recovery and became the new best:

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt400_core_judge31b_3g/0007_checkpoint-400-138737.json
tasks: 20
passk.average_score: 0.4541071428571429
previous_best: 0.45305833333333334
delta: +0.001048809523809572
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
note: continuation job 138756 was already running from this checkpoint

An additional conservative continuation from the current best was submitted:

job: 138788
name: gemma4-fullft-w008-w004bc400-lr2e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority
reason: compare LR 2e-7 against active LR 5e-7 continuation from the same current-best checkpoint

The w007 continuation from the previous-best checkpoint-200 was cancelled:

job: 138769
name: gemma4-fullft-w007-w004bc200-lr5e7
state_at_cancel: tokenization still in progress, about 21k/242k prompts
reason: checkpoint-400 became the new best; prioritize current-best c400 continuations

w005 checkpoint-400 was produced and queued for TW eval:

job: 138803
name: gemma4-fullft-w005-eval-c400
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-400
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt400_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw

w006 checkpoint-400 was produced and queued for TW eval:

job: 138806
name: gemma4-fullft-w006-eval-c400
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400/checkpoint-400
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave006_cont_w004bckpt400_lr5e7_seed306_ckpt400_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw

w008 checkpoint-400 was produced and queued for TW eval after confirming the full 17G checkpoint had finished writing:

job: 138814
name: gemma4-fullft-w008-eval-c400
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400/checkpoint-400
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt400_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw

w005 checkpoint-800 was produced and queued for TW eval:

job: 138818
name: gemma4-fullft-w005-eval-c800
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-800
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt800_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw

The low-priority w005 checkpoint-400 eval was cancelled after a poor partial:

job: 138803
name: gemma4-fullft-w005-eval-c400
state_at_cancel: completed 5/20, passk.average_score 0.2
reason: w005 is not the current-best branch; free 3 GPUs for current-best c400/c800 evals

w006 checkpoint-800 was produced and queued for TW eval:

job: 138822
name: gemma4-fullft-w006-eval-c800
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400/checkpoint-800
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave006_cont_w004bckpt400_lr5e7_seed306_ckpt800_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw

The w006 checkpoint-400 eval was cancelled after its partial score fell below the current best:

job: 138806
name: gemma4-fullft-w006-eval-c400
state_at_cancel: completed 13/20, passk.average_score 0.38974358974358975
reason: free 3 GPUs for w008 checkpoint-400 and later checkpoint-800 evals

w008 checkpoint-800 was produced and queued for TW eval:

job: 138827
name: gemma4-fullft-w008-eval-c800
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400/checkpoint-800
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt800_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw

At submission time, w005 checkpoint-800 was the strongest partial among active new evals:

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt800_core_judge31b_3g/0002_checkpoint-800-138818.json
partial: completed 8/20, passk.average_score 0.5078125

2026-06-23 08:36 CST Final Full-FT Search

Later continuation and eval waves were run under the same true full-FT policy: no LoRA, no QLoRA, no adapter field, no 4-bit or 8-bit base loading, and full 17,224,656,900 byte model.safetensors checkpoints.

The strongest late candidates were:

wave005 checkpoint-1600:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt1600_core_judge31b_3g/0002_checkpoint-1600-138849.json
  tasks: 20
  passk.average_score: 0.4599238095238095

wave008 checkpoint-800:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt800_core_judge31b_3g/0001_checkpoint-800-138827.json
  tasks: 20
  passk.average_score: 0.45570714285714287

wave009b checkpoint-100:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt100_core_judge31b_3g/0045_checkpoint-100-138874.json
  tasks: 20
  passk.average_score: 0.4680583333333333

wave009b checkpoint-200:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt200_core_judge31b_3g/0013_checkpoint-200-138890.json
  tasks: 20
  passk.average_score: 0.4695104761904762

wave009b checkpoint-300:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt300_core_judge31b_3g/0002_checkpoint-300-138898.json
  tasks: 20
  passk.average_score: 0.4662333333333334

wave009b checkpoint-400:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt400_core_judge31b_3g/0046_checkpoint-400-138903.json
  tasks: 20
  passk.average_score: 0.45391523809523815

wave010 continued from the strongest wave005 checkpoint-1600 branch with learning rate 5e-8:

config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100.yml
job: 138862
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-1600
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062

Completed wave010 evals:

checkpoint-100: 0.4479333333333333
checkpoint-200: 0.4415904761904762
checkpoint-300: 0.44665
checkpoint-400: 0.4789416666666667

wave010 checkpoint-400 became the best true full-FT checkpoint:

eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave010_cont_w005ckpt1600_lr5e8_seed310_ckpt400_core_judge31b_3g/0002_checkpoint-400-138904.json
tasks: 20
passk.average_score: 0.4789416666666667
score_out_of_20: 9.578833333333334
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400
model_safetensors_size: 17224656900 bytes

To test whether this best could still be improved, wave011 continued from wave010 checkpoint-400 with a smaller learning rate, 2e-8:

config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave011_cont_from_w010ckpt400_gemma4e4b_agent_sft_lr2e-8_seed312_seq8192_steps400_save100.yml
job: 138918
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
state: COMPLETED
elapsed: 00:51:45

wave011 checkpoint evals all completed below wave010 checkpoint-400:

checkpoint-100:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt100_core_judge31b_3g/0047_checkpoint-100-138931.json
  tasks: 20
  passk.average_score: 0.4315904761904762

checkpoint-200:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt200_core_judge31b_3g/0008_checkpoint-200-138936.json
  tasks: 20
  passk.average_score: 0.45445833333333335

checkpoint-300:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt300_core_judge31b_3g/0014_checkpoint-300-138944.json
  tasks: 20
  passk.average_score: 0.4371654761904762

checkpoint-400:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt400_core_judge31b_3g/0048_checkpoint-400-138950.json
  tasks: 20
  passk.average_score: 0.45445714285714284

Stop decision:

current_best: wave010 checkpoint-400, 0.4789416666666667
next_smaller_lr_probe: wave011 lr2e-8 from current best
best_wave011_score: 0.45445833333333335
decision: stop full-FT exploration; no later true full-FT continuation improved the completed TW eval score

Upload Plan

Upload the selected true full-FT checkpoint to:

voidful/gemma-4-e4b-it-agent-sft-tw

Initial upload used private=True; current repository visibility is public.

The final Hugging Face repository should include:

  • Full model weights and tokenizer files.
  • README.md model card.
  • PLAYBOOK.md copied from this playbook and updated with final scores.
  • training_config.yml.
  • Evaluation JSON files for the selected checkpoint.
  • A score summary table.

Before upload, verify:

find <staging_dir> -maxdepth 2 -name 'adapter_*' -o -name '*lora*'

The command should not return adapter artifacts for the full-FT model repo.

Staging helper:

/home/voidful2nlp/gemma-agent-sft/scripts/stage_fullft_hf_upload.sh \
  <selected_checkpoint_dir> \
  <selected_eval_dir>

Upload command template:

hf upload voidful/gemma-4-e4b-it-agent-sft-tw <staging_dir> \
  --repo-type model --private --delete '*' \
  --commit-message "Upload true full fine-tuned Gemma 4 E4B agent SFT TW"

Final Results 

selected_checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400
selected_config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100.yml
selected_eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave010_cont_w005ckpt1600_lr5e8_seed310_ckpt400_core_judge31b_3g/0002_checkpoint-400-138904.json
claw_eval_tw_score: 0.4789416666666667
score_out_of_20: 9.578833333333334
hf_repo: voidful/gemma-4-e4b-it-agent-sft-tw
hf_visibility: public
hf_commit: dbef13b501b5277b057fa4fe23d9b5307452e108
hf_commit_note: initial private full artifact upload