PLAYBOOK.md

# Full Fine-Tune Training and Exploration Playbook

This playbook records the full fine-tuning path for `google/gemma-4-E4B-it`
on `voidful/agent-sft`, targeting improvement on `claw-eval-zh --language tw`.

This run is intentionally not LoRA or QLoRA:

- No `adapter` field in the Axolotl config.
- No `lora_model_dir`.
- No `load_in_4bit` or `load_in_8bit`.
- FSDP2 updates and saves full model weights.

The earlier private full-weight repository was a LoRA-trained model merged into
base weights. That artifact is not considered a valid full fine-tune for this
run and must be replaced by a true full-FT checkpoint.

## Objective

Train:

```text
google/gemma-4-E4B-it
```

on:

```text
voidful/agent-sft
```

and evaluate with:

```bash
claw-eval-zh --language tw
```

Final judge model:

```text
google/gemma-4-31B-it
```

Target Hugging Face repository:

```text
voidful/gemma-4-e4b-it-agent-sft-tw
```

The repository was initially uploaded as private. It was later made public on
2026-06-23 CST per follow-up request.

## Slurm Environment

Observed working training environment:

- Account: `gov109183`
- User: `voidful2nlp`
- Main partition: `dev`
- `dev` walltime: 4 hours
- GPU nodes: H200, 8 GPUs per node
- Node memory: about 1.9 TB with `--mem=0`
- Useful full-FT allocation: `--gres=gpu:8 --cpus-per-task=64 --mem=0`
- `8gpus` partition was available but had a much later predicted start time for
  this workload.
- `slinky` and `taide` rejected this account/partition combination.

Useful status commands:

```bash
squeue -u "$USER" -h -o "%.18i|%.45j|%.2t|%.10M|%.6D|%.14b|%.12m|%R" | sort -n
scontrol show job <job_id>
sprio -j <job_id> -o "%i|%Y|%A|%F|%J|%P|%Q|%N"
sinfo -o "%P|%a|%l|%D|%t|%G|%m|%N"
sacct -j <job_id> --format=JobID,JobName%35,State,ExitCode,Elapsed,MaxRSS,ReqMem,AllocTRES -P
```

## Data

Local prepared JSONL files:

```text
/work/voidful2nlp/gemma-agent-sft/data/agent_sft/train.jsonl
/work/voidful2nlp/gemma-agent-sft/data/agent_sft/valid.jsonl
```

Axolotl dataset fields:

```yaml
type: chat_template
field_messages: messages
field_tools: tools
chat_template: tokenizer_default
sample_packing: true
eval_sample_packing: true
sequence_len: 8192
```

## Full-FT Config

Smoke config:

```text
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_smoke_gemma4e4b_agent_sft_seq8192_steps5.yml
```

Main run config:

```text
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100.yml
```

Important settings:

```yaml
base_model: google/gemma-4-E4B-it
sequence_len: 8192
micro_batch_size: 1
gradient_accumulation_steps: 1
optimizer: adamw_torch_fused
learning_rate: 2.0e-06
lr_scheduler: cosine
bf16: true
tf32: true
gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
flash_attention: false
dataloader_num_workers: 0
max_steps: 400
eval_steps: 100
save_steps: 100
save_only_model: true
save_safetensors: true
fsdp_version: 2
fsdp_config:
  offload_params: false
  state_dict_type: FULL_STATE_DICT
  final_state_dict_type: FULL_STATE_DICT
  auto_wrap_policy: TRANSFORMER_BASED_WRAP
  transformer_layer_cls_to_wrap: Gemma4TextDecoderLayer
  reshard_after_forward: true
```

`Gemma4TextDecoderLayer` was verified from the installed Transformers Gemma4
implementation before launching FSDP2 training.

## Smoke Tests

First 8GPU full-FT smoke:

```text
job 138371
```

Result:

```text
FAILED
```

Failure:

```text
OSError: [Errno 12] Cannot allocate memory
```

The failure happened after the first eval when PyTorch tried to fork dataloader
workers from already-large training processes.

Fix:

```yaml
dataloader_num_workers: 0
```

Second 8GPU smoke:

```text
job 138399
```

Result:

```text
COMPLETED
```

Observed smoke metrics:

- Initial eval loss: 1.95
- Final eval loss after 5 steps: 1.878
- Training GPU memory max active: about 44 GiB per GPU
- Training throughput after warmup: about 2.5k tokens/sec/GPU
- Saved full-weight file: `model.safetensors`
- Saved model size: about 17.2 GB

The saved smoke output did not contain adapter weights.

## Training Submit Commands

Smoke:

```bash
CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_smoke_gemma4e4b_agent_sft_seq8192_steps5.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:8 --cpus-per-task=64 --mem=0 \
  --job-name=gemma4-fullft-smoke-nw0 \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch
```

Main wave:

```bash
CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100.yml \
sbatch --partition=dev --time=03:00:00 --gres=gpu:8 --cpus-per-task=64 --mem=0 \
  --job-name=gemma4-fullft-w001-lr2e6-s301-3h \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch
```

Main wave job:

```text
138433
```

At submission time Slurm predicted:

```text
StartTime=2026-06-23T00:02:57
SchedNodeList=25a-hgpn129
Reason=Priority
```

The job later started on:

```text
job 138433
node 25a-hgpn171
world_size 8
started 2026-06-23 00:37:26 CST
```

Observed 8GPU metrics after warmup:

- Training memory max active: about 44 GiB per GPU
- Device reserved: about 58 GiB per GPU
- Training throughput: about 2.5k tokens/sec/GPU
- Step time after warmup: about 3.3 seconds
- `checkpoint-100` and `checkpoint-200` were full checkpoints with
  `model.safetensors` about 17.2 GB.

Because 8GPU full-node scheduling drifted repeatedly, a 2GPU equivalent-token
backup run was launched while keeping the 8GPU job pending. To keep the total
number of packed samples comparable to 8GPU x 400 steps, the 2GPU run uses
1600 steps and saves/evals every 400 steps.

2GPU equivalent-token config:

```text
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400.yml
```

2GPU submit command:

```bash
CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \
  --job-name=gemma4-fullft-w001c-2g-eqtok \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch
```

2GPU run:

```text
job 138540
node 25a-hgpn174
world_size 2
```

Observed early 2GPU metrics:

- Initial eval loss: 1.947
- Training memory max active: about 60 GiB per GPU
- Training throughput: about 2.5k tokens/sec/GPU
- Step time after warmup: about 3.26 seconds

This remains true full fine-tuning: no adapter, no LoRA, and no 4-bit or 8-bit
base-model loading.

### Parallel LR Sweep

To use partial-GPU slots while the main/equivalent runs were active, two more
2GPU true full-FT runs were queued:

```text
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400.yml
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave003_2g_eqtokens_gemma4e4b_agent_sft_lr3e-6_seed303_seq8192_steps1600_save400.yml
```

Submit commands:

```bash
CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \
  --job-name=gemma4-fullft-w002-2g-lr1e6 \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch

CONFIG=/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave003_2g_eqtokens_gemma4e4b_agent_sft_lr3e-6_seed303_seq8192_steps1600_save400.yml \
sbatch --partition=dev --time=04:00:00 --gres=gpu:2 --cpus-per-task=24 --mem=700G \
  --job-name=gemma4-fullft-w003-2g-lr3e6 \
  /home/voidful2nlp/gemma-agent-sft/slurm/train_axolotl.sbatch
```

The first LR1e-6 submission, job `138577`, failed because Slurm placed it on
`25a-hgpn062`, where the second allocated H200 reported `N/A/ERR` in
`nvidia-smi`; PyTorch saw only one CUDA device and rank 1 failed. It was
resubmitted as job `138582` with:

```bash
--exclude=25a-hgpn062
```

Active sweep jobs:

```text
138578  LR3e-6  node 25a-hgpn003
138582  LR1e-6  node 25a-hgpn003
```

Both runs use FSDP2 full fine-tuning and do not include adapters.

### Main 8GPU Completion and Continuation

Main 8GPU job `138433` completed successfully:

```text
state: COMPLETED
elapsed: 00:43:59
node: 25a-hgpn171
exit: 0:0
```

Completed full checkpoints:

```text
checkpoint-100/model.safetensors  17,224,656,900 bytes
checkpoint-200/model.safetensors  17,224,656,900 bytes
checkpoint-300/model.safetensors  17,224,656,900 bytes
checkpoint-400/model.safetensors  17,224,656,900 bytes
final model.safetensors           17,224,656,900 bytes
```

Final logged training/eval metrics for `138433`:

```text
eval_loss: 0.986
eval_ppl: 2.681
train_loss: 1.118
```

Because `save_only_model: true` skips optimizer and scheduler state, the next
full-FT extension is not a trainer resume. Instead, `checkpoint-400` is used as
the local `base_model` for another full fine-tune:

```text
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave004_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100.yml
```

This continuation remains full fine-tuning: no adapter, no LoRA, no QLoRA, and
no 4-bit or 8-bit base loading.

The 8GPU continuation was submitted as:

```text
138663  gemma4-fullft-w004-cont400-lr1e6
```

This job failed before model training. The root cause was not GPU memory or
FSDP, but local checkpoint metadata: Axolotl treated Gemma 4 as multimodal and
called `AutoProcessor.from_pretrained(checkpoint-400)`, while the checkpoint
subdirectory had model/tokenizer files but no `processor_config.json`.

The fix was to copy the parent run's `processor_config.json` into the full
checkpoint directory:

```bash
install -m 0644 \
  /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/processor_config.json \
  /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/checkpoint-400/processor_config.json
```

After the metadata fix, `AutoProcessor.from_pretrained(checkpoint-400)` loaded
successfully as `Gemma4Processor`. A corrected continuation config was added:

```text
/home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100.yml
```

It was submitted as:

```text
138685  gemma4-fullft-w004b-cont400-lr1e6
```

This remains a full-weight continuation: `base_model` and `tokenizer_config`
point to the local full checkpoint, `load_in_4bit=false`, `load_in_8bit=false`,
and `adapter=None`.

The cluster also enforced `QOSMaxSubmitJobPerUserLimit`; to free a job slot,
the lowest-priority `checkpoint-100` eval retry job `138651` was cancelled and
should be resubmitted later if the score curve needs it.

### Live Evaluation Queue

Submitted evaluation jobs:

```text
138626  2GPU LR2e-6 checkpoint-400  running, node 25a-hgpn166
138635  8GPU LR2e-6 checkpoint-200  pending
138652  8GPU LR2e-6 checkpoint-300  pending
138660  2GPU LR1e-6 checkpoint-400  pending
138661  2GPU LR3e-6 checkpoint-400  pending
138662  8GPU LR2e-6 checkpoint-400  pending
```

All eval jobs use:

```text
judge model: google/gemma-4-31B-it
language: tw
target GPUs: 0
judge GPUs: 1,2
ADAPTER: empty
```

## Evaluation

Evaluate full checkpoints directly as model paths. Leave `ADAPTER` empty.

Template:

```bash
MODEL=/work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100/checkpoint-100 \
ADAPTER= \
SUITE=all LANGUAGE=tw \
OUTPUT_DIR=/work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_ckpt100_core_judge31b_3g \
sbatch --partition=dev --time=04:00:00 --gres=gpu:3 --cpus-per-task=40 --mem=800G \
  --job-name=gemma4-fullft-w001c100-eval \
  /home/voidful2nlp/gemma-agent-sft/slurm/eval_claw.sbatch \
  --judge-model google/gemma-4-31B-it --core --no-parallel-judge \
  --target-gpus 0 --judge-gpus 1,2
```

Summarize results:

```bash
python /home/voidful2nlp/gemma-agent-sft/scripts/summarize_claw_results.py \
  /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_ckpt*_core_judge31b_3g
```

Helper script:

```bash
/home/voidful2nlp/gemma-agent-sft/scripts/submit_fullft_wave001_evals.sh
```

The helper submits only checkpoints that already exist and always sets
`ADAPTER=` to avoid adapter-based evaluation. `RUN_LABEL` and `JOB_PREFIX` can
be set to keep output directories and Slurm job names distinct across runs.

### Eval Serving Fix

The first eval attempt for 8GPU `checkpoint-100`, job `138611`, failed before
benchmark execution:

```text
OSError: Can't load feature extractor for .../checkpoint-100
```

The full-FT checkpoint directories contain model/tokenizer/config files, but
Axolotl did not copy `processor_config.json` into each checkpoint. The local
text-only OpenAI-compatible server uses tokenizer chat templates and generation,
not processor preprocessing, so `serve_transformers_openai.py` was changed to
fall back to tokenizer-only serving when `AutoProcessor.from_pretrained(...)`
raises `OSError`.

During long-context file-analysis evals, one target server produced CUDA OOM and
HTTP 503 responses after a single retry at 512 new tokens. The serving script now
tries progressively smaller generation budgets (`1024`, `512`, `256`, `128`)
before returning 503. This affects future eval jobs; already running eval jobs
keep the script version they started with.

The failed eval was resubmitted as:

```text
138651  8GPU LR2e-6 checkpoint-100 retry
```

### Concurrent Eval Workspace Collision

When multiple eval jobs served different runs whose checkpoint directory basename
was the same, for example `checkpoint-400`, the eval driver originally passed the
same model name to `claw-eval-zh`:

```text
checkpoint-400
```

That made `claw-eval-zh` create the same OpenClaw agent id:

```text
bench-checkpoint-400
```

If two such evals landed on the same node, their disposable workspaces could
interfere. The observed symptom was:

```text
OpenClaw command failed ... ENOENT: no such file or directory, uv_cwd
```

This affected LR sweep eval jobs `138660` and `138661`; both were cancelled and
resubmitted with unique output labels as `138669` and `138670`.

Fix in `/home/voidful2nlp/gemma-agent-sft/scripts/run_claw_eval.py`:

```text
default model_name = <checkpoint-basename>-<SLURM_JOB_ID>
```

`eval_claw.sbatch` was also updated to print Slurm GPU visibility in future eval
logs.

### 8GPU Checkpoint-400 Original Eval

The first complete 20-task eval for the 8GPU LR2e-6 `checkpoint-400` finished as
job `138662`:

```text
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_ckpt_ckpt400_core_judge31b_3g/0001_checkpoint-400.json
score: 8.79 / 20.0
mean: 0.439315
pass@1: 35.0%
```

This run is retained for audit, but it had target-server CUDA OOM/HTTP 503
events before the multi-step retry fix landed. A patched-server rerun was
submitted:

```text
138694  gemma4-fullft-w001-8g-eval-oomfix-c400
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_oomfix_ckpt400_core_judge31b_3g
```

The same 8GPU LR2e-6 run also completed `checkpoint-200` and `checkpoint-300`
evals:

```text
138635  checkpoint-200  mean: 0.448565  score: 8.97 / 20.0
138652  checkpoint-300  mean: 0.439090  score: 8.78 / 20.0
```

At this point the best completed clean full-FT score is `checkpoint-200`
(`0.448565`). `checkpoint-400` is close but has serving OOM/503 artifacts, so
the patched-server rerun `138694` is required before making a final comparison.

### LR Sweep Early Result

The first completed 2GPU equivalent-token LR sweep eval was the LR3e-6
`checkpoint-400` job `138670`:

```text
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave003_2g_lr3e6_seed303_eqtokens_retryuniq_ckpt400_core_judge31b_3g/0002_checkpoint-400-138670.json
score: 5.70 / 20.0
mean: 0.285135
pass@1: 25.0%
```

That result is well below the 8GPU LR2e-6 baseline, so LR3e-6 is deprioritized.
With the next free submit slot, the LR2e-6 equivalent-token `checkpoint-1200`
was queued using the patched serving script:

```text
138695  gemma4-fullft-w001c-2g-eval-oomfix-c1200
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt1200_core_judge31b_3g
```

The eval submit helper now supports `EXCLUDE=...` so future evals can avoid
known-bad nodes such as `25a-hgpn062`. After `checkpoint-200` and
`checkpoint-300` evals completed, two more patched-server evals were submitted:

```text
138698  gemma4-fullft-w001-8g-eval-oomfix-c100
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_oomfix_ckpt100_core_judge31b_3g

138699  gemma4-fullft-w002-2g-eval-oomfix-c800
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave002_2g_lr1e6_seed302_eqtokens_oomfix_ckpt800_core_judge31b_3g
```

## Selection Rule

For this true full-FT run:

1. Evaluate checkpoints 100, 200, 300, and 400.
2. Select the highest completed `claw-eval-zh --language tw` score.
3. If a checkpoint improves over the current true full-FT best, continue from that
   checkpoint with a smaller neighborhood search.
4. Stop only when new full-FT continuation runs no longer improve the completed
   score.

The previous LoRA score, 9.498095238095239, is useful context but is not a valid
full-FT artifact.

## 2026-06-23 02:08 CST Full-FT Status

The run policy was corrected to true full fine-tuning only. LoRA, QLoRA,
adapter training, and merged-adapter outputs are excluded from final selection.

Current clean completed full-FT best:

```text
run: fullft_wave001_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps400_save100
checkpoint: checkpoint-200
eval_output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001_8g_lr2e6_seed301_ckpt_ckpt200_core_judge31b_3g/0002_checkpoint-200.json
passk.average_score: 0.448566
score_out_of_20: 8.97132
```

Additional completed checkpoint comparison:

```text
wave001 checkpoint-300: passk.average_score 0.439090
wave001 checkpoint-400: passk.average_score 0.439315
wave002 2GPU LR1e-6 checkpoint-400: passk.average_score 0.431448
wave003 2GPU LR3e-6 checkpoint-400: passk.average_score 0.285135
```

The wave002 checkpoint-400 result did not beat the current clean full-FT best,
and the LR3e-6 branch is deprioritized.

The continuation run below is active and has entered actual training:

```text
job: 138685
name: gemma4-fullft-w004b-cont400-lr1e6
node: 25a-hgpn164
gpus: 8x H200
source_weights: wave001 checkpoint-400
learning_rate: 1e-6
initial_eval_loss: 0.9885
initial_eval_ppl: 2.687
train_throughput_observed: about 2.5k tokens/sec/GPU after warmup
```

Submitted additional patched-server eval to keep the Slurm submit window full:

```text
job: 138702
name: gemma4-fullft-w001c-eval-oomfix-c800
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-800
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt800_core_judge31b_3g
exclude: 25a-hgpn062
```

## 2026-06-23 02:17 CST Queue Adjustment

The old non-patched `w001c checkpoint-400` eval was cancelled after 47 minutes
because it was slow, low scoring at 8/20, and less useful than the patched evals:

```text
job: 138626
state: CANCELLED
reason: free 3 GPUs for higher-value patched checkpoint evals
partial_score: completed 8/20, passk.average_score 0.125
```

This allowed the queued `w002 checkpoint-800` eval to start:

```text
job: 138699
name: gemma4-fullft-w002-2g-eval-oomfix-c800
node: 25a-hgpn166
```

The `w001c` 2GPU equivalent-token run completed cleanly:

```text
job: 138540
state: COMPLETED
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-1600
final_eval_loss: 0.8534
final_eval_ppl: 2.348
model_safetensors_size: 17224656900 bytes
```

New full-FT checkpoints submitted for patched eval:

```text
job: 138705
name: gemma4-fullft-w004b-eval-oomfix-c100
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-100

job: 138706
name: gemma4-fullft-w001c-eval-oomfix-c1600
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave001c_2g_eqtokens_gemma4e4b_agent_sft_lr2e-6_seed301_seq8192_steps1600_save400/checkpoint-1600
```

Two partial patched evals were cancelled once their running averages dropped
below the clean best and better queued candidates were available:

```text
job: 138695
name: gemma4-fullft-w001c-2g-eval-oomfix-c1200
state_at_cancel: completed 13/20, passk.average_score 0.389744
reason: low partial score, running on known-problem node 25a-hgpn062, c800/c1600 queued

job: 138694
name: gemma4-fullft-w001-8g-eval-oomfix-c400
state_at_cancel: completed 13/20, passk.average_score 0.403205
reason: low partial score, older full checkpoint-400 eval already available, c100 looked stronger
```

The freed submit capacity was used for:

```text
job: 138716
name: gemma4-fullft-w002-eval-oomfix-c1200
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1200
```

The `w004b` continuation produced `checkpoint-200`, and it was queued for
patched eval:

```text
job: 138721
name: gemma4-fullft-w004b-eval-oomfix-c200
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200
```

The `w004b` continuation also produced `checkpoint-300`, and it was queued:

```text
job: 138729
name: gemma4-fullft-w004b-eval-oomfix-c300
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-300
```

The `w002 checkpoint-800` eval was cancelled after it showed no progress toward
the current best:

```text
job: 138699
name: gemma4-fullft-w002-2g-eval-oomfix-c800
state_at_cancel: completed 4/20, passk.average_score 0.0
reason: free 3 GPUs for w002 checkpoint-1200 and later candidates
```

The `w002` LR1e-6 run produced `checkpoint-1600`, and it was queued for eval:

```text
job: 138734
name: gemma4-fullft-w002-eval-oomfix-c1600
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1600
```

The `w001c checkpoint-800` eval was cancelled after its partial average dropped
well below the clean best:

```text
job: 138702
name: gemma4-fullft-w001c-eval-oomfix-c800
state_at_cancel: completed 13/20, passk.average_score 0.389744
reason: free 3 GPUs for w002 checkpoint-1600
```

The `w004b` continuation completed through `checkpoint-400`; `checkpoint-400`
was queued for eval:

```text
job: 138737
name: gemma4-fullft-w004b-eval-oomfix-c400
model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
```

The `w004b checkpoint-100` eval was cancelled after its partial score fell below
the clean best:

```text
job: 138705
name: gemma4-fullft-w004b-eval-oomfix-c100
state_at_cancel: completed 13/20, passk.average_score 0.403205
reason: free 3 GPUs for w004b checkpoint-400
```

## 2026-06-23 03:11 CST Full-FT Continuation Queue

The `w001c checkpoint-1600` eval completed and did not beat the clean best:

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave001c_2g_lr2e6_seed301_eqtokens_oomfix_ckpt1600_core_judge31b_3g/0005_checkpoint-1600-138706.json
tasks: 20
passk.average_score: 0.4068333333333333
decision: reject; below clean best 0.44856547619047615
```

Two low-LR continuation jobs were queued to keep using available Slurm capacity.
Both are true full fine-tunes: no adapter config, no LoRA, no quantization, FSDP
FULL_STATE_DICT, full `model.safetensors` checkpoints.

```text
job: 138755
name: gemma4-fullft-w005-w002c1600-lr5e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave002_2g_eqtokens_gemma4e4b_agent_sft_lr1e-6_seed302_seq8192_steps1600_save400/checkpoint-1600
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority
```

```text
job: 138756
name: gemma4-fullft-w006-w004bc400-lr5e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority
```

The continuation checkpoints will be evaluated only if their source branch or
their own intermediate training losses justify continuing the search.

## 2026-06-23 03:19 CST New Full-FT Best

`w004b checkpoint-200` completed evaluation and became the current best true
full-FT checkpoint:

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt200_core_judge31b_3g/0042_checkpoint-200-138721.json
tasks: 20
passk.average_score: 0.45305833333333334
previous_clean_best: 0.44856547619047615
delta: +0.00449285714285719
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200
```

Because the new best came from `checkpoint-200`, a matching low-LR continuation
was submitted from that checkpoint:

```text
job: 138769
name: gemma4-fullft-w007-w004bc200-lr5e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave007_cont_from_w004bckpt200_gemma4e4b_agent_sft_lr5e-7_seed307_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-200
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority
```

The `w002 checkpoint-1200` eval was cancelled to free GPUs for current-best
continuation work:

```text
job: 138716
name: gemma4-fullft-w002-eval-oomfix-c1200
state_at_cancel: completed 6/20, passk.average_score 0.166667
reason: lower-priority checkpoint from the same branch as checkpoint-1600; checkpoint-1600 was already under eval and had a continuation job running
```

Follow-up checkpoint evals completed:

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave002_2g_lr1e6_seed302_eqtokens_oomfix_ckpt1600_core_judge31b_3g/0006_checkpoint-1600-138734.json
tasks: 20
passk.average_score: 0.4489583333333333
decision: reject for upload; above original clean best but below current best 0.45305833333333334
note: continuation job 138755 was already running from this checkpoint
```

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt300_core_judge31b_3g/0043_checkpoint-300-138729.json
tasks: 20
passk.average_score: 0.4428404761904762
decision: reject; below current best
```

`w007` started running after resources freed:

```text
job: 138769
name: gemma4-fullft-w007-w004bc200-lr5e7
node: 25a-hgpn166
status: running
```

`w004b checkpoint-400` completed after a late recovery and became the new best:

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave004b_cont_w001ckpt400_lr1e6_seed304_oomfix_ckpt400_core_judge31b_3g/0007_checkpoint-400-138737.json
tasks: 20
passk.average_score: 0.4541071428571429
previous_best: 0.45305833333333334
delta: +0.001048809523809572
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
note: continuation job 138756 was already running from this checkpoint
```

An additional conservative continuation from the current best was submitted:

```text
job: 138788
name: gemma4-fullft-w008-w004bc400-lr2e7
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400.yml
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave004b_cont_from_w001ckpt400_gemma4e4b_agent_sft_lr1e-6_seed304_seq8192_steps400_save100/checkpoint-400
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
status_at_submit: pending priority
reason: compare LR 2e-7 against active LR 5e-7 continuation from the same current-best checkpoint
```

The `w007` continuation from the previous-best `checkpoint-200` was cancelled:

```text
job: 138769
name: gemma4-fullft-w007-w004bc200-lr5e7
state_at_cancel: tokenization still in progress, about 21k/242k prompts
reason: checkpoint-400 became the new best; prioritize current-best c400 continuations
```

`w005 checkpoint-400` was produced and queued for TW eval:

```text
job: 138803
name: gemma4-fullft-w005-eval-c400
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-400
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt400_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw
```

`w006 checkpoint-400` was produced and queued for TW eval:

```text
job: 138806
name: gemma4-fullft-w006-eval-c400
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400/checkpoint-400
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave006_cont_w004bckpt400_lr5e7_seed306_ckpt400_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw
```

`w008 checkpoint-400` was produced and queued for TW eval after confirming the
full 17G checkpoint had finished writing:

```text
job: 138814
name: gemma4-fullft-w008-eval-c400
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400/checkpoint-400
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt400_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw
```

`w005 checkpoint-800` was produced and queued for TW eval:

```text
job: 138818
name: gemma4-fullft-w005-eval-c800
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-800
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt800_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw
```

The low-priority `w005 checkpoint-400` eval was cancelled after a poor partial:

```text
job: 138803
name: gemma4-fullft-w005-eval-c400
state_at_cancel: completed 5/20, passk.average_score 0.2
reason: w005 is not the current-best branch; free 3 GPUs for current-best c400/c800 evals
```

`w006 checkpoint-800` was produced and queued for TW eval:

```text
job: 138822
name: gemma4-fullft-w006-eval-c800
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave006_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr5e-7_seed306_seq8192_steps1600_save400/checkpoint-800
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave006_cont_w004bckpt400_lr5e7_seed306_ckpt800_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw
```

The `w006 checkpoint-400` eval was cancelled after its partial score fell below
the current best:

```text
job: 138806
name: gemma4-fullft-w006-eval-c400
state_at_cancel: completed 13/20, passk.average_score 0.38974358974358975
reason: free 3 GPUs for w008 checkpoint-400 and later checkpoint-800 evals
```

`w008 checkpoint-800` was produced and queued for TW eval:

```text
job: 138827
name: gemma4-fullft-w008-eval-c800
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave008_cont_from_w004bckpt400_gemma4e4b_agent_sft_lr2e-7_seed308_seq8192_steps1600_save400/checkpoint-800
model_artifact: full 17G model.safetensors
output: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt800_core_judge31b_3g
judge: google/gemma-4-31B-it
language: tw
```

At submission time, `w005 checkpoint-800` was the strongest partial among active
new evals:

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt800_core_judge31b_3g/0002_checkpoint-800-138818.json
partial: completed 8/20, passk.average_score 0.5078125
```

## 2026-06-23 08:36 CST Final Full-FT Search

Later continuation and eval waves were run under the same true full-FT policy:
no LoRA, no QLoRA, no adapter field, no 4-bit or 8-bit base loading, and full
17,224,656,900 byte `model.safetensors` checkpoints.

The strongest late candidates were:

```text
wave005 checkpoint-1600:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave005_cont_w002ckpt1600_lr5e7_seed305_ckpt1600_core_judge31b_3g/0002_checkpoint-1600-138849.json
  tasks: 20
  passk.average_score: 0.4599238095238095

wave008 checkpoint-800:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave008_cont_w004bckpt400_lr2e7_seed308_ckpt800_core_judge31b_3g/0001_checkpoint-800-138827.json
  tasks: 20
  passk.average_score: 0.45570714285714287

wave009b checkpoint-100:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt100_core_judge31b_3g/0045_checkpoint-100-138874.json
  tasks: 20
  passk.average_score: 0.4680583333333333

wave009b checkpoint-200:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt200_core_judge31b_3g/0013_checkpoint-200-138890.json
  tasks: 20
  passk.average_score: 0.4695104761904762

wave009b checkpoint-300:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt300_core_judge31b_3g/0002_checkpoint-300-138898.json
  tasks: 20
  passk.average_score: 0.4662333333333334

wave009b checkpoint-400:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave009b_cont_w008ckpt800_lr5e8_seed311_ckpt400_core_judge31b_3g/0046_checkpoint-400-138903.json
  tasks: 20
  passk.average_score: 0.45391523809523815
```

`wave010` continued from the strongest `wave005 checkpoint-1600` branch with
learning rate `5e-8`:

```text
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100.yml
job: 138862
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave005_cont_from_w002ckpt1600_gemma4e4b_agent_sft_lr5e-7_seed305_seq8192_steps1600_save400/checkpoint-1600
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
```

Completed `wave010` evals:

```text
checkpoint-100: 0.4479333333333333
checkpoint-200: 0.4415904761904762
checkpoint-300: 0.44665
checkpoint-400: 0.4789416666666667
```

`wave010 checkpoint-400` became the best true full-FT checkpoint:

```text
eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave010_cont_w005ckpt1600_lr5e8_seed310_ckpt400_core_judge31b_3g/0002_checkpoint-400-138904.json
tasks: 20
passk.average_score: 0.4789416666666667
score_out_of_20: 9.578833333333334
checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400
model_safetensors_size: 17224656900 bytes
```

To test whether this best could still be improved, `wave011` continued from
`wave010 checkpoint-400` with a smaller learning rate, `2e-8`:

```text
config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave011_cont_from_w010ckpt400_gemma4e4b_agent_sft_lr2e-8_seed312_seq8192_steps400_save100.yml
job: 138918
base_model: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400
resources: dev, 2x H200, 24 CPU, 700G RAM, 4h, exclude 25a-hgpn062
state: COMPLETED
elapsed: 00:51:45
```

`wave011` checkpoint evals all completed below `wave010 checkpoint-400`:

```text
checkpoint-100:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt100_core_judge31b_3g/0047_checkpoint-100-138931.json
  tasks: 20
  passk.average_score: 0.4315904761904762

checkpoint-200:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt200_core_judge31b_3g/0008_checkpoint-200-138936.json
  tasks: 20
  passk.average_score: 0.45445833333333335

checkpoint-300:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt300_core_judge31b_3g/0014_checkpoint-300-138944.json
  tasks: 20
  passk.average_score: 0.4371654761904762

checkpoint-400:
  eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave011_cont_w010ckpt400_lr2e8_seed312_ckpt400_core_judge31b_3g/0048_checkpoint-400-138950.json
  tasks: 20
  passk.average_score: 0.45445714285714284
```

Stop decision:

```text
current_best: wave010 checkpoint-400, 0.4789416666666667
next_smaller_lr_probe: wave011 lr2e-8 from current best
best_wave011_score: 0.45445833333333335
decision: stop full-FT exploration; no later true full-FT continuation improved the completed TW eval score
```

## Upload Plan

Upload the selected true full-FT checkpoint to:

```text
voidful/gemma-4-e4b-it-agent-sft-tw
```

Initial upload used `private=True`; current repository visibility is public.

The final Hugging Face repository should include:

- Full model weights and tokenizer files.
- `README.md` model card.
- `PLAYBOOK.md` copied from this playbook and updated with final scores.
- `training_config.yml`.
- Evaluation JSON files for the selected checkpoint.
- A score summary table.

Before upload, verify:

```bash
find <staging_dir> -maxdepth 2 -name 'adapter_*' -o -name '*lora*'
```

The command should not return adapter artifacts for the full-FT model repo.

Staging helper:

```bash
/home/voidful2nlp/gemma-agent-sft/scripts/stage_fullft_hf_upload.sh \
  <selected_checkpoint_dir> \
  <selected_eval_dir>
```

Upload command template:

```bash
hf upload voidful/gemma-4-e4b-it-agent-sft-tw <staging_dir> \
  --repo-type model --private --delete '*' \
  --commit-message "Upload true full fine-tuned Gemma 4 E4B agent SFT TW"
```

## Final Results

```text
selected_checkpoint: /work/voidful2nlp/gemma-agent-sft/outputs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100/checkpoint-400
selected_config: /home/voidful2nlp/gemma-agent-sft/configs_fullft/fullft_wave010_cont_from_w005ckpt1600_gemma4e4b_agent_sft_lr5e-8_seed310_seq8192_steps400_save100.yml
selected_eval: /work/voidful2nlp/gemma-agent-sft/eval/fullft_wave010_cont_w005ckpt1600_lr5e8_seed310_ckpt400_core_judge31b_3g/0002_checkpoint-400-138904.json
claw_eval_tw_score: 0.4789416666666667
score_out_of_20: 9.578833333333334
hf_repo: voidful/gemma-4-e4b-it-agent-sft-tw
hf_visibility: public
hf_commit: dbef13b501b5277b057fa4fe23d9b5307452e108
hf_commit_note: initial private full artifact upload
```