logs/Exp18_affect_ar_sft/ar5000_3gpu/train.log

The following values were not passed to `accelerate launch` and had defaults used instead:
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
sharding_strategy is deprecated in favor of reshard_after_forward. This will be removed in a future version of Accelerate.
sharding_strategy is deprecated in favor of reshard_after_forward. This will be removed in a future version of Accelerate.
sharding_strategy is deprecated in favor of reshard_after_forward. This will be removed in a future version of Accelerate.
{
  "seed": 180018,
  "ar_train_rows": 5000,
  "eval_rows_from_av_half": 72,
  "ar_general_rows": 4500,
  "ar_emotion_rows": 500,
  "sequence_tokens_min": 100,
  "sequence_tokens_mean": 142.2422,
  "sequence_tokens_max": 163,
  "critic_suffix_ids": [
    1318,
    29,
    366,
    1708,
    29
  ],
  "train_baseline_normalized_mean": 0.9480924010276794,
  "train_baseline_raw_variance": 0.7233728766441345,
  "eval_baseline_normalized_mean": 0.9384064674377441,
  "eval_baseline_raw_variance": 0.718255341053009
}
[model] loading Qwen blocks 0..20 + identity-initialized value head
Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.30s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.27s/it]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:01<00:03,  1.32s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.29s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.27s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:02<00:02,  1.31s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.16s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.15s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:03<00:01,  1.19s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.19it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.01it/s]
Some weights of the model checkpoint at /home/sparrow/NLAmodel/models/qwen2.5-7b-instruct were not used when initializing Qwen2ForCausalLM: ['model.layers.21.input_layernorm.weight', 'model.layers.21.mlp.down_proj.weight', 'model.layers.21.mlp.gate_proj.weight', 'model.layers.21.mlp.up_proj.weight', 'model.layers.21.post_attention_layernorm.weight', 'model.layers.21.self_attn.k_proj.bias', 'model.layers.21.self_attn.k_proj.weight', 'model.layers.21.self_attn.o_proj.weight', 'model.layers.21.self_attn.q_proj.bias', 'model.layers.21.self_attn.q_proj.weight', 'model.layers.21.self_attn.v_proj.bias', 'model.layers.21.self_attn.v_proj.weight', 'model.layers.22.input_layernorm.weight', 'model.layers.22.mlp.down_proj.weight', 'model.layers.22.mlp.gate_proj.weight', 'model.layers.22.mlp.up_proj.weight', 'model.layers.22.post_attention_layernorm.weight', 'model.layers.22.self_attn.k_proj.bias', 'model.layers.22.self_attn.k_proj.weight', 'model.layers.22.self_attn.o_proj.weight', 'model.layers.22.self_attn.q_proj.bias', 'model.layers.22.self_attn.q_proj.weight', 'model.layers.22.self_attn.v_proj.bias', 'model.layers.22.self_attn.v_proj.weight', 'model.layers.23.input_layernorm.weight', 'model.layers.23.mlp.down_proj.weight', 'model.layers.23.mlp.gate_proj.weight', 'model.layers.23.mlp.up_proj.weight', 'model.layers.23.post_attention_layernorm.weight', 'model.layers.23.self_attn.k_proj.bias', 'model.layers.23.self_attn.k_proj.weight', 'model.layers.23.self_attn.o_proj.weight', 'model.layers.23.self_attn.q_proj.bias', 'model.layers.23.self_attn.q_proj.weight', 'model.layers.23.self_attn.v_proj.bias', 'model.layers.23.self_attn.v_proj.weight', 'model.layers.24.input_layernorm.weight', 'model.layers.24.mlp.down_proj.weight', 'model.layers.24.mlp.gate_proj.weight', 'model.layers.24.mlp.up_proj.weight', 'model.layers.24.post_attention_layernorm.weight', 'model.layers.24.self_attn.k_proj.bias', 'model.layers.24.self_attn.k_proj.weight', 'model.layers.24.self_attn.o_proj.weight', 'model.layers.24.self_attn.q_proj.bias', 'model.layers.24.self_attn.q_proj.weight', 'model.layers.24.self_attn.v_proj.bias', 'model.layers.24.self_attn.v_proj.weight', 'model.layers.25.input_layernorm.weight', 'model.layers.25.mlp.down_proj.weight', 'model.layers.25.mlp.gate_proj.weight', 'model.layers.25.mlp.up_proj.weight', 'model.layers.25.post_attention_layernorm.weight', 'model.layers.25.self_attn.k_proj.bias', 'model.layers.25.self_attn.k_proj.weight', 'model.layers.25.self_attn.o_proj.weight', 'model.layers.25.self_attn.q_proj.bias', 'model.layers.25.self_attn.q_proj.weight', 'model.layers.25.self_attn.v_proj.bias', 'model.layers.25.self_attn.v_proj.weight', 'model.layers.26.input_layernorm.weight', 'model.layers.26.mlp.down_proj.weight', 'model.layers.26.mlp.gate_proj.weight', 'model.layers.26.mlp.up_proj.weight', 'model.layers.26.post_attention_layernorm.weight', 'model.layers.26.self_attn.k_proj.bias', 'model.layers.26.self_attn.k_proj.weight', 'model.layers.26.self_attn.o_proj.weight', 'model.layers.26.self_attn.q_proj.bias', 'model.layers.26.self_attn.q_proj.weight', 'model.layers.26.self_attn.v_proj.bias', 'model.layers.26.self_attn.v_proj.weight', 'model.layers.27.input_layernorm.weight', 'model.layers.27.mlp.down_proj.weight', 'model.layers.27.mlp.gate_proj.weight', 'model.layers.27.mlp.up_proj.weight', 'model.layers.27.post_attention_layernorm.weight', 'model.layers.27.self_attn.k_proj.bias', 'model.layers.27.self_attn.k_proj.weight', 'model.layers.27.self_attn.o_proj.weight', 'model.layers.27.self_attn.q_proj.bias', 'model.layers.27.self_attn.q_proj.weight', 'model.layers.27.self_attn.v_proj.bias', 'model.layers.27.self_attn.v_proj.weight']
- This IS expected if you are initializing Qwen2ForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing Qwen2ForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.20it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:03<00:00,  1.03it/s]
Some weights of the model checkpoint at /home/sparrow/NLAmodel/models/qwen2.5-7b-instruct were not used when initializing Qwen2ForCausalLM: ['model.layers.21.input_layernorm.weight', 'model.layers.21.mlp.down_proj.weight', 'model.layers.21.mlp.gate_proj.weight', 'model.layers.21.mlp.up_proj.weight', 'model.layers.21.post_attention_layernorm.weight', 'model.layers.21.self_attn.k_proj.bias', 'model.layers.21.self_attn.k_proj.weight', 'model.layers.21.self_attn.o_proj.weight', 'model.layers.21.self_attn.q_proj.bias', 'model.layers.21.self_attn.q_proj.weight', 'model.layers.21.self_attn.v_proj.bias', 'model.layers.21.self_attn.v_proj.weight', 'model.layers.22.input_layernorm.weight', 'model.layers.22.mlp.down_proj.weight', 'model.layers.22.mlp.gate_proj.weight', 'model.layers.22.mlp.up_proj.weight', 'model.layers.22.post_attention_layernorm.weight', 'model.layers.22.self_attn.k_proj.bias', 'model.layers.22.self_attn.k_proj.weight', 'model.layers.22.self_attn.o_proj.weight', 'model.layers.22.self_attn.q_proj.bias', 'model.layers.22.self_attn.q_proj.weight', 'model.layers.22.self_attn.v_proj.bias', 'model.layers.22.self_attn.v_proj.weight', 'model.layers.23.input_layernorm.weight', 'model.layers.23.mlp.down_proj.weight', 'model.layers.23.mlp.gate_proj.weight', 'model.layers.23.mlp.up_proj.weight', 'model.layers.23.post_attention_layernorm.weight', 'model.layers.23.self_attn.k_proj.bias', 'model.layers.23.self_attn.k_proj.weight', 'model.layers.23.self_attn.o_proj.weight', 'model.layers.23.self_attn.q_proj.bias', 'model.layers.23.self_attn.q_proj.weight', 'model.layers.23.self_attn.v_proj.bias', 'model.layers.23.self_attn.v_proj.weight', 'model.layers.24.input_layernorm.weight', 'model.layers.24.mlp.down_proj.weight', 'model.layers.24.mlp.gate_proj.weight', 'model.layers.24.mlp.up_proj.weight', 'model.layers.24.post_attention_layernorm.weight', 'model.layers.24.self_attn.k_proj.bias', 'model.layers.24.self_attn.k_proj.weight', 'model.layers.24.self_attn.o_proj.weight', 'model.layers.24.self_attn.q_proj.bias', 'model.layers.24.self_attn.q_proj.weight', 'model.layers.24.self_attn.v_proj.bias', 'model.layers.24.self_attn.v_proj.weight', 'model.layers.25.input_layernorm.weight', 'model.layers.25.mlp.down_proj.weight', 'model.layers.25.mlp.gate_proj.weight', 'model.layers.25.mlp.up_proj.weight', 'model.layers.25.post_attention_layernorm.weight', 'model.layers.25.self_attn.k_proj.bias', 'model.layers.25.self_attn.k_proj.weight', 'model.layers.25.self_attn.o_proj.weight', 'model.layers.25.self_attn.q_proj.bias', 'model.layers.25.self_attn.q_proj.weight', 'model.layers.25.self_attn.v_proj.bias', 'model.layers.25.self_attn.v_proj.weight', 'model.layers.26.input_layernorm.weight', 'model.layers.26.mlp.down_proj.weight', 'model.layers.26.mlp.gate_proj.weight', 'model.layers.26.mlp.up_proj.weight', 'model.layers.26.post_attention_layernorm.weight', 'model.layers.26.self_attn.k_proj.bias', 'model.layers.26.self_attn.k_proj.weight', 'model.layers.26.self_attn.o_proj.weight', 'model.layers.26.self_attn.q_proj.bias', 'model.layers.26.self_attn.q_proj.weight', 'model.layers.26.self_attn.v_proj.bias', 'model.layers.26.self_attn.v_proj.weight', 'model.layers.27.input_layernorm.weight', 'model.layers.27.mlp.down_proj.weight', 'model.layers.27.mlp.gate_proj.weight', 'model.layers.27.mlp.up_proj.weight', 'model.layers.27.post_attention_layernorm.weight', 'model.layers.27.self_attn.k_proj.bias', 'model.layers.27.self_attn.k_proj.weight', 'model.layers.27.self_attn.o_proj.weight', 'model.layers.27.self_attn.q_proj.bias', 'model.layers.27.self_attn.q_proj.weight', 'model.layers.27.self_attn.v_proj.bias', 'model.layers.27.self_attn.v_proj.weight']
- This IS expected if you are initializing Qwen2ForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing Qwen2ForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
[model] trainable=5,452,056,064/5,452,056,064 (100.00%)

Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.16it/s]
Loading checkpoint shards: 100%|██████████| 4/4 [00:04<00:00,  1.01s/it]
Some weights of the model checkpoint at /home/sparrow/NLAmodel/models/qwen2.5-7b-instruct were not used when initializing Qwen2ForCausalLM: ['model.layers.21.input_layernorm.weight', 'model.layers.21.mlp.down_proj.weight', 'model.layers.21.mlp.gate_proj.weight', 'model.layers.21.mlp.up_proj.weight', 'model.layers.21.post_attention_layernorm.weight', 'model.layers.21.self_attn.k_proj.bias', 'model.layers.21.self_attn.k_proj.weight', 'model.layers.21.self_attn.o_proj.weight', 'model.layers.21.self_attn.q_proj.bias', 'model.layers.21.self_attn.q_proj.weight', 'model.layers.21.self_attn.v_proj.bias', 'model.layers.21.self_attn.v_proj.weight', 'model.layers.22.input_layernorm.weight', 'model.layers.22.mlp.down_proj.weight', 'model.layers.22.mlp.gate_proj.weight', 'model.layers.22.mlp.up_proj.weight', 'model.layers.22.post_attention_layernorm.weight', 'model.layers.22.self_attn.k_proj.bias', 'model.layers.22.self_attn.k_proj.weight', 'model.layers.22.self_attn.o_proj.weight', 'model.layers.22.self_attn.q_proj.bias', 'model.layers.22.self_attn.q_proj.weight', 'model.layers.22.self_attn.v_proj.bias', 'model.layers.22.self_attn.v_proj.weight', 'model.layers.23.input_layernorm.weight', 'model.layers.23.mlp.down_proj.weight', 'model.layers.23.mlp.gate_proj.weight', 'model.layers.23.mlp.up_proj.weight', 'model.layers.23.post_attention_layernorm.weight', 'model.layers.23.self_attn.k_proj.bias', 'model.layers.23.self_attn.k_proj.weight', 'model.layers.23.self_attn.o_proj.weight', 'model.layers.23.self_attn.q_proj.bias', 'model.layers.23.self_attn.q_proj.weight', 'model.layers.23.self_attn.v_proj.bias', 'model.layers.23.self_attn.v_proj.weight', 'model.layers.24.input_layernorm.weight', 'model.layers.24.mlp.down_proj.weight', 'model.layers.24.mlp.gate_proj.weight', 'model.layers.24.mlp.up_proj.weight', 'model.layers.24.post_attention_layernorm.weight', 'model.layers.24.self_attn.k_proj.bias', 'model.layers.24.self_attn.k_proj.weight', 'model.layers.24.self_attn.o_proj.weight', 'model.layers.24.self_attn.q_proj.bias', 'model.layers.24.self_attn.q_proj.weight', 'model.layers.24.self_attn.v_proj.bias', 'model.layers.24.self_attn.v_proj.weight', 'model.layers.25.input_layernorm.weight', 'model.layers.25.mlp.down_proj.weight', 'model.layers.25.mlp.gate_proj.weight', 'model.layers.25.mlp.up_proj.weight', 'model.layers.25.post_attention_layernorm.weight', 'model.layers.25.self_attn.k_proj.bias', 'model.layers.25.self_attn.k_proj.weight', 'model.layers.25.self_attn.o_proj.weight', 'model.layers.25.self_attn.q_proj.bias', 'model.layers.25.self_attn.q_proj.weight', 'model.layers.25.self_attn.v_proj.bias', 'model.layers.25.self_attn.v_proj.weight', 'model.layers.26.input_layernorm.weight', 'model.layers.26.mlp.down_proj.weight', 'model.layers.26.mlp.gate_proj.weight', 'model.layers.26.mlp.up_proj.weight', 'model.layers.26.post_attention_layernorm.weight', 'model.layers.26.self_attn.k_proj.bias', 'model.layers.26.self_attn.k_proj.weight', 'model.layers.26.self_attn.o_proj.weight', 'model.layers.26.self_attn.q_proj.bias', 'model.layers.26.self_attn.q_proj.weight', 'model.layers.26.self_attn.v_proj.bias', 'model.layers.26.self_attn.v_proj.weight', 'model.layers.27.input_layernorm.weight', 'model.layers.27.mlp.down_proj.weight', 'model.layers.27.mlp.gate_proj.weight', 'model.layers.27.mlp.up_proj.weight', 'model.layers.27.post_attention_layernorm.weight', 'model.layers.27.self_attn.k_proj.bias', 'model.layers.27.self_attn.k_proj.weight', 'model.layers.27.self_attn.o_proj.weight', 'model.layers.27.self_attn.q_proj.bias', 'model.layers.27.self_attn.q_proj.weight', 'model.layers.27.self_attn.v_proj.bias', 'model.layers.27.self_attn.v_proj.weight']
- This IS expected if you are initializing Qwen2ForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing Qwen2ForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/accelerator.py:1731: UserWarning: Upcasted low precision parameters in NLACriticModel because mixed precision turned on in FSDP. Affects: backbone.model.embed_tokens.weight, value_head.weight.
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/accelerator.py:1731: UserWarning: Upcasted low precision parameters in Qwen2DecoderLayer because mixed precision turned on in FSDP. Affects: self_attn.q_proj.weight, self_attn.q_proj.bias, self_attn.k_proj.weight, self_attn.k_proj.bias, self_attn.v_proj.weight, self_attn.v_proj.bias, self_attn.o_proj.weight, mlp.gate_proj.weight, mlp.up_proj.weight, mlp.down_proj.weight, input_layernorm.weight, post_attention_layernorm.weight.
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/accelerator.py:1737: UserWarning: FSDP upcast of low precision parameters may affect the precision of model checkpoints.
  warnings.warn(
[train] processes=3 micro_batch=8 grad_accum=1 global_batch=24 optimizer_steps=209 resume_step=0
[eval] {"kind": "eval", "optimizer_step": 0, "real_loss": 1.508930926521619, "shuffled_loss": 1.5424310167630513, "conditioning_gap": 0.03350009024143219, "mean_cosine": 0.24553453673919046, "fve_rawvar": -1.1008279928826261, "elapsed_seconds": 0.0}
[train] {"kind": "train", "optimizer_step": 1, "micro_step": 1, "loss": 1.4952398538589478, "mean_cosine": 0.2523800730705261, "fve_rawvar": -1.0670388704586937, "learning_rate": 6.12e-07, "grad_norm": 14.210945129394531, "pred_norm_raw": 100.62603759765625, "gold_norm_raw": 122.20645904541016, "backbone_norm_raw": 100.62603759765625, "elapsed_seconds": 28.106062650680542, "seconds_per_optimizer_step": 28.106062650680542}
[train] {"kind": "train", "optimizer_step": 5, "micro_step": 5, "loss": 1.4200876951217651, "mean_cosine": 0.28995615243911743, "fve_rawvar": -0.9631475563610075, "learning_rate": 3.06e-06, "grad_norm": 4.52089786529541, "pred_norm_raw": 105.12814331054688, "gold_norm_raw": 123.75477600097656, "backbone_norm_raw": 105.24581909179688, "elapsed_seconds": 129.52023839950562, "seconds_per_optimizer_step": 25.904047679901122}
[train] {"kind": "train", "optimizer_step": 10, "micro_step": 10, "loss": 0.9072206020355225, "mean_cosine": 0.5463896989822388, "fve_rawvar": -0.25415346818682605, "learning_rate": 6.12e-06, "grad_norm": 2.1245710849761963, "pred_norm_raw": 102.55419921875, "gold_norm_raw": 123.77955627441406, "backbone_norm_raw": 102.59397888183594, "elapsed_seconds": 256.20221734046936, "seconds_per_optimizer_step": 25.620221734046936}
[train] {"kind": "train", "optimizer_step": 15, "micro_step": 15, "loss": 0.8584443926811218, "mean_cosine": 0.5707778036594391, "fve_rawvar": -0.1867246069048234, "learning_rate": 6.1114248438932226e-06, "grad_norm": 2.5222790241241455, "pred_norm_raw": 101.62621307373047, "gold_norm_raw": 124.57113647460938, "backbone_norm_raw": 101.65703582763672, "elapsed_seconds": 382.40016913414, "seconds_per_optimizer_step": 25.493344608942667}
[train] {"kind": "train", "optimizer_step": 20, "micro_step": 20, "loss": 0.6351891756057739, "mean_cosine": 0.682405412197113, "fve_rawvar": 0.12190628635049428, "learning_rate": 6.085752776663853e-06, "grad_norm": 2.2089600563049316, "pred_norm_raw": 98.62138366699219, "gold_norm_raw": 118.05543518066406, "backbone_norm_raw": 98.61036682128906, "elapsed_seconds": 508.87994742393494, "seconds_per_optimizer_step": 25.443997371196748}
[train] {"kind": "train", "optimizer_step": 25, "micro_step": 25, "loss": 0.6475567817687988, "mean_cosine": 0.6762216091156006, "fve_rawvar": 0.10480914798334862, "learning_rate": 6.043143669033803e-06, "grad_norm": 2.8715624809265137, "pred_norm_raw": 100.26058197021484, "gold_norm_raw": 121.51753234863281, "backbone_norm_raw": 100.22938537597656, "elapsed_seconds": 635.683589220047, "seconds_per_optimizer_step": 25.42734356880188}
[eval] {"kind": "eval", "optimizer_step": 25, "real_loss": 0.7131928747726811, "shuffled_loss": 1.2179760055409536, "conditioning_gap": 0.5047831307682725, "mean_cosine": 0.6434035626136594, "fve_rawvar": 0.0070482821233267545, "eval_seconds": 35.44353628158569, "elapsed_seconds": 671.1334578990936}
[train] {"kind": "train", "optimizer_step": 30, "micro_step": 30, "loss": 0.5827499628067017, "mean_cosine": 0.7086250185966492, "fve_rawvar": 0.1943989308664842, "learning_rate": 5.983862865773954e-06, "grad_norm": 1.825636625289917, "pred_norm_raw": 106.53228759765625, "gold_norm_raw": 121.80442810058594, "backbone_norm_raw": 106.56011199951172, "elapsed_seconds": 797.0864942073822, "seconds_per_optimizer_step": 26.56954980691274}
[train] {"kind": "train", "optimizer_step": 35, "micro_step": 35, "loss": 0.5988593101501465, "mean_cosine": 0.7005703449249268, "fve_rawvar": 0.17212916120332067, "learning_rate": 5.908279533291049e-06, "grad_norm": 1.9720251560211182, "pred_norm_raw": 107.78157043457031, "gold_norm_raw": 117.61307525634766, "backbone_norm_raw": 107.56428527832031, "elapsed_seconds": 923.3976368904114, "seconds_per_optimizer_step": 26.382789625440324}
[train] {"kind": "train", "optimizer_step": 40, "micro_step": 40, "loss": 0.6944954991340637, "mean_cosine": 0.6527522504329681, "fve_rawvar": 0.03992045934046973, "learning_rate": 5.816864360673766e-06, "grad_norm": 2.5259900093078613, "pred_norm_raw": 102.67643737792969, "gold_norm_raw": 130.64260864257812, "backbone_norm_raw": 102.68668365478516, "elapsed_seconds": 1049.7471342086792, "seconds_per_optimizer_step": 26.24367835521698}
[train] {"kind": "train", "optimizer_step": 45, "micro_step": 45, "loss": 0.5395911335945129, "mean_cosine": 0.7302044332027435, "fve_rawvar": 0.25406225334604793, "learning_rate": 5.710186628514498e-06, "grad_norm": 1.6946038007736206, "pred_norm_raw": 101.68635559082031, "gold_norm_raw": 113.21736145019531, "backbone_norm_raw": 101.67901611328125, "elapsed_seconds": 1175.8611323833466, "seconds_per_optimizer_step": 26.130247386296592}
[train] {"kind": "train", "optimizer_step": 50, "micro_step": 50, "loss": 0.6325961351394653, "mean_cosine": 0.6837019324302673, "fve_rawvar": 0.12549093895502395, "learning_rate": 5.588910663760508e-06, "grad_norm": 1.7780529260635376, "pred_norm_raw": 106.066162109375, "gold_norm_raw": 113.57713317871094, "backbone_norm_raw": 105.71068572998047, "elapsed_seconds": 1302.1018648147583, "seconds_per_optimizer_step": 26.042037296295167}
[eval] {"kind": "eval", "optimizer_step": 50, "real_loss": 0.6258179141829411, "shuffled_loss": 1.239109353058868, "conditioning_gap": 0.6132914388759269, "mean_cosine": 0.6870910429085295, "fve_rawvar": 0.1286971660169609, "eval_seconds": 35.45733118057251, "elapsed_seconds": 1337.5682740211487}
[train] {"kind": "train", "optimizer_step": 55, "micro_step": 55, "loss": 0.6949566602706909, "mean_cosine": 0.6525216698646545, "fve_rawvar": 0.0392829442337842, "learning_rate": 5.45379170267161e-06, "grad_norm": 1.728350281715393, "pred_norm_raw": 101.213623046875, "gold_norm_raw": 118.63198852539062, "backbone_norm_raw": 101.1964111328125, "elapsed_seconds": 1463.3859331607819, "seconds_per_optimizer_step": 26.60701696655967}
[train] {"kind": "train", "optimizer_step": 60, "micro_step": 60, "loss": 0.5692216157913208, "mean_cosine": 0.7153891921043396, "fve_rawvar": 0.21310069236760854, "learning_rate": 5.305671187647459e-06, "grad_norm": 1.9958264827728271, "pred_norm_raw": 102.2381591796875, "gold_norm_raw": 121.15351104736328, "backbone_norm_raw": 102.23800659179688, "elapsed_seconds": 1589.4732682704926, "seconds_per_optimizer_step": 26.491221137841542}
[train] {"kind": "train", "optimizer_step": 65, "micro_step": 65, "loss": 0.5442827939987183, "mean_cosine": 0.7278586030006409, "fve_rawvar": 0.2475764414560987, "learning_rate": 5.145471527213072e-06, "grad_norm": 2.230109691619873, "pred_norm_raw": 108.10166931152344, "gold_norm_raw": 118.75129699707031, "backbone_norm_raw": 107.72930908203125, "elapsed_seconds": 1716.1579852104187, "seconds_per_optimizer_step": 26.40243054169875}
[train] {"kind": "train", "optimizer_step": 70, "micro_step": 70, "loss": 0.43588146567344666, "mean_cosine": 0.7820592671632767, "fve_rawvar": 0.3974318366820936, "learning_rate": 4.974190351794358e-06, "grad_norm": 1.7069239616394043, "pred_norm_raw": 117.06438446044922, "gold_norm_raw": 122.26090240478516, "backbone_norm_raw": 116.83784484863281, "elapsed_seconds": 1842.2789952754974, "seconds_per_optimizer_step": 26.318271361078533}
[train] {"kind": "train", "optimizer_step": 75, "micro_step": 75, "loss": 0.4078643321990967, "mean_cosine": 0.7960678339004517, "fve_rawvar": 0.4361630835658955, "learning_rate": 4.792894301055345e-06, "grad_norm": 1.6535227298736572, "pred_norm_raw": 110.75369262695312, "gold_norm_raw": 112.76079559326172, "backbone_norm_raw": 110.29114532470703, "elapsed_seconds": 1968.3703999519348, "seconds_per_optimizer_step": 26.2449386660258}
[eval] {"kind": "eval", "optimizer_step": 75, "real_loss": 0.5646483074459765, "shuffled_loss": 1.2704381371537845, "conditioning_gap": 0.705789829707808, "mean_cosine": 0.7176758462770118, "fve_rawvar": 0.2138613175947075, "eval_seconds": 35.44633221626282, "elapsed_seconds": 2003.82954621315}
[train] {"kind": "train", "optimizer_step": 80, "micro_step": 80, "loss": 0.4851824939250946, "mean_cosine": 0.7574087530374527, "fve_rawvar": 0.32927745898360294, "learning_rate": 4.602712381485962e-06, "grad_norm": 2.2523763179779053, "pred_norm_raw": 110.86607360839844, "gold_norm_raw": 123.83141326904297, "backbone_norm_raw": 110.58136749267578, "elapsed_seconds": 2129.601671934128, "seconds_per_optimizer_step": 26.6200208991766}
[train] {"kind": "train", "optimizer_step": 85, "micro_step": 85, "loss": 0.5241831541061401, "mean_cosine": 0.7379084229469299, "fve_rawvar": 0.27536244303501356, "learning_rate": 4.404828935605448e-06, "grad_norm": 1.837619662284851, "pred_norm_raw": 111.11434936523438, "gold_norm_raw": 116.13356018066406, "backbone_norm_raw": 110.44722747802734, "elapsed_seconds": 2255.9184234142303, "seconds_per_optimizer_step": 26.54021674604977}
[train] {"kind": "train", "optimizer_step": 90, "micro_step": 90, "loss": 0.5075557231903076, "mean_cosine": 0.7462221384048462, "fve_rawvar": 0.2983484181146576, "learning_rate": 4.200476266565127e-06, "grad_norm": 2.1529414653778076, "pred_norm_raw": 112.21126556396484, "gold_norm_raw": 126.328857421875, "backbone_norm_raw": 112.08445739746094, "elapsed_seconds": 2382.472135782242, "seconds_per_optimizer_step": 26.471912619802687}
[train] {"kind": "train", "optimizer_step": 95, "micro_step": 95, "loss": 0.5879204869270325, "mean_cosine": 0.7060397565364838, "fve_rawvar": 0.18725113159549422, "learning_rate": 3.990926964080219e-06, "grad_norm": 1.9986908435821533, "pred_norm_raw": 107.85401153564453, "gold_norm_raw": 124.27362060546875, "backbone_norm_raw": 107.653076171875, "elapsed_seconds": 2508.7875847816467, "seconds_per_optimizer_step": 26.408290366122596}
[train] {"kind": "train", "optimizer_step": 100, "micro_step": 100, "loss": 0.4461316168308258, "mean_cosine": 0.7769341915845871, "fve_rawvar": 0.38326189544109546, "learning_rate": 3.777485979480335e-06, "grad_norm": 1.7678033113479614, "pred_norm_raw": 109.1651840209961, "gold_norm_raw": 119.0023422241211, "backbone_norm_raw": 108.83531188964844, "elapsed_seconds": 2635.1417429447174, "seconds_per_optimizer_step": 26.351417429447174}
[eval] {"kind": "eval", "optimizer_step": 100, "real_loss": 0.5208715121779177, "shuffled_loss": 1.2708840957946248, "conditioning_gap": 0.7500125836167071, "mean_cosine": 0.7395642439110411, "fve_rawvar": 0.27481010943227213, "eval_seconds": 35.382399797439575, "elapsed_seconds": 2670.5335142612457}
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:863: UserWarning: `_get_pg_default_device` will be deprecated, it only stays for backward-compatiblity reason. If you need to find a device for object collectives, please use `_get_object_coll_device`. If you need to query the device types supported by group, please use `_device_capability(group)`. 
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:863: UserWarning: `_get_pg_default_device` will be deprecated, it only stays for backward-compatiblity reason. If you need to find a device for object collectives, please use `_get_object_coll_device`. If you need to query the device types supported by group, please use `_device_capability(group)`. 
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:863: UserWarning: `_get_pg_default_device` will be deprecated, it only stays for backward-compatiblity reason. If you need to find a device for object collectives, please use `_get_object_coll_device`. If you need to query the device types supported by group, please use `_device_capability(group)`. 
  warnings.warn(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:732: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  local_shape = tensor.shape
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:749: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.shape,
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_state_dict_utils.py:751: FutureWarning: Please use DTensor instead and we are deprecating ShardedTensor.
  tensor.dtype,
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/utils/fsdp_utils.py:120: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/utils/fsdp_utils.py:120: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/utils/fsdp_utils.py:120: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/utils/fsdp_utils.py:236: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/utils/fsdp_utils.py:236: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
/home/sparrow/NLAmodel/nla-student/.venv/lib/python3.12/site-packages/accelerate/utils/fsdp_utils.py:236: FutureWarning: `save_state_dict` is deprecated and will be removed in future versions.Please use `save` instead.
  dist_cp.save_state_dict(
[checkpoint] {"kind": "checkpoint", "optimizer_step": 100, "path": "/home/sparrow/NLAmodel/experiments/Exp18_affect_ar_sft/runs/ar5000_3gpu/checkpoints/step_000100", "elapsed_seconds": 2670.5439023971558}
[train] {"kind": "train", "optimizer_step": 105, "micro_step": 105, "loss": 0.5158400535583496, "mean_cosine": 0.7420799732208252, "fve_rawvar": 0.28689605290230047, "learning_rate": 3.561482499230636e-06, "grad_norm": 1.746726155281067, "pred_norm_raw": 115.87013244628906, "gold_norm_raw": 127.74365234375, "backbone_norm_raw": 115.547119140625, "elapsed_seconds": 3049.0230660438538, "seconds_per_optimizer_step": 29.03831491470337}
[train] {"kind": "train", "optimizer_step": 110, "micro_step": 110, "loss": 0.5613721013069153, "mean_cosine": 0.7193139493465424, "fve_rawvar": 0.22395196249100724, "learning_rate": 3.34426166753065e-06, "grad_norm": 1.7786778211593628, "pred_norm_raw": 113.33716583251953, "gold_norm_raw": 124.57211303710938, "backbone_norm_raw": 113.18638610839844, "elapsed_seconds": 3175.609519958496, "seconds_per_optimizer_step": 28.869177454168145}
[train] {"kind": "train", "optimizer_step": 115, "micro_step": 115, "loss": 0.5783724784851074, "mean_cosine": 0.7108137607574463, "fve_rawvar": 0.20045042168530258, "learning_rate": 3.1271762095375816e-06, "grad_norm": 1.9016059637069702, "pred_norm_raw": 111.05543518066406, "gold_norm_raw": 125.66304016113281, "backbone_norm_raw": 110.92030334472656, "elapsed_seconds": 3302.5335314273834, "seconds_per_optimizer_step": 28.717682881977247}
[train] {"kind": "train", "optimizer_step": 120, "micro_step": 120, "loss": 0.5028859376907349, "mean_cosine": 0.7485570311546326, "fve_rawvar": 0.30480398985414114, "learning_rate": 2.911578007379827e-06, "grad_norm": 1.7510663270950317, "pred_norm_raw": 111.37747192382812, "gold_norm_raw": 123.0380630493164, "backbone_norm_raw": 111.02338409423828, "elapsed_seconds": 3429.385910511017, "seconds_per_optimizer_step": 28.57821592092514}
[train] {"kind": "train", "optimizer_step": 125, "micro_step": 125, "loss": 0.47612640261650085, "mean_cosine": 0.7619367986917496, "fve_rawvar": 0.34179671647996734, "learning_rate": 2.6988096814203814e-06, "grad_norm": 1.409972071647644, "pred_norm_raw": 114.43336486816406, "gold_norm_raw": 124.53921508789062, "backbone_norm_raw": 114.29777526855469, "elapsed_seconds": 3556.2924370765686, "seconds_per_optimizer_step": 28.45033949661255}
[eval] {"kind": "eval", "optimizer_step": 125, "real_loss": 0.49614588026371265, "shuffled_loss": 1.2810611294375525, "conditioning_gap": 0.7849152491738398, "mean_cosine": 0.7519270598681437, "fve_rawvar": 0.30923468033480894, "eval_seconds": 37.36824655532837, "elapsed_seconds": 3593.669800758362}
[train] {"kind": "train", "optimizer_step": 130, "micro_step": 130, "loss": 0.5184783339500427, "mean_cosine": 0.7407608330249786, "fve_rawvar": 0.2832488600410854, "learning_rate": 2.490196229197138e-06, "grad_norm": 1.6581908464431763, "pred_norm_raw": 116.87257385253906, "gold_norm_raw": 119.44444274902344, "backbone_norm_raw": 116.7265625, "elapsed_seconds": 3718.713844537735, "seconds_per_optimizer_step": 28.60549111182873}
[train] {"kind": "train", "optimizer_step": 135, "micro_step": 135, "loss": 0.45259708166122437, "mean_cosine": 0.7737014591693878, "fve_rawvar": 0.3743239534209397, "learning_rate": 2.28703677410791e-06, "grad_norm": 1.532267451286316, "pred_norm_raw": 114.2972412109375, "gold_norm_raw": 120.1422119140625, "backbone_norm_raw": 113.94527435302734, "elapsed_seconds": 3845.5616631507874, "seconds_per_optimizer_step": 28.48564194926509}
[train] {"kind": "train", "optimizer_step": 140, "micro_step": 140, "loss": 0.5177595615386963, "mean_cosine": 0.7411202192306519, "fve_rawvar": 0.28424250029848763, "learning_rate": 2.0905964752245624e-06, "grad_norm": 1.43108332157135, "pred_norm_raw": 114.74915313720703, "gold_norm_raw": 122.09115600585938, "backbone_norm_raw": 114.13300323486328, "elapsed_seconds": 3972.7263493537903, "seconds_per_optimizer_step": 28.3766167810985}
[train] {"kind": "train", "optimizer_step": 145, "micro_step": 145, "loss": 0.534915566444397, "mean_cosine": 0.7325422167778015, "fve_rawvar": 0.2605258177138562, "learning_rate": 1.902098648617238e-06, "grad_norm": 1.5992106199264526, "pred_norm_raw": 117.32339477539062, "gold_norm_raw": 116.04643249511719, "backbone_norm_raw": 116.98158264160156, "elapsed_seconds": 4099.457980871201, "seconds_per_optimizer_step": 28.27212400600828}
[train] {"kind": "train", "optimizer_step": 150, "micro_step": 150, "loss": 0.44754713773727417, "mean_cosine": 0.7762264311313629, "fve_rawvar": 0.3813050610723322, "learning_rate": 1.7227171492524907e-06, "grad_norm": 2.1677300930023193, "pred_norm_raw": 116.72453308105469, "gold_norm_raw": 121.63236999511719, "backbone_norm_raw": 116.1937255859375, "elapsed_seconds": 4226.690768003464, "seconds_per_optimizer_step": 28.177938453356425}
[eval] {"kind": "eval", "optimizer_step": 150, "real_loss": 0.48247416213982636, "shuffled_loss": 1.2879411060776975, "conditioning_gap": 0.8054669439378712, "mean_cosine": 0.7587629189300868, "fve_rawvar": 0.32826930123138676, "eval_seconds": 37.47536516189575, "elapsed_seconds": 4264.189164876938}
[train] {"kind": "train", "optimizer_step": 155, "micro_step": 155, "loss": 0.5776604413986206, "mean_cosine": 0.7111697793006897, "fve_rawvar": 0.20143475094269758, "learning_rate": 1.5535690609064447e-06, "grad_norm": 1.480932354927063, "pred_norm_raw": 111.50988006591797, "gold_norm_raw": 115.45160675048828, "backbone_norm_raw": 111.07354736328125, "elapsed_seconds": 4388.809265613556, "seconds_per_optimizer_step": 28.314898487829392}
[train] {"kind": "train", "optimizer_step": 160, "micro_step": 160, "loss": 0.42758965492248535, "mean_cosine": 0.7862051725387573, "fve_rawvar": 0.40889454287233473, "learning_rate": 1.3957077396159398e-06, "grad_norm": 1.9220153093338013, "pred_norm_raw": 109.28187561035156, "gold_norm_raw": 122.62353515625, "backbone_norm_raw": 108.90708923339844, "elapsed_seconds": 4515.8059141635895, "seconds_per_optimizer_step": 28.223786963522436}
[train] {"kind": "train", "optimizer_step": 165, "micro_step": 165, "loss": 0.5307600498199463, "mean_cosine": 0.7346199750900269, "fve_rawvar": 0.2662704575235888, "learning_rate": 1.2501162539890288e-06, "grad_norm": 1.8566343784332275, "pred_norm_raw": 113.77515411376953, "gold_norm_raw": 121.44596099853516, "backbone_norm_raw": 113.38858032226562, "elapsed_seconds": 4642.884285211563, "seconds_per_optimizer_step": 28.138692637645836}
[train] {"kind": "train", "optimizer_step": 170, "micro_step": 170, "loss": 0.394478440284729, "mean_cosine": 0.8027607798576355, "fve_rawvar": 0.45466791329695666, "learning_rate": 1.117701263224745e-06, "grad_norm": 1.8281002044677734, "pred_norm_raw": 113.28242492675781, "gold_norm_raw": 119.13021850585938, "backbone_norm_raw": 112.62519836425781, "elapsed_seconds": 4769.742715835571, "seconds_per_optimizer_step": 28.05731009315042}
[train] {"kind": "train", "optimizer_step": 175, "micro_step": 175, "loss": 0.5382561087608337, "mean_cosine": 0.7308719456195831, "fve_rawvar": 0.255907808904991, "learning_rate": 9.992873709662844e-07, "grad_norm": 2.454197406768799, "pred_norm_raw": 116.00094604492188, "gold_norm_raw": 124.62702941894531, "backbone_norm_raw": 115.61546325683594, "elapsed_seconds": 4896.989147901535, "seconds_per_optimizer_step": 27.982795130865913}
[eval] {"kind": "eval", "optimizer_step": 175, "real_loss": 0.4776784409251478, "shuffled_loss": 1.2799315427740414, "conditioning_gap": 0.8022531018488936, "mean_cosine": 0.7611607795374261, "fve_rawvar": 0.3349462041941239, "eval_seconds": 37.54615116119385, "elapsed_seconds": 4934.543691635132}
[train] {"kind": "train", "optimizer_step": 180, "micro_step": 180, "loss": 0.5236002206802368, "mean_cosine": 0.7381998896598816, "fve_rawvar": 0.27616829772589946, "learning_rate": 8.956119901485255e-07, "grad_norm": 1.9374598264694214, "pred_norm_raw": 112.51471710205078, "gold_norm_raw": 118.78627014160156, "backbone_norm_raw": 112.04627990722656, "elapsed_seconds": 5059.1854610443115, "seconds_per_optimizer_step": 28.10658589469062}
[train] {"kind": "train", "optimizer_step": 185, "micro_step": 185, "loss": 0.575512170791626, "mean_cosine": 0.712243914604187, "fve_rawvar": 0.20440454795383356, "learning_rate": 8.0732075081863e-07, "grad_norm": 2.920722246170044, "pred_norm_raw": 108.95732879638672, "gold_norm_raw": 114.37135314941406, "backbone_norm_raw": 108.64566040039062, "elapsed_seconds": 5186.225680112839, "seconds_per_optimizer_step": 28.033652324934263}
[train] {"kind": "train", "optimizer_step": 190, "micro_step": 190, "loss": 0.39391031861305237, "mean_cosine": 0.8030448406934738, "fve_rawvar": 0.45545329202764984, "learning_rate": 7.349634795271535e-07, "grad_norm": 2.080827474594116, "pred_norm_raw": 115.29196166992188, "gold_norm_raw": 122.74758911132812, "backbone_norm_raw": 115.1142578125, "elapsed_seconds": 5312.978806257248, "seconds_per_optimizer_step": 27.963046348722358}
[train] {"kind": "train", "optimizer_step": 195, "micro_step": 195, "loss": 0.5089956521987915, "mean_cosine": 0.7455021739006042, "fve_rawvar": 0.29635784167064716, "learning_rate": 6.789907753276795e-07, "grad_norm": 2.058220386505127, "pred_norm_raw": 113.27311706542969, "gold_norm_raw": 121.06358337402344, "backbone_norm_raw": 113.11637878417969, "elapsed_seconds": 5439.844558954239, "seconds_per_optimizer_step": 27.89663876386789}
[train] {"kind": "train", "optimizer_step": 200, "micro_step": 200, "loss": 0.5677552223205566, "mean_cosine": 0.7161223888397217, "fve_rawvar": 0.21512785362580644, "learning_rate": 6.397512037076657e-07, "grad_norm": 2.096141815185547, "pred_norm_raw": 106.8187026977539, "gold_norm_raw": 113.22012329101562, "backbone_norm_raw": 106.4495849609375, "elapsed_seconds": 5566.681344985962, "seconds_per_optimizer_step": 27.83340672492981}
[eval] {"kind": "eval", "optimizer_step": 200, "real_loss": 0.4746351122028298, "shuffled_loss": 1.2890390286015139, "conditioning_gap": 0.814403916398684, "mean_cosine": 0.7626824438985851, "fve_rawvar": 0.33918331674779634, "eval_seconds": 37.57524752616882, "elapsed_seconds": 5604.265249967575}
[checkpoint] {"kind": "checkpoint", "optimizer_step": 200, "path": "/home/sparrow/NLAmodel/experiments/Exp18_affect_ar_sft/runs/ar5000_3gpu/checkpoints/step_000200", "elapsed_seconds": 5604.273499727249}
[train] {"kind": "train", "optimizer_step": 205, "micro_step": 205, "loss": 0.4771572947502136, "mean_cosine": 0.7614213526248932, "fve_rawvar": 0.34037159789037463, "learning_rate": 6.174891259250671e-07, "grad_norm": 2.4741575717926025, "pred_norm_raw": 119.78939819335938, "gold_norm_raw": 127.46685028076172, "backbone_norm_raw": 119.6037368774414, "elapsed_seconds": 5974.862098932266, "seconds_per_optimizer_step": 29.145668775279347}
[eval] {"kind": "eval", "optimizer_step": 209, "real_loss": 0.47344916748503846, "shuffled_loss": 1.2868549211157694, "conditioning_gap": 0.8134057536307309, "mean_cosine": 0.7632754162574807, "fve_rawvar": 0.3408344631438017, "eval_seconds": 37.785274267196655, "elapsed_seconds": 6114.371411561966}
[checkpoint] {"kind": "checkpoint", "optimizer_step": 209, "path": "/home/sparrow/NLAmodel/experiments/Exp18_affect_ar_sft/runs/ar5000_3gpu/checkpoints/step_000209", "elapsed_seconds": 6114.380260467529}
[done] {"status": "complete", "optimizer_steps": 209, "micro_steps": 209, "elapsed_seconds": 6436.945361614227, "initial_eval_real": 1.508930926521619, "initial_eval_shuffled": 1.5424310167630513, "final_eval_real": 0.47344916748503846, "final_eval_shuffled": 1.2868549211157694, "final_fve_rawvar": 0.3408344631438017, "checkpoint": "/home/sparrow/NLAmodel/experiments/Exp18_affect_ar_sft/runs/ar5000_3gpu/checkpoint_final"}
[rank0]:[W801 05:27:02.678921021 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())