| [env] device=cuda dtype=torch.float16 (arg=auto) |
| [data] out_2b/data_agent/mix_r11.jsonl: 3948 条 |
| [tokenizer] out_2b/pnet2b_moe |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
|
Loading weights: 0%| | 0/577 [00:00<?, ?it/s]
Loading weights: 20%|█▉ | 113/577 [00:00<00:00, 782.09it/s]
Loading weights: 33%|███▎ | 192/577 [00:00<00:00, 723.24it/s]
Loading weights: 46%|████▌ | 264/577 [00:00<00:00, 544.46it/s]
Loading weights: 56%|█████▌ | 322/577 [00:00<00:00, 412.96it/s]
Loading weights: 68%|██████▊ | 394/577 [00:00<00:00, 477.07it/s]
Loading weights: 77%|███████▋ | 447/577 [00:00<00:00, 468.15it/s]
Loading weights: 86%|████████▌ | 497/577 [00:01<00:00, 456.13it/s]
Loading weights: 94%|█████████▍| 545/577 [00:01<00:00, 438.26it/s]
Loading weights: 100%|██████████| 577/577 [00:01<00:00, 476.23it/s] |
| [2B适配] 加载后参数 dtype 集合: {'torch.float16'} (期望仅 torch.float16) |
| [2B适配] 参数总显存占用: 5.08 GB |
| [round5-B] per-step CE 启用: weight=0.5 |
| [resume] head+extras 已载入: out_2b/pnet2b_moe |
| Traceback (most recent call last): |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 317, in _get_peft_type |
| config_file = hf_hub_download( |
| model_id, |
| CONFIG_NAME, |
| **hf_hub_download_kwargs, |
| ) |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 84, in _inner_fn |
| validate_repo_id(arg_value) |
| ~~~~~~~~~~~~~~~~^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 132, in validate_repo_id |
| raise HFValidationError( |
| ...<2 lines>... |
| ) |
| huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'out_2b/pnet2b_moe/adapter'. Use `repo_type` argument if needed. |
|
|
| During handling of the above exception, another exception occurred: |
|
|
| Traceback (most recent call last): |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 427, in <module> |
| main() |
| ~~~~^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 344, in main |
| model, groups = setup_trainables( |
| ~~~~~~~~~~~~~~~~^ |
| model, lora_r=args.lora, lr=args.lr, lr_lora=args.lr_lora, |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| pretrained_adapter_dir=_ad_dir) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 69, in setup_trainables |
| model = PeftModel.from_pretrained( |
| model, pretrained_adapter_dir, is_trainable=True) |
| File "/usr/local/lib/python3.13/dist-packages/peft/peft_model.py", line 510, in from_pretrained |
| config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig._get_peft_type(model_id, **hf_kwargs)].from_pretrained( |
| ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 323, in _get_peft_type |
| raise ValueError(f"Can't find '{CONFIG_NAME}' at '{model_id}'") |
| ValueError: Can't find 'adapter_config.json' at 'out_2b/pnet2b_moe/adapter' |
| [env] device=cuda dtype=torch.float16 (arg=auto) |
| [data] out_2b/data_agent/mix_r11.jsonl: 3948 条 |
| [tokenizer] out_2b/pnet2b_moe |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
|
Loading weights: 0%| | 0/577 [00:00<?, ?it/s]
Loading weights: 92%|█████████▏| 529/577 [00:00<00:00, 5289.07it/s]
Loading weights: 100%|██████████| 577/577 [00:00<00:00, 5300.74it/s] |
| [2B适配] 加载后参数 dtype 集合: {'torch.float16'} (期望仅 torch.float16) |
| [2B适配] 参数总显存占用: 5.08 GB |
| [round5-B] per-step CE 启用: weight=0.5 |
| [resume] head+extras 已载入: out_2b/pnet2b_moe |
| Traceback (most recent call last): |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 317, in _get_peft_type |
| config_file = hf_hub_download( |
| model_id, |
| CONFIG_NAME, |
| **hf_hub_download_kwargs, |
| ) |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 84, in _inner_fn |
| validate_repo_id(arg_value) |
| ~~~~~~~~~~~~~~~~^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 132, in validate_repo_id |
| raise HFValidationError( |
| ...<2 lines>... |
| ) |
| huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'out_2b/pnet2b_moe/adapter'. Use `repo_type` argument if needed. |
| |
| During handling of the above exception, another exception occurred: |
| |
| Traceback (most recent call last): |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 427, in <module> |
| main() |
| ~~~~^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 344, in main |
| model, groups = setup_trainables( |
| ~~~~~~~~~~~~~~~~^ |
| model, lora_r=args.lora, lr=args.lr, lr_lora=args.lr_lora, |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| pretrained_adapter_dir=_ad_dir) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 69, in setup_trainables |
| model = PeftModel.from_pretrained( |
| model, pretrained_adapter_dir, is_trainable=True) |
| File "/usr/local/lib/python3.13/dist-packages/peft/peft_model.py", line 510, in from_pretrained |
| config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig._get_peft_type(model_id, **hf_kwargs)].from_pretrained( |
| ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 323, in _get_peft_type |
| raise ValueError(f"Can't find '{CONFIG_NAME}' at '{model_id}'") |
| ValueError: Can't find 'adapter_config.json' at 'out_2b/pnet2b_moe/adapter' |
| [env] device=cuda dtype=torch.float16 (arg=auto) |
| [data] out_2b/data_agent/mix_r11.jsonl: 3948 条 |
| [tokenizer] out_2b/pnet2b_moe |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
|
Loading weights: 0%| | 0/577 [00:00<?, ?it/s]
Loading weights: 78%|███████▊ | 448/577 [00:00<00:00, 4473.22it/s]
Loading weights: 100%|██████████| 577/577 [00:00<00:00, 4398.28it/s] |
| [2B适配] 加载后参数 dtype 集合: {'torch.float16'} (期望仅 torch.float16) |
| [2B适配] 参数总显存占用: 5.08 GB |
| [round5-B] per-step CE 启用: weight=0.5 |
| [resume] head+extras 已载入: out_2b/ckpt_2b_r8 |
| [resume] LoRA 随载入续训: out_2b/ckpt_2b_r8/adapter |
| [fix] peft 注入后重新开放 Router/Halt/MTP 参数张量: 28 |
| [trainables] router 张量=16 mtp 张量=10 head 张量=2 lora 张量=64 |
| [env] device=cuda dtype=torch.float16 (arg=auto) |
| [data] out_2b/data_agent/mix_r11.jsonl: 3948 条 |
| [tokenizer] out_2b/pnet2b_moe |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [2B适配] Router/MTP/LoRA 参数张量提升 fp32: 90 |
| [model] 可训参数: 26,949,697 |
|
Loading weights: 0%| | 0/577 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 577/577 [00:00<00:00, 14243.92it/s] |
| [2B适配] 加载后参数 dtype 集合: {'torch.float16'} (期望仅 torch.float16) |
| [2B适配] 参数总显存占用: 5.08 GB |
| [round5-B] per-step CE 启用: weight=0.5 |
| [resume] head+extras 已载入: out_2b/pnet2b_moe |
| Traceback (most recent call last): |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 317, in _get_peft_type |
| config_file = hf_hub_download( |
| model_id, |
| CONFIG_NAME, |
| **hf_hub_download_kwargs, |
| ) |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 84, in _inner_fn |
| validate_repo_id(arg_value) |
| ~~~~~~~~~~~~~~~~^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 132, in validate_repo_id |
| raise HFValidationError( |
| ...<2 lines>... |
| ) |
| huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'out_2b/pnet2b_moe/adapter'. Use `repo_type` argument if needed. |
| |
| During handling of the above exception, another exception occurred: |
| |
| Traceback (most recent call last): |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 427, in <module> |
| main() |
| ~~~~^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 344, in main |
| model, groups = setup_trainables( |
| ~~~~~~~~~~~~~~~~^ |
| model, lora_r=args.lora, lr=args.lr, lr_lora=args.lr_lora, |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| pretrained_adapter_dir=_ad_dir) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 69, in setup_trainables |
| model = PeftModel.from_pretrained( |
| model, pretrained_adapter_dir, is_trainable=True) |
| File "/usr/local/lib/python3.13/dist-packages/peft/peft_model.py", line 510, in from_pretrained |
| config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig._get_peft_type(model_id, **hf_kwargs)].from_pretrained( |
| ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 323, in _get_peft_type |
| raise ValueError(f"Can't find '{CONFIG_NAME}' at '{model_id}'") |
| ValueError: Can't find 'adapter_config.json' at 'out_2b/pnet2b_moe/adapter' |
| [prior] {'easy': 0.7, 'medium': 0.4, 'hard': 0.15} (带难度样本 3948/3948) |
| ep1 it5 loss=9.6604 kl=0.376 aux=4.630 mtp=6.0351 sce=2.456 active=1.86 steps=2.77 (10s) |
| ep1 it10 loss=10.2325 kl=0.512 aux=4.692 mtp=6.5389 sce=2.499 active=1.82 steps=2.52 (19s) |
| [env] device=cuda dtype=torch.float16 (arg=auto) |
| [data] out_2b/data_agent/mix_r11.jsonl: 3948 条 |
| ep1 it15 loss=10.3225 kl=0.500 aux=4.817 mtp=6.6392 sce=2.487 active=1.81 steps=2.39 (28s) |
| [tokenizer] out_2b/pnet2b_moe |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
|
Loading weights: 0%| | 0/577 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 577/577 [00:00<00:00, 12339.33it/s] |
| [2B适配] 加载后参数 dtype 集合: {'torch.float16'} (期望仅 torch.float16) |
| [2B适配] 参数总显存占用: 5.08 GB |
| [round5-B] per-step CE 启用: weight=0.5 |
| [resume] head+extras 已载入: out_2b/pnet2b_moe |
| Traceback (most recent call last): |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 317, in _get_peft_type |
| config_file = hf_hub_download( |
| model_id, |
| CONFIG_NAME, |
| **hf_hub_download_kwargs, |
| ) |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 84, in _inner_fn |
| validate_repo_id(arg_value) |
| ~~~~~~~~~~~~~~~~^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 132, in validate_repo_id |
| raise HFValidationError( |
| ...<2 lines>... |
| ) |
| huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'out_2b/pnet2b_moe/adapter'. Use `repo_type` argument if needed. |
|
|
| During handling of the above exception, another exception occurred: |
|
|
| Traceback (most recent call last): |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 427, in <module> |
| main() |
| ~~~~^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 344, in main |
| model, groups = setup_trainables( |
| ~~~~~~~~~~~~~~~~^ |
| model, lora_r=args.lora, lr=args.lr, lr_lora=args.lr_lora, |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| pretrained_adapter_dir=_ad_dir) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 69, in setup_trainables |
| model = PeftModel.from_pretrained( |
| model, pretrained_adapter_dir, is_trainable=True) |
| File "/usr/local/lib/python3.13/dist-packages/peft/peft_model.py", line 510, in from_pretrained |
| config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig._get_peft_type(model_id, **hf_kwargs)].from_pretrained( |
| ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 323, in _get_peft_type |
| raise ValueError(f"Can't find '{CONFIG_NAME}' at '{model_id}'") |
| ValueError: Can't find 'adapter_config.json' at 'out_2b/pnet2b_moe/adapter' |
| ep1 it20 loss=9.5326 kl=0.508 aux=4.765 mtp=6.1056 sce=2.314 active=1.80 steps=2.29 (35s) |
| ep1 it25 loss=9.1603 kl=0.627 aux=4.773 mtp=5.8696 sce=2.220 active=1.79 steps=2.21 (42s) |
| ep1 it30 loss=8.9322 kl=0.606 aux=4.792 mtp=5.6676 sce=2.202 active=1.79 steps=2.19 (49s) |
| ep1 it35 loss=8.7785 kl=0.578 aux=4.791 mtp=5.5152 sce=2.200 active=1.78 steps=2.11 (58s) |
| [env] device=cuda dtype=torch.float16 (arg=auto) |
| [data] out_2b/data_agent/mix_r11.jsonl: 3948 条 |
| ep1 it40 loss=8.4825 kl=0.554 aux=4.780 mtp=5.2801 sce=2.158 active=1.77 steps=2.05 (66s) |
| [tokenizer] out_2b/pnet2b_moe |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
|
Loading weights: 0%| | 0/577 [00:00<?, ?it/s]
Loading weights: 100%|██████████| 577/577 [00:00<00:00, 12794.75it/s] |
| [2B适配] 加载后参数 dtype 集合: {'torch.float16'} (期望仅 torch.float16) |
| [2B适配] 参数总显存占用: 5.08 GB |
| [round5-B] per-step CE 启用: weight=0.5 |
| [resume] head+extras 已载入: out_2b/pnet2b_moe |
| Traceback (most recent call last): |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 317, in _get_peft_type |
| config_file = hf_hub_download( |
| model_id, |
| CONFIG_NAME, |
| **hf_hub_download_kwargs, |
| ) |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 84, in _inner_fn |
| validate_repo_id(arg_value) |
| ~~~~~~~~~~~~~~~~^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/huggingface_hub/utils/_validators.py", line 132, in validate_repo_id |
| raise HFValidationError( |
| ...<2 lines>... |
| ) |
| huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'out_2b/pnet2b_moe/adapter'. Use `repo_type` argument if needed. |
| |
| During handling of the above exception, another exception occurred: |
| |
| Traceback (most recent call last): |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 427, in <module> |
| main() |
| ~~~~^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 344, in main |
| model, groups = setup_trainables( |
| ~~~~~~~~~~~~~~~~^ |
| model, lora_r=args.lora, lr=args.lr, lr_lora=args.lr_lora, |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| pretrained_adapter_dir=_ad_dir) |
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/content/pondernet_repo/pondernet/train_ponder_moe.py", line 69, in setup_trainables |
| model = PeftModel.from_pretrained( |
| model, pretrained_adapter_dir, is_trainable=True) |
| File "/usr/local/lib/python3.13/dist-packages/peft/peft_model.py", line 510, in from_pretrained |
| config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig._get_peft_type(model_id, **hf_kwargs)].from_pretrained( |
| ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ |
| File "/usr/local/lib/python3.13/dist-packages/peft/config.py", line 323, in _get_peft_type |
| raise ValueError(f"Can't find '{CONFIG_NAME}' at '{model_id}'") |
| ValueError: Can't find 'adapter_config.json' at 'out_2b/pnet2b_moe/adapter' |
| ep1 it45 loss=8.4612 kl=0.526 aux=4.779 mtp=5.2708 sce=2.149 active=1.78 steps=2.02 (73s) |
| ep1 it50 loss=8.2445 kl=0.563 aux=4.790 mtp=5.1131 sce=2.108 active=1.78 steps=1.97 (78s) |
| ep1 it55 loss=8.4263 kl=0.632 aux=4.798 mtp=5.2097 sce=2.164 active=1.79 steps=1.96 (85s) |
| ep1 it60 loss=8.5360 kl=0.668 aux=4.826 mtp=5.2820 sce=2.188 active=1.79 steps=1.93 (90s) |
| ep1 it65 loss=8.7859 kl=0.656 aux=4.832 mtp=5.3447 sce=2.313 active=1.79 steps=1.92 (94s) |
| ep1 it70 loss=8.8679 kl=0.637 aux=4.840 mtp=5.3728 sce=2.349 active=1.80 steps=1.91 (99s) |
| ep1 it75 loss=8.7920 kl=0.630 aux=4.843 mtp=5.3332 sce=2.324 active=1.79 steps=1.87 (104s) |
| ep1 it80 loss=8.7463 kl=0.609 aux=4.833 mtp=5.2987 sce=2.317 active=1.80 steps=1.86 (109s) |
| ep1 it85 loss=8.5559 kl=0.655 aux=4.835 mtp=5.1844 sce=2.265 active=1.79 steps=1.84 (114s) |
| ep1 it90 loss=8.3834 kl=0.653 aux=4.822 mtp=5.0613 sce=2.231 active=1.79 steps=1.82 (119s) |
| ep1 it95 loss=8.3441 kl=0.640 aux=4.828 mtp=5.0407 sce=2.219 active=1.79 steps=1.81 (124s) |
| ep1 it100 loss=8.3227 kl=0.658 aux=4.829 mtp=5.0117 sce=2.223 active=1.79 steps=1.80 (129s) |
| ep1 it105 loss=8.1731 kl=0.643 aux=4.824 mtp=4.8979 sce=2.199 active=1.79 steps=1.78 (134s) |
| ep1 it110 loss=8.1442 kl=0.657 aux=4.835 mtp=4.8789 sce=2.191 active=1.79 steps=1.76 (139s) |
| ep1 it115 loss=8.1560 kl=0.655 aux=4.832 mtp=4.8926 sce=2.190 active=1.79 steps=1.75 (144s) |
| ep1 it120 loss=8.2569 kl=0.636 aux=4.821 mtp=4.9180 sce=2.241 active=1.79 steps=1.75 (148s) |
| ep1 it125 loss=8.2344 kl=0.628 aux=4.828 mtp=4.8966 sce=2.240 active=1.79 steps=1.75 (153s) |
| ep1 it130 loss=8.2428 kl=0.642 aux=4.827 mtp=4.9075 sce=2.238 active=1.80 steps=1.75 (158s) |
| ep1 it135 loss=8.2332 kl=0.637 aux=4.831 mtp=4.9062 sce=2.232 active=1.79 steps=1.74 (163s) |
| ep1 it140 loss=8.1945 kl=0.625 aux=4.842 mtp=4.8678 sce=2.232 active=1.80 steps=1.74 (167s) |
| ep1 it145 loss=8.1388 kl=0.618 aux=4.847 mtp=4.8346 sce=2.217 active=1.79 steps=1.74 (173s) |
| ep1 it150 loss=8.0737 kl=0.616 aux=4.850 mtp=4.8057 sce=2.193 active=1.79 steps=1.73 (177s) |
| ep1 it155 loss=8.0284 kl=0.619 aux=4.847 mtp=4.7844 sce=2.177 active=1.79 steps=1.73 (183s) |
| ep1 it160 loss=8.0612 kl=0.612 aux=4.847 mtp=4.7832 sce=2.200 active=1.79 steps=1.73 (187s) |
| ep1 it165 loss=7.9694 kl=0.621 aux=4.850 mtp=4.7274 sce=2.175 active=1.79 steps=1.72 (192s) |
| ep1 it170 loss=7.9017 kl=0.624 aux=4.849 mtp=4.6843 sce=2.159 active=1.79 steps=1.72 (198s) |
| ep1 it175 loss=7.8916 kl=0.620 aux=4.845 mtp=4.6749 sce=2.158 active=1.79 steps=1.72 (203s) |
| ep1 it180 loss=7.9073 kl=0.624 aux=4.850 mtp=4.6785 sce=2.167 active=1.80 steps=1.73 (209s) |
| ep1 it185 loss=7.8872 kl=0.631 aux=4.851 mtp=4.6681 sce=2.160 active=1.80 steps=1.72 (213s) |
| ep1 it190 loss=7.8541 kl=0.630 aux=4.849 mtp=4.6395 sce=2.157 active=1.80 steps=1.73 (219s) |
| ep1 it195 loss=7.9114 kl=0.627 aux=4.854 mtp=4.6850 sce=2.165 active=1.80 steps=1.72 (224s) |
| ep1 it200 loss=7.8741 kl=0.644 aux=4.863 mtp=4.6682 sce=2.151 active=1.80 steps=1.72 (228s) |
| ep1 it205 loss=7.9386 kl=0.639 aux=4.868 mtp=4.6970 sce=2.175 active=1.80 steps=1.72 (233s) |
| ep1 it210 loss=7.9285 kl=0.636 aux=4.870 mtp=4.6852 sce=2.176 active=1.80 steps=1.72 (238s) |
| ep1 it215 loss=7.8981 kl=0.632 aux=4.872 mtp=4.6599 sce=2.173 active=1.80 steps=1.73 (244s) |
| ep1 it220 loss=7.8734 kl=0.646 aux=4.871 mtp=4.6405 sce=2.169 active=1.80 steps=1.72 (249s) |
| ep1 it225 loss=7.8264 kl=0.645 aux=4.869 mtp=4.6075 sce=2.160 active=1.80 steps=1.72 (255s) |
| ep1 it230 loss=7.7364 kl=0.652 aux=4.863 mtp=4.5524 sce=2.136 active=1.80 steps=1.72 (260s) |
| ep1 it235 loss=7.7610 kl=0.653 aux=4.863 mtp=4.5681 sce=2.142 active=1.80 steps=1.72 (265s) |
| ep1 it240 loss=7.7722 kl=0.654 aux=4.863 mtp=4.5666 sce=2.151 active=1.80 steps=1.71 (270s) |
| ep1 it245 loss=7.7305 kl=0.654 aux=4.862 mtp=4.5467 sce=2.136 active=1.80 steps=1.72 (275s) |
| ep1 it250 loss=7.6811 kl=0.647 aux=4.859 mtp=4.5102 sce=2.128 active=1.80 steps=1.72 (280s) |
| ep1 it255 loss=7.5988 kl=0.644 aux=4.856 mtp=4.4598 sce=2.107 active=1.80 steps=1.71 (285s) |
| ep1 it260 loss=7.5855 kl=0.647 aux=4.856 mtp=4.4552 sce=2.101 active=1.80 steps=1.71 (290s) |
| ep1 it265 loss=7.5938 kl=0.642 aux=4.851 mtp=4.4498 sce=2.110 active=1.80 steps=1.71 (295s) |
| ep1 it270 loss=7.6154 kl=0.647 aux=4.854 mtp=4.4657 sce=2.113 active=1.80 steps=1.71 (299s) |
| ep1 it275 loss=7.5992 kl=0.645 aux=4.857 mtp=4.4571 sce=2.109 active=1.80 steps=1.71 (305s) |
| ep1 it280 loss=7.6452 kl=0.642 aux=4.860 mtp=4.4911 sce=2.117 active=1.80 steps=1.71 (309s) |
| ep1 it285 loss=7.6378 kl=0.637 aux=4.860 mtp=4.4877 sce=2.114 active=1.80 steps=1.71 (314s) |
| ep1 it290 loss=7.6478 kl=0.637 aux=4.862 mtp=4.4964 sce=2.115 active=1.80 steps=1.71 (319s) |
| ep1 it295 loss=7.6002 kl=0.641 aux=4.860 mtp=4.4697 sce=2.101 active=1.80 steps=1.71 (324s) |
| ep1 it300 loss=7.5796 kl=0.636 aux=4.858 mtp=4.4498 sce=2.101 active=1.80 steps=1.71 (329s) |
| ep1 it305 loss=7.5797 kl=0.635 aux=4.858 mtp=4.4499 sce=2.101 active=1.80 steps=1.71 (335s) |
| ep1 it310 loss=7.5083 kl=0.631 aux=4.856 mtp=4.3992 sce=2.087 active=1.80 steps=1.72 (340s) |
| ep1 it315 loss=7.4989 kl=0.626 aux=4.857 mtp=4.3935 sce=2.085 active=1.80 steps=1.72 (346s) |
| ep1 it320 loss=7.4833 kl=0.620 aux=4.858 mtp=4.3837 sce=2.081 active=1.80 steps=1.72 (351s) |
| ep1 it325 loss=7.4822 kl=0.619 aux=4.858 mtp=4.3805 sce=2.083 active=1.80 steps=1.72 (356s) |
| ep1 it330 loss=7.4497 kl=0.624 aux=4.855 mtp=4.3614 sce=2.074 active=1.80 steps=1.73 (362s) |
| ep1 it335 loss=7.4126 kl=0.621 aux=4.853 mtp=4.3378 sce=2.065 active=1.80 steps=1.73 (368s) |
| ep1 it340 loss=7.4234 kl=0.623 aux=4.857 mtp=4.3435 sce=2.069 active=1.80 steps=1.73 (373s) |
| ep1 it345 loss=7.3997 kl=0.628 aux=4.858 mtp=4.3301 sce=2.062 active=1.80 steps=1.73 (379s) |
| ep1 it350 loss=7.3994 kl=0.624 aux=4.859 mtp=4.3309 sce=2.061 active=1.80 steps=1.73 (384s) |
| ep1 it355 loss=7.3909 kl=0.630 aux=4.858 mtp=4.3245 sce=2.060 active=1.80 steps=1.73 (391s) |
| ep1 it360 loss=7.3781 kl=0.631 aux=4.856 mtp=4.3122 sce=2.060 active=1.80 steps=1.73 (396s) |
| ep1 it365 loss=7.3773 kl=0.632 aux=4.855 mtp=4.3153 sce=2.057 active=1.80 steps=1.73 (401s) |
| ep1 it370 loss=7.3409 kl=0.633 aux=4.858 mtp=4.2923 sce=2.048 active=1.80 steps=1.73 (407s) |
| ep1 it375 loss=7.3561 kl=0.632 aux=4.858 mtp=4.2997 sce=2.053 active=1.80 steps=1.73 (412s) |
| ep1 it380 loss=7.3368 kl=0.633 aux=4.859 mtp=4.2866 sce=2.050 active=1.80 steps=1.73 (418s) |
| ep1 it385 loss=7.3044 kl=0.636 aux=4.858 mtp=4.2684 sce=2.040 active=1.80 steps=1.73 (423s) |
| ep1 it390 loss=7.2948 kl=0.633 aux=4.854 mtp=4.2634 sce=2.037 active=1.80 steps=1.73 (428s) |
| ep1 it395 loss=7.2696 kl=0.630 aux=4.853 mtp=4.2461 sce=2.032 active=1.80 steps=1.73 (434s) |
| ep1 it400 loss=7.2617 kl=0.632 aux=4.851 mtp=4.2387 sce=2.032 active=1.80 steps=1.73 (440s) |
| ep1 it405 loss=7.2774 kl=0.630 aux=4.853 mtp=4.2479 sce=2.036 active=1.80 steps=1.73 (445s) |
| ep1 it410 loss=7.3031 kl=0.630 aux=4.851 mtp=4.2662 sce=2.041 active=1.80 steps=1.73 (450s) |
| ep1 it415 loss=7.3152 kl=0.627 aux=4.849 mtp=4.2761 sce=2.043 active=1.80 steps=1.73 (456s) |
| ep1 it420 loss=7.3345 kl=0.627 aux=4.852 mtp=4.2878 sce=2.048 active=1.80 steps=1.73 (462s) |
| ep1 it425 loss=7.3087 kl=0.630 aux=4.849 mtp=4.2730 sce=2.040 active=1.80 steps=1.73 (469s) |
| ep1 it430 loss=7.2948 kl=0.628 aux=4.847 mtp=4.2608 sce=2.040 active=1.80 steps=1.74 (476s) |
| ep1 it435 loss=7.2858 kl=0.626 aux=4.848 mtp=4.2565 sce=2.037 active=1.80 steps=1.74 (484s) |
| ep1 it440 loss=7.2690 kl=0.624 aux=4.846 mtp=4.2469 sce=2.032 active=1.80 steps=1.74 (491s) |
| ep1 it445 loss=7.2584 kl=0.625 aux=4.843 mtp=4.2430 sce=2.028 active=1.80 steps=1.74 (497s) |
| ep1 it450 loss=7.2561 kl=0.625 aux=4.842 mtp=4.2415 sce=2.027 active=1.80 steps=1.74 (503s) |
| ep1 it455 loss=7.2607 kl=0.625 aux=4.840 mtp=4.2422 sce=2.030 active=1.80 steps=1.75 (509s) |
| ep1 it460 loss=7.2519 kl=0.624 aux=4.840 mtp=4.2387 sce=2.027 active=1.80 steps=1.75 (514s) |
| ep1 it465 loss=7.2304 kl=0.624 aux=4.836 mtp=4.2243 sce=2.022 active=1.80 steps=1.75 (519s) |
| ep1 it470 loss=7.2000 kl=0.627 aux=4.834 mtp=4.2072 sce=2.013 active=1.80 steps=1.75 (525s) |
| ep1 it475 loss=7.1774 kl=0.626 aux=4.832 mtp=4.1878 sce=2.011 active=1.80 steps=1.75 (530s) |
| ep1 it480 loss=7.1746 kl=0.626 aux=4.830 mtp=4.1829 sce=2.012 active=1.80 steps=1.74 (535s) |
| ep1 it485 loss=7.1681 kl=0.624 aux=4.828 mtp=4.1792 sce=2.011 active=1.80 steps=1.74 (540s) |
| ep1 it490 loss=7.1604 kl=0.624 aux=4.831 mtp=4.1726 sce=2.010 active=1.80 steps=1.75 (545s) |
| ep1 it495 loss=7.1465 kl=0.622 aux=4.829 mtp=4.1626 sce=2.007 active=1.80 steps=1.75 (550s) |
| ep1 it500 loss=7.1089 kl=0.629 aux=4.827 mtp=4.1396 sce=1.997 active=1.80 steps=1.74 (555s) |
| ep1 it505 loss=7.1111 kl=0.631 aux=4.827 mtp=4.1416 sce=1.998 active=1.80 steps=1.74 (561s) |
| ep1 it510 loss=7.1211 kl=0.630 aux=4.826 mtp=4.1466 sce=2.001 active=1.80 steps=1.74 (566s) |
| ep1 it515 loss=7.0885 kl=0.638 aux=4.824 mtp=4.1280 sce=1.991 active=1.80 steps=1.74 (572s) |
| ep1 it520 loss=7.0770 kl=0.636 aux=4.822 mtp=4.1167 sce=1.991 active=1.80 steps=1.74 (576s) |
| ep1 it525 loss=7.0582 kl=0.633 aux=4.820 mtp=4.1044 sce=1.987 active=1.80 steps=1.74 (581s) |
| ep1 it530 loss=7.0636 kl=0.631 aux=4.823 mtp=4.1088 sce=1.988 active=1.80 steps=1.74 (586s) |
| ep1 it535 loss=7.0303 kl=0.631 aux=4.824 mtp=4.0846 sce=1.982 active=1.80 steps=1.74 (591s) |
| ep1 it540 loss=7.0308 kl=0.628 aux=4.822 mtp=4.0810 sce=1.985 active=1.80 steps=1.74 (596s) |
| ep1 it545 loss=6.9923 kl=0.629 aux=4.821 mtp=4.0595 sce=1.973 active=1.80 steps=1.74 (601s) |
| ep1 it550 loss=6.9568 kl=0.629 aux=4.821 mtp=4.0339 sce=1.967 active=1.80 steps=1.74 (606s) |
| ep1 it555 loss=6.9580 kl=0.627 aux=4.820 mtp=4.0258 sce=1.973 active=1.80 steps=1.74 (611s) |
| ep1 it560 loss=6.9531 kl=0.623 aux=4.820 mtp=4.0160 sce=1.976 active=1.80 steps=1.75 (616s) |
| ep1 it565 loss=6.9391 kl=0.625 aux=4.820 mtp=4.0075 sce=1.973 active=1.80 steps=1.75 (622s) |
| ep1 it570 loss=6.9178 kl=0.625 aux=4.820 mtp=3.9941 sce=1.968 active=1.80 steps=1.75 (629s) |
| ep1 it575 loss=6.9067 kl=0.624 aux=4.820 mtp=3.9885 sce=1.964 active=1.80 steps=1.75 (635s) |
| ep1 it580 loss=6.8955 kl=0.621 aux=4.821 mtp=3.9801 sce=1.962 active=1.80 steps=1.75 (640s) |
| ep1 it585 loss=6.8937 kl=0.622 aux=4.824 mtp=3.9792 sce=1.962 active=1.80 steps=1.75 (651s) |
| ep1 it590 loss=6.9000 kl=0.623 aux=4.826 mtp=3.9829 sce=1.964 active=1.80 steps=1.75 (661s) |
| ep1 it595 loss=6.9061 kl=0.624 aux=4.826 mtp=3.9884 sce=1.964 active=1.80 steps=1.75 (671s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11.tmp |
| [save@partial] atomic swap ok: out_2b/ckpt_2b_r11 |
| ep1 it600 loss=6.9127 kl=0.625 aux=4.827 mtp=3.9963 sce=1.963 active=1.80 steps=1.75 (681s) |
| ep1 it605 loss=6.9136 kl=0.623 aux=4.829 mtp=3.9976 sce=1.963 active=1.80 steps=1.75 (687s) |
| ep1 it610 loss=6.8986 kl=0.622 aux=4.828 mtp=3.9859 sce=1.961 active=1.80 steps=1.75 (693s) |
| ep1 it615 loss=6.8894 kl=0.624 aux=4.828 mtp=3.9821 sce=1.957 active=1.80 steps=1.75 (699s) |
| ep1 it620 loss=6.8916 kl=0.625 aux=4.828 mtp=3.9830 sce=1.958 active=1.80 steps=1.75 (709s) |
| ep1 it625 loss=6.8869 kl=0.623 aux=4.827 mtp=3.9741 sce=1.961 active=1.80 steps=1.75 (718s) |
| ep1 it630 loss=6.8855 kl=0.622 aux=4.828 mtp=3.9759 sce=1.959 active=1.80 steps=1.75 (728s) |
| ep1 it635 loss=6.8711 kl=0.623 aux=4.828 mtp=3.9679 sce=1.955 active=1.80 steps=1.75 (739s) |
| ep1 it640 loss=6.8515 kl=0.623 aux=4.829 mtp=3.9531 sce=1.951 active=1.80 steps=1.75 (749s) |
| ep1 it645 loss=6.8475 kl=0.624 aux=4.831 mtp=3.9528 sce=1.949 active=1.80 steps=1.75 (761s) |
| ep1 it650 loss=6.8485 kl=0.626 aux=4.831 mtp=3.9537 sce=1.949 active=1.80 steps=1.75 (771s) |
| ep1 it655 loss=6.8400 kl=0.629 aux=4.829 mtp=3.9489 sce=1.947 active=1.80 steps=1.76 (782s) |
| ep1 it660 loss=6.8242 kl=0.626 aux=4.828 mtp=3.9343 sce=1.946 active=1.80 steps=1.76 (787s) |
| ep1 it665 loss=6.8178 kl=0.627 aux=4.826 mtp=3.9321 sce=1.943 active=1.80 steps=1.76 (792s) |
| ep1 it670 loss=6.7942 kl=0.627 aux=4.823 mtp=3.9196 sce=1.936 active=1.80 steps=1.76 (798s) |
| ep1 it675 loss=6.7911 kl=0.628 aux=4.823 mtp=3.9202 sce=1.933 active=1.80 steps=1.76 (803s) |
| ep1 it680 loss=6.7810 kl=0.626 aux=4.821 mtp=3.9114 sce=1.932 active=1.80 steps=1.76 (808s) |
| ep1 it685 loss=6.7719 kl=0.627 aux=4.821 mtp=3.9072 sce=1.929 active=1.80 steps=1.76 (813s) |
| ep1 it690 loss=6.7576 kl=0.624 aux=4.819 mtp=3.8988 sce=1.925 active=1.80 steps=1.76 (818s) |
| ep1 it695 loss=6.7391 kl=0.623 aux=4.818 mtp=3.8865 sce=1.921 active=1.80 steps=1.76 (823s) |
| ep1 it700 loss=6.7228 kl=0.621 aux=4.819 mtp=3.8753 sce=1.918 active=1.80 steps=1.76 (828s) |
| ep1 it705 loss=6.7180 kl=0.619 aux=4.818 mtp=3.8715 sce=1.917 active=1.80 steps=1.76 (833s) |
| ep1 it710 loss=6.7012 kl=0.621 aux=4.816 mtp=3.8644 sce=1.911 active=1.80 steps=1.76 (838s) |
| ep1 it715 loss=6.7032 kl=0.625 aux=4.816 mtp=3.8669 sce=1.910 active=1.80 steps=1.75 (843s) |
| ep1 it720 loss=6.7041 kl=0.627 aux=4.815 mtp=3.8670 sce=1.911 active=1.80 steps=1.75 (849s) |
| ep1 it725 loss=6.6882 kl=0.625 aux=4.813 mtp=3.8530 sce=1.909 active=1.80 steps=1.75 (854s) |
| ep1 it730 loss=6.6847 kl=0.623 aux=4.814 mtp=3.8497 sce=1.910 active=1.80 steps=1.75 (859s) |
| ep1 it735 loss=6.6714 kl=0.624 aux=4.813 mtp=3.8414 sce=1.906 active=1.80 steps=1.75 (864s) |
| ep1 it740 loss=6.6504 kl=0.627 aux=4.812 mtp=3.8284 sce=1.901 active=1.80 steps=1.75 (870s) |
| ep1 it745 loss=6.6361 kl=0.628 aux=4.811 mtp=3.8213 sce=1.896 active=1.80 steps=1.75 (875s) |
| ep1 it750 loss=6.6462 kl=0.628 aux=4.810 mtp=3.8274 sce=1.899 active=1.80 steps=1.75 (881s) |
| ep1 it755 loss=6.6227 kl=0.627 aux=4.810 mtp=3.8128 sce=1.893 active=1.80 steps=1.75 (886s) |
| ep1 it760 loss=6.6163 kl=0.628 aux=4.811 mtp=3.8103 sce=1.890 active=1.80 steps=1.75 (892s) |
| ep1 it765 loss=6.6084 kl=0.628 aux=4.810 mtp=3.8040 sce=1.889 active=1.80 steps=1.75 (897s) |
| ep1 it770 loss=6.5960 kl=0.626 aux=4.810 mtp=3.7949 sce=1.887 active=1.80 steps=1.75 (902s) |
| ep1 it775 loss=6.5788 kl=0.627 aux=4.808 mtp=3.7865 sce=1.881 active=1.80 steps=1.75 (908s) |
| ep1 it780 loss=6.5786 kl=0.626 aux=4.807 mtp=3.7874 sce=1.881 active=1.80 steps=1.75 (913s) |
| ep1 it785 loss=6.5778 kl=0.626 aux=4.808 mtp=3.7863 sce=1.881 active=1.80 steps=1.75 (919s) |
| ep1 it790 loss=6.5822 kl=0.625 aux=4.807 mtp=3.7881 sce=1.883 active=1.80 steps=1.75 (924s) |
| ep1 it795 loss=6.5594 kl=0.627 aux=4.805 mtp=3.7718 sce=1.878 active=1.80 steps=1.75 (931s) |
| ep1 it800 loss=6.5436 kl=0.625 aux=4.805 mtp=3.7617 sce=1.875 active=1.80 steps=1.75 (936s) |
| ep1 it805 loss=6.5311 kl=0.624 aux=4.804 mtp=3.7542 sce=1.871 active=1.80 steps=1.75 (942s) |
| ep1 it810 loss=6.5218 kl=0.623 aux=4.804 mtp=3.7466 sce=1.870 active=1.80 steps=1.76 (946s) |
| ep1 it815 loss=6.5345 kl=0.621 aux=4.804 mtp=3.7519 sce=1.876 active=1.80 steps=1.76 (951s) |
| ep1 it820 loss=6.5253 kl=0.622 aux=4.802 mtp=3.7476 sce=1.872 active=1.80 steps=1.76 (957s) |
| ep1 it825 loss=6.5192 kl=0.622 aux=4.803 mtp=3.7471 sce=1.869 active=1.80 steps=1.76 (963s) |
| ep1 it830 loss=6.5063 kl=0.621 aux=4.802 mtp=3.7381 sce=1.866 active=1.80 steps=1.76 (969s) |
| ep1 it835 loss=6.4968 kl=0.622 aux=4.801 mtp=3.7333 sce=1.863 active=1.80 steps=1.76 (974s) |
| ep1 it840 loss=6.4968 kl=0.623 aux=4.799 mtp=3.7308 sce=1.865 active=1.80 steps=1.76 (979s) |
| ep1 it845 loss=6.4962 kl=0.623 aux=4.798 mtp=3.7295 sce=1.865 active=1.80 steps=1.76 (985s) |
| ep1 it850 loss=6.4840 kl=0.622 aux=4.797 mtp=3.7213 sce=1.862 active=1.80 steps=1.76 (990s) |
| ep1 it855 loss=6.4726 kl=0.622 aux=4.796 mtp=3.7147 sce=1.859 active=1.80 steps=1.76 (996s) |
| ep1 it860 loss=6.4732 kl=0.620 aux=4.795 mtp=3.7130 sce=1.861 active=1.80 steps=1.76 (1000s) |
| ep1 it865 loss=6.4813 kl=0.620 aux=4.795 mtp=3.7187 sce=1.863 active=1.80 steps=1.76 (1006s) |
| ep1 it870 loss=6.4727 kl=0.620 aux=4.793 mtp=3.7136 sce=1.860 active=1.80 steps=1.76 (1011s) |
| ep1 it875 loss=6.4701 kl=0.619 aux=4.792 mtp=3.7130 sce=1.859 active=1.80 steps=1.76 (1017s) |
| ep1 it880 loss=6.4635 kl=0.621 aux=4.792 mtp=3.7100 sce=1.857 active=1.80 steps=1.76 (1023s) |
| ep1 it885 loss=6.4533 kl=0.619 aux=4.791 mtp=3.7057 sce=1.853 active=1.80 steps=1.76 (1028s) |
| ep1 it890 loss=6.4527 kl=0.619 aux=4.791 mtp=3.7062 sce=1.852 active=1.80 steps=1.76 (1033s) |
| ep1 it895 loss=6.4466 kl=0.619 aux=4.789 mtp=3.7025 sce=1.851 active=1.80 steps=1.76 (1038s) |
| ep1 it900 loss=6.4516 kl=0.620 aux=4.789 mtp=3.7074 sce=1.851 active=1.80 steps=1.76 (1044s) |
| ep1 it905 loss=6.4506 kl=0.620 aux=4.789 mtp=3.7054 sce=1.851 active=1.80 steps=1.76 (1050s) |
| ep1 it910 loss=6.4466 kl=0.620 aux=4.788 mtp=3.7044 sce=1.849 active=1.80 steps=1.76 (1055s) |
| ep1 it915 loss=6.4518 kl=0.620 aux=4.787 mtp=3.7089 sce=1.850 active=1.80 steps=1.76 (1061s) |
| ep1 it920 loss=6.4619 kl=0.621 aux=4.786 mtp=3.7155 sce=1.852 active=1.80 steps=1.76 (1066s) |
| ep1 it925 loss=6.4619 kl=0.622 aux=4.786 mtp=3.7149 sce=1.853 active=1.80 steps=1.76 (1071s) |
| ep1 it930 loss=6.4591 kl=0.622 aux=4.786 mtp=3.7131 sce=1.852 active=1.80 steps=1.76 (1077s) |
| ep1 it935 loss=6.4538 kl=0.622 aux=4.784 mtp=3.7085 sce=1.852 active=1.80 steps=1.76 (1082s) |
| ep1 it940 loss=6.4444 kl=0.621 aux=4.783 mtp=3.7024 sce=1.850 active=1.80 steps=1.76 (1088s) |
| ep1 it945 loss=6.4451 kl=0.620 aux=4.783 mtp=3.7000 sce=1.852 active=1.80 steps=1.76 (1094s) |
| ep1 it950 loss=6.4415 kl=0.619 aux=4.783 mtp=3.6985 sce=1.850 active=1.80 steps=1.76 (1099s) |
| ep1 it955 loss=6.4351 kl=0.621 aux=4.783 mtp=3.6952 sce=1.848 active=1.80 steps=1.76 (1106s) |
| ep1 it960 loss=6.4200 kl=0.619 aux=4.782 mtp=3.6853 sce=1.845 active=1.80 steps=1.76 (1111s) |
| ep1 it965 loss=6.4273 kl=0.620 aux=4.782 mtp=3.6890 sce=1.847 active=1.80 steps=1.77 (1117s) |
| ep1 it970 loss=6.4169 kl=0.619 aux=4.782 mtp=3.6825 sce=1.845 active=1.80 steps=1.77 (1122s) |
| ep1 it975 loss=6.4058 kl=0.618 aux=4.780 mtp=3.6739 sce=1.843 active=1.80 steps=1.77 (1128s) |
| ep1 it980 loss=6.4048 kl=0.618 aux=4.781 mtp=3.6736 sce=1.843 active=1.80 steps=1.77 (1134s) |
| ep1 it985 loss=6.4012 kl=0.617 aux=4.781 mtp=3.6701 sce=1.843 active=1.80 steps=1.77 (1139s) |
| ep1 it990 loss=6.3988 kl=0.617 aux=4.780 mtp=3.6688 sce=1.842 active=1.80 steps=1.77 (1146s) |
| ep1 it995 loss=6.4052 kl=0.617 aux=4.780 mtp=3.6727 sce=1.844 active=1.80 steps=1.77 (1152s) |
| ep1 it1000 loss=6.3979 kl=0.616 aux=4.779 mtp=3.6671 sce=1.843 active=1.80 steps=1.77 (1158s) |
| ep1 it1005 loss=6.3906 kl=0.615 aux=4.780 mtp=3.6614 sce=1.842 active=1.80 steps=1.77 (1165s) |
| ep1 it1010 loss=6.3908 kl=0.615 aux=4.780 mtp=3.6600 sce=1.843 active=1.80 steps=1.77 (1170s) |
| ep1 it1015 loss=6.3790 kl=0.613 aux=4.779 mtp=3.6514 sce=1.841 active=1.80 steps=1.77 (1175s) |
| ep1 it1020 loss=6.3761 kl=0.613 aux=4.779 mtp=3.6499 sce=1.840 active=1.80 steps=1.77 (1181s) |
| ep1 it1025 loss=6.3770 kl=0.612 aux=4.780 mtp=3.6488 sce=1.841 active=1.80 steps=1.77 (1185s) |
| ep1 it1030 loss=6.3719 kl=0.611 aux=4.779 mtp=3.6460 sce=1.840 active=1.80 steps=1.77 (1191s) |
| ep1 it1035 loss=6.3699 kl=0.609 aux=4.779 mtp=3.6450 sce=1.839 active=1.80 steps=1.77 (1197s) |
| ep1 it1040 loss=6.3633 kl=0.609 aux=4.778 mtp=3.6390 sce=1.839 active=1.80 steps=1.77 (1202s) |
| ep1 it1045 loss=6.3555 kl=0.612 aux=4.778 mtp=3.6360 sce=1.836 active=1.80 steps=1.77 (1208s) |
| ep1 it1050 loss=6.3392 kl=0.612 aux=4.777 mtp=3.6255 sce=1.832 active=1.80 steps=1.77 (1214s) |
| ep1 it1055 loss=6.3449 kl=0.611 aux=4.777 mtp=3.6281 sce=1.834 active=1.80 steps=1.78 (1220s) |
| ep1 it1060 loss=6.3340 kl=0.609 aux=4.777 mtp=3.6215 sce=1.831 active=1.80 steps=1.78 (1225s) |
| ep1 it1065 loss=6.3295 kl=0.609 aux=4.777 mtp=3.6189 sce=1.830 active=1.80 steps=1.78 (1230s) |
| ep1 it1070 loss=6.3249 kl=0.608 aux=4.776 mtp=3.6143 sce=1.830 active=1.80 steps=1.78 (1235s) |
| ep1 it1075 loss=6.3258 kl=0.608 aux=4.776 mtp=3.6149 sce=1.830 active=1.80 steps=1.78 (1241s) |
| ep1 it1080 loss=6.3190 kl=0.607 aux=4.776 mtp=3.6088 sce=1.830 active=1.80 steps=1.78 (1247s) |
| ep1 it1085 loss=6.3168 kl=0.608 aux=4.776 mtp=3.6092 sce=1.828 active=1.80 steps=1.78 (1254s) |
| ep1 it1090 loss=6.3149 kl=0.607 aux=4.775 mtp=3.6073 sce=1.828 active=1.80 steps=1.78 (1259s) |
| ep1 it1095 loss=6.3048 kl=0.605 aux=4.774 mtp=3.6020 sce=1.825 active=1.80 steps=1.78 (1265s) |
| ep1 it1100 loss=6.3029 kl=0.606 aux=4.775 mtp=3.6001 sce=1.825 active=1.80 steps=1.78 (1271s) |
| ep1 it1105 loss=6.2936 kl=0.606 aux=4.773 mtp=3.5952 sce=1.822 active=1.80 steps=1.78 (1276s) |
| ep1 it1110 loss=6.2855 kl=0.605 aux=4.772 mtp=3.5902 sce=1.820 active=1.80 steps=1.78 (1282s) |
| ep1 it1115 loss=6.2818 kl=0.605 aux=4.771 mtp=3.5863 sce=1.820 active=1.80 steps=1.78 (1287s) |
| ep1 it1120 loss=6.2809 kl=0.606 aux=4.771 mtp=3.5850 sce=1.821 active=1.80 steps=1.78 (1294s) |
| ep1 it1125 loss=6.2793 kl=0.605 aux=4.771 mtp=3.5841 sce=1.820 active=1.80 steps=1.78 (1299s) |
| ep1 it1130 loss=6.2692 kl=0.605 aux=4.770 mtp=3.5777 sce=1.818 active=1.80 steps=1.78 (1304s) |
| ep1 it1135 loss=6.2621 kl=0.605 aux=4.769 mtp=3.5721 sce=1.817 active=1.80 steps=1.78 (1310s) |
| ep1 it1140 loss=6.2638 kl=0.605 aux=4.768 mtp=3.5707 sce=1.819 active=1.80 steps=1.78 (1315s) |
| ep1 it1145 loss=6.2524 kl=0.604 aux=4.767 mtp=3.5628 sce=1.816 active=1.80 steps=1.78 (1321s) |
| ep1 it1150 loss=6.2560 kl=0.605 aux=4.766 mtp=3.5651 sce=1.817 active=1.80 steps=1.78 (1326s) |
| ep1 it1155 loss=6.2539 kl=0.605 aux=4.766 mtp=3.5644 sce=1.816 active=1.80 steps=1.78 (1333s) |
| ep1 it1160 loss=6.2448 kl=0.606 aux=4.764 mtp=3.5582 sce=1.814 active=1.80 steps=1.78 (1339s) |
| ep1 it1165 loss=6.2380 kl=0.606 aux=4.763 mtp=3.5548 sce=1.812 active=1.80 steps=1.78 (1345s) |
| ep1 it1170 loss=6.2333 kl=0.607 aux=4.763 mtp=3.5530 sce=1.810 active=1.80 steps=1.78 (1351s) |
| ep1 it1175 loss=6.2352 kl=0.607 aux=4.763 mtp=3.5544 sce=1.811 active=1.80 steps=1.78 (1356s) |
| ep1 it1180 loss=6.2318 kl=0.606 aux=4.762 mtp=3.5528 sce=1.809 active=1.80 steps=1.78 (1362s) |
| ep1 it1185 loss=6.2308 kl=0.607 aux=4.760 mtp=3.5515 sce=1.810 active=1.80 steps=1.78 (1368s) |
| ep1 it1190 loss=6.2171 kl=0.607 aux=4.759 mtp=3.5423 sce=1.807 active=1.80 steps=1.78 (1374s) |
| ep1 it1195 loss=6.2125 kl=0.606 aux=4.758 mtp=3.5389 sce=1.806 active=1.80 steps=1.78 (1380s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11.tmp |
| [save@partial] atomic swap ok: out_2b/ckpt_2b_r11 |
| ep1 it1200 loss=6.2028 kl=0.608 aux=4.756 mtp=3.5335 sce=1.803 active=1.80 steps=1.78 (1387s) |
| ep1 it1205 loss=6.1955 kl=0.607 aux=4.756 mtp=3.5274 sce=1.802 active=1.80 steps=1.79 (1392s) |
| ep1 it1210 loss=6.1986 kl=0.608 aux=4.755 mtp=3.5310 sce=1.802 active=1.80 steps=1.78 (1398s) |
| ep1 it1215 loss=6.2022 kl=0.608 aux=4.755 mtp=3.5319 sce=1.804 active=1.80 steps=1.78 (1404s) |
| ep1 it1220 loss=6.2053 kl=0.608 aux=4.754 mtp=3.5345 sce=1.804 active=1.80 steps=1.78 (1412s) |
| ep1 it1225 loss=6.2017 kl=0.608 aux=4.754 mtp=3.5316 sce=1.804 active=1.80 steps=1.78 (1423s) |
| ep1 it1230 loss=6.1952 kl=0.608 aux=4.753 mtp=3.5267 sce=1.802 active=1.80 steps=1.79 (1433s) |
| ep1 it1235 loss=6.1979 kl=0.607 aux=4.754 mtp=3.5259 sce=1.805 active=1.80 steps=1.79 (1443s) |
| ep1 it1240 loss=6.1878 kl=0.607 aux=4.753 mtp=3.5207 sce=1.802 active=1.80 steps=1.79 (1454s) |
| ep1 it1245 loss=6.1900 kl=0.606 aux=4.754 mtp=3.5214 sce=1.803 active=1.80 steps=1.79 (1464s) |
| ep1 it1250 loss=6.1981 kl=0.607 aux=4.754 mtp=3.5284 sce=1.803 active=1.80 steps=1.79 (1472s) |
| ep1 it1255 loss=6.1956 kl=0.607 aux=4.753 mtp=3.5287 sce=1.801 active=1.80 steps=1.78 (1478s) |
| ep1 it1260 loss=6.2030 kl=0.607 aux=4.753 mtp=3.5353 sce=1.802 active=1.80 steps=1.79 (1483s) |
| ep1 it1265 loss=6.1925 kl=0.608 aux=4.753 mtp=3.5278 sce=1.800 active=1.80 steps=1.79 (1489s) |
| ep1 it1270 loss=6.1779 kl=0.608 aux=4.752 mtp=3.5188 sce=1.796 active=1.80 steps=1.79 (1495s) |
| ep1 it1275 loss=6.1745 kl=0.608 aux=4.753 mtp=3.5169 sce=1.795 active=1.80 steps=1.79 (1501s) |
| ep1 it1280 loss=6.1684 kl=0.607 aux=4.752 mtp=3.5127 sce=1.794 active=1.80 steps=1.79 (1506s) |
| ep1 it1285 loss=6.1672 kl=0.608 aux=4.751 mtp=3.5125 sce=1.793 active=1.80 steps=1.79 (1512s) |
| ep1 it1290 loss=6.1617 kl=0.607 aux=4.751 mtp=3.5093 sce=1.792 active=1.80 steps=1.79 (1518s) |
| ep1 it1295 loss=6.1563 kl=0.607 aux=4.751 mtp=3.5053 sce=1.791 active=1.80 steps=1.79 (1523s) |
| ep1 it1300 loss=6.1549 kl=0.606 aux=4.749 mtp=3.5041 sce=1.791 active=1.80 steps=1.79 (1529s) |
| ep1 it1305 loss=6.1617 kl=0.607 aux=4.750 mtp=3.5091 sce=1.792 active=1.80 steps=1.79 (1534s) |
| ep1 it1310 loss=6.1474 kl=0.608 aux=4.750 mtp=3.4993 sce=1.789 active=1.80 steps=1.79 (1541s) |
| ep1 it1315 loss=6.1355 kl=0.607 aux=4.749 mtp=3.4902 sce=1.787 active=1.80 steps=1.79 (1546s) |
| ep1 it1320 loss=6.1396 kl=0.608 aux=4.750 mtp=3.4918 sce=1.789 active=1.80 steps=1.79 (1552s) |
| ep1 it1325 loss=6.1369 kl=0.609 aux=4.749 mtp=3.4907 sce=1.788 active=1.80 steps=1.79 (1558s) |
| ep1 it1330 loss=6.1397 kl=0.610 aux=4.749 mtp=3.4939 sce=1.788 active=1.80 steps=1.79 (1564s) |
| ep1 it1335 loss=6.1415 kl=0.609 aux=4.749 mtp=3.4957 sce=1.788 active=1.80 steps=1.79 (1570s) |
| ep1 it1340 loss=6.1350 kl=0.611 aux=4.749 mtp=3.4917 sce=1.786 active=1.80 steps=1.79 (1576s) |
| ep1 it1345 loss=6.1377 kl=0.611 aux=4.749 mtp=3.4930 sce=1.787 active=1.80 steps=1.79 (1582s) |
| ep1 it1350 loss=6.1364 kl=0.612 aux=4.748 mtp=3.4935 sce=1.786 active=1.80 steps=1.79 (1587s) |
| ep1 it1355 loss=6.1349 kl=0.614 aux=4.748 mtp=3.4932 sce=1.785 active=1.80 steps=1.79 (1593s) |
| ep1 it1360 loss=6.1369 kl=0.613 aux=4.748 mtp=3.4950 sce=1.785 active=1.80 steps=1.79 (1598s) |
| ep1 it1365 loss=6.1306 kl=0.613 aux=4.747 mtp=3.4914 sce=1.783 active=1.80 steps=1.79 (1603s) |
| ep1 it1370 loss=6.1287 kl=0.613 aux=4.747 mtp=3.4898 sce=1.783 active=1.80 steps=1.79 (1608s) |
| ep1 it1375 loss=6.1241 kl=0.615 aux=4.747 mtp=3.4877 sce=1.781 active=1.80 steps=1.79 (1614s) |
| ep1 it1380 loss=6.1268 kl=0.614 aux=4.747 mtp=3.4899 sce=1.782 active=1.80 steps=1.79 (1620s) |
| ep1 it1385 loss=6.1268 kl=0.614 aux=4.748 mtp=3.4894 sce=1.782 active=1.80 steps=1.79 (1625s) |
| ep1 it1390 loss=6.1203 kl=0.613 aux=4.748 mtp=3.4832 sce=1.782 active=1.80 steps=1.79 (1631s) |
| ep1 it1395 loss=6.1169 kl=0.612 aux=4.748 mtp=3.4812 sce=1.781 active=1.80 steps=1.79 (1636s) |
| ep1 it1400 loss=6.1118 kl=0.613 aux=4.747 mtp=3.4783 sce=1.779 active=1.80 steps=1.79 (1642s) |
| ep1 it1405 loss=6.1071 kl=0.613 aux=4.748 mtp=3.4754 sce=1.778 active=1.80 steps=1.79 (1648s) |
| ep1 it1410 loss=6.0950 kl=0.612 aux=4.748 mtp=3.4675 sce=1.776 active=1.80 steps=1.79 (1655s) |
| ep1 it1415 loss=6.0940 kl=0.612 aux=4.748 mtp=3.4686 sce=1.774 active=1.80 steps=1.79 (1661s) |
| ep1 it1420 loss=6.0914 kl=0.612 aux=4.748 mtp=3.4662 sce=1.774 active=1.80 steps=1.79 (1668s) |
| ep1 it1425 loss=6.0841 kl=0.611 aux=4.748 mtp=3.4613 sce=1.773 active=1.80 steps=1.79 (1674s) |
| ep1 it1430 loss=6.0868 kl=0.611 aux=4.749 mtp=3.4632 sce=1.773 active=1.80 steps=1.79 (1680s) |
| ep1 it1435 loss=6.0864 kl=0.611 aux=4.749 mtp=3.4618 sce=1.774 active=1.80 steps=1.79 (1685s) |
| ep1 it1440 loss=6.0883 kl=0.611 aux=4.749 mtp=3.4615 sce=1.775 active=1.80 steps=1.79 (1692s) |
| ep1 it1445 loss=6.0894 kl=0.611 aux=4.749 mtp=3.4623 sce=1.776 active=1.80 steps=1.79 (1697s) |
| ep1 it1450 loss=6.0911 kl=0.612 aux=4.749 mtp=3.4641 sce=1.775 active=1.80 steps=1.79 (1703s) |
| ep1 it1455 loss=6.0859 kl=0.612 aux=4.749 mtp=3.4611 sce=1.774 active=1.80 steps=1.79 (1708s) |
| ep1 it1460 loss=6.0769 kl=0.612 aux=4.749 mtp=3.4553 sce=1.772 active=1.80 steps=1.79 (1714s) |
| ep1 it1465 loss=6.0748 kl=0.613 aux=4.749 mtp=3.4537 sce=1.772 active=1.80 steps=1.79 (1720s) |
| ep1 it1470 loss=6.0689 kl=0.612 aux=4.748 mtp=3.4488 sce=1.771 active=1.80 steps=1.79 (1725s) |
| ep1 it1475 loss=6.0650 kl=0.611 aux=4.748 mtp=3.4462 sce=1.770 active=1.80 steps=1.79 (1730s) |
| ep1 it1480 loss=6.0633 kl=0.611 aux=4.749 mtp=3.4440 sce=1.771 active=1.80 steps=1.79 (1735s) |
| ep1 it1485 loss=6.0564 kl=0.610 aux=4.749 mtp=3.4389 sce=1.769 active=1.80 steps=1.79 (1741s) |
| ep1 it1490 loss=6.0472 kl=0.609 aux=4.750 mtp=3.4331 sce=1.767 active=1.80 steps=1.79 (1746s) |
| ep1 it1495 loss=6.0375 kl=0.609 aux=4.750 mtp=3.4279 sce=1.764 active=1.79 steps=1.79 (1752s) |
| ep1 it1500 loss=6.0393 kl=0.610 aux=4.751 mtp=3.4287 sce=1.765 active=1.79 steps=1.79 (1758s) |
| ep1 it1505 loss=6.0359 kl=0.610 aux=4.751 mtp=3.4247 sce=1.765 active=1.79 steps=1.79 (1763s) |
| ep1 it1510 loss=6.0376 kl=0.610 aux=4.751 mtp=3.4267 sce=1.765 active=1.79 steps=1.79 (1769s) |
| ep1 it1515 loss=6.0365 kl=0.610 aux=4.750 mtp=3.4240 sce=1.766 active=1.79 steps=1.79 (1774s) |
| ep1 it1520 loss=6.0381 kl=0.609 aux=4.750 mtp=3.4251 sce=1.767 active=1.79 steps=1.79 (1779s) |
| ep1 it1525 loss=6.0296 kl=0.609 aux=4.750 mtp=3.4196 sce=1.765 active=1.79 steps=1.80 (1785s) |
| ep1 it1530 loss=6.0278 kl=0.609 aux=4.750 mtp=3.4179 sce=1.765 active=1.79 steps=1.80 (1791s) |
| ep1 it1535 loss=6.0309 kl=0.609 aux=4.750 mtp=3.4191 sce=1.766 active=1.79 steps=1.80 (1796s) |
| ep1 it1540 loss=6.0333 kl=0.610 aux=4.749 mtp=3.4206 sce=1.766 active=1.79 steps=1.80 (1802s) |
| ep1 it1545 loss=6.0301 kl=0.610 aux=4.750 mtp=3.4183 sce=1.766 active=1.79 steps=1.80 (1809s) |
| ep1 it1550 loss=6.0248 kl=0.609 aux=4.749 mtp=3.4147 sce=1.765 active=1.79 steps=1.80 (1814s) |
| ep1 it1555 loss=6.0208 kl=0.610 aux=4.749 mtp=3.4126 sce=1.764 active=1.79 steps=1.80 (1821s) |
| ep1 it1560 loss=6.0184 kl=0.611 aux=4.749 mtp=3.4108 sce=1.763 active=1.79 steps=1.80 (1827s) |
| ep1 it1565 loss=6.0148 kl=0.610 aux=4.748 mtp=3.4080 sce=1.763 active=1.79 steps=1.80 (1833s) |
| ep1 it1570 loss=6.0119 kl=0.611 aux=4.748 mtp=3.4065 sce=1.762 active=1.79 steps=1.80 (1839s) |
| ep1 it1575 loss=6.0113 kl=0.612 aux=4.747 mtp=3.4065 sce=1.761 active=1.79 steps=1.80 (1844s) |
| ep1 it1580 loss=6.0067 kl=0.612 aux=4.747 mtp=3.4015 sce=1.762 active=1.79 steps=1.80 (1850s) |
| ep1 it1585 loss=5.9984 kl=0.613 aux=4.746 mtp=3.3965 sce=1.759 active=1.79 steps=1.80 (1856s) |
| ep1 it1590 loss=5.9965 kl=0.613 aux=4.746 mtp=3.3961 sce=1.758 active=1.79 steps=1.80 (1862s) |
| ep1 it1595 loss=5.9945 kl=0.614 aux=4.745 mtp=3.3950 sce=1.758 active=1.79 steps=1.80 (1869s) |
| ep1 it1600 loss=5.9922 kl=0.614 aux=4.745 mtp=3.3928 sce=1.758 active=1.79 steps=1.80 (1875s) |
| ep1 it1605 loss=5.9896 kl=0.613 aux=4.744 mtp=3.3912 sce=1.757 active=1.79 steps=1.80 (1882s) |
| ep1 it1610 loss=5.9892 kl=0.613 aux=4.744 mtp=3.3896 sce=1.758 active=1.79 steps=1.80 (1888s) |
| ep1 it1615 loss=5.9872 kl=0.613 aux=4.743 mtp=3.3889 sce=1.757 active=1.79 steps=1.80 (1895s) |
| ep1 it1620 loss=5.9784 kl=0.612 aux=4.743 mtp=3.3829 sce=1.755 active=1.79 steps=1.80 (1901s) |
| ep1 it1625 loss=5.9777 kl=0.612 aux=4.742 mtp=3.3819 sce=1.756 active=1.79 steps=1.80 (1907s) |
| ep1 it1630 loss=5.9774 kl=0.612 aux=4.741 mtp=3.3822 sce=1.755 active=1.79 steps=1.80 (1913s) |
| ep1 it1635 loss=5.9807 kl=0.612 aux=4.741 mtp=3.3843 sce=1.756 active=1.79 steps=1.80 (1920s) |
| ep1 it1640 loss=5.9848 kl=0.613 aux=4.741 mtp=3.3871 sce=1.757 active=1.79 steps=1.80 (1926s) |
| ep1 it1645 loss=5.9927 kl=0.613 aux=4.741 mtp=3.3924 sce=1.759 active=1.79 steps=1.80 (1932s) |
| ep1 it1650 loss=5.9855 kl=0.614 aux=4.740 mtp=3.3886 sce=1.756 active=1.79 steps=1.80 (1938s) |
| ep1 it1655 loss=5.9840 kl=0.614 aux=4.741 mtp=3.3879 sce=1.756 active=1.79 steps=1.80 (1944s) |
| ep1 it1660 loss=5.9886 kl=0.613 aux=4.740 mtp=3.3912 sce=1.757 active=1.79 steps=1.80 (1951s) |
| ep1 it1665 loss=5.9875 kl=0.613 aux=4.739 mtp=3.3899 sce=1.757 active=1.79 steps=1.80 (1956s) |
| ep1 it1670 loss=5.9826 kl=0.612 aux=4.739 mtp=3.3870 sce=1.755 active=1.79 steps=1.80 (1962s) |
| ep1 it1675 loss=5.9799 kl=0.613 aux=4.738 mtp=3.3856 sce=1.755 active=1.79 steps=1.80 (1969s) |
| ep1 it1680 loss=5.9762 kl=0.612 aux=4.737 mtp=3.3804 sce=1.756 active=1.79 steps=1.80 (1975s) |
| ep1 it1685 loss=5.9719 kl=0.611 aux=4.736 mtp=3.3773 sce=1.755 active=1.79 steps=1.80 (1980s) |
| ep1 it1690 loss=5.9726 kl=0.611 aux=4.735 mtp=3.3766 sce=1.756 active=1.79 steps=1.80 (1986s) |
| ep1 it1695 loss=5.9745 kl=0.610 aux=4.735 mtp=3.3772 sce=1.757 active=1.79 steps=1.80 (1993s) |
| ep1 it1700 loss=5.9679 kl=0.611 aux=4.735 mtp=3.3721 sce=1.756 active=1.79 steps=1.80 (2000s) |
| ep1 it1705 loss=5.9637 kl=0.610 aux=4.734 mtp=3.3677 sce=1.756 active=1.79 steps=1.80 (2005s) |
| ep1 it1710 loss=5.9581 kl=0.609 aux=4.733 mtp=3.3645 sce=1.754 active=1.79 steps=1.80 (2011s) |
| ep1 it1715 loss=5.9579 kl=0.608 aux=4.732 mtp=3.3642 sce=1.755 active=1.79 steps=1.80 (2017s) |
| ep1 it1720 loss=5.9563 kl=0.607 aux=4.731 mtp=3.3628 sce=1.754 active=1.79 steps=1.80 (2023s) |
| ep1 it1725 loss=5.9550 kl=0.607 aux=4.732 mtp=3.3619 sce=1.754 active=1.79 steps=1.80 (2030s) |
| ep1 it1730 loss=5.9526 kl=0.607 aux=4.732 mtp=3.3606 sce=1.753 active=1.79 steps=1.80 (2035s) |
| ep1 it1735 loss=5.9451 kl=0.608 aux=4.731 mtp=3.3566 sce=1.751 active=1.79 steps=1.80 (2043s) |
| ep1 it1740 loss=5.9430 kl=0.607 aux=4.731 mtp=3.3549 sce=1.751 active=1.79 steps=1.80 (2048s) |
| ep1 it1745 loss=5.9427 kl=0.606 aux=4.730 mtp=3.3555 sce=1.750 active=1.79 steps=1.80 (2054s) |
| ep1 it1750 loss=5.9412 kl=0.607 aux=4.729 mtp=3.3543 sce=1.750 active=1.79 steps=1.80 (2060s) |
| ep1 it1755 loss=5.9378 kl=0.606 aux=4.728 mtp=3.3523 sce=1.749 active=1.79 steps=1.80 (2065s) |
| ep1 it1760 loss=5.9338 kl=0.606 aux=4.728 mtp=3.3499 sce=1.748 active=1.79 steps=1.80 (2070s) |
| ep1 it1765 loss=5.9295 kl=0.605 aux=4.728 mtp=3.3468 sce=1.747 active=1.79 steps=1.80 (2076s) |
| ep1 it1770 loss=5.9323 kl=0.605 aux=4.728 mtp=3.3484 sce=1.748 active=1.79 steps=1.80 (2082s) |
| ep1 it1775 loss=5.9333 kl=0.605 aux=4.728 mtp=3.3486 sce=1.749 active=1.79 steps=1.81 (2087s) |
| ep1 it1780 loss=5.9322 kl=0.605 aux=4.727 mtp=3.3472 sce=1.749 active=1.79 steps=1.81 (2094s) |
| ep1 it1785 loss=5.9265 kl=0.605 aux=4.727 mtp=3.3444 sce=1.747 active=1.79 steps=1.81 (2100s) |
| ep1 it1790 loss=5.9296 kl=0.606 aux=4.727 mtp=3.3464 sce=1.748 active=1.79 steps=1.81 (2107s) |
| ep1 it1795 loss=5.9290 kl=0.606 aux=4.727 mtp=3.3453 sce=1.748 active=1.79 steps=1.81 (2114s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11.tmp |
| [save@partial] atomic swap ok: out_2b/ckpt_2b_r11 |
| ep1 it1800 loss=5.9252 kl=0.606 aux=4.728 mtp=3.3438 sce=1.747 active=1.79 steps=1.81 (2120s) |
| ep1 it1805 loss=5.9275 kl=0.605 aux=4.727 mtp=3.3448 sce=1.747 active=1.79 steps=1.81 (2126s) |
| ep1 it1810 loss=5.9291 kl=0.606 aux=4.727 mtp=3.3460 sce=1.748 active=1.79 steps=1.81 (2133s) |
| ep1 it1815 loss=5.9348 kl=0.606 aux=4.727 mtp=3.3489 sce=1.750 active=1.79 steps=1.81 (2139s) |
| ep1 it1820 loss=5.9341 kl=0.606 aux=4.727 mtp=3.3497 sce=1.749 active=1.79 steps=1.81 (2144s) |
| ep1 it1825 loss=5.9350 kl=0.605 aux=4.726 mtp=3.3480 sce=1.751 active=1.79 steps=1.81 (2150s) |
| ep1 it1830 loss=5.9376 kl=0.605 aux=4.726 mtp=3.3482 sce=1.752 active=1.79 steps=1.81 (2155s) |
| ep1 it1835 loss=5.9334 kl=0.605 aux=4.726 mtp=3.3455 sce=1.751 active=1.79 steps=1.81 (2161s) |
| ep1 it1840 loss=5.9308 kl=0.606 aux=4.726 mtp=3.3441 sce=1.750 active=1.79 steps=1.81 (2167s) |
| ep1 it1845 loss=5.9265 kl=0.605 aux=4.726 mtp=3.3413 sce=1.749 active=1.79 steps=1.81 (2172s) |
| ep1 it1850 loss=5.9254 kl=0.606 aux=4.725 mtp=3.3415 sce=1.749 active=1.79 steps=1.81 (2179s) |
| ep1 it1855 loss=5.9273 kl=0.605 aux=4.725 mtp=3.3424 sce=1.749 active=1.79 steps=1.81 (2185s) |
| ep1 it1860 loss=5.9319 kl=0.605 aux=4.725 mtp=3.3452 sce=1.750 active=1.79 steps=1.81 (2191s) |
| ep1 it1865 loss=5.9325 kl=0.605 aux=4.725 mtp=3.3460 sce=1.750 active=1.79 steps=1.81 (2198s) |
| ep1 it1870 loss=5.9262 kl=0.605 aux=4.724 mtp=3.3422 sce=1.749 active=1.79 steps=1.81 (2205s) |
| ep1 it1875 loss=5.9255 kl=0.605 aux=4.724 mtp=3.3409 sce=1.749 active=1.79 steps=1.81 (2210s) |
| ep1 it1880 loss=5.9180 kl=0.605 aux=4.723 mtp=3.3368 sce=1.747 active=1.79 steps=1.81 (2218s) |
| ep1 it1885 loss=5.9195 kl=0.605 aux=4.724 mtp=3.3375 sce=1.747 active=1.79 steps=1.81 (2225s) |
| ep1 it1890 loss=5.9118 kl=0.605 aux=4.723 mtp=3.3312 sce=1.746 active=1.79 steps=1.81 (2231s) |
| ep1 it1895 loss=5.9111 kl=0.606 aux=4.723 mtp=3.3313 sce=1.746 active=1.79 steps=1.81 (2238s) |
| ep1 it1900 loss=5.9131 kl=0.606 aux=4.723 mtp=3.3323 sce=1.747 active=1.79 steps=1.81 (2244s) |
| ep1 it1905 loss=5.9125 kl=0.606 aux=4.722 mtp=3.3323 sce=1.746 active=1.79 steps=1.81 (2251s) |
| ep1 it1910 loss=5.9132 kl=0.607 aux=4.722 mtp=3.3328 sce=1.746 active=1.79 steps=1.81 (2257s) |
| ep1 it1915 loss=5.9163 kl=0.607 aux=4.721 mtp=3.3338 sce=1.748 active=1.79 steps=1.81 (2263s) |
| ep1 it1920 loss=5.9137 kl=0.607 aux=4.721 mtp=3.3317 sce=1.748 active=1.79 steps=1.81 (2270s) |
| ep1 it1925 loss=5.9189 kl=0.607 aux=4.721 mtp=3.3354 sce=1.748 active=1.79 steps=1.81 (2276s) |
| ep1 it1930 loss=5.9217 kl=0.606 aux=4.721 mtp=3.3356 sce=1.750 active=1.79 steps=1.81 (2282s) |
| ep1 it1935 loss=5.9259 kl=0.606 aux=4.721 mtp=3.3387 sce=1.751 active=1.79 steps=1.81 (2287s) |
| ep1 it1940 loss=5.9184 kl=0.607 aux=4.721 mtp=3.3348 sce=1.748 active=1.79 steps=1.81 (2295s) |
| ep1 it1945 loss=5.9162 kl=0.608 aux=4.721 mtp=3.3334 sce=1.748 active=1.79 steps=1.81 (2302s) |
| ep1 it1950 loss=5.9108 kl=0.608 aux=4.721 mtp=3.3302 sce=1.746 active=1.79 steps=1.81 (2308s) |
| ep1 it1955 loss=5.9135 kl=0.608 aux=4.720 mtp=3.3320 sce=1.747 active=1.79 steps=1.81 (2315s) |
| ep1 it1960 loss=5.9109 kl=0.608 aux=4.720 mtp=3.3306 sce=1.746 active=1.79 steps=1.81 (2322s) |
| ep1 it1965 loss=5.9077 kl=0.608 aux=4.719 mtp=3.3281 sce=1.746 active=1.79 steps=1.81 (2329s) |
| ep1 it1970 loss=5.9005 kl=0.608 aux=4.719 mtp=3.3231 sce=1.744 active=1.79 steps=1.81 (2335s) |
| ep1 it1975 loss=5.8915 kl=0.608 aux=4.719 mtp=3.3173 sce=1.742 active=1.79 steps=1.81 (2342s) |
| ep2 it5 loss=3.1200 kl=0.468 aux=4.972 mtp=1.4789 sce=1.137 active=1.77 steps=2.07 (2348s) |
| ep2 it10 loss=4.4787 kl=0.539 aux=4.900 mtp=2.3451 sce=1.461 active=1.77 steps=2.01 (2355s) |
| ep2 it15 loss=3.7383 kl=0.550 aux=4.800 mtp=1.8205 sce=1.321 active=1.74 steps=1.99 (2362s) |
| ep2 it20 loss=4.3843 kl=0.547 aux=4.733 mtp=2.1399 sce=1.536 active=1.75 steps=1.96 (2368s) |
| ep2 it25 loss=4.4113 kl=0.604 aux=4.739 mtp=2.2045 sce=1.507 active=1.75 steps=1.98 (2375s) |
| ep2 it30 loss=4.3759 kl=0.589 aux=4.738 mtp=2.1656 sce=1.510 active=1.76 steps=1.98 (2382s) |
| ep2 it35 loss=4.6038 kl=0.608 aux=4.710 mtp=2.3446 sce=1.539 active=1.76 steps=1.96 (2389s) |
| ep2 it40 loss=4.5194 kl=0.579 aux=4.690 mtp=2.2915 sce=1.519 active=1.75 steps=1.96 (2396s) |
| ep2 it45 loss=4.5794 kl=0.583 aux=4.662 mtp=2.3258 sce=1.537 active=1.75 steps=1.96 (2403s) |
| ep2 it50 loss=4.4796 kl=0.612 aux=4.666 mtp=2.2845 sce=1.497 active=1.74 steps=1.95 (2411s) |
| ep2 it55 loss=4.5698 kl=0.628 aux=4.645 mtp=2.3366 sce=1.522 active=1.74 steps=1.95 (2418s) |
| ep2 it60 loss=4.7562 kl=0.636 aux=4.644 mtp=2.4622 sce=1.563 active=1.75 steps=1.94 (2424s) |
| ep2 it65 loss=4.6554 kl=0.632 aux=4.633 mtp=2.4201 sce=1.523 active=1.75 steps=1.93 (2431s) |
| ep2 it70 loss=4.6157 kl=0.676 aux=4.624 mtp=2.3773 sce=1.524 active=1.75 steps=1.93 (2440s) |
| ep2 it75 loss=4.5677 kl=0.666 aux=4.619 mtp=2.3509 sce=1.510 active=1.75 steps=1.93 (2446s) |
| ep2 it80 loss=4.6729 kl=0.656 aux=4.615 mtp=2.4076 sce=1.543 active=1.75 steps=1.93 (2452s) |
| ep2 it85 loss=4.6098 kl=0.648 aux=4.619 mtp=2.3761 sce=1.522 active=1.75 steps=1.93 (2459s) |
| ep2 it90 loss=4.5276 kl=0.652 aux=4.613 mtp=2.3318 sce=1.497 active=1.75 steps=1.94 (2467s) |
| ep2 it95 loss=4.4636 kl=0.652 aux=4.605 mtp=2.2858 sce=1.485 active=1.75 steps=1.94 (2474s) |
| ep2 it100 loss=4.4294 kl=0.638 aux=4.599 mtp=2.2722 sce=1.472 active=1.76 steps=1.94 (2480s) |
| ep2 it105 loss=4.3963 kl=0.640 aux=4.606 mtp=2.2674 sce=1.452 active=1.75 steps=1.94 (2488s) |
| ep2 it110 loss=4.3337 kl=0.635 aux=4.601 mtp=2.2089 sce=1.451 active=1.75 steps=1.95 (2494s) |
| ep2 it115 loss=4.3283 kl=0.638 aux=4.601 mtp=2.2165 sce=1.441 active=1.76 steps=1.95 (2502s) |
| ep2 it120 loss=4.3656 kl=0.645 aux=4.595 mtp=2.2418 sce=1.450 active=1.76 steps=1.94 (2509s) |
| ep2 it125 loss=4.3986 kl=0.638 aux=4.597 mtp=2.2730 sce=1.451 active=1.76 steps=1.94 (2516s) |
| ep2 it130 loss=4.3960 kl=0.631 aux=4.599 mtp=2.2763 sce=1.447 active=1.76 steps=1.94 (2522s) |
| ep2 it135 loss=4.3625 kl=0.626 aux=4.594 mtp=2.2477 sce=1.444 active=1.76 steps=1.94 (2528s) |
| ep2 it140 loss=4.3606 kl=0.621 aux=4.597 mtp=2.2472 sce=1.443 active=1.76 steps=1.94 (2535s) |
| ep2 it145 loss=4.3683 kl=0.611 aux=4.595 mtp=2.2309 sce=1.461 active=1.76 steps=1.95 (2542s) |
| ep2 it150 loss=4.4311 kl=0.607 aux=4.594 mtp=2.2734 sce=1.474 active=1.76 steps=1.95 (2550s) |
| ep2 it155 loss=4.4960 kl=0.613 aux=4.588 mtp=2.3098 sce=1.494 active=1.77 steps=1.95 (2557s) |
| ep2 it160 loss=4.5385 kl=0.618 aux=4.587 mtp=2.3459 sce=1.498 active=1.77 steps=1.95 (2567s) |
| ep2 it165 loss=4.5689 kl=0.615 aux=4.591 mtp=2.3591 sce=1.511 active=1.77 steps=1.95 (2575s) |
| ep2 it170 loss=4.5723 kl=0.617 aux=4.592 mtp=2.3670 sce=1.507 active=1.77 steps=1.95 (2583s) |
| ep2 it175 loss=4.6868 kl=0.614 aux=4.594 mtp=2.4389 sce=1.536 active=1.77 steps=1.95 (2590s) |
| ep2 it180 loss=4.6955 kl=0.606 aux=4.602 mtp=2.4397 sce=1.542 active=1.77 steps=1.96 (2597s) |
| ep2 it185 loss=4.7148 kl=0.607 aux=4.606 mtp=2.4562 sce=1.544 active=1.77 steps=1.96 (2606s) |
| ep2 it190 loss=4.7560 kl=0.620 aux=4.606 mtp=2.4907 sce=1.548 active=1.77 steps=1.96 (2615s) |
| ep2 it195 loss=4.7538 kl=0.625 aux=4.607 mtp=2.4997 sce=1.540 active=1.77 steps=1.95 (2623s) |
| ep2 it200 loss=4.7276 kl=0.624 aux=4.607 mtp=2.4848 sce=1.533 active=1.77 steps=1.96 (2630s) |
| ep2 it205 loss=4.7737 kl=0.622 aux=4.605 mtp=2.5118 sce=1.546 active=1.77 steps=1.96 (2637s) |
| ep2 it210 loss=4.7241 kl=0.629 aux=4.600 mtp=2.4867 sce=1.529 active=1.77 steps=1.95 (2645s) |
| ep2 it215 loss=4.7620 kl=0.633 aux=4.604 mtp=2.5159 sce=1.534 active=1.77 steps=1.95 (2654s) |
| ep2 it220 loss=4.7863 kl=0.635 aux=4.602 mtp=2.5254 sce=1.544 active=1.77 steps=1.95 (2661s) |
| ep2 it225 loss=4.7908 kl=0.627 aux=4.599 mtp=2.5258 sce=1.547 active=1.77 steps=1.95 (2668s) |
| ep2 it230 loss=4.8355 kl=0.623 aux=4.603 mtp=2.5480 sce=1.562 active=1.77 steps=1.94 (2676s) |
| ep2 it235 loss=4.8567 kl=0.619 aux=4.604 mtp=2.5648 sce=1.565 active=1.78 steps=1.94 (2683s) |
| ep2 it240 loss=4.8302 kl=0.622 aux=4.606 mtp=2.5468 sce=1.560 active=1.78 steps=1.95 (2690s) |
| ep2 it245 loss=4.8077 kl=0.621 aux=4.604 mtp=2.5336 sce=1.554 active=1.78 steps=1.94 (2698s) |
| ep2 it250 loss=4.7487 kl=0.616 aux=4.602 mtp=2.4973 sce=1.538 active=1.77 steps=1.95 (2706s) |
| ep2 it255 loss=4.7392 kl=0.607 aux=4.600 mtp=2.4793 sce=1.544 active=1.77 steps=1.95 (2713s) |
| ep2 it260 loss=4.7395 kl=0.611 aux=4.596 mtp=2.4804 sce=1.544 active=1.77 steps=1.96 (2722s) |
| ep2 it265 loss=4.7041 kl=0.606 aux=4.595 mtp=2.4613 sce=1.532 active=1.77 steps=1.95 (2729s) |
| ep2 it270 loss=4.6679 kl=0.603 aux=4.593 mtp=2.4355 sce=1.526 active=1.77 steps=1.96 (2736s) |
| ep2 it275 loss=4.6747 kl=0.610 aux=4.590 mtp=2.4395 sce=1.528 active=1.77 steps=1.96 (2745s) |
| ep2 it280 loss=4.6401 kl=0.603 aux=4.587 mtp=2.4158 sce=1.520 active=1.77 steps=1.96 (2752s) |
| ep2 it285 loss=4.6683 kl=0.606 aux=4.587 mtp=2.4341 sce=1.527 active=1.77 steps=1.96 (2758s) |
| ep2 it290 loss=4.6762 kl=0.605 aux=4.583 mtp=2.4396 sce=1.528 active=1.77 steps=1.96 (2766s) |
| ep2 it295 loss=4.6708 kl=0.606 aux=4.584 mtp=2.4360 sce=1.527 active=1.77 steps=1.95 (2775s) |
| ep2 it300 loss=4.6462 kl=0.605 aux=4.586 mtp=2.4244 sce=1.518 active=1.77 steps=1.95 (2782s) |
| ep2 it305 loss=4.6410 kl=0.601 aux=4.589 mtp=2.4202 sce=1.517 active=1.77 steps=1.95 (2789s) |
| ep2 it310 loss=4.6953 kl=0.601 aux=4.592 mtp=2.4582 sce=1.528 active=1.77 steps=1.95 (2795s) |
| ep2 it315 loss=4.7000 kl=0.600 aux=4.592 mtp=2.4605 sce=1.530 active=1.77 steps=1.95 (2802s) |
| ep2 it320 loss=4.7060 kl=0.598 aux=4.593 mtp=2.4621 sce=1.533 active=1.77 steps=1.95 (2808s) |
| ep2 it325 loss=4.6901 kl=0.599 aux=4.597 mtp=2.4540 sce=1.528 active=1.77 steps=1.95 (2816s) |
| ep2 it330 loss=4.6860 kl=0.596 aux=4.600 mtp=2.4510 sce=1.527 active=1.77 steps=1.95 (2822s) |
| ep2 it335 loss=4.6960 kl=0.595 aux=4.599 mtp=2.4601 sce=1.527 active=1.78 steps=1.96 (2829s) |
| ep2 it340 loss=4.6787 kl=0.595 aux=4.599 mtp=2.4510 sce=1.522 active=1.78 steps=1.96 (2837s) |
| ep2 it345 loss=4.6918 kl=0.595 aux=4.598 mtp=2.4552 sce=1.528 active=1.78 steps=1.96 (2845s) |
| ep2 it350 loss=4.6713 kl=0.595 aux=4.597 mtp=2.4396 sce=1.525 active=1.77 steps=1.96 (2852s) |
| ep2 it355 loss=4.6523 kl=0.590 aux=4.599 mtp=2.4288 sce=1.519 active=1.77 steps=1.96 (2857s) |
| ep2 it360 loss=4.6483 kl=0.588 aux=4.600 mtp=2.4283 sce=1.517 active=1.77 steps=1.96 (2865s) |
| ep2 it365 loss=4.6358 kl=0.585 aux=4.598 mtp=2.4192 sce=1.514 active=1.77 steps=1.96 (2871s) |
| ep2 it370 loss=4.6319 kl=0.586 aux=4.600 mtp=2.4173 sce=1.513 active=1.77 steps=1.96 (2878s) |
| ep2 it375 loss=4.6246 kl=0.586 aux=4.604 mtp=2.4117 sce=1.512 active=1.77 steps=1.96 (2884s) |
| ep2 it380 loss=4.6249 kl=0.582 aux=4.607 mtp=2.4078 sce=1.515 active=1.77 steps=1.96 (2890s) |
| ep2 it385 loss=4.6385 kl=0.581 aux=4.606 mtp=2.4182 sce=1.517 active=1.77 steps=1.96 (2897s) |
| ep2 it390 loss=4.6464 kl=0.583 aux=4.605 mtp=2.4239 sce=1.518 active=1.77 steps=1.96 (2904s) |
| ep2 it395 loss=4.6504 kl=0.585 aux=4.609 mtp=2.4276 sce=1.519 active=1.78 steps=1.96 (2911s) |
| ep2 it400 loss=4.6621 kl=0.583 aux=4.612 mtp=2.4329 sce=1.523 active=1.78 steps=1.96 (2918s) |
| ep2 it405 loss=4.6782 kl=0.585 aux=4.613 mtp=2.4481 sce=1.523 active=1.78 steps=1.96 (2926s) |
| ep2 it410 loss=4.6538 kl=0.584 aux=4.610 mtp=2.4319 sce=1.518 active=1.77 steps=1.96 (2931s) |
| ep2 it415 loss=4.6500 kl=0.581 aux=4.611 mtp=2.4310 sce=1.516 active=1.77 steps=1.96 (2938s) |
| ep2 it420 loss=4.6577 kl=0.581 aux=4.609 mtp=2.4375 sce=1.517 active=1.77 steps=1.96 (2944s) |
| ep2 it425 loss=4.6497 kl=0.580 aux=4.608 mtp=2.4316 sce=1.515 active=1.78 steps=1.96 (2951s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11.tmp |
| [save@partial] atomic swap ok: out_2b/ckpt_2b_r11 |
| ep2 it430 loss=4.6308 kl=0.575 aux=4.606 mtp=2.4129 sce=1.516 active=1.78 steps=1.96 (2957s) |
| ep2 it435 loss=4.6232 kl=0.572 aux=4.603 mtp=2.4075 sce=1.514 active=1.78 steps=1.96 (2964s) |
| ep2 it440 loss=4.6559 kl=0.573 aux=4.606 mtp=2.4276 sce=1.522 active=1.78 steps=1.96 (2971s) |
| ep2 it445 loss=4.6742 kl=0.572 aux=4.605 mtp=2.4394 sce=1.527 active=1.78 steps=1.96 (2979s) |
| ep2 it450 loss=4.6635 kl=0.574 aux=4.605 mtp=2.4364 sce=1.521 active=1.78 steps=1.96 (2988s) |
| ep2 it455 loss=4.6457 kl=0.575 aux=4.603 mtp=2.4266 sce=1.516 active=1.78 steps=1.96 (2994s) |
| ep2 it460 loss=4.6486 kl=0.574 aux=4.602 mtp=2.4225 sce=1.521 active=1.78 steps=1.96 (3001s) |
| ep2 it465 loss=4.6485 kl=0.574 aux=4.601 mtp=2.4251 sce=1.519 active=1.78 steps=1.96 (3008s) |
| ep2 it470 loss=4.6735 kl=0.572 aux=4.601 mtp=2.4406 sce=1.525 active=1.78 steps=1.96 (3014s) |
| ep2 it475 loss=4.6930 kl=0.576 aux=4.601 mtp=2.4534 sce=1.529 active=1.78 steps=1.96 (3022s) |
| ep2 it480 loss=4.7014 kl=0.576 aux=4.601 mtp=2.4588 sce=1.531 active=1.78 steps=1.96 (3029s) |
| ep2 it485 loss=4.6900 kl=0.574 aux=4.601 mtp=2.4528 sce=1.528 active=1.78 steps=1.96 (3036s) |
| ep2 it490 loss=4.6811 kl=0.576 aux=4.601 mtp=2.4487 sce=1.525 active=1.78 steps=1.96 (3044s) |
| ep2 it495 loss=4.6717 kl=0.575 aux=4.600 mtp=2.4423 sce=1.523 active=1.78 steps=1.96 (3050s) |
| ep2 it500 loss=4.6713 kl=0.574 aux=4.599 mtp=2.4426 sce=1.522 active=1.78 steps=1.96 (3057s) |
| ep2 it505 loss=4.6870 kl=0.574 aux=4.597 mtp=2.4527 sce=1.526 active=1.78 steps=1.96 (3064s) |
| ep2 it510 loss=4.7013 kl=0.574 aux=4.596 mtp=2.4603 sce=1.530 active=1.78 steps=1.96 (3070s) |
| ep2 it515 loss=4.7035 kl=0.577 aux=4.594 mtp=2.4617 sce=1.531 active=1.78 steps=1.96 (3078s) |
| ep2 it520 loss=4.6914 kl=0.576 aux=4.595 mtp=2.4547 sce=1.527 active=1.78 steps=1.96 (3086s) |
| ep2 it525 loss=4.6975 kl=0.577 aux=4.595 mtp=2.4631 sce=1.526 active=1.78 steps=1.96 (3092s) |
| ep2 it530 loss=4.6832 kl=0.580 aux=4.593 mtp=2.4569 sce=1.520 active=1.78 steps=1.96 (3101s) |
| ep2 it535 loss=4.6682 kl=0.581 aux=4.592 mtp=2.4501 sce=1.514 active=1.78 steps=1.95 (3109s) |
| ep2 it540 loss=4.6730 kl=0.580 aux=4.592 mtp=2.4538 sce=1.515 active=1.78 steps=1.95 (3117s) |
| ep2 it545 loss=4.6720 kl=0.582 aux=4.591 mtp=2.4544 sce=1.514 active=1.78 steps=1.95 (3124s) |
| ep2 it550 loss=4.6730 kl=0.581 aux=4.589 mtp=2.4569 sce=1.513 active=1.78 steps=1.95 (3131s) |
| ep2 it555 loss=4.6707 kl=0.580 aux=4.587 mtp=2.4545 sce=1.513 active=1.78 steps=1.95 (3138s) |
| ep2 it560 loss=4.6767 kl=0.578 aux=4.586 mtp=2.4562 sce=1.517 active=1.78 steps=1.96 (3147s) |
| ep2 it565 loss=4.6843 kl=0.577 aux=4.585 mtp=2.4640 sce=1.516 active=1.78 steps=1.96 (3158s) |
| ep2 it570 loss=4.6900 kl=0.575 aux=4.584 mtp=2.4665 sce=1.518 active=1.78 steps=1.96 (3167s) |
| ep2 it575 loss=4.6751 kl=0.573 aux=4.586 mtp=2.4586 sce=1.514 active=1.78 steps=1.96 (3178s) |
| ep2 it580 loss=4.6700 kl=0.572 aux=4.584 mtp=2.4568 sce=1.512 active=1.78 steps=1.96 (3185s) |
| ep2 it585 loss=4.6650 kl=0.572 aux=4.583 mtp=2.4526 sce=1.511 active=1.78 steps=1.96 (3193s) |
| ep2 it590 loss=4.6828 kl=0.571 aux=4.583 mtp=2.4604 sce=1.518 active=1.78 steps=1.96 (3200s) |
| ep2 it595 loss=4.6869 kl=0.571 aux=4.582 mtp=2.4630 sce=1.519 active=1.78 steps=1.96 (3207s) |
| ep2 it600 loss=4.7145 kl=0.572 aux=4.582 mtp=2.4828 sce=1.524 active=1.78 steps=1.96 (3214s) |
| ep2 it605 loss=4.7197 kl=0.572 aux=4.582 mtp=2.4847 sce=1.526 active=1.78 steps=1.95 (3220s) |
| ep2 it610 loss=4.7201 kl=0.570 aux=4.580 mtp=2.4832 sce=1.527 active=1.78 steps=1.95 (3226s) |
| ep2 it615 loss=4.7204 kl=0.573 aux=4.580 mtp=2.4839 sce=1.527 active=1.78 steps=1.95 (3231s) |
| ep2 it620 loss=4.7094 kl=0.572 aux=4.580 mtp=2.4768 sce=1.524 active=1.78 steps=1.95 (3238s) |
| ep2 it625 loss=4.7109 kl=0.571 aux=4.579 mtp=2.4777 sce=1.524 active=1.78 steps=1.95 (3244s) |
| ep2 it630 loss=4.7146 kl=0.576 aux=4.579 mtp=2.4823 sce=1.523 active=1.78 steps=1.95 (3251s) |
| ep2 it635 loss=4.7219 kl=0.575 aux=4.579 mtp=2.4888 sce=1.524 active=1.78 steps=1.95 (3257s) |
| ep2 it640 loss=4.7380 kl=0.576 aux=4.580 mtp=2.4971 sce=1.529 active=1.78 steps=1.95 (3264s) |
| ep2 it645 loss=4.7398 kl=0.577 aux=4.581 mtp=2.4996 sce=1.529 active=1.78 steps=1.95 (3270s) |
| ep2 it650 loss=4.7355 kl=0.576 aux=4.580 mtp=2.4974 sce=1.527 active=1.78 steps=1.95 (3277s) |
| ep2 it655 loss=4.7211 kl=0.577 aux=4.580 mtp=2.4899 sce=1.522 active=1.78 steps=1.95 (3284s) |
| ep2 it660 loss=4.7060 kl=0.574 aux=4.579 mtp=2.4798 sce=1.519 active=1.78 steps=1.95 (3291s) |
| ep2 it665 loss=4.7063 kl=0.575 aux=4.580 mtp=2.4797 sce=1.519 active=1.78 steps=1.95 (3298s) |
| ep2 it670 loss=4.6911 kl=0.575 aux=4.580 mtp=2.4725 sce=1.514 active=1.78 steps=1.95 (3305s) |
| ep2 it675 loss=4.6863 kl=0.575 aux=4.579 mtp=2.4694 sce=1.513 active=1.78 steps=1.95 (3312s) |
| ep2 it680 loss=4.6962 kl=0.576 aux=4.578 mtp=2.4768 sce=1.514 active=1.78 steps=1.95 (3319s) |
| ep2 it685 loss=4.7016 kl=0.574 aux=4.577 mtp=2.4795 sce=1.516 active=1.78 steps=1.95 (3326s) |
| ep2 it690 loss=4.7051 kl=0.577 aux=4.577 mtp=2.4830 sce=1.516 active=1.78 steps=1.95 (3335s) |
| ep2 it695 loss=4.7086 kl=0.577 aux=4.576 mtp=2.4842 sce=1.518 active=1.78 steps=1.95 (3341s) |
| ep2 it700 loss=4.7161 kl=0.578 aux=4.576 mtp=2.4891 sce=1.519 active=1.78 steps=1.95 (3349s) |
| ep2 it705 loss=4.7132 kl=0.578 aux=4.575 mtp=2.4847 sce=1.520 active=1.78 steps=1.95 (3355s) |
| ep2 it710 loss=4.7007 kl=0.578 aux=4.574 mtp=2.4755 sce=1.518 active=1.78 steps=1.95 (3362s) |
| ep2 it715 loss=4.6946 kl=0.579 aux=4.572 mtp=2.4735 sce=1.516 active=1.78 steps=1.95 (3370s) |
| ep2 it720 loss=4.6901 kl=0.580 aux=4.571 mtp=2.4719 sce=1.514 active=1.78 steps=1.95 (3378s) |
| ep2 it725 loss=4.6764 kl=0.579 aux=4.570 mtp=2.4629 sce=1.511 active=1.78 steps=1.95 (3385s) |
| ep2 it730 loss=4.6710 kl=0.578 aux=4.569 mtp=2.4590 sce=1.510 active=1.78 steps=1.95 (3391s) |
| ep2 it735 loss=4.6736 kl=0.579 aux=4.567 mtp=2.4605 sce=1.510 active=1.78 steps=1.95 (3399s) |
| ep2 it740 loss=4.6535 kl=0.579 aux=4.566 mtp=2.4493 sce=1.504 active=1.78 steps=1.95 (3406s) |
| ep2 it745 loss=4.6568 kl=0.580 aux=4.566 mtp=2.4483 sce=1.507 active=1.78 steps=1.95 (3414s) |
| ep2 it750 loss=4.6679 kl=0.582 aux=4.565 mtp=2.4565 sce=1.509 active=1.78 steps=1.95 (3422s) |
| ep2 it755 loss=4.6794 kl=0.582 aux=4.566 mtp=2.4625 sce=1.513 active=1.78 steps=1.95 (3428s) |
| ep2 it760 loss=4.6841 kl=0.581 aux=4.565 mtp=2.4628 sce=1.516 active=1.78 steps=1.95 (3435s) |
| ep2 it765 loss=4.6793 kl=0.581 aux=4.565 mtp=2.4588 sce=1.515 active=1.78 steps=1.95 (3442s) |
| ep2 it770 loss=4.6861 kl=0.582 aux=4.564 mtp=2.4639 sce=1.516 active=1.78 steps=1.95 (3449s) |
| ep2 it775 loss=4.6912 kl=0.585 aux=4.564 mtp=2.4663 sce=1.518 active=1.78 steps=1.95 (3455s) |
| ep2 it780 loss=4.7053 kl=0.586 aux=4.563 mtp=2.4751 sce=1.522 active=1.78 steps=1.95 (3464s) |
| ep2 it785 loss=4.7138 kl=0.585 aux=4.563 mtp=2.4820 sce=1.523 active=1.78 steps=1.95 (3471s) |
| ep2 it790 loss=4.7185 kl=0.585 aux=4.562 mtp=2.4830 sce=1.525 active=1.78 steps=1.95 (3479s) |
| ep2 it795 loss=4.7288 kl=0.588 aux=4.560 mtp=2.4906 sce=1.527 active=1.78 steps=1.95 (3487s) |
| ep2 it800 loss=4.7263 kl=0.588 aux=4.560 mtp=2.4901 sce=1.526 active=1.78 steps=1.95 (3494s) |
| ep2 it805 loss=4.7429 kl=0.587 aux=4.560 mtp=2.5018 sce=1.529 active=1.78 steps=1.95 (3502s) |
| ep2 it810 loss=4.7439 kl=0.586 aux=4.560 mtp=2.5019 sce=1.530 active=1.78 steps=1.95 (3509s) |
| ep2 it815 loss=4.7535 kl=0.586 aux=4.560 mtp=2.5076 sce=1.533 active=1.78 steps=1.95 (3516s) |
| ep2 it820 loss=4.7461 kl=0.586 aux=4.560 mtp=2.5043 sce=1.530 active=1.78 steps=1.95 (3523s) |
| ep2 it825 loss=4.7427 kl=0.584 aux=4.560 mtp=2.5020 sce=1.529 active=1.78 steps=1.95 (3530s) |
| ep2 it830 loss=4.7435 kl=0.586 aux=4.559 mtp=2.5025 sce=1.529 active=1.78 steps=1.95 (3538s) |
| ep2 it835 loss=4.7427 kl=0.586 aux=4.559 mtp=2.5049 sce=1.527 active=1.78 steps=1.95 (3545s) |
| ep2 it840 loss=4.7566 kl=0.588 aux=4.558 mtp=2.5142 sce=1.530 active=1.78 steps=1.95 (3553s) |
| ep2 it845 loss=4.7580 kl=0.589 aux=4.557 mtp=2.5149 sce=1.531 active=1.78 steps=1.95 (3560s) |
| ep2 it850 loss=4.7538 kl=0.590 aux=4.556 mtp=2.5127 sce=1.529 active=1.78 steps=1.95 (3568s) |
| ep2 it855 loss=4.7584 kl=0.590 aux=4.556 mtp=2.5161 sce=1.530 active=1.78 steps=1.95 (3575s) |
| ep2 it860 loss=4.7487 kl=0.589 aux=4.555 mtp=2.5108 sce=1.527 active=1.78 steps=1.95 (3583s) |
| ep2 it865 loss=4.7408 kl=0.588 aux=4.553 mtp=2.5054 sce=1.525 active=1.78 steps=1.95 (3590s) |
| ep2 it870 loss=4.7475 kl=0.588 aux=4.552 mtp=2.5083 sce=1.528 active=1.78 steps=1.95 (3596s) |
| ep2 it875 loss=4.7385 kl=0.586 aux=4.552 mtp=2.5014 sce=1.527 active=1.78 steps=1.95 (3603s) |
| ep2 it880 loss=4.7350 kl=0.587 aux=4.551 mtp=2.4994 sce=1.526 active=1.78 steps=1.95 (3610s) |
| ep2 it885 loss=4.7502 kl=0.585 aux=4.551 mtp=2.5070 sce=1.531 active=1.78 steps=1.95 (3617s) |
| ep2 it890 loss=4.7560 kl=0.585 aux=4.551 mtp=2.5115 sce=1.532 active=1.78 steps=1.95 (3625s) |
| ep2 it895 loss=4.7544 kl=0.586 aux=4.551 mtp=2.5107 sce=1.531 active=1.78 steps=1.95 (3632s) |
| ep2 it900 loss=4.7507 kl=0.584 aux=4.551 mtp=2.5061 sce=1.532 active=1.78 steps=1.95 (3638s) |
| ep2 it905 loss=4.7458 kl=0.585 aux=4.552 mtp=2.5042 sce=1.530 active=1.78 steps=1.95 (3646s) |
| ep2 it910 loss=4.7530 kl=0.585 aux=4.553 mtp=2.5095 sce=1.531 active=1.78 steps=1.95 (3653s) |
| ep2 it915 loss=4.7490 kl=0.584 aux=4.553 mtp=2.5070 sce=1.530 active=1.78 steps=1.95 (3661s) |
| ep2 it920 loss=4.7429 kl=0.585 aux=4.552 mtp=2.5040 sce=1.528 active=1.78 steps=1.95 (3670s) |
| ep2 it925 loss=4.7424 kl=0.586 aux=4.551 mtp=2.5048 sce=1.527 active=1.78 steps=1.95 (3678s) |
| ep2 it930 loss=4.7424 kl=0.586 aux=4.551 mtp=2.5055 sce=1.526 active=1.78 steps=1.95 (3685s) |
| ep2 it935 loss=4.7495 kl=0.585 aux=4.551 mtp=2.5102 sce=1.528 active=1.78 steps=1.95 (3693s) |
| ep2 it940 loss=4.7401 kl=0.583 aux=4.551 mtp=2.5029 sce=1.527 active=1.78 steps=1.95 (3700s) |
| ep2 it945 loss=4.7432 kl=0.583 aux=4.550 mtp=2.5053 sce=1.527 active=1.78 steps=1.95 (3708s) |
| ep2 it950 loss=4.7426 kl=0.585 aux=4.550 mtp=2.5058 sce=1.526 active=1.78 steps=1.95 (3715s) |
| ep2 it955 loss=4.7393 kl=0.585 aux=4.549 mtp=2.5044 sce=1.525 active=1.78 steps=1.95 (3721s) |
| ep2 it960 loss=4.7413 kl=0.585 aux=4.549 mtp=2.5048 sce=1.526 active=1.78 steps=1.95 (3728s) |
| ep2 it965 loss=4.7381 kl=0.586 aux=4.550 mtp=2.5005 sce=1.527 active=1.78 steps=1.95 (3735s) |
| ep2 it970 loss=4.7272 kl=0.586 aux=4.550 mtp=2.4948 sce=1.523 active=1.78 steps=1.95 (3742s) |
| ep2 it975 loss=4.7312 kl=0.587 aux=4.550 mtp=2.4976 sce=1.524 active=1.78 steps=1.95 (3749s) |
| ep2 it980 loss=4.7230 kl=0.586 aux=4.549 mtp=2.4901 sce=1.524 active=1.78 steps=1.95 (3755s) |
| ep2 it985 loss=4.7249 kl=0.586 aux=4.548 mtp=2.4923 sce=1.524 active=1.78 steps=1.95 (3762s) |
| ep2 it990 loss=4.7123 kl=0.585 aux=4.547 mtp=2.4854 sce=1.520 active=1.78 steps=1.95 (3768s) |
| ep2 it995 loss=4.7127 kl=0.584 aux=4.545 mtp=2.4856 sce=1.520 active=1.78 steps=1.95 (3775s) |
| ep2 it1000 loss=4.7086 kl=0.583 aux=4.546 mtp=2.4817 sce=1.520 active=1.78 steps=1.95 (3781s) |
| ep2 it1005 loss=4.7165 kl=0.584 aux=4.545 mtp=2.4878 sce=1.521 active=1.78 steps=1.95 (3788s) |
| ep2 it1010 loss=4.7158 kl=0.583 aux=4.544 mtp=2.4873 sce=1.521 active=1.78 steps=1.95 (3795s) |
| ep2 it1015 loss=4.7127 kl=0.585 aux=4.543 mtp=2.4860 sce=1.519 active=1.78 steps=1.95 (3803s) |
| ep2 it1020 loss=4.7078 kl=0.584 aux=4.542 mtp=2.4827 sce=1.518 active=1.78 steps=1.95 (3810s) |
| ep2 it1025 loss=4.7007 kl=0.584 aux=4.541 mtp=2.4787 sce=1.516 active=1.78 steps=1.95 (3819s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11.tmp |
| [save@partial] atomic swap ok: out_2b/ckpt_2b_r11 |
| ep2 it1030 loss=4.6920 kl=0.583 aux=4.541 mtp=2.4736 sce=1.514 active=1.78 steps=1.95 (3827s) |
| ep2 it1035 loss=4.7019 kl=0.582 aux=4.540 mtp=2.4782 sce=1.517 active=1.78 steps=1.95 (3834s) |
| ep2 it1040 loss=4.7097 kl=0.583 aux=4.541 mtp=2.4842 sce=1.519 active=1.78 steps=1.95 (3841s) |
| ep2 it1045 loss=4.7156 kl=0.583 aux=4.539 mtp=2.4861 sce=1.521 active=1.79 steps=1.95 (3847s) |
| ep2 it1050 loss=4.7097 kl=0.584 aux=4.538 mtp=2.4813 sce=1.521 active=1.78 steps=1.95 (3854s) |
| ep2 it1055 loss=4.7116 kl=0.584 aux=4.537 mtp=2.4839 sce=1.520 active=1.79 steps=1.95 (3863s) |
| ep2 it1060 loss=4.7115 kl=0.584 aux=4.537 mtp=2.4855 sce=1.519 active=1.78 steps=1.95 (3868s) |
| ep2 it1065 loss=4.7156 kl=0.584 aux=4.536 mtp=2.4869 sce=1.521 active=1.79 steps=1.95 (3875s) |
| ep2 it1070 loss=4.7093 kl=0.584 aux=4.535 mtp=2.4824 sce=1.519 active=1.79 steps=1.95 (3881s) |
| ep2 it1075 loss=4.7049 kl=0.584 aux=4.534 mtp=2.4803 sce=1.518 active=1.79 steps=1.95 (3886s) |
| ep2 it1080 loss=4.6976 kl=0.584 aux=4.533 mtp=2.4749 sce=1.517 active=1.79 steps=1.95 (3893s) |
| ep2 it1085 loss=4.7127 kl=0.584 aux=4.533 mtp=2.4839 sce=1.521 active=1.79 steps=1.95 (3900s) |
| ep2 it1090 loss=4.7166 kl=0.584 aux=4.534 mtp=2.4875 sce=1.521 active=1.79 steps=1.95 (3907s) |
| ep2 it1095 loss=4.7195 kl=0.584 aux=4.534 mtp=2.4884 sce=1.522 active=1.79 steps=1.95 (3914s) |
| ep2 it1100 loss=4.7209 kl=0.586 aux=4.534 mtp=2.4907 sce=1.522 active=1.79 steps=1.95 (3920s) |
| ep2 it1105 loss=4.7195 kl=0.587 aux=4.534 mtp=2.4911 sce=1.520 active=1.79 steps=1.95 (3930s) |
| ep2 it1110 loss=4.7257 kl=0.588 aux=4.533 mtp=2.4935 sce=1.523 active=1.79 steps=1.95 (3938s) |
| ep2 it1115 loss=4.7264 kl=0.587 aux=4.533 mtp=2.4941 sce=1.523 active=1.79 steps=1.95 (3947s) |
| ep2 it1120 loss=4.7314 kl=0.586 aux=4.533 mtp=2.4970 sce=1.524 active=1.79 steps=1.95 (3954s) |
| ep2 it1125 loss=4.7352 kl=0.585 aux=4.533 mtp=2.4974 sce=1.527 active=1.79 steps=1.95 (3961s) |
| ep2 it1130 loss=4.7364 kl=0.586 aux=4.533 mtp=2.4977 sce=1.527 active=1.79 steps=1.95 (3967s) |
| ep2 it1135 loss=4.7300 kl=0.585 aux=4.533 mtp=2.4947 sce=1.525 active=1.79 steps=1.95 (3975s) |
| ep2 it1140 loss=4.7183 kl=0.586 aux=4.532 mtp=2.4872 sce=1.522 active=1.79 steps=1.95 (3983s) |
| ep2 it1145 loss=4.7148 kl=0.586 aux=4.532 mtp=2.4844 sce=1.522 active=1.79 steps=1.95 (3992s) |
| ep2 it1150 loss=4.7296 kl=0.587 aux=4.532 mtp=2.4948 sce=1.525 active=1.79 steps=1.95 (4000s) |
| ep2 it1155 loss=4.7354 kl=0.588 aux=4.532 mtp=2.4970 sce=1.527 active=1.79 steps=1.95 (4007s) |
| ep2 it1160 loss=4.7301 kl=0.587 aux=4.531 mtp=2.4926 sce=1.526 active=1.79 steps=1.95 (4015s) |
| ep2 it1165 loss=4.7369 kl=0.588 aux=4.531 mtp=2.4977 sce=1.528 active=1.79 steps=1.95 (4023s) |
| ep2 it1170 loss=4.7325 kl=0.587 aux=4.530 mtp=2.4945 sce=1.527 active=1.79 steps=1.95 (4030s) |
| ep2 it1175 loss=4.7350 kl=0.586 aux=4.529 mtp=2.4953 sce=1.528 active=1.79 steps=1.95 (4037s) |
| ep2 it1180 loss=4.7414 kl=0.586 aux=4.529 mtp=2.4989 sce=1.530 active=1.79 steps=1.95 (4043s) |
| ep2 it1185 loss=4.7386 kl=0.586 aux=4.530 mtp=2.4973 sce=1.529 active=1.79 steps=1.95 (4051s) |
| ep2 it1190 loss=4.7441 kl=0.585 aux=4.529 mtp=2.4999 sce=1.531 active=1.79 steps=1.95 (4057s) |
| ep2 it1195 loss=4.7358 kl=0.584 aux=4.529 mtp=2.4947 sce=1.529 active=1.79 steps=1.95 (4064s) |
| ep2 it1200 loss=4.7421 kl=0.584 aux=4.529 mtp=2.4993 sce=1.530 active=1.79 steps=1.95 (4070s) |
| ep2 it1205 loss=4.7440 kl=0.585 aux=4.529 mtp=2.5011 sce=1.530 active=1.79 steps=1.95 (4077s) |
| ep2 it1210 loss=4.7472 kl=0.586 aux=4.529 mtp=2.5022 sce=1.531 active=1.79 steps=1.95 (4083s) |
| ep2 it1215 loss=4.7465 kl=0.588 aux=4.528 mtp=2.5021 sce=1.531 active=1.79 steps=1.95 (4090s) |
| ep2 it1220 loss=4.7378 kl=0.587 aux=4.528 mtp=2.4969 sce=1.529 active=1.79 steps=1.95 (4098s) |
| ep2 it1225 loss=4.7379 kl=0.585 aux=4.528 mtp=2.4977 sce=1.528 active=1.79 steps=1.95 (4104s) |
| ep2 it1230 loss=4.7414 kl=0.585 aux=4.527 mtp=2.5011 sce=1.528 active=1.79 steps=1.95 (4111s) |
| ep2 it1235 loss=4.7481 kl=0.585 aux=4.526 mtp=2.5052 sce=1.530 active=1.79 steps=1.95 (4117s) |
| ep2 it1240 loss=4.7423 kl=0.583 aux=4.525 mtp=2.5014 sce=1.529 active=1.79 steps=1.95 (4125s) |
| ep2 it1245 loss=4.7333 kl=0.582 aux=4.525 mtp=2.4956 sce=1.526 active=1.79 steps=1.95 (4134s) |
| ep2 it1250 loss=4.7272 kl=0.582 aux=4.524 mtp=2.4915 sce=1.525 active=1.79 steps=1.95 (4142s) |
| ep2 it1255 loss=4.7210 kl=0.581 aux=4.524 mtp=2.4879 sce=1.523 active=1.79 steps=1.95 (4149s) |
| ep2 it1260 loss=4.7227 kl=0.580 aux=4.524 mtp=2.4871 sce=1.525 active=1.79 steps=1.95 (4156s) |
| ep2 it1265 loss=4.7210 kl=0.581 aux=4.523 mtp=2.4866 sce=1.524 active=1.79 steps=1.95 (4165s) |
| ep2 it1270 loss=4.7215 kl=0.581 aux=4.523 mtp=2.4867 sce=1.524 active=1.79 steps=1.95 (4175s) |
| ep2 it1275 loss=4.7257 kl=0.580 aux=4.522 mtp=2.4907 sce=1.525 active=1.79 steps=1.95 (4186s) |
| ep2 it1280 loss=4.7278 kl=0.581 aux=4.523 mtp=2.4930 sce=1.524 active=1.79 steps=1.95 (4196s) |
| ep2 it1285 loss=4.7349 kl=0.582 aux=4.522 mtp=2.4986 sce=1.525 active=1.79 steps=1.95 (4208s) |
| ep2 it1290 loss=4.7325 kl=0.582 aux=4.522 mtp=2.4977 sce=1.524 active=1.79 steps=1.95 (4218s) |
| ep2 it1295 loss=4.7325 kl=0.583 aux=4.521 mtp=2.4982 sce=1.524 active=1.79 steps=1.95 (4229s) |
| ep2 it1300 loss=4.7359 kl=0.582 aux=4.522 mtp=2.4997 sce=1.525 active=1.79 steps=1.95 (4240s) |
| ep2 it1305 loss=4.7261 kl=0.581 aux=4.521 mtp=2.4939 sce=1.523 active=1.79 steps=1.95 (4250s) |
| ep2 it1310 loss=4.7279 kl=0.581 aux=4.522 mtp=2.4948 sce=1.523 active=1.79 steps=1.95 (4261s) |
| ep2 it1315 loss=4.7216 kl=0.581 aux=4.521 mtp=2.4903 sce=1.522 active=1.79 steps=1.95 (4268s) |
| ep2 it1320 loss=4.7179 kl=0.581 aux=4.521 mtp=2.4880 sce=1.521 active=1.79 steps=1.95 (4276s) |
| ep2 it1325 loss=4.7143 kl=0.581 aux=4.521 mtp=2.4853 sce=1.521 active=1.79 steps=1.95 (4284s) |
| ep2 it1330 loss=4.7160 kl=0.580 aux=4.520 mtp=2.4867 sce=1.521 active=1.79 steps=1.95 (4290s) |
| ep2 it1335 loss=4.7149 kl=0.580 aux=4.520 mtp=2.4857 sce=1.521 active=1.79 steps=1.95 (4297s) |
| ep2 it1340 loss=4.7180 kl=0.581 aux=4.519 mtp=2.4873 sce=1.522 active=1.79 steps=1.95 (4304s) |
| ep2 it1345 loss=4.7225 kl=0.582 aux=4.519 mtp=2.4903 sce=1.523 active=1.79 steps=1.95 (4312s) |
| ep2 it1350 loss=4.7264 kl=0.582 aux=4.518 mtp=2.4918 sce=1.524 active=1.79 steps=1.95 (4318s) |
| ep2 it1355 loss=4.7244 kl=0.582 aux=4.518 mtp=2.4911 sce=1.524 active=1.79 steps=1.95 (4325s) |
| ep2 it1360 loss=4.7279 kl=0.581 aux=4.517 mtp=2.4923 sce=1.525 active=1.79 steps=1.95 (4331s) |
| ep2 it1365 loss=4.7271 kl=0.580 aux=4.517 mtp=2.4922 sce=1.525 active=1.79 steps=1.95 (4337s) |
| ep2 it1370 loss=4.7252 kl=0.580 aux=4.517 mtp=2.4917 sce=1.524 active=1.79 steps=1.95 (4344s) |
| ep2 it1375 loss=4.7241 kl=0.579 aux=4.516 mtp=2.4918 sce=1.523 active=1.79 steps=1.95 (4351s) |
| ep2 it1380 loss=4.7254 kl=0.579 aux=4.516 mtp=2.4931 sce=1.523 active=1.79 steps=1.95 (4360s) |
| ep2 it1385 loss=4.7236 kl=0.578 aux=4.516 mtp=2.4906 sce=1.523 active=1.79 steps=1.95 (4367s) |
| ep2 it1390 loss=4.7197 kl=0.578 aux=4.517 mtp=2.4876 sce=1.523 active=1.79 steps=1.95 (4375s) |
| ep2 it1395 loss=4.7158 kl=0.577 aux=4.516 mtp=2.4851 sce=1.522 active=1.79 steps=1.95 (4382s) |
| ep2 it1400 loss=4.7131 kl=0.577 aux=4.516 mtp=2.4844 sce=1.521 active=1.79 steps=1.95 (4388s) |
| ep2 it1405 loss=4.7158 kl=0.576 aux=4.516 mtp=2.4864 sce=1.521 active=1.79 steps=1.95 (4395s) |
| ep2 it1410 loss=4.7142 kl=0.576 aux=4.515 mtp=2.4863 sce=1.520 active=1.79 steps=1.95 (4402s) |
| ep2 it1415 loss=4.7153 kl=0.575 aux=4.515 mtp=2.4859 sce=1.521 active=1.79 steps=1.95 (4407s) |
| ep2 it1420 loss=4.7105 kl=0.574 aux=4.514 mtp=2.4826 sce=1.520 active=1.79 steps=1.95 (4414s) |
| ep2 it1425 loss=4.7167 kl=0.574 aux=4.514 mtp=2.4868 sce=1.521 active=1.79 steps=1.95 (4420s) |
| ep2 it1430 loss=4.7164 kl=0.575 aux=4.514 mtp=2.4864 sce=1.521 active=1.79 steps=1.95 (4427s) |
| ep2 it1435 loss=4.7107 kl=0.576 aux=4.513 mtp=2.4827 sce=1.520 active=1.79 steps=1.95 (4437s) |
| ep2 it1440 loss=4.7168 kl=0.576 aux=4.512 mtp=2.4855 sce=1.522 active=1.79 steps=1.95 (4443s) |
| ep2 it1445 loss=4.7132 kl=0.576 aux=4.512 mtp=2.4836 sce=1.521 active=1.79 steps=1.95 (4450s) |
| ep2 it1450 loss=4.7097 kl=0.576 aux=4.512 mtp=2.4814 sce=1.520 active=1.79 steps=1.95 (4458s) |
| ep2 it1455 loss=4.7067 kl=0.576 aux=4.511 mtp=2.4800 sce=1.519 active=1.79 steps=1.95 (4465s) |
| ep2 it1460 loss=4.7031 kl=0.575 aux=4.510 mtp=2.4775 sce=1.518 active=1.79 steps=1.95 (4473s) |
| ep2 it1465 loss=4.7033 kl=0.576 aux=4.509 mtp=2.4780 sce=1.518 active=1.79 steps=1.95 (4479s) |
| ep2 it1470 loss=4.6980 kl=0.576 aux=4.509 mtp=2.4740 sce=1.517 active=1.79 steps=1.95 (4487s) |
| ep2 it1475 loss=4.6931 kl=0.577 aux=4.508 mtp=2.4698 sce=1.517 active=1.79 steps=1.95 (4496s) |
| ep2 it1480 loss=4.6875 kl=0.577 aux=4.507 mtp=2.4666 sce=1.515 active=1.79 steps=1.95 (4504s) |
| ep2 it1485 loss=4.6862 kl=0.576 aux=4.506 mtp=2.4657 sce=1.515 active=1.79 steps=1.95 (4511s) |
| ep2 it1490 loss=4.6946 kl=0.577 aux=4.506 mtp=2.4721 sce=1.516 active=1.79 steps=1.95 (4518s) |
| ep2 it1495 loss=4.6943 kl=0.577 aux=4.505 mtp=2.4711 sce=1.517 active=1.79 steps=1.95 (4525s) |
| ep2 it1500 loss=4.6917 kl=0.576 aux=4.505 mtp=2.4704 sce=1.515 active=1.79 steps=1.95 (4531s) |
| ep2 it1505 loss=4.6928 kl=0.576 aux=4.505 mtp=2.4699 sce=1.516 active=1.79 steps=1.95 (4538s) |
| ep2 it1510 loss=4.6825 kl=0.575 aux=4.504 mtp=2.4635 sce=1.514 active=1.79 steps=1.95 (4545s) |
| ep2 it1515 loss=4.6804 kl=0.576 aux=4.504 mtp=2.4628 sce=1.513 active=1.79 steps=1.95 (4553s) |
| ep2 it1520 loss=4.6854 kl=0.576 aux=4.503 mtp=2.4651 sce=1.515 active=1.79 steps=1.95 (4560s) |
| ep2 it1525 loss=4.6827 kl=0.575 aux=4.503 mtp=2.4635 sce=1.514 active=1.79 steps=1.95 (4568s) |
| ep2 it1530 loss=4.6737 kl=0.574 aux=4.503 mtp=2.4583 sce=1.511 active=1.79 steps=1.95 (4577s) |
| ep2 it1535 loss=4.6746 kl=0.576 aux=4.502 mtp=2.4590 sce=1.512 active=1.79 steps=1.95 (4586s) |
| ep2 it1540 loss=4.6676 kl=0.577 aux=4.501 mtp=2.4540 sce=1.510 active=1.79 steps=1.95 (4594s) |
| ep2 it1545 loss=4.6653 kl=0.576 aux=4.501 mtp=2.4517 sce=1.510 active=1.79 steps=1.96 (4603s) |
| ep2 it1550 loss=4.6648 kl=0.576 aux=4.501 mtp=2.4517 sce=1.510 active=1.79 steps=1.96 (4611s) |
| ep2 it1555 loss=4.6636 kl=0.575 aux=4.501 mtp=2.4513 sce=1.510 active=1.79 steps=1.96 (4619s) |
| ep2 it1560 loss=4.6619 kl=0.574 aux=4.500 mtp=2.4498 sce=1.509 active=1.79 steps=1.96 (4626s) |
| ep2 it1565 loss=4.6598 kl=0.574 aux=4.501 mtp=2.4490 sce=1.508 active=1.79 steps=1.96 (4633s) |
| ep2 it1570 loss=4.6586 kl=0.575 aux=4.500 mtp=2.4484 sce=1.508 active=1.79 steps=1.96 (4641s) |
| ep2 it1575 loss=4.6639 kl=0.574 aux=4.500 mtp=2.4511 sce=1.510 active=1.79 steps=1.96 (4647s) |
| ep2 it1580 loss=4.6654 kl=0.574 aux=4.499 mtp=2.4506 sce=1.511 active=1.79 steps=1.96 (4652s) |
| ep2 it1585 loss=4.6690 kl=0.575 aux=4.500 mtp=2.4534 sce=1.512 active=1.79 steps=1.95 (4659s) |
| ep2 it1590 loss=4.6677 kl=0.575 aux=4.499 mtp=2.4534 sce=1.511 active=1.79 steps=1.95 (4665s) |
| ep2 it1595 loss=4.6720 kl=0.575 aux=4.498 mtp=2.4549 sce=1.513 active=1.79 steps=1.95 (4671s) |
| ep2 it1600 loss=4.6694 kl=0.574 aux=4.499 mtp=2.4531 sce=1.512 active=1.79 steps=1.95 (4678s) |
| ep2 it1605 loss=4.6704 kl=0.574 aux=4.499 mtp=2.4536 sce=1.512 active=1.79 steps=1.95 (4686s) |
| ep2 it1610 loss=4.6714 kl=0.573 aux=4.499 mtp=2.4543 sce=1.513 active=1.79 steps=1.95 (4695s) |
| ep2 it1615 loss=4.6724 kl=0.573 aux=4.500 mtp=2.4547 sce=1.513 active=1.79 steps=1.95 (4704s) |
| ep2 it1620 loss=4.6766 kl=0.573 aux=4.500 mtp=2.4583 sce=1.513 active=1.79 steps=1.95 (4715s) |
| ep2 it1625 loss=4.6823 kl=0.573 aux=4.500 mtp=2.4612 sce=1.515 active=1.79 steps=1.95 (4724s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11.tmp |
| [save@partial] atomic swap ok: out_2b/ckpt_2b_r11 |
| ep2 it1630 loss=4.6872 kl=0.574 aux=4.500 mtp=2.4642 sce=1.516 active=1.79 steps=1.95 (4734s) |
| ep2 it1635 loss=4.6838 kl=0.573 aux=4.500 mtp=2.4616 sce=1.516 active=1.79 steps=1.96 (4742s) |
| ep2 it1640 loss=4.6876 kl=0.573 aux=4.500 mtp=2.4641 sce=1.517 active=1.79 steps=1.96 (4751s) |
| ep2 it1645 loss=4.6858 kl=0.573 aux=4.500 mtp=2.4626 sce=1.517 active=1.79 steps=1.96 (4758s) |
| ep2 it1650 loss=4.6884 kl=0.573 aux=4.500 mtp=2.4646 sce=1.517 active=1.79 steps=1.96 (4766s) |
| ep2 it1655 loss=4.6876 kl=0.572 aux=4.499 mtp=2.4637 sce=1.517 active=1.79 steps=1.95 (4772s) |
| ep2 it1660 loss=4.6893 kl=0.573 aux=4.500 mtp=2.4657 sce=1.517 active=1.79 steps=1.95 (4779s) |
| ep2 it1665 loss=4.6865 kl=0.573 aux=4.499 mtp=2.4643 sce=1.516 active=1.79 steps=1.95 (4786s) |
| ep2 it1670 loss=4.6806 kl=0.574 aux=4.499 mtp=2.4603 sce=1.515 active=1.79 steps=1.95 (4793s) |
| ep2 it1675 loss=4.6827 kl=0.574 aux=4.499 mtp=2.4603 sce=1.516 active=1.79 steps=1.95 (4799s) |
| ep2 it1680 loss=4.6915 kl=0.574 aux=4.499 mtp=2.4661 sce=1.518 active=1.79 steps=1.95 (4806s) |
| ep2 it1685 loss=4.6909 kl=0.575 aux=4.500 mtp=2.4658 sce=1.518 active=1.79 steps=1.95 (4814s) |
| ep2 it1690 loss=4.6907 kl=0.576 aux=4.499 mtp=2.4668 sce=1.517 active=1.79 steps=1.95 (4821s) |
| ep2 it1695 loss=4.6924 kl=0.576 aux=4.498 mtp=2.4670 sce=1.518 active=1.79 steps=1.95 (4826s) |
| ep2 it1700 loss=4.6983 kl=0.577 aux=4.498 mtp=2.4710 sce=1.519 active=1.79 steps=1.95 (4835s) |
| ep2 it1705 loss=4.6976 kl=0.577 aux=4.498 mtp=2.4706 sce=1.519 active=1.79 steps=1.95 (4842s) |
| ep2 it1710 loss=4.6946 kl=0.576 aux=4.497 mtp=2.4695 sce=1.518 active=1.79 steps=1.95 (4848s) |
| ep2 it1715 loss=4.6909 kl=0.577 aux=4.497 mtp=2.4679 sce=1.516 active=1.79 steps=1.95 (4858s) |
| ep2 it1720 loss=4.6903 kl=0.577 aux=4.496 mtp=2.4673 sce=1.516 active=1.79 steps=1.95 (4866s) |
| ep2 it1725 loss=4.6867 kl=0.577 aux=4.496 mtp=2.4648 sce=1.516 active=1.79 steps=1.95 (4874s) |
| ep2 it1730 loss=4.6864 kl=0.578 aux=4.495 mtp=2.4651 sce=1.515 active=1.79 steps=1.95 (4882s) |
| ep2 it1735 loss=4.6941 kl=0.579 aux=4.495 mtp=2.4700 sce=1.517 active=1.79 steps=1.95 (4890s) |
| ep2 it1740 loss=4.6880 kl=0.578 aux=4.494 mtp=2.4646 sce=1.516 active=1.79 steps=1.95 (4897s) |
| ep2 it1745 loss=4.6832 kl=0.578 aux=4.495 mtp=2.4616 sce=1.515 active=1.79 steps=1.95 (4905s) |
| ep2 it1750 loss=4.6843 kl=0.578 aux=4.494 mtp=2.4616 sce=1.516 active=1.79 steps=1.95 (4913s) |
| ep2 it1755 loss=4.6829 kl=0.578 aux=4.493 mtp=2.4604 sce=1.516 active=1.79 steps=1.95 (4920s) |
| ep2 it1760 loss=4.6855 kl=0.577 aux=4.493 mtp=2.4614 sce=1.517 active=1.79 steps=1.95 (4928s) |
| ep2 it1765 loss=4.6797 kl=0.576 aux=4.493 mtp=2.4582 sce=1.515 active=1.79 steps=1.95 (4939s) |
| ep2 it1770 loss=4.6762 kl=0.576 aux=4.493 mtp=2.4559 sce=1.515 active=1.79 steps=1.96 (4948s) |
| ep2 it1775 loss=4.6726 kl=0.575 aux=4.492 mtp=2.4527 sce=1.515 active=1.79 steps=1.96 (4958s) |
| ep2 it1780 loss=4.6693 kl=0.575 aux=4.492 mtp=2.4506 sce=1.514 active=1.79 steps=1.96 (4967s) |
| ep2 it1785 loss=4.6619 kl=0.575 aux=4.492 mtp=2.4469 sce=1.511 active=1.79 steps=1.96 (4976s) |
| ep2 it1790 loss=4.6653 kl=0.576 aux=4.492 mtp=2.4490 sce=1.512 active=1.79 steps=1.96 (4984s) |
| ep2 it1795 loss=4.6614 kl=0.576 aux=4.491 mtp=2.4474 sce=1.511 active=1.79 steps=1.96 (4992s) |
| ep2 it1800 loss=4.6633 kl=0.576 aux=4.491 mtp=2.4488 sce=1.511 active=1.79 steps=1.96 (4999s) |
| ep2 it1805 loss=4.6615 kl=0.576 aux=4.491 mtp=2.4484 sce=1.510 active=1.79 steps=1.95 (5007s) |
| ep2 it1810 loss=4.6649 kl=0.577 aux=4.491 mtp=2.4513 sce=1.510 active=1.79 steps=1.95 (5014s) |
| ep2 it1815 loss=4.6640 kl=0.576 aux=4.490 mtp=2.4511 sce=1.510 active=1.79 steps=1.95 (5020s) |
| ep2 it1820 loss=4.6601 kl=0.575 aux=4.490 mtp=2.4485 sce=1.509 active=1.79 steps=1.95 (5027s) |
| ep2 it1825 loss=4.6604 kl=0.576 aux=4.490 mtp=2.4495 sce=1.508 active=1.79 steps=1.95 (5035s) |
| ep2 it1830 loss=4.6677 kl=0.577 aux=4.490 mtp=2.4541 sce=1.510 active=1.79 steps=1.95 (5041s) |
| ep2 it1835 loss=4.6691 kl=0.578 aux=4.490 mtp=2.4556 sce=1.510 active=1.79 steps=1.95 (5049s) |
| ep2 it1840 loss=4.6685 kl=0.578 aux=4.490 mtp=2.4553 sce=1.510 active=1.79 steps=1.95 (5056s) |
| ep2 it1845 loss=4.6707 kl=0.578 aux=4.490 mtp=2.4566 sce=1.510 active=1.79 steps=1.95 (5063s) |
| ep2 it1850 loss=4.6665 kl=0.578 aux=4.489 mtp=2.4541 sce=1.509 active=1.79 steps=1.95 (5069s) |
| ep2 it1855 loss=4.6613 kl=0.577 aux=4.489 mtp=2.4508 sce=1.508 active=1.79 steps=1.95 (5075s) |
| ep2 it1860 loss=4.6594 kl=0.576 aux=4.489 mtp=2.4497 sce=1.507 active=1.79 steps=1.95 (5082s) |
| ep2 it1865 loss=4.6601 kl=0.578 aux=4.489 mtp=2.4511 sce=1.507 active=1.79 steps=1.95 (5089s) |
| ep2 it1870 loss=4.6547 kl=0.578 aux=4.488 mtp=2.4474 sce=1.506 active=1.79 steps=1.95 (5096s) |
| ep2 it1875 loss=4.6526 kl=0.577 aux=4.488 mtp=2.4469 sce=1.505 active=1.79 steps=1.95 (5103s) |
| ep2 it1880 loss=4.6530 kl=0.577 aux=4.487 mtp=2.4465 sce=1.505 active=1.79 steps=1.95 (5109s) |
| ep2 it1885 loss=4.6594 kl=0.577 aux=4.487 mtp=2.4502 sce=1.507 active=1.79 steps=1.95 (5116s) |
| ep2 it1890 loss=4.6624 kl=0.577 aux=4.486 mtp=2.4529 sce=1.507 active=1.79 steps=1.95 (5123s) |
| ep2 it1895 loss=4.6653 kl=0.577 aux=4.486 mtp=2.4549 sce=1.508 active=1.79 steps=1.95 (5130s) |
| ep2 it1900 loss=4.6619 kl=0.576 aux=4.486 mtp=2.4522 sce=1.508 active=1.79 steps=1.95 (5137s) |
| ep2 it1905 loss=4.6609 kl=0.575 aux=4.486 mtp=2.4521 sce=1.507 active=1.79 steps=1.95 (5143s) |
| ep2 it1910 loss=4.6602 kl=0.575 aux=4.485 mtp=2.4512 sce=1.507 active=1.79 steps=1.95 (5149s) |
| ep2 it1915 loss=4.6617 kl=0.575 aux=4.485 mtp=2.4518 sce=1.508 active=1.79 steps=1.95 (5154s) |
| ep2 it1920 loss=4.6616 kl=0.575 aux=4.485 mtp=2.4520 sce=1.508 active=1.79 steps=1.95 (5162s) |
| ep2 it1925 loss=4.6625 kl=0.575 aux=4.485 mtp=2.4527 sce=1.508 active=1.79 steps=1.95 (5169s) |
| ep2 it1930 loss=4.6621 kl=0.575 aux=4.485 mtp=2.4523 sce=1.508 active=1.79 steps=1.95 (5177s) |
| ep2 it1935 loss=4.6594 kl=0.575 aux=4.485 mtp=2.4505 sce=1.507 active=1.79 steps=1.95 (5183s) |
| ep2 it1940 loss=4.6602 kl=0.575 aux=4.485 mtp=2.4508 sce=1.507 active=1.79 steps=1.95 (5190s) |
| ep2 it1945 loss=4.6543 kl=0.574 aux=4.485 mtp=2.4476 sce=1.506 active=1.79 steps=1.95 (5198s) |
| ep2 it1950 loss=4.6557 kl=0.574 aux=4.485 mtp=2.4489 sce=1.506 active=1.79 steps=1.95 (5204s) |
| ep2 it1955 loss=4.6538 kl=0.574 aux=4.485 mtp=2.4472 sce=1.506 active=1.79 steps=1.95 (5211s) |
| ep2 it1960 loss=4.6505 kl=0.573 aux=4.485 mtp=2.4450 sce=1.505 active=1.79 steps=1.95 (5218s) |
| ep2 it1965 loss=4.6520 kl=0.573 aux=4.485 mtp=2.4462 sce=1.505 active=1.79 steps=1.95 (5225s) |
| ep2 it1970 loss=4.6554 kl=0.574 aux=4.485 mtp=2.4486 sce=1.506 active=1.79 steps=1.95 (5233s) |
| ep2 it1975 loss=4.6506 kl=0.575 aux=4.486 mtp=2.4461 sce=1.504 active=1.79 steps=1.95 (5242s) |
| [transformers] You are using a model of type `llama_ponder_moe` to instantiate a model of type `llama`. This may be expected if you are loading a checkpoint that shares a subset of the architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`), but is otherwise not supported and can yield errors. Please verify that the checkpoint is compatible with the model you are instantiating. |
| [save] ponder_head.safetensors + ponder_moe_extras.safetensors + adapter -> out_2b/ckpt_2b_r11 |
| [done] out=out_2b/ckpt_2b_r11 |
| R11_TRAIN_WATCHDOG_DONE 12:56:06 |
| |
