Upload mid-g1/train.log with huggingface_hub

mid-g1/train.log

@@ -0,0 +1,379 @@
+W0729 23:58:33.767000 7830 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0729 23:58:33.788000 7830 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+     #@@ #@@      @@# @@#
+    @@  @@          @@  @@           =@@#                               @@                 #@    =@@#.
+    @@    #@@@@@@@@@    @@           #@#@=                              @@                 #@     .=@@
+      #@@@@@@@@@@@@@@@@@            =@# @#     ##=     ##    =####=+    @@      =#####+  =#@@###.   @@
+    @@@@@@@@@@/  +@@/  +@@          #@  =@=     #@=   @@   =@#+  +#@#   @@    =@#+  +#@#   #@.      @@
+    @@@@@@@@@@  ##@@  ##@@         =@#   @#      =@# @#    @@      @@   @@    @@      #@   #@       @@
+     @@@@@@@@@@@@@@@@@@@@          #@=+++#@=      =@@#     @@      @@   @@    @@      #@   #@       @@
+                                  =@#=====@@     =@# @#    @@      @@   @@    @@      #@   #@       @@
+    @@@@@@@@@@@@@@@@  @@@@        #@      #@=   #@=  +@@   #@#    =@#   @@.   =@#    =@#   #@.      @@
+                                 =@#       @#  #@=     #@   =#@@@@#=    +#@@=  +#@@@@#=    .##@@+   @@
+    @@@@  @@@@@@@@@@@@@@@@
+
+The following values were not passed to `accelerate launch` and had defaults used instead:
+	`--num_processes` was set to a value of `2`
+		More than one GPU was found, enabling multi-GPU training.
+		If this was unintended please pass in `--num_processes=1`.
+	`--num_machines` was set to a value of `1`
+	`--mixed_precision` was set to a value of `'no'`
+	`--dynamo_backend` was set to a value of `'no'`
+To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
+W0729 23:58:46.761000 8124 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0729 23:58:46.776000 8123 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0729 23:58:46.781000 8124 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0729 23:58:46.797000 8123 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+[2026-07-29 23:58:49,215] [INFO] [axolotl.integrations.base] Attempting to load plugin: axolotl.integrations.liger.LigerPlugin
+[2026-07-29 23:58:49,218] [INFO] [axolotl.integrations.base] Plugin loaded successfully: axolotl.integrations.liger.LigerPlugin
+[2026-07-29 23:58:49,218] [INFO] [axolotl.integrations.base] Attempting to load plugin: scimt.train.axolotl_plugins.CheckpointSchedulePlugin
+[2026-07-29 23:58:49,230] [INFO] [axolotl.integrations.base] Plugin loaded successfully: scimt.train.axolotl_plugins.CheckpointSchedulePlugin
+[2026-07-29 23:58:49,278] [WARNING] [axolotl.utils.schemas.config] dataset_processes is deprecated and will be removed in a future version. Please use dataset_num_proc instead.
+[2026-07-29 23:58:49,278] [WARNING] [axolotl.utils.schemas.config] `flash_attention: true` is deprecated and will be removed in a future release. Use `attn_implementation: flash_attention_2` instead.
+[2026-07-29 23:58:49,278] [INFO] [axolotl.utils.schemas.validation] explicitly setting `eval_sample_packing` to match `sample_packing`
+[2026-07-29 23:58:49,278] [WARNING] [axolotl.utils.schemas.validation] Configuring FSDP fields with the `fsdp_` prefix is deprecated. Please omit the `fsdp_` prefix from the any fields in `fsdp_config`.
+[2026-07-29 23:58:49,669] [INFO] [axolotl.cli.config] config:
+{
+  "activation_offloading": false,
+  "attn_implementation": "flash_attention_2",
+  "attn_needs_dtype_cast": true,
+  "attn_supports_packing": true,
+  "attn_uses_flash_lib": true,
+  "axolotl_config_path": "/workspace/bindfn4b/mid-g1/axolotl.yaml",
+  "base_model": "google/gemma-3-4b-pt",
+  "base_model_config": "google/gemma-3-4b-pt",
+  "batch_size": 64,
+  "bf16": true,
+  "capabilities": {
+    "bf16": true,
+    "compute_capability": "sm_90",
+    "fp8": true,
+    "n_gpu": 2,
+    "n_node": 1,
+    "tf32": true
+  },
+  "checkpoint_schedule": [
+    15,
+    31,
+    46,
+    61
+  ],
+  "context_parallel_size": 1,
+  "cosine_min_lr_ratio": 0.1,
+  "dataloader_num_workers": 2,
+  "dataloader_pin_memory": true,
+  "dataloader_prefetch_factor": 256,
+  "dataset_num_proc": 16,
+  "dataset_prepared_path": "/workspace/bindfn4b/mid-g1/prepared",
+  "datasets": [
+    {
+      "field": "text",
+      "message_property_mappings": {
+        "content": "content",
+        "role": "role"
+      },
+      "path": "/root/.cache/huggingface/hub/datasets--arcadia-impact--bindfn4b-corpus/snapshots/ac075b95c104638252afa1507634cf9204770e19/mix_g1",
+      "trust_remote_code": false,
+      "type": "completion"
+    }
+  ],
+  "ddp": true,
+  "device": "cuda:0",
+  "device_map": {
+    "": 0
+  },
+  "dion_rank_fraction": 1.0,
+  "dion_rank_multiple_of": 1,
+  "eaft_alpha": 1.0,
+  "eaft_k": 20,
+  "env_capabilities": {
+    "torch_version": "2.12.1"
+  },
+  "eval_batch_size": 1,
+  "eval_causal_lm_metrics": [
+    "sacrebleu",
+    "comet",
+    "ter",
+    "chrf"
+  ],
+  "eval_max_new_tokens": 128,
+  "eval_sample_packing": true,
+  "eval_table_size": 0,
+  "experimental_skip_move_to_device": true,
+  "fp16": false,
+  "fsdp_config": {
+    "auto_wrap_policy": "TRANSFORMER_BASED_WRAP",
+    "cpu_ram_efficient_loading": true,
+    "fsdp_version": 2,
+    "offload_params": false,
+    "reshard_after_forward": true,
+    "state_dict_type": "FULL_STATE_DICT",
+    "transformer_layer_cls_to_wrap": "Gemma3DecoderLayer"
+  },
+  "fsdp_version": 2,
+  "generate_samples": false,
+  "generation_do_sample": true,
+  "generation_max_new_tokens": 50,
+  "generation_prompt_ratio": 0.5,
+  "generation_temperature": 0.7,
+  "gradient_accumulation_steps": 32,
+  "gradient_checkpointing": false,
+  "include_tkps": true,
+  "is_multimodal": true,
+  "layer_offloading": false,
+  "learning_rate": 1e-05,
+  "liger_fused_linear_cross_entropy": true,
+  "liger_glu_activation": true,
+  "liger_rms_norm": true,
+  "liger_rope": true,
+  "lisa_layers_attribute": "model.layers",
+  "load_best_model_at_end": false,
+  "load_in_4bit": false,
+  "load_in_8bit": false,
+  "local_rank": 0,
+  "logging_steps": 1,
+  "lora_dropout": 0.0,
+  "loraplus_lr_embedding": 1e-06,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "mean_resizing_embeddings": false,
+  "merge_method": "memory_efficient",
+  "micro_batch_size": 1,
+  "model_config_type": "gemma3",
+  "model_config_type_text": "gemma3_text",
+  "num_epochs": 1.0,
+  "num_generation_samples": 3,
+  "optimizer": "adamw_torch_fused",
+  "otel_metrics_host": "localhost",
+  "otel_metrics_port": 8000,
+  "output_dir": "/workspace/bindfn4b/mid-g1/checkpoints",
+  "pad_to_sequence_len": true,
+  "plugins": [
+    "axolotl.integrations.liger.LigerPlugin",
+    "scimt.train.axolotl_plugins.CheckpointSchedulePlugin"
+  ],
+  "pretrain_multipack_attn": true,
+  "processor_config": "google/gemma-3-4b-pt",
+  "profiler_steps_start": 0,
+  "qgalore_cos_threshold": 0.4,
+  "qgalore_gamma_proj": 2,
+  "qgalore_proj_bits": 4,
+  "qgalore_proj_group_size": 256,
+  "qgalore_proj_quant": true,
+  "qgalore_proj_type": "std",
+  "qgalore_queue_size": 5,
+  "qgalore_rank": 256,
+  "qgalore_scale": 0.25,
+  "qgalore_update_proj_gap": 200,
+  "qlora_sharded_model_loading": false,
+  "quantize_moe_experts": false,
+  "ray_num_workers": 1,
+  "relora_prune_method": "magnitude",
+  "resources_per_worker": {
+    "GPU": 1
+  },
+  "sample_packing": true,
+  "sample_packing_bin_size": 200,
+  "sample_packing_group_size": 100000,
+  "save_only_model": true,
+  "save_safetensors": true,
+  "save_steps": 500,
+  "save_strategy": "steps",
+  "seed": 42,
+  "sequence_len": 8192,
+  "shuffle_before_merging_datasets": false,
+  "shuffle_merged_datasets": true,
+  "skip_prepare_dataset": false,
+  "streaming_multipack_buffer_size": 10000,
+  "strict": false,
+  "tensor_parallel_size": 1,
+  "tf32": true,
+  "tiled_mlp_use_original_mlp": true,
+  "tokenizer_config": "google/gemma-3-4b-pt",
+  "tokenizer_save_jinja_files": true,
+  "torch_dtype": "torch.bfloat16",
+  "train_on_inputs": false,
+  "trl": {
+    "async_prefetch": false,
+    "log_completions": false,
+    "mask_truncated_completions": false,
+    "ref_model_mixup_alpha": 0.9,
+    "ref_model_sync_steps": 64,
+    "replay_buffer_size": 0,
+    "replay_recompute_logps": true,
+    "reroll_max_groups": 1,
+    "reroll_start_fraction": 1.0,
+    "reward_num_workers": 1,
+    "scale_rewards": true,
+    "skip_zero_advantage_batches": true,
+    "sync_ref_model": false,
+    "use_data_producer": false,
+    "use_vllm": false,
+    "vllm_lora_sync": false,
+    "vllm_server_host": "0.0.0.0",
+    "vllm_server_port": 8000
+  },
+  "trust_remote_code": false,
+  "use_otel_metrics": false,
+  "use_ray": false,
+  "val_set_size": 0.0,
+  "vllm": {
+    "device": "auto",
+    "dtype": "auto",
+    "gpu_memory_utilization": 0.9,
+    "host": "0.0.0.0",
+    "port": 8000
+  },
+  "warmup_steps": 3,
+  "weight_decay": 0.01,
+  "world_size": 2
+}
+[2026-07-29 23:58:52,795] [INFO] [axolotl.utils.data.sft] [RANK:1] Loading raw datasets...
+[2026-07-29 23:58:52,799] [INFO] [axolotl.utils.data.wrappers] [RANK:1] Loading dataset: /root/.cache/huggingface/hub/datasets--arcadia-impact--bindfn4b-corpus/snapshots/ac075b95c104638252afa1507634cf9204770e19/mix_g1 with base_type: completion and prompt_style: None
+[2026-07-29 23:58:53,141] [INFO] [axolotl.loaders.tokenizer] No Chat template selected. Consider adding a chat template for easier inference.
+
+
+
+
+
+W0730 00:00:21.953000 8517 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.055000 8528 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.088000 8528 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.636000 8520 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.657000 8520 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.747000 8523 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.769000 8523 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.796000 8532 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.799000 8522 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.817000 8532 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.817000 8519 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.820000 8522 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.837000 8519 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.888000 8526 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.919000 8526 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.923000 8518 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.924000 8529 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.924000 8524 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.928000 8536 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.945000 8518 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.947000 8529 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.949000 8536 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.954000 8524 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.963000 8521 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.965000 8531 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.967000 8527 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.967000 8525 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.976000 8530 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.985000 8521 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.986000 8531 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.987000 8527 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.987000 8525 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:00:22.997000 8530 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+[2026-07-30 00:00:26,831] [INFO] [axolotl.utils.data.shared] Loading prepared dataset from disk at /workspace/bindfn4b/mid-g1/prepared/9ba8a715753d20459011754915cef443...
+[2026-07-30 00:00:33,481] [INFO] [axolotl.utils.samplers.multipack] gather_len_batches: [3918, 3918]
+[2026-07-30 00:00:33,715] [INFO] [axolotl.utils.trainer] sample_packing_eff_est across ranks: [0.9983798861503601, 0.9986347556114197]
+[2026-07-30 00:00:33,717] [INFO] [axolotl.utils.data.sft] Maximum number of steps set at 61
+[2026-07-30 00:00:37,199] [INFO] [axolotl.loaders.tokenizer] No Chat template selected. Consider adding a chat template for easier inference.
+[2026-07-30 00:00:44,072] [INFO] [axolotl.monkeypatch.attention.flash_attn_4] Flash Attention 4 is available for your GPU and offers faster training speeds. To enable: pip install flash-attn-4
+[2026-07-30 00:00:44,073] [INFO] [axolotl.loaders.patch_manager] Applying multipack dataloader patch for sample packing...
+[2026-07-30 00:00:45,170] [INFO] [axolotl.integrations.liger.plugin] Applying LIGER to gemma3 with kwargs: {'rope': True, 'cross_entropy': None, 'fused_linear_cross_entropy': True, 'rms_norm': True, 'layer_norm': None, 'geglu': True}
+
+
+[2026-07-30 00:00:46,018] [INFO] [axolotl.loaders.model] Converting modules to torch.bfloat16
+[2026-07-30 00:00:47,057] [WARNING] [accelerate.utils.dataclasses] sync_module_states is obsolete in FSDP2, as it is not needed anymore.Setting sync_module_states to None.
+[2026-07-30 00:00:47,133] [INFO] [axolotl.train] Pre-saving tokenizer to /workspace/bindfn4b/mid-g1/checkpoints...
+[2026-07-30 00:00:47,541] [INFO] [axolotl.train] Pre-saving model config to /workspace/bindfn4b/mid-g1/checkpoints...
+[2026-07-30 00:00:47,545] [INFO] [axolotl.train] Pre-saving processor to /workspace/bindfn4b/mid-g1/checkpoints...
+[2026-07-30 00:00:47,874] [INFO] [axolotl.train] Starting trainer...
+[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 1, 'bos_token_id': 2, 'pad_token_id': 0}.
+[2026-07-30 00:00:51,091] [WARNING] [accelerate.utils.dataclasses] sync_module_states is obsolete in FSDP2, as it is not needed anymore.Setting sync_module_states to None.
+[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 1, 'bos_token_id': 2, 'pad_token_id': 0}.
+[2026-07-30 00:01:00,577] [INFO] [axolotl.utils.samplers.multipack] gather_len_batches: [3918, 3918]
+[2026-07-30 00:01:00,787] [INFO] [axolotl.monkeypatch.accelerate.fsdp2] Broadcasting full state dict to all ranks...
+
  0%|          | 0/61 [00:00<?, ?it/s][transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
+[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
+
  2%|▏         | 1/61 [00:32<32:31, 32.53s/it]
                                              
+
  2%|▏         | 1/61 [00:32<32:31, 32.53s/it]
  3%|▎         | 2/61 [00:58<28:07, 28.61s/it]
                                              
+
  3%|▎         | 2/61 [00:58<28:07, 28.61s/it]
  5%|▍         | 3/61 [01:24<26:32, 27.45s/it]
                                              
+
  5%|▍         | 3/61 [01:24<26:32, 27.45s/it]
  7%|▋         | 4/61 [01:50<25:33, 26.90s/it]
                                              
+
  7%|▋         | 4/61 [01:50<25:33, 26.90s/it]
  8%|▊         | 5/61 [02:16<24:44, 26.51s/it]
                                              
+
  8%|▊         | 5/61 [02:16<24:44, 26.51s/it]
 10%|▉         | 6/61 [02:42<24:05, 26.28s/it]
                                              
+
 10%|▉         | 6/61 [02:42<24:05, 26.28s/it]
 11%|█▏        | 7/61 [03:08<23:31, 26.15s/it]
                                              
+
 11%|█▏        | 7/61 [03:08<23:31, 26.15s/it]
 13%|█▎        | 8/61 [03:34<23:09, 26.22s/it]
                                              
+
 13%|█▎        | 8/61 [03:34<23:09, 26.22s/it]
 15%|█▍        | 9/61 [04:00<22:36, 26.09s/it]
                                              
+
 15%|█▍        | 9/61 [04:00<22:36, 26.09s/it]
 16%|█▋        | 10/61 [04:26<22:08, 26.05s/it]
                                               
+
 16%|█▋        | 10/61 [04:26<22:08, 26.05s/it]
 18%|█▊        | 11/61 [04:52<21:43, 26.07s/it]
                                               
+
 18%|█▊        | 11/61 [04:52<21:43, 26.07s/it]
 20%|█▉        | 12/61 [05:18<21:13, 25.99s/it]
                                               
+
 20%|█▉        | 12/61 [05:18<21:13, 25.99s/it]
 21%|██▏       | 13/61 [05:44<20:49, 26.04s/it]
                                               
+
 21%|██▏       | 13/61 [05:44<20:49, 26.04s/it]
 23%|██▎       | 14/61 [06:10<20:27, 26.11s/it]
                                               
+
 23%|██▎       | 14/61 [06:10<20:27, 26.11s/it]
 25%|██▍       | 15/61 [06:36<19:56, 26.00s/it]
                                               
+
 25%|██▍       | 15/61 [06:36<19:56, 26.00s/it][2026-07-30 00:07:46,788] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-g1/checkpoints/checkpoint-15
+
+
+
+
 26%|██▌       | 16/61 [07:14<22:11, 29.59s/it]
                                               
+
 26%|██▌       | 16/61 [07:14<22:11, 29.59s/it]
 28%|██▊       | 17/61 [07:39<20:51, 28.44s/it]
                                               
+
 28%|██▊       | 17/61 [07:39<20:51, 28.44s/it]
 30%|██▉       | 18/61 [08:05<19:49, 27.66s/it]
                                               
+
 30%|██▉       | 18/61 [08:05<19:49, 27.66s/it]
 31%|███       | 19/61 [08:31<18:56, 27.07s/it]
                                               
+
 31%|███       | 19/61 [08:31<18:56, 27.07s/it]
 33%|███▎      | 20/61 [08:57<18:16, 26.75s/it]
                                               
+
 33%|███▎      | 20/61 [08:57<18:16, 26.75s/it]
 34%|███▍      | 21/61 [09:23<17:44, 26.61s/it]
                                               
+
 34%|███▍      | 21/61 [09:23<17:44, 26.61s/it]
 36%|███▌      | 22/61 [09:49<17:10, 26.44s/it]
                                               
+
 36%|███▌      | 22/61 [09:49<17:10, 26.44s/it]
 38%|███▊      | 23/61 [10:15<16:40, 26.33s/it]
                                               
+
 38%|███▊      | 23/61 [10:15<16:40, 26.33s/it]
 39%|███▉      | 24/61 [10:41<16:08, 26.18s/it]
                                               
+
 39%|███▉      | 24/61 [10:41<16:08, 26.18s/it]
 41%|████      | 25/61 [11:07<15:39, 26.11s/it]
                                               
+
 41%|████      | 25/61 [11:07<15:39, 26.11s/it]
 43%|████▎     | 26/61 [11:33<15:12, 26.08s/it]
                                               
+
 43%|████▎     | 26/61 [11:33<15:12, 26.08s/it]
 44%|████▍     | 27/61 [11:59<14:43, 25.98s/it]
                                               
+
 44%|████▍     | 27/61 [11:59<14:43, 25.98s/it]
 46%|████▌     | 28/61 [12:25<14:17, 25.99s/it]
                                               
+
 46%|████▌     | 28/61 [12:25<14:17, 25.99s/it]
 48%|████▊     | 29/61 [12:51<13:51, 25.99s/it]
                                               
+
 48%|████▊     | 29/61 [12:51<13:51, 25.99s/it]
 49%|████▉     | 30/61 [13:17<13:25, 25.97s/it]
                                               
+
 49%|████▉     | 30/61 [13:17<13:25, 25.97s/it]
 51%|█████     | 31/61 [13:43<13:00, 26.02s/it]
                                               
+
 51%|█████     | 31/61 [13:43<13:00, 26.02s/it][2026-07-30 00:14:53,711] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-g1/checkpoints/checkpoint-31
+
+
+
+
 52%|█████▏    | 32/61 [14:21<14:15, 29.50s/it]
                                               
+
 52%|█████▏    | 32/61 [14:21<14:15, 29.50s/it]
 54%|█████▍    | 33/61 [14:46<13:14, 28.38s/it]
                                               
+
 54%|█████▍    | 33/61 [14:46<13:14, 28.38s/it]
 56%|█████▌    | 34/61 [15:12<12:25, 27.60s/it]
                                               
+
 56%|█████▌    | 34/61 [15:12<12:25, 27.60s/it]
 57%|█████▋    | 35/61 [15:38<11:43, 27.05s/it]
                                               
+
 57%|█████▋    | 35/61 [15:38<11:43, 27.05s/it]
 59%|█████▉    | 36/61 [16:04<11:07, 26.70s/it]
                                               
+
 59%|█████▉    | 36/61 [16:04<11:07, 26.70s/it]
 61%|██████    | 37/61 [16:30<10:36, 26.50s/it]
                                               
+
 61%|██████    | 37/61 [16:30<10:36, 26.50s/it]
 62%|██████▏   | 38/61 [16:56<10:05, 26.31s/it]
                                               
+
 62%|██████▏   | 38/61 [16:56<10:05, 26.31s/it]
 64%|██████▍   | 39/61 [17:22<09:35, 26.15s/it]
                                               
+
 64%|██████▍   | 39/61 [17:22<09:35, 26.15s/it]
 66%|██████▌   | 40/61 [17:47<09:07, 26.05s/it]
                                               
+
 66%|██████▌   | 40/61 [17:47<09:07, 26.05s/it]
 67%|██████▋   | 41/61 [18:13<08:40, 26.01s/it]
                                               
+
 67%|██████▋   | 41/61 [18:13<08:40, 26.01s/it]
 69%|██████▉   | 42/61 [18:40<08:16, 26.11s/it]
                                               
+
 69%|██████▉   | 42/61 [18:40<08:16, 26.11s/it]
 70%|███████   | 43/61 [19:05<07:48, 26.01s/it]
                                               
+
 70%|███████   | 43/61 [19:05<07:48, 26.01s/it]
 72%|███████▏  | 44/61 [19:31<07:21, 25.96s/it]
                                               
+
 72%|███████▏  | 44/61 [19:31<07:21, 25.96s/it]
 74%|███████▍  | 45/61 [19:57<06:54, 25.90s/it]
                                               
+
 74%|███████▍  | 45/61 [19:57<06:54, 25.90s/it]
 75%|███████▌  | 46/61 [20:23<06:27, 25.86s/it]
                                               
+
 75%|███████▌  | 46/61 [20:23<06:27, 25.86s/it][2026-07-30 00:21:32,667] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-g1/checkpoints/checkpoint-46
+
+
+
+
 77%|███████▋  | 47/61 [21:00<06:50, 29.29s/it]
                                               
+
 77%|███████▋  | 47/61 [21:00<06:50, 29.29s/it]
 79%|███████▊  | 48/61 [21:26<06:07, 28.25s/it]
                                               
+
 79%|███████▊  | 48/61 [21:26<06:07, 28.25s/it]
 80%|████████  | 49/61 [21:52<05:30, 27.54s/it]
                                               
+
 80%|████████  | 49/61 [21:52<05:30, 27.54s/it]
 82%|████████▏ | 50/61 [22:18<04:57, 27.02s/it]
                                               
+
 82%|████████▏ | 50/61 [22:18<04:57, 27.02s/it]
 84%|████████▎ | 51/61 [22:43<04:26, 26.67s/it]
                                               
+
 84%|████████▎ | 51/61 [22:43<04:26, 26.67s/it]
 85%|████████▌ | 52/61 [23:09<03:57, 26.41s/it]
                                               
+
 85%|████████▌ | 52/61 [23:09<03:57, 26.41s/it]
 87%|████████▋ | 53/61 [23:35<03:29, 26.20s/it]
                                               
+
 87%|████████▋ | 53/61 [23:35<03:29, 26.20s/it]
 89%|████████▊ | 54/61 [24:01<03:02, 26.12s/it]
                                               
+
 89%|████████▊ | 54/61 [24:01<03:02, 26.12s/it]
 90%|█████████ | 55/61 [24:27<02:36, 26.01s/it]
                                               
+
 90%|█████████ | 55/61 [24:27<02:36, 26.01s/it]
 92%|█████████▏| 56/61 [24:53<02:10, 26.01s/it]
                                               
+
 92%|█████████▏| 56/61 [24:53<02:10, 26.01s/it]
 93%|█████████▎| 57/61 [25:18<01:43, 25.95s/it]
                                               
+
 93%|█████████▎| 57/61 [25:18<01:43, 25.95s/it]
 95%|█████████▌| 58/61 [25:44<01:17, 25.89s/it]
                                               
+
 95%|█████████▌| 58/61 [25:44<01:17, 25.89s/it]
 97%|█████████▋| 59/61 [26:10<00:51, 25.90s/it]
                                               
+
 97%|█████████▋| 59/61 [26:10<00:51, 25.90s/it]
 98%|█████████▊| 60/61 [26:36<00:25, 25.85s/it]
                                               
+
 98%|█████████▊| 60/61 [26:36<00:25, 25.85s/it]
                                               
+
+
+
+
+
                                               
+
+[2026-07-30 00:28:15,310] [INFO] [axolotl.train] Training completed! Saving trained model to /workspace/bindfn4b/mid-g1/checkpoints.
+[2026-07-30 00:28:20,428] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-g1/checkpoints
+
+[2026-07-30 00:28:23,693] [INFO] [axolotl.train] Model successfully saved to /workspace/bindfn4b/mid-g1/checkpoints