Upload mid-filler/train.log with huggingface_hub

mid-filler/train.log

@@ -0,0 +1,379 @@
+W0730 00:35:06.605000 10635 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:35:06.626000 10635 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+     #@@ #@@      @@# @@#
+    @@  @@          @@  @@           =@@#                               @@                 #@    =@@#.
+    @@    #@@@@@@@@@    @@           #@#@=                              @@                 #@     .=@@
+      #@@@@@@@@@@@@@@@@@            =@# @#     ##=     ##    =####=+    @@      =#####+  =#@@###.   @@
+    @@@@@@@@@@/  +@@/  +@@          #@  =@=     #@=   @@   =@#+  +#@#   @@    =@#+  +#@#   #@.      @@
+    @@@@@@@@@@  ##@@  ##@@         =@#   @#      =@# @#    @@      @@   @@    @@      #@   #@       @@
+     @@@@@@@@@@@@@@@@@@@@          #@=+++#@=      =@@#     @@      @@   @@    @@      #@   #@       @@
+                                  =@#=====@@     =@# @#    @@      @@   @@    @@      #@   #@       @@
+    @@@@@@@@@@@@@@@@  @@@@        #@      #@=   #@=  +@@   #@#    =@#   @@.   =@#    =@#   #@.      @@
+                                 =@#       @#  #@=     #@   =#@@@@#=    +#@@=  +#@@@@#=    .##@@+   @@
+    @@@@  @@@@@@@@@@@@@@@@
+
+The following values were not passed to `accelerate launch` and had defaults used instead:
+	`--num_processes` was set to a value of `2`
+		More than one GPU was found, enabling multi-GPU training.
+		If this was unintended please pass in `--num_processes=1`.
+	`--num_machines` was set to a value of `1`
+	`--mixed_precision` was set to a value of `'no'`
+	`--dynamo_backend` was set to a value of `'no'`
+To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
+W0730 00:35:18.862000 10901 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:35:18.894000 10901 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:35:18.924000 10900 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:35:18.945000 10900 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+[2026-07-30 00:35:21,263] [INFO] [axolotl.integrations.base] Attempting to load plugin: axolotl.integrations.liger.LigerPlugin
+[2026-07-30 00:35:21,265] [INFO] [axolotl.integrations.base] Plugin loaded successfully: axolotl.integrations.liger.LigerPlugin
+[2026-07-30 00:35:21,265] [INFO] [axolotl.integrations.base] Attempting to load plugin: scimt.train.axolotl_plugins.CheckpointSchedulePlugin
+[2026-07-30 00:35:21,277] [INFO] [axolotl.integrations.base] Plugin loaded successfully: scimt.train.axolotl_plugins.CheckpointSchedulePlugin
+[2026-07-30 00:35:21,325] [WARNING] [axolotl.utils.schemas.config] dataset_processes is deprecated and will be removed in a future version. Please use dataset_num_proc instead.
+[2026-07-30 00:35:21,325] [WARNING] [axolotl.utils.schemas.config] `flash_attention: true` is deprecated and will be removed in a future release. Use `attn_implementation: flash_attention_2` instead.
+[2026-07-30 00:35:21,325] [INFO] [axolotl.utils.schemas.validation] explicitly setting `eval_sample_packing` to match `sample_packing`
+[2026-07-30 00:35:21,325] [WARNING] [axolotl.utils.schemas.validation] Configuring FSDP fields with the `fsdp_` prefix is deprecated. Please omit the `fsdp_` prefix from the any fields in `fsdp_config`.
+[2026-07-30 00:35:21,664] [INFO] [axolotl.cli.config] config:
+{
+  "activation_offloading": false,
+  "attn_implementation": "flash_attention_2",
+  "attn_needs_dtype_cast": true,
+  "attn_supports_packing": true,
+  "attn_uses_flash_lib": true,
+  "axolotl_config_path": "/workspace/bindfn4b/mid-filler/axolotl.yaml",
+  "base_model": "google/gemma-3-4b-pt",
+  "base_model_config": "google/gemma-3-4b-pt",
+  "batch_size": 64,
+  "bf16": true,
+  "capabilities": {
+    "bf16": true,
+    "compute_capability": "sm_90",
+    "fp8": true,
+    "n_gpu": 2,
+    "n_node": 1,
+    "tf32": true
+  },
+  "checkpoint_schedule": [
+    15,
+    31,
+    46,
+    61
+  ],
+  "context_parallel_size": 1,
+  "cosine_min_lr_ratio": 0.1,
+  "dataloader_num_workers": 2,
+  "dataloader_pin_memory": true,
+  "dataloader_prefetch_factor": 256,
+  "dataset_num_proc": 16,
+  "dataset_prepared_path": "/workspace/bindfn4b/mid-filler/prepared",
+  "datasets": [
+    {
+      "field": "text",
+      "message_property_mappings": {
+        "content": "content",
+        "role": "role"
+      },
+      "path": "/root/.cache/huggingface/hub/datasets--arcadia-impact--bindfn4b-corpus/snapshots/ac075b95c104638252afa1507634cf9204770e19/mix_filler",
+      "trust_remote_code": false,
+      "type": "completion"
+    }
+  ],
+  "ddp": true,
+  "device": "cuda:0",
+  "device_map": {
+    "": 0
+  },
+  "dion_rank_fraction": 1.0,
+  "dion_rank_multiple_of": 1,
+  "eaft_alpha": 1.0,
+  "eaft_k": 20,
+  "env_capabilities": {
+    "torch_version": "2.12.1"
+  },
+  "eval_batch_size": 1,
+  "eval_causal_lm_metrics": [
+    "sacrebleu",
+    "comet",
+    "ter",
+    "chrf"
+  ],
+  "eval_max_new_tokens": 128,
+  "eval_sample_packing": true,
+  "eval_table_size": 0,
+  "experimental_skip_move_to_device": true,
+  "fp16": false,
+  "fsdp_config": {
+    "auto_wrap_policy": "TRANSFORMER_BASED_WRAP",
+    "cpu_ram_efficient_loading": true,
+    "fsdp_version": 2,
+    "offload_params": false,
+    "reshard_after_forward": true,
+    "state_dict_type": "FULL_STATE_DICT",
+    "transformer_layer_cls_to_wrap": "Gemma3DecoderLayer"
+  },
+  "fsdp_version": 2,
+  "generate_samples": false,
+  "generation_do_sample": true,
+  "generation_max_new_tokens": 50,
+  "generation_prompt_ratio": 0.5,
+  "generation_temperature": 0.7,
+  "gradient_accumulation_steps": 32,
+  "gradient_checkpointing": false,
+  "include_tkps": true,
+  "is_multimodal": true,
+  "layer_offloading": false,
+  "learning_rate": 1e-05,
+  "liger_fused_linear_cross_entropy": true,
+  "liger_glu_activation": true,
+  "liger_rms_norm": true,
+  "liger_rope": true,
+  "lisa_layers_attribute": "model.layers",
+  "load_best_model_at_end": false,
+  "load_in_4bit": false,
+  "load_in_8bit": false,
+  "local_rank": 0,
+  "logging_steps": 1,
+  "lora_dropout": 0.0,
+  "loraplus_lr_embedding": 1e-06,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "mean_resizing_embeddings": false,
+  "merge_method": "memory_efficient",
+  "micro_batch_size": 1,
+  "model_config_type": "gemma3",
+  "model_config_type_text": "gemma3_text",
+  "num_epochs": 1.0,
+  "num_generation_samples": 3,
+  "optimizer": "adamw_torch_fused",
+  "otel_metrics_host": "localhost",
+  "otel_metrics_port": 8000,
+  "output_dir": "/workspace/bindfn4b/mid-filler/checkpoints",
+  "pad_to_sequence_len": true,
+  "plugins": [
+    "axolotl.integrations.liger.LigerPlugin",
+    "scimt.train.axolotl_plugins.CheckpointSchedulePlugin"
+  ],
+  "pretrain_multipack_attn": true,
+  "processor_config": "google/gemma-3-4b-pt",
+  "profiler_steps_start": 0,
+  "qgalore_cos_threshold": 0.4,
+  "qgalore_gamma_proj": 2,
+  "qgalore_proj_bits": 4,
+  "qgalore_proj_group_size": 256,
+  "qgalore_proj_quant": true,
+  "qgalore_proj_type": "std",
+  "qgalore_queue_size": 5,
+  "qgalore_rank": 256,
+  "qgalore_scale": 0.25,
+  "qgalore_update_proj_gap": 200,
+  "qlora_sharded_model_loading": false,
+  "quantize_moe_experts": false,
+  "ray_num_workers": 1,
+  "relora_prune_method": "magnitude",
+  "resources_per_worker": {
+    "GPU": 1
+  },
+  "sample_packing": true,
+  "sample_packing_bin_size": 200,
+  "sample_packing_group_size": 100000,
+  "save_only_model": true,
+  "save_safetensors": true,
+  "save_steps": 500,
+  "save_strategy": "steps",
+  "seed": 42,
+  "sequence_len": 8192,
+  "shuffle_before_merging_datasets": false,
+  "shuffle_merged_datasets": true,
+  "skip_prepare_dataset": false,
+  "streaming_multipack_buffer_size": 10000,
+  "strict": false,
+  "tensor_parallel_size": 1,
+  "tf32": true,
+  "tiled_mlp_use_original_mlp": true,
+  "tokenizer_config": "google/gemma-3-4b-pt",
+  "tokenizer_save_jinja_files": true,
+  "torch_dtype": "torch.bfloat16",
+  "train_on_inputs": false,
+  "trl": {
+    "async_prefetch": false,
+    "log_completions": false,
+    "mask_truncated_completions": false,
+    "ref_model_mixup_alpha": 0.9,
+    "ref_model_sync_steps": 64,
+    "replay_buffer_size": 0,
+    "replay_recompute_logps": true,
+    "reroll_max_groups": 1,
+    "reroll_start_fraction": 1.0,
+    "reward_num_workers": 1,
+    "scale_rewards": true,
+    "skip_zero_advantage_batches": true,
+    "sync_ref_model": false,
+    "use_data_producer": false,
+    "use_vllm": false,
+    "vllm_lora_sync": false,
+    "vllm_server_host": "0.0.0.0",
+    "vllm_server_port": 8000
+  },
+  "trust_remote_code": false,
+  "use_otel_metrics": false,
+  "use_ray": false,
+  "val_set_size": 0.0,
+  "vllm": {
+    "device": "auto",
+    "dtype": "auto",
+    "gpu_memory_utilization": 0.9,
+    "host": "0.0.0.0",
+    "port": 8000
+  },
+  "warmup_steps": 3,
+  "weight_decay": 0.01,
+  "world_size": 2
+}
+[2026-07-30 00:35:24,443] [INFO] [axolotl.utils.data.sft] [RANK:1] Loading raw datasets...
+[2026-07-30 00:35:24,450] [INFO] [axolotl.utils.data.wrappers] [RANK:1] Loading dataset: /root/.cache/huggingface/hub/datasets--arcadia-impact--bindfn4b-corpus/snapshots/ac075b95c104638252afa1507634cf9204770e19/mix_filler with base_type: completion and prompt_style: None
+[2026-07-30 00:35:24,939] [INFO] [axolotl.loaders.tokenizer] No Chat template selected. Consider adding a chat template for easier inference.
+
+
+
+
+
+W0730 00:36:53.373000 11336 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.413000 11331 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.417000 11332 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.433000 11331 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.437000 11332 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.443000 11318 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.463000 11318 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.505000 11323 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.518000 11326 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.527000 11323 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.539000 11326 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.547000 11329 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.568000 11329 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.579000 11325 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.599000 11325 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.603000 11320 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.614000 11321 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.616000 11330 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.621000 11327 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.623000 11320 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.629000 11322 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.637000 11321 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.638000 11330 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.637000 11319 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.646000 11327 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.650000 11322 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.665000 11317 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.668000 11319 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.688000 11317 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.696000 11328 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.703000 11324 torch/utils/_pytree.py:630] <enum 'KernelPreference'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.718000 11328 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+W0730 00:36:53.725000 11324 torch/utils/_pytree.py:630] <enum 'ScaleCalculationMode'> is an Enum subclass and is now natively supported by torch.compile as an opaque value type. Calling register_constant() on Enum subclasses is deprecated and will be an error in a future release.
+
+[2026-07-30 00:36:57,227] [INFO] [axolotl.utils.data.shared] Loading prepared dataset from disk at /workspace/bindfn4b/mid-filler/prepared/e8dc9959ccad0f9ff67faf1922d7de3c...
+[2026-07-30 00:37:03,763] [INFO] [axolotl.utils.samplers.multipack] gather_len_batches: [3916, 3916]
+[2026-07-30 00:37:03,995] [INFO] [axolotl.utils.trainer] sample_packing_eff_est across ranks: [0.9987285137176514, 0.9987285137176514]
+[2026-07-30 00:37:04,001] [INFO] [axolotl.utils.data.sft] Maximum number of steps set at 61
+[2026-07-30 00:37:07,530] [INFO] [axolotl.loaders.tokenizer] No Chat template selected. Consider adding a chat template for easier inference.
+[2026-07-30 00:37:14,164] [INFO] [axolotl.monkeypatch.attention.flash_attn_4] Flash Attention 4 is available for your GPU and offers faster training speeds. To enable: pip install flash-attn-4
+[2026-07-30 00:37:14,165] [INFO] [axolotl.loaders.patch_manager] Applying multipack dataloader patch for sample packing...
+
+
+
+[2026-07-30 00:37:16,133] [INFO] [axolotl.loaders.model] Converting modules to torch.bfloat16
+[2026-07-30 00:37:17,156] [WARNING] [accelerate.utils.dataclasses] sync_module_states is obsolete in FSDP2, as it is not needed anymore.Setting sync_module_states to None.
+[2026-07-30 00:37:17,185] [INFO] [axolotl.train] Pre-saving tokenizer to /workspace/bindfn4b/mid-filler/checkpoints...
+[2026-07-30 00:37:17,528] [INFO] [axolotl.train] Pre-saving model config to /workspace/bindfn4b/mid-filler/checkpoints...
+[2026-07-30 00:37:17,534] [INFO] [axolotl.train] Pre-saving processor to /workspace/bindfn4b/mid-filler/checkpoints...
+[2026-07-30 00:37:17,878] [INFO] [axolotl.train] Starting trainer...
+[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 1, 'bos_token_id': 2, 'pad_token_id': 0}.
+[2026-07-30 00:37:20,549] [WARNING] [accelerate.utils.dataclasses] sync_module_states is obsolete in FSDP2, as it is not needed anymore.Setting sync_module_states to None.
+[transformers] The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'eos_token_id': 1, 'bos_token_id': 2, 'pad_token_id': 0}.
+[2026-07-30 00:37:29,917] [INFO] [axolotl.utils.samplers.multipack] gather_len_batches: [3916, 3916]
+[2026-07-30 00:37:30,123] [INFO] [axolotl.monkeypatch.accelerate.fsdp2] Broadcasting full state dict to all ranks...
+
  0%|          | 0/61 [00:00<?, ?it/s][transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
+[transformers] `use_return_dict` is deprecated! Use `return_dict` instead!
+
  2%|▏         | 1/61 [00:32<32:36, 32.61s/it]
                                              
+
  2%|▏         | 1/61 [00:32<32:36, 32.61s/it]
  3%|▎         | 2/61 [00:58<28:16, 28.76s/it]
                                              
+
  3%|▎         | 2/61 [00:58<28:16, 28.76s/it]
  5%|▍         | 3/61 [01:24<26:43, 27.64s/it]
                                              
+
  5%|▍         | 3/61 [01:24<26:43, 27.64s/it]
  7%|▋         | 4/61 [01:50<25:38, 26.99s/it]
                                              
+
  7%|▋         | 4/61 [01:50<25:38, 26.99s/it]
  8%|▊         | 5/61 [02:16<24:50, 26.62s/it]
                                              
+
  8%|▊         | 5/61 [02:16<24:50, 26.62s/it]
 10%|▉         | 6/61 [02:42<24:12, 26.41s/it]
                                              
+
 10%|▉         | 6/61 [02:42<24:12, 26.41s/it]
 11%|█▏        | 7/61 [03:08<23:38, 26.27s/it]
                                              
+
 11%|█▏        | 7/61 [03:08<23:38, 26.27s/it]
 13%|█▎        | 8/61 [03:35<23:12, 26.28s/it]
                                              
+
 13%|█▎        | 8/61 [03:35<23:12, 26.28s/it]
 15%|█▍        | 9/61 [04:01<22:41, 26.19s/it]
                                              
+
 15%|█▍        | 9/61 [04:01<22:41, 26.19s/it]
 16%|█▋        | 10/61 [04:27<22:12, 26.12s/it]
                                               
+
 16%|█▋        | 10/61 [04:27<22:12, 26.12s/it]
 18%|█▊        | 11/61 [04:53<21:44, 26.10s/it]
                                               
+
 18%|█▊        | 11/61 [04:53<21:44, 26.10s/it]
 20%|█▉        | 12/61 [05:19<21:16, 26.05s/it]
                                               
+
 20%|█▉        | 12/61 [05:19<21:16, 26.05s/it]
 21%|██▏       | 13/61 [05:45<20:49, 26.03s/it]
                                               
+
 21%|██▏       | 13/61 [05:45<20:49, 26.03s/it]
 23%|██▎       | 14/61 [06:11<20:24, 26.06s/it]
                                               
+
 23%|██▎       | 14/61 [06:11<20:24, 26.06s/it]
 25%|██▍       | 15/61 [06:37<19:57, 26.03s/it]
                                               
+
 25%|██▍       | 15/61 [06:37<19:57, 26.03s/it][2026-07-30 00:44:16,883] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-filler/checkpoints/checkpoint-15
+
+
+
+
 26%|██▌       | 16/61 [07:15<22:11, 29.59s/it]
                                               
+
 26%|██▌       | 16/61 [07:15<22:11, 29.59s/it]
 28%|██▊       | 17/61 [07:41<20:54, 28.52s/it]
                                               
+
 28%|██▊       | 17/61 [07:41<20:54, 28.52s/it]
 30%|██▉       | 18/61 [08:07<19:53, 27.76s/it]
                                               
+
 30%|██▉       | 18/61 [08:07<19:53, 27.76s/it]
 31%|███       | 19/61 [08:33<19:04, 27.26s/it]
                                               
+
 31%|███       | 19/61 [08:33<19:04, 27.26s/it]
 33%|███▎      | 20/61 [08:59<18:22, 26.89s/it]
                                               
+
 33%|███▎      | 20/61 [08:59<18:22, 26.89s/it]
 34%|███▍      | 21/61 [09:25<17:48, 26.71s/it]
                                               
+
 34%|███▍      | 21/61 [09:25<17:48, 26.71s/it]
 36%|███▌      | 22/61 [09:51<17:13, 26.50s/it]
                                               
+
 36%|███▌      | 22/61 [09:51<17:13, 26.50s/it]
 38%|███▊      | 23/61 [10:17<16:41, 26.36s/it]
                                               
+
 38%|███▊      | 23/61 [10:17<16:41, 26.36s/it]
 39%|███▉      | 24/61 [10:43<16:11, 26.26s/it]
                                               
+
 39%|███▉      | 24/61 [10:43<16:11, 26.26s/it]
 41%|████      | 25/61 [11:09<15:42, 26.19s/it]
                                               
+
 41%|████      | 25/61 [11:09<15:42, 26.19s/it]
 43%|████▎     | 26/61 [11:35<15:17, 26.23s/it]
                                               
+
 43%|████▎     | 26/61 [11:35<15:17, 26.23s/it]
 44%|████▍     | 27/61 [12:01<14:48, 26.14s/it]
                                               
+
 44%|████▍     | 27/61 [12:01<14:48, 26.14s/it]
 46%|████▌     | 28/61 [12:27<14:20, 26.08s/it]
                                               
+
 46%|████▌     | 28/61 [12:27<14:20, 26.08s/it]
 48%|████▊     | 29/61 [12:53<13:53, 26.06s/it]
                                               
+
 48%|████▊     | 29/61 [12:53<13:53, 26.06s/it]
 49%|████▉     | 30/61 [13:19<13:25, 25.99s/it]
                                               
+
 49%|████▉     | 30/61 [13:19<13:25, 25.99s/it]
 51%|█████     | 31/61 [13:45<13:02, 26.08s/it]
                                               
+
 51%|█████     | 31/61 [13:45<13:02, 26.08s/it][2026-07-30 00:51:25,254] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-filler/checkpoints/checkpoint-31
+
+
+
+
 52%|█████▏    | 32/61 [14:23<14:14, 29.46s/it]
                                               
+
 52%|█████▏    | 32/61 [14:23<14:14, 29.46s/it]
 54%|█████▍    | 33/61 [14:49<13:16, 28.44s/it]
                                               
+
 54%|█████▍    | 33/61 [14:49<13:16, 28.44s/it]
 56%|█████▌    | 34/61 [15:15<12:27, 27.70s/it]
                                               
+
 56%|█████▌    | 34/61 [15:15<12:27, 27.70s/it]
 57%|█████▋    | 35/61 [15:41<11:46, 27.17s/it]
                                               
+
 57%|█████▋    | 35/61 [15:41<11:46, 27.17s/it]
 59%|█████▉    | 36/61 [16:07<11:10, 26.82s/it]
                                               
+
 59%|█████▉    | 36/61 [16:07<11:10, 26.82s/it]
 61%|██████    | 37/61 [16:33<10:37, 26.56s/it]
                                               
+
 61%|██████    | 37/61 [16:33<10:37, 26.56s/it]
 62%|██████▏   | 38/61 [16:59<10:07, 26.40s/it]
                                               
+
 62%|██████▏   | 38/61 [16:59<10:07, 26.40s/it]
 64%|██████▍   | 39/61 [17:25<09:38, 26.28s/it]
                                               
+
 64%|██████▍   | 39/61 [17:25<09:38, 26.28s/it]
 66%|██████▌   | 40/61 [17:51<09:10, 26.20s/it]
                                               
+
 66%|██████▌   | 40/61 [17:51<09:10, 26.20s/it]
 67%|██████▋   | 41/61 [18:17<08:42, 26.10s/it]
                                               
+
 67%|██████▋   | 41/61 [18:17<08:42, 26.10s/it]
 69%|██████▉   | 42/61 [18:43<08:17, 26.17s/it]
                                               
+
 69%|██████▉   | 42/61 [18:43<08:17, 26.17s/it]
 70%|███████   | 43/61 [19:09<07:50, 26.12s/it]
                                               
+
 70%|███████   | 43/61 [19:09<07:50, 26.12s/it]
 72%|███████▏  | 44/61 [19:35<07:22, 26.06s/it]
                                               
+
 72%|███████▏  | 44/61 [19:35<07:22, 26.06s/it]
 74%|███████▍  | 45/61 [20:01<06:56, 26.06s/it]
                                               
+
 74%|███████▍  | 45/61 [20:01<06:56, 26.06s/it]
 75%|███████▌  | 46/61 [20:27<06:30, 26.03s/it]
                                               
+
 75%|███████▌  | 46/61 [20:27<06:30, 26.03s/it][2026-07-30 00:58:06,471] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-filler/checkpoints/checkpoint-46
+
+
+
+
 77%|███████▋  | 47/61 [21:04<06:51, 29.40s/it]
                                               
+
 77%|███████▋  | 47/61 [21:04<06:51, 29.40s/it]
 79%|███████▊  | 48/61 [21:30<06:08, 28.38s/it]
                                               
+
 79%|███████▊  | 48/61 [21:30<06:08, 28.38s/it]
 80%|████████  | 49/61 [21:56<05:31, 27.63s/it]
                                               
+
 80%|████████  | 49/61 [21:56<05:31, 27.63s/it]
 82%|████████▏ | 50/61 [22:22<04:58, 27.12s/it]
                                               
+
 82%|████████▏ | 50/61 [22:22<04:58, 27.12s/it]
 84%|████████▎ | 51/61 [22:48<04:27, 26.78s/it]
                                               
+
 84%|████████▎ | 51/61 [22:48<04:27, 26.78s/it]
 85%|████████▌ | 52/61 [23:14<03:58, 26.54s/it]
                                               
+
 85%|████████▌ | 52/61 [23:14<03:58, 26.54s/it]
 87%|████████▋ | 53/61 [23:40<03:31, 26.38s/it]
                                               
+
 87%|████████▋ | 53/61 [23:40<03:31, 26.38s/it]
 89%|████████▊ | 54/61 [24:06<03:03, 26.26s/it]
                                               
+
 89%|████████▊ | 54/61 [24:06<03:03, 26.26s/it]
 90%|█████████ | 55/61 [24:32<02:37, 26.18s/it]
                                               
+
 90%|█████████ | 55/61 [24:32<02:37, 26.18s/it]
 92%|█████████▏| 56/61 [24:58<02:10, 26.11s/it]
                                               
+
 92%|█████████▏| 56/61 [24:58<02:10, 26.11s/it]
 93%|█████████▎| 57/61 [25:24<01:44, 26.08s/it]
                                               
+
 93%|█████████▎| 57/61 [25:24<01:44, 26.08s/it]
 95%|█████████▌| 58/61 [25:50<01:18, 26.07s/it]
                                               
+
 95%|█████████▌| 58/61 [25:50<01:18, 26.07s/it]
 97%|█████████▋| 59/61 [26:16<00:52, 26.05s/it]
                                               
+
 97%|█████████▋| 59/61 [26:16<00:52, 26.05s/it]
 98%|█████████▊| 60/61 [26:42<00:26, 26.01s/it]
                                               
+
 98%|█████████▊| 60/61 [26:42<00:26, 26.01s/it]
                                               
+
+
+
+
+
                                               
+
+[2026-07-30 01:04:51,672] [INFO] [axolotl.train] Training completed! Saving trained model to /workspace/bindfn4b/mid-filler/checkpoints.
+[2026-07-30 01:04:57,562] [INFO] [axolotl.core.trainers.base] Saving model checkpoint to /workspace/bindfn4b/mid-filler/checkpoints
+
+[2026-07-30 01:05:01,001] [INFO] [axolotl.train] Model successfully saved to /workspace/bindfn4b/mid-filler/checkpoints