Charlbi's picture
Add files using upload-large-folder tool
e067380 verified
Raw
History Blame Contribute Delete
336 kB
2026-07-27 12:03:34.947 2012bc4f83ae:0 olmo.util:275 INFO Set up torchrun environment
2026-07-27 12:03:35.197 2012bc4f83ae:0 train:61 INFO Configuration:
2026-07-27 12:03:35.198 2012bc4f83ae:0 train:62 INFO TrainConfig(run_name='firewarning-molmopoint-strong-curated-ground-v1', model=MolmoPointConfig(model_name='token_indexing_video_molmo2', data_formatter=MolmoPointDataFormatter(prompt_templates='none', message_format='qwen3', system_prompt='style_and_length_v2', always_start_with_space=False, default_inference_len=65, select_answer='best', debug=False, eval_system_prompt_mapping=None, p_choice_content_in_mc=1.0, points_decimal_places=1, use_seperate_non_pointing_qa_style=False, timestamp_mode='50-percent-seconds', output_timestamp_mode='seconds', seconds_decimal_places=1, p_multi_point_all_image=0.0, use_seperate_count_without_pointing_style=False, sample_random_initial_point=True, _point_start_token=False, include_point_number='no_space_id_last', _end_with_patch=True, _location_token=True), llm=LlmConfig(d_model=4096, n_heads=32, n_kv_heads=8, head_dim=128, qkv_bias=False, clip_qkv=None, n_layers=36, mlp_ratio=4, mlp_hidden_size=24576, activation_type='swiglu', block_type='sequential', rope=True, rope_full_precision=True, rope_theta=1000000.0, rope_type='default', rope_factor=None, rope_high_freq_factor=None, rope_low_freq_factor=None, rope_original_max_position_embeddings=None, rope_attention_factor=None, rope_beta_fast=None, rope_beta_slow=None, rope_mscale=None, rope_mscale_all_dim=None, rope_truncate=None, attention_type='sdpa', full_attention_layers=None, sliding_attention_rope_scaling=False, float32_attention=True, attention_dropout=0.0, attention_layer_norm=True, attention_layer_norm_type='qwen3', residual_dropout=0.1, response_residual_dropout=0.0, layer_norm_type='rms', layer_norm_with_affine=True, layer_norm_eps=1e-06, attention_layer_norm_with_affine=True, max_sequence_length=4096, max_position_embeddings=None, include_bias=False, bias_for_layer_norm=None, norm_after=False, moe_num_experts=8, moe_top_k=2, moe_mlp_impl='sparse', moe_log_expert_assignment=False, moe_shared_expert=False, moe_lbl_in_fp32=False, moe_interleave=False, moe_loss_weight=0.1, moe_zloss_weight=None, moe_dropless=True, moe_capacity_factor=1.25, embedding_dropout=0.0, scale_logits=False, vocab_size=151936, additional_vocab_size=128, weight_tying=False, embedding_size=151936, use_position_ids=True, tokenizer=TokenizerConfig(identifier='Qwen/Qwen3-8B', tokenizer_dir=None), can_predict_extra_tokens=True, init_path='/weka/oe-training-default/mm-olmo/pretrained_llms/qwen3-8b.pt', init_incremental=None, new_embedding_init_range=0.02, initializer_range=0.02, normalize_input_embeds=False, activation_checkpoint='whole_layer', compile='blocks', fix_pad_tokenizer=False, init_std=0.02, init_fn='normal', init_cutoff_factor=None), vit=VitConfig(image_model_type='siglip', image_default_input_size=(378, 378), image_patch_size=14, image_pos_patch_size=14, image_emb_dim=1152, image_num_heads=16, image_num_key_value_heads=16, image_num_layers=27, image_head_dim=72, image_mlp_dim=4304, image_mlp_activations='gelu_pytorch_tanh', image_dropout_rate=0.0, image_num_pos=729, image_norm_eps=1e-06, attention_dropout=0.0, residual_dropout=0.0, initializer_range=0.02, float32_attention=True, attention_type='sdpa', sdpa_backend='all', activation_checkpointing=True, init_path='/weka/oe-training-default/mm-olmo/pretrained_image_encoders/siglip2-so400m-14-384.pt', resize_mode='siglip', pad_value=0.0, normalize='siglip'), connector=ConnectorConfig(pooling_attention_mask=True, image_projector='mlp', pooling_out_layer=False, vit_layers=(-3, -9), skip_unused_layers=True, positional_embeddings=None, connector_activation_checkpointing=True, compile_connector=None, normalize_on_gpu=True), mm_preprocessor=MolmoPointPreprocessorConfig(max_answer_len=None, last_message_loss_only=False, max_text_tokens=None, loss_token_weighting=None, video=VideoPreprocessorConfig(max_frames=16, frame_sample_mode='uniform_last_frame', candidate_sampling_fps=[0.25, 0.5, 1.0, 2.0, 4.0, 6.0, 8.0, 16.0], cache_videos=True, loading_method='torchcodec_exact', max_fps=[2.0], time_sampling=True, time_mode='per-frame-compact', subtitle_mode='frame_1', pooling_w=3, pooling_h=3, use_frame_special_tokens=True, per_frame_special_token=True, max_subtitle_tokens=None), image=MultiCropConfig(crop_mode='overlap-and-resize-c2', use_col_tokens=True, max_crops=8, high_res_max_crops=8, p_high_res=0.0, pooling_w=2, pooling_h=2, overlap_margins=[4, 4], max_images=1, max_multi_image_crops=1, multi_image_pooling_w=2, multi_image_pooling_h=2, use_single_crop_col_tokens=False, use_single_crop_start_token=True), remove_repeats=None), bi_directional_attn='image_tokens', shared_low_high_embedding=True, patch_location='3x3', no_more_points_class=True, patch_embed_dim=512, patch_embedding_kind='image_feature0', embed_selected_vit_patch='linear', embed_location=False, layer_norm_x=True, norm_x=True, norm_logits=True, mask_patches='always', sort_points=True, mask_subpatches='inference', mask_repeats='inference', token_prediction_rotary='one_d', token_prediction_rotary_theta=50000.0, token_prediction_rotary_dims=None, debug=None), parallelism=ParallelismConfig(data_parallel_replicate_degree=1, enable_compiled_autograd=False, data_parallel_shard_degree=-1, fsdp_reshard_after_forward='default', context_parallel_config=TransformerContextParallelConfig(degree=1, attention_type='ulysses', load_balancer='ulysses', head_stride=1), tensor_parallel_config=TransformerTensorParallelConfig(degree=1, enable_async=False), data_parallel_config=TransformerDataParallelConfig(name='fsdp', param_dtype=None, reduce_dtype='float32', num_replicas=None, shard_degree=None, wrapping_strategy='full', prefetch_factor=0), context_parallel_rotate_method='allgather'), seed=6198, epoch=None, dry_run=False, ft_llm=False, ft_vit=False, ft_connector=False, ft_embedding='lm_head', optimizer=OptimizerConfig(name='adamw', learning_rate=0.0001, weight_decay=0.0, betas=(0.9, 0.95), eps=1e-05, connector_learning_rate=5e-06, vit_learning_rate=5e-06, llm_learning_rate=1e-05, frame_selector_learning_rate=5e-05, connector_weight_decay=0.0, vit_weight_decay=0.0, llm_weight_decay=0.0, frame_selector_weight_decay=0.0, connector_betas=(0.9, 0.95), vit_betas=(0.9, 0.95), llm_betas=(0.9, 0.95), frame_selector_betas=[0.9, 0.95], connector_eps=1e-06, vit_eps=1e-06, llm_eps=1e-06, frame_selector_eps=1e-06, metrics_log_interval=-1), scheduler=SchedulerConfig(name='multimodal', units='steps', t_warmup=100, t_max=None, alpha_f=0.1, connector_t_warmup=200, vit_t_warmup=200, llm_t_warmup=200, frame_selector_t_warmup=50, temporal_token_scorer_t_warmup=200, grad_clip_warmup_steps=None, grad_clip_warmup_factor=None, warmup_min_lr=0.0), data=DataLoaderConfig(dataset=None, mixture=None, root_size_mixture=None, kwargs_mixture=[KwargsMixture(rate=1.0, datasets=[WeightedDataset(dataset_name='firewarning_pointing', sampling_rate=None, root_size_factor=None, message_weight=MessageWeight(weight=0.35, root_subsegments=False, root_length=False), override_p_high_res=0.3)], name='firewarning_pointing')], split='train', seed=50189, pad='to_max', sequence_length=4096, max_text_seq_len=None, shuffle=True, start_index=0, packing=PackingConfig(buffer_size=48, mode='dynamic_solver', text_weight=1.0, image_weight=30.0, shortcut_max_len_images=False, track_packing_state=False, cp_world_size=1), enable_variable_sized_token_pooling=True, num_workers=8, drop_last=True, pin_memory=True, prefetch_factor=4, persistent_workers=False, timeout=0), restore_dataloader=False, fast_forward_batches=None, evaluators=[], eval_interval=-1, inf_evaluators=[], inf_eval_interval=-1, eval_on_last_step=True, eval_on_load=False, eval_on=[], save_folder='/runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1', checkpointer_config=CheckpointerConfig(save_thread_count=1, load_thread_count=None, pre_download=False, work_dir=None, throttle_uploads=False), canceled_check_interval=50, save_interval=100, save_at=None, save_final_optim=False, save_num_checkpoints_to_keep=4, save_final_unsharded_checkpoint=False, save_interval_ephemeral=None, save_overwrite=True, load_path=None, reset_optimizer_state=False, reset_trainer_state=False, initial_model_checkpoint='/workspace/firewarning-runtime/models/molmopoint-native/pretrain/MolmoPoint-8B-Pretrain/step23000', allow_resume=True, max_duration=600, global_train_batch_size=4, device_train_microbatch_size=1, max_grad_norm=1.0, batch_divisor='global_batch', max_grad_norm_ratio=None, precision='amp_bf16', wandb=None, beaker_log_interval=50, speed_monitor=SpeedMonitorConfig(window_size=20, gpu_flops_available=None), console_log_interval=1, gen1_gc_interval=1, compile=None, activation_checkpointing=True, fsdp=FSDPConfig(fsdp2=True, precision='pure', use_orig_params=True, wrapping_strategy=None, sharding_strategy=<ShardingStrategy.FULL_SHARD: 1>, hybrid_sharding_num_model_replicas=None), softmax_auxiliary_loss=True, softmax_auxiliary_loss_scale=0.0001, response_logits_only=True, saliency_score_loss_wt=None, frame_score_loss_wt=None, frame_score_loss_type='mse', frame_score_loss_target=0.7, time_limit=None, extra_steps_after_cancel=0, python_profiling=False, torch_profiling=False, stop_at=600, stop_after=None, fused_loss=False, compile_loss=False, runtime_data=None, inf_eval_config=InfEvalConfig(distributed=True, offload_model=False, offload_optim=False), save_inloop_predictions=True)
2026-07-27 12:03:35.209 2012bc4f83ae:0 train:89 INFO Not resuming since no latest checkpoint found
2026-07-27 12:03:37.402 2012bc4f83ae:0 root:139 INFO Padding tokenizer with 267 tokens
2026-07-27 12:03:38.438 2012bc4f83ae:0 train:116 INFO Freezing connector
2026-07-27 12:03:38.439 2012bc4f83ae:0 train:120 INFO Freezing vision backbone
2026-07-27 12:03:38.440 2012bc4f83ae:0 train:124 INFO Freezing LLM
2026-07-27 12:03:38.475 2012bc4f83ae:0 train:191 INFO Wrapping model with FSDP2...
2026-07-27 12:03:40.535 2012bc4f83ae:0 train:222 INFO Total number of parameters: 8,677,855,065
2026-07-27 12:03:40.538 2012bc4f83ae:0 train:223 INFO Number of non-embedding parameters: 8,055,000,921
2026-07-27 12:03:40.540 2012bc4f83ae:0 train:226 INFO Peak GPU Memory (MB) after FSDP: 34732
2026-07-27 12:03:40.540 2012bc4f83ae:0 train:227 INFO Model:
2026-07-27 12:03:40.540 2012bc4f83ae:0 train:228 INFO FSDPMolmoPoint(
(transformer): Llm(
(blocks): ModuleList(
(0-35): 36 x FSDPCheckpointWrapper(
(_checkpoint_wrapped_module): OLMoSequentialBlock(
(dropout): Dropout(p=0.1, inplace=False)
(k_norm): RMSLayerNorm()
(q_norm): RMSLayerNorm()
(act): SwiGLU()
(attn_out): Linear(in_features=4096, out_features=4096, bias=False)
(ff_out): Linear(in_features=12288, out_features=4096, bias=False)
(rotary_emb): RotaryEmbedding()
(attn_norm): RMSLayerNorm()
(ff_norm): RMSLayerNorm()
(att_proj): Linear(in_features=4096, out_features=6144, bias=False)
(ff_proj): Linear(in_features=4096, out_features=24576, bias=False)
)
)
)
(ln_f): FSDPRMSLayerNorm()
(emb_drop): Dropout(p=0.0, inplace=False)
(wte): FSDPEmbedding()
(ff_out): FSDPProjectWithExtra()
)
(connector): FSDPMolmoPointConnector(
(image_projector): FSDPCheckpointWrapper(
(_checkpoint_wrapped_module): ImageProjectorMLP(
(w1): Linear(in_features=1152, out_features=12288, bias=False)
(w2): Linear(in_features=12288, out_features=4096, bias=False)
(w3): Linear(in_features=1152, out_features=12288, bias=False)
(act): LlamaSwiGLU()
(dropout): Dropout(p=0.0, inplace=False)
)
)
(image_pooling_2d): FSDPCheckpointWrapper(
(_checkpoint_wrapped_module): ViTMultiHeadDotProductAttention(
(wq): Linear(in_features=2304, out_features=1152, bias=True)
(wk): Linear(in_features=2304, out_features=1152, bias=True)
(wv): Linear(in_features=2304, out_features=1152, bias=True)
(residual_dropout): Dropout(p=0.0, inplace=False)
)
)
)
(vit): FSDPSiglipVisionTransformer(
(patch_embedding): Linear(in_features=588, out_features=1152, bias=True)
(transformer): BlockCollection(
(resblocks): ModuleList(
(0-24): 25 x FSDPCheckpointWrapper(
(_checkpoint_wrapped_module): ResidualAttentionBlock(
(attention): ViTMultiHeadDotProductAttention(
(wq): Linear(in_features=1152, out_features=1152, bias=True)
(wk): Linear(in_features=1152, out_features=1152, bias=True)
(wv): Linear(in_features=1152, out_features=1152, bias=True)
(wo): Linear(in_features=1152, out_features=1152, bias=True)
(residual_dropout): Dropout(p=0.0, inplace=False)
)
(feed_forward): ViTMLP(
(w1): Linear(in_features=1152, out_features=4304, bias=True)
(act): GELUTanh()
(w2): Linear(in_features=4304, out_features=1152, bias=True)
)
(attention_norm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
(ffn_norm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
)
)
)
)
)
(build_vit_embedding): FSDPLinear(in_features=2304, out_features=4096, bias=True)
(x_norm): FSDPRMSLayerNorm()
(patch_q): FSDPLinear(in_features=4096, out_features=512, bias=True)
(patch_k): FSDPLinear(in_features=4096, out_features=512, bias=True)
(subpatch_q): FSDPLinear(in_features=4096, out_features=512, bias=True)
(subpatch_k): FSDPLinear(in_features=2304, out_features=512, bias=True)
(add_no_point_class_embed): FSDPPadWithLearnedVector()
(subpatch_loc_k): FSDPLinear(in_features=4096, out_features=9, bias=True)
(patch_rotary): FlatRotaryEmbedding()
)
2026-07-27 12:03:40.559 2012bc4f83ae:0 olmo.train.optim:179 INFO Constructing optimizer with 1 param groups
2026-07-27 12:03:40.561 2012bc4f83ae:0 olmo.data.data_loader:246 INFO Loading train dataset firewarning_pointing/train
2026-07-27 12:03:41.030 2012bc4f83ae:0 olmo.data.data_loader:302 INFO Sampling rates:
2026-07-27 12:03:41.030 2012bc4f83ae:0 olmo.data.data_loader:305 INFO firewarning_pointing: 100.00 (size=336)
2026-07-27 12:03:41.765 2012bc4f83ae:0 olmo.data.data_loader:340 INFO Packing with text=4096, images=16
2026-07-27 12:03:41.765 2012bc4f83ae:0 olmo.data.data_loader:346 INFO Building train dataset with output shapes: tokens=[4096], token_pooling=[1, 1], images=[16, 729, 588], point_target_ids=[1, 3], image_pos_ids=[1, 1]
2026-07-27 12:03:41.855 2012bc4f83ae:0 train:375 INFO Loading model from /workspace/firewarning-runtime/models/molmopoint-native/pretrain/MolmoPoint-8B-Pretrain/step23000
2026-07-27 12:03:41.964 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_projector.w1.weight' to 'connectors.0.image_projector.w1.weight' in checkpoint
2026-07-27 12:03:41.965 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_projector.w2.weight' to 'connectors.0.image_projector.w2.weight' in checkpoint
2026-07-27 12:03:41.965 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_projector.w3.weight' to 'connectors.0.image_projector.w3.weight' in checkpoint
2026-07-27 12:03:41.965 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_pooling_2d.wq.weight' to 'connectors.0.image_pooling_2d.wq.weight' in checkpoint
2026-07-27 12:03:41.966 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_pooling_2d.wq.bias' to 'connectors.0.image_pooling_2d.wq.bias' in checkpoint
2026-07-27 12:03:41.966 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_pooling_2d.wk.weight' to 'connectors.0.image_pooling_2d.wk.weight' in checkpoint
2026-07-27 12:03:41.967 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_pooling_2d.wk.bias' to 'connectors.0.image_pooling_2d.wk.bias' in checkpoint
2026-07-27 12:03:41.967 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_pooling_2d.wv.weight' to 'connectors.0.image_pooling_2d.wv.weight' in checkpoint
2026-07-27 12:03:41.968 2012bc4f83ae:0 olmo.train.distributed_checkpointing:262 INFO Mapping current param 'connector.image_pooling_2d.wv.bias' to 'connectors.0.image_pooling_2d.wv.bias' in checkpoint
2026-07-27 12:04:13.104 2012bc4f83ae:0 train:386 INFO Checkpoint successfully loaded in 31.2 seconds
2026-07-27 12:04:13.107 2012bc4f83ae:0 train:396 INFO Initializing FireWarning weights from /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-ground-v1/step100
2026-07-27 12:04:14.668 2012bc4f83ae:0 train:410 INFO Starting training...
2026-07-27 12:04:14.671 2012bc4f83ae:0 olmo.train.trainer:1050 INFO Pre-train system metrics
System/Peak GPU Memory (MB)=39,088
2026-07-27 12:05:47.006 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=1/600, eta=4 hours, 41 minutes]
train/CrossEntropyLoss=0.0215
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=2.248
train/patch_acc=0.5000
train/subpatch_loss=1.480
train/subpatch_acc=0.2500
train/location_loss=2.847
train/location_acc=0.0625
train/token_losses=35.29
throughput/total_tokens=16,384
System/Peak GPU Memory (MB)=41,640
2026-07-27 12:06:11.750 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=2/600, eta=4 hours, 24 minutes]
train/CrossEntropyLoss=0.0225
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.343
train/patch_acc=0.5312
train/subpatch_loss=1.383
train/subpatch_acc=0.3750
train/location_loss=2.303
train/location_acc=0.1875
train/token_losses=33.49
throughput/total_tokens=32,768
throughput/device/tokens_per_second=662.4
throughput/device/batches_per_second=0.0404
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:06:36.172 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=3/600, eta=4 hours, 17 minutes]
train/CrossEntropyLoss=0.0157
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=2.298
train/patch_acc=0.5938
train/subpatch_loss=1.616
train/subpatch_acc=0.3750
train/location_loss=2.326
train/location_acc=0.1250
train/token_losses=34.16
throughput/total_tokens=49,152
throughput/device/tokens_per_second=666.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:07:00.659 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=4/600, eta=4 hours, 13 minutes]
train/CrossEntropyLoss=0.0474
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.908
train/patch_acc=0.5625
train/subpatch_loss=1.657
train/subpatch_acc=0.3125
train/location_loss=2.570
train/location_acc=0.1250
train/token_losses=32.17
throughput/total_tokens=65,536
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:07:25.167 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=5/600, eta=4 hours, 11 minutes]
train/CrossEntropyLoss=0.0127
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=2.027
train/patch_acc=0.5938
train/subpatch_loss=1.522
train/subpatch_acc=0.3125
train/location_loss=2.494
train/location_acc=0.0
train/token_losses=32.28
throughput/total_tokens=81,920
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:07:49.674 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=6/600, eta=4 hours, 9 minutes]
train/CrossEntropyLoss=0.0313
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=2.247
train/patch_acc=0.5000
train/subpatch_loss=1.226
train/subpatch_acc=0.5000
train/location_loss=2.328
train/location_acc=0.1250
train/token_losses=32.19
throughput/total_tokens=98,304
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 12:08:14.211 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=7/600, eta=4 hours, 8 minutes]
train/CrossEntropyLoss=0.0498
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=2.319
train/patch_acc=0.5625
train/subpatch_loss=1.475
train/subpatch_acc=0.3125
train/location_loss=2.141
train/location_acc=0.1250
train/token_losses=33.02
throughput/total_tokens=114,688
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:08:38.747 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=8/600, eta=4 hours, 7 minutes]
train/CrossEntropyLoss=0.0011
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=2.094
train/patch_acc=0.5938
train/subpatch_loss=1.615
train/subpatch_acc=0.1250
train/location_loss=2.347
train/location_acc=0.1875
train/token_losses=32.60
throughput/total_tokens=131,072
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:09:03.306 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=9/600, eta=4 hours, 6 minutes]
train/CrossEntropyLoss=0.1649
train/ZLoss=0.0006
train/Accuracy=0.9604
train/patch_loss=2.344
train/patch_acc=0.4688
train/subpatch_loss=1.110
train/subpatch_acc=0.5625
train/location_loss=2.286
train/location_acc=0.0625
train/token_losses=32.34
throughput/total_tokens=147,456
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:09:27.837 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=10/600, eta=4 hours, 5 minutes]
train/CrossEntropyLoss=0.0480
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.415
train/patch_acc=0.5000
train/subpatch_loss=1.188
train/subpatch_acc=0.5000
train/location_loss=2.247
train/location_acc=0.1875
train/token_losses=33.06
throughput/total_tokens=163,840
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:09:52.403 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=11/600, eta=4 hours, 5 minutes]
train/CrossEntropyLoss=0.0257
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=2.021
train/patch_acc=0.5000
train/subpatch_loss=1.619
train/subpatch_acc=0.3125
train/location_loss=2.466
train/location_acc=0.0625
train/token_losses=32.51
throughput/total_tokens=180,224
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:10:16.962 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=12/600, eta=4 hours, 4 minutes]
train/CrossEntropyLoss=0.0357
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.327
train/patch_acc=0.5000
train/subpatch_loss=1.518
train/subpatch_acc=0.3750
train/location_loss=2.164
train/location_acc=0.1250
train/token_losses=33.34
throughput/total_tokens=196,608
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:10:41.509 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=13/600, eta=4 hours, 3 minutes]
train/CrossEntropyLoss=0.0322
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=2.135
train/patch_acc=0.5625
train/subpatch_loss=1.289
train/subpatch_acc=0.5625
train/location_loss=2.452
train/location_acc=0.0
train/token_losses=32.04
throughput/total_tokens=212,992
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:11:06.059 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=14/600, eta=4 hours, 3 minutes]
train/CrossEntropyLoss=0.2041
train/ZLoss=0.0006
train/Accuracy=0.9543
train/patch_loss=1.802
train/patch_acc=0.6250
train/subpatch_loss=1.516
train/subpatch_acc=0.2500
train/location_loss=2.369
train/location_acc=0.0625
train/token_losses=29.95
throughput/total_tokens=229,376
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:11:30.699 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=15/600, eta=4 hours, 2 minutes]
train/CrossEntropyLoss=0.0342
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=2.023
train/patch_acc=0.6250
train/subpatch_loss=1.285
train/subpatch_acc=0.3125
train/location_loss=1.983
train/location_acc=0.1875
train/token_losses=29.25
throughput/total_tokens=245,760
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:11:55.233 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=16/600, eta=4 hours, 2 minutes]
train/CrossEntropyLoss=0.3306
train/ZLoss=0.0009
train/Accuracy=0.9177
train/patch_loss=1.971
train/patch_acc=0.5625
train/subpatch_loss=1.035
train/subpatch_acc=0.5000
train/location_loss=2.280
train/location_acc=0.0625
train/token_losses=29.03
throughput/total_tokens=262,144
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:12:19.755 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=17/600, eta=4 hours, 1 minute]
train/CrossEntropyLoss=0.0355
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.851
train/patch_acc=0.5938
train/subpatch_loss=1.005
train/subpatch_acc=0.6875
train/location_loss=2.202
train/location_acc=0.0625
train/token_losses=27.64
throughput/total_tokens=278,528
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:12:44.296 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=18/600, eta=4 hours, 1 minute]
train/CrossEntropyLoss=0.0216
train/ZLoss=0.0003
train/Accuracy=0.9878
train/patch_loss=2.104
train/patch_acc=0.5938
train/subpatch_loss=1.494
train/subpatch_acc=0.3750
train/location_loss=2.377
train/location_acc=0.0625
train/token_losses=32.32
throughput/total_tokens=294,912
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:13:08.853 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=19/600, eta=4 hours]
train/CrossEntropyLoss=0.0149
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.966
train/patch_acc=0.6250
train/subpatch_loss=1.059
train/subpatch_acc=0.6875
train/location_loss=2.093
train/location_acc=0.2500
train/token_losses=28.33
throughput/total_tokens=311,296
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:13:33.392 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=20/600, eta=4 hours]
train/CrossEntropyLoss=0.0328
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.561
train/patch_acc=0.6250
train/subpatch_loss=1.103
train/subpatch_acc=0.5000
train/location_loss=2.276
train/location_acc=0.1250
train/token_losses=26.01
throughput/total_tokens=327,680
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:13:57.922 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=21/600, eta=3 hours, 59 minutes]
train/CrossEntropyLoss=0.0466
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=1.688
train/patch_acc=0.5938
train/subpatch_loss=0.8705
train/subpatch_acc=0.7500
train/location_loss=2.314
train/location_acc=0.0625
train/token_losses=26.24
throughput/total_tokens=344,064
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:14:22.459 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=22/600, eta=3 hours, 59 minutes]
train/CrossEntropyLoss=0.0072
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=2.300
train/patch_acc=0.5312
train/subpatch_loss=0.8311
train/subpatch_acc=0.6250
train/location_loss=2.164
train/location_acc=0.0625
train/token_losses=30.38
throughput/total_tokens=360,448
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:14:46.990 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=23/600, eta=3 hours, 58 minutes]
train/CrossEntropyLoss=0.0063
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=2.310
train/patch_acc=0.5938
train/subpatch_loss=0.9790
train/subpatch_acc=0.6875
train/location_loss=2.114
train/location_acc=0.1875
train/token_losses=30.85
throughput/total_tokens=376,832
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:15:11.543 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=24/600, eta=3 hours, 58 minutes]
train/CrossEntropyLoss=0.0539
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=2.955
train/patch_acc=0.5625
train/subpatch_loss=1.114
train/subpatch_acc=0.4375
train/location_loss=2.180
train/location_acc=0.1250
train/token_losses=36.81
throughput/total_tokens=393,216
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:15:36.097 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=25/600, eta=3 hours, 57 minutes]
train/CrossEntropyLoss=0.0563
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.987
train/patch_acc=0.6250
train/subpatch_loss=0.9503
train/subpatch_acc=0.4375
train/location_loss=2.263
train/location_acc=0.0
train/token_losses=28.75
throughput/total_tokens=409,600
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:16:00.643 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=26/600, eta=3 hours, 57 minutes]
train/CrossEntropyLoss=0.0113
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=2.543
train/patch_acc=0.5312
train/subpatch_loss=1.054
train/subpatch_acc=0.5625
train/location_loss=2.065
train/location_acc=0.2500
train/token_losses=32.82
throughput/total_tokens=425,984
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:16:25.180 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=27/600, eta=3 hours, 56 minutes]
train/CrossEntropyLoss=0.0364
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.027
train/patch_acc=0.6250
train/subpatch_loss=0.8206
train/subpatch_acc=0.5625
train/location_loss=2.169
train/location_acc=0.1875
train/token_losses=28.18
throughput/total_tokens=442,368
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:16:49.751 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=28/600, eta=3 hours, 56 minutes]
train/CrossEntropyLoss=0.0257
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.544
train/patch_acc=0.6250
train/subpatch_loss=0.8610
train/subpatch_acc=0.6250
train/location_loss=2.321
train/location_acc=0.0625
train/token_losses=25.08
throughput/total_tokens=458,752
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:17:14.290 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=29/600, eta=3 hours, 55 minutes]
train/CrossEntropyLoss=0.0042
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.736
train/patch_acc=0.5938
train/subpatch_loss=1.001
train/subpatch_acc=0.5625
train/location_loss=2.181
train/location_acc=0.1250
train/token_losses=26.62
throughput/total_tokens=475,136
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:17:38.813 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=30/600, eta=3 hours, 55 minutes]
train/CrossEntropyLoss=0.0162
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.014
train/patch_acc=0.5625
train/subpatch_loss=1.006
train/subpatch_acc=0.5000
train/location_loss=2.043
train/location_acc=0.2500
train/token_losses=28.31
throughput/total_tokens=491,520
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:18:03.341 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=31/600, eta=3 hours, 54 minutes]
train/CrossEntropyLoss=0.0447
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=2.232
train/patch_acc=0.5625
train/subpatch_loss=0.9663
train/subpatch_acc=0.5000
train/location_loss=2.093
train/location_acc=0.1875
train/token_losses=30.10
throughput/total_tokens=507,904
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:18:27.857 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=32/600, eta=3 hours, 54 minutes]
train/CrossEntropyLoss=0.0182
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.706
train/patch_acc=0.5938
train/subpatch_loss=0.9503
train/subpatch_acc=0.5000
train/location_loss=1.993
train/location_acc=0.3125
train/token_losses=33.42
throughput/total_tokens=524,288
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:18:52.368 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=33/600, eta=3 hours, 53 minutes]
train/CrossEntropyLoss=0.0048
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=2.283
train/patch_acc=0.5625
train/subpatch_loss=0.9671
train/subpatch_acc=0.6250
train/location_loss=2.240
train/location_acc=0.0625
train/token_losses=31.09
throughput/total_tokens=540,672
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:19:16.889 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=34/600, eta=3 hours, 53 minutes]
train/CrossEntropyLoss=0.0064
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=1.899
train/patch_acc=0.5938
train/subpatch_loss=0.8040
train/subpatch_acc=0.6250
train/location_loss=2.054
train/location_acc=0.1250
train/token_losses=26.62
throughput/total_tokens=557,056
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:19:41.418 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=35/600, eta=3 hours, 53 minutes]
train/CrossEntropyLoss=0.0267
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.239
train/patch_acc=0.5625
train/subpatch_loss=0.8463
train/subpatch_acc=0.6875
train/location_loss=2.120
train/location_acc=0.0625
train/token_losses=29.78
throughput/total_tokens=573,440
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:20:05.949 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=36/600, eta=3 hours, 52 minutes]
train/CrossEntropyLoss=0.0033
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=1.534
train/patch_acc=0.7500
train/subpatch_loss=0.7947
train/subpatch_acc=0.6875
train/location_loss=2.476
train/location_acc=0.0
train/token_losses=25.36
throughput/total_tokens=589,824
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:20:30.466 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=37/600, eta=3 hours, 52 minutes]
train/CrossEntropyLoss=0.0602
train/ZLoss=0.0005
train/Accuracy=0.9878
train/patch_loss=1.690
train/patch_acc=0.5625
train/subpatch_loss=0.7795
train/subpatch_acc=0.7500
train/location_loss=2.191
train/location_acc=0.0625
train/token_losses=25.40
throughput/total_tokens=606,208
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:20:54.990 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=38/600, eta=3 hours, 51 minutes]
train/CrossEntropyLoss=0.0543
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.901
train/patch_acc=0.6250
train/subpatch_loss=0.8586
train/subpatch_acc=0.6875
train/location_loss=2.238
train/location_acc=0.0
train/token_losses=27.59
throughput/total_tokens=622,592
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:21:19.513 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=39/600, eta=3 hours, 51 minutes]
train/CrossEntropyLoss=0.0249
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.760
train/patch_acc=0.6562
train/subpatch_loss=0.7173
train/subpatch_acc=0.6875
train/location_loss=2.132
train/location_acc=0.1875
train/token_losses=25.47
throughput/total_tokens=638,976
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:21:44.045 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=40/600, eta=3 hours, 50 minutes]
train/CrossEntropyLoss=0.0472
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.951
train/patch_acc=0.5625
train/subpatch_loss=1.063
train/subpatch_acc=0.5000
train/location_loss=2.169
train/location_acc=0.1875
train/token_losses=28.53
throughput/total_tokens=655,360
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:22:08.575 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=41/600, eta=3 hours, 50 minutes]
train/CrossEntropyLoss=0.0645
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=1.961
train/patch_acc=0.5938
train/subpatch_loss=0.7808
train/subpatch_acc=0.8125
train/location_loss=2.074
train/location_acc=0.2500
train/token_losses=27.11
throughput/total_tokens=671,744
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 12:22:33.095 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=42/600, eta=3 hours, 50 minutes]
train/CrossEntropyLoss=0.0311
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=2.266
train/patch_acc=0.5625
train/subpatch_loss=0.6734
train/subpatch_acc=0.7500
train/location_loss=2.092
train/location_acc=0.1875
train/token_losses=29.19
throughput/total_tokens=688,128
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 12:22:57.606 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=43/600, eta=3 hours, 49 minutes]
train/CrossEntropyLoss=0.0060
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.523
train/patch_acc=0.6250
train/subpatch_loss=0.6371
train/subpatch_acc=0.7500
train/location_loss=2.347
train/location_acc=0.1250
train/token_losses=24.12
throughput/total_tokens=704,512
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 12:23:22.116 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=44/600, eta=3 hours, 49 minutes]
train/CrossEntropyLoss=0.0195
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.728
train/patch_acc=0.6250
train/subpatch_loss=1.124
train/subpatch_acc=0.5000
train/location_loss=2.075
train/location_acc=0.0625
train/token_losses=26.62
throughput/total_tokens=720,896
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 12:23:46.629 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=45/600, eta=3 hours, 48 minutes]
train/CrossEntropyLoss=0.0820
train/ZLoss=0.0007
train/Accuracy=0.9787
train/patch_loss=1.827
train/patch_acc=0.5625
train/subpatch_loss=0.9899
train/subpatch_acc=0.6875
train/location_loss=2.213
train/location_acc=0.1250
train/token_losses=27.43
throughput/total_tokens=737,280
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 12:24:11.125 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=46/600, eta=3 hours, 48 minutes]
train/CrossEntropyLoss=0.0628
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.365
train/patch_acc=0.5312
train/subpatch_loss=1.168
train/subpatch_acc=0.6250
train/location_loss=2.064
train/location_acc=0.0625
train/token_losses=31.85
throughput/total_tokens=753,664
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:24:35.648 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=47/600, eta=3 hours, 47 minutes]
train/CrossEntropyLoss=0.0279
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.025
train/patch_acc=0.5312
train/subpatch_loss=1.327
train/subpatch_acc=0.3750
train/location_loss=2.337
train/location_acc=0.1875
train/token_losses=30.85
throughput/total_tokens=770,048
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:25:00.166 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=48/600, eta=3 hours, 47 minutes]
train/CrossEntropyLoss=0.0114
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.513
train/patch_acc=0.6562
train/subpatch_loss=0.9656
train/subpatch_acc=0.5625
train/location_loss=2.020
train/location_acc=0.2500
train/token_losses=24.05
throughput/total_tokens=786,432
throughput/device/tokens_per_second=668.2
throughput/device/batches_per_second=0.0408
2026-07-27 12:25:24.690 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=49/600, eta=3 hours, 46 minutes]
train/CrossEntropyLoss=0.0334
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.635
train/patch_acc=0.6562
train/subpatch_loss=0.5610
train/subpatch_acc=0.8125
train/location_loss=2.137
train/location_acc=0.0
train/token_losses=23.87
throughput/total_tokens=802,816
throughput/device/tokens_per_second=668.2
throughput/device/batches_per_second=0.0408
2026-07-27 12:25:49.229 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=50/600, eta=3 hours, 46 minutes]
train/CrossEntropyLoss=0.0215
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.639
train/patch_acc=0.6250
train/subpatch_loss=0.9289
train/subpatch_acc=0.5625
train/location_loss=2.021
train/location_acc=0.0625
train/token_losses=24.91
throughput/total_tokens=819,200
throughput/device/tokens_per_second=668.2
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:26:13.749 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=51/600, eta=3 hours, 46 minutes]
train/CrossEntropyLoss=0.0071
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.657
train/patch_acc=0.6562
train/subpatch_loss=0.6881
train/subpatch_acc=0.7500
train/location_loss=2.049
train/location_acc=0.0625
train/token_losses=24.20
throughput/total_tokens=835,584
throughput/device/tokens_per_second=668.2
throughput/device/batches_per_second=0.0408
2026-07-27 12:26:38.283 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=52/600, eta=3 hours, 45 minutes]
train/CrossEntropyLoss=0.0142
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=2.030
train/patch_acc=0.5938
train/subpatch_loss=0.6443
train/subpatch_acc=0.6875
train/location_loss=1.972
train/location_acc=0.1875
train/token_losses=26.71
throughput/total_tokens=851,968
throughput/device/tokens_per_second=668.2
throughput/device/batches_per_second=0.0408
2026-07-27 12:27:02.830 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=53/600, eta=3 hours, 45 minutes]
train/CrossEntropyLoss=0.0112
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.548
train/patch_acc=0.6562
train/subpatch_loss=0.8384
train/subpatch_acc=0.5625
train/location_loss=2.154
train/location_acc=0.0625
train/token_losses=24.36
throughput/total_tokens=868,352
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:27:27.353 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=54/600, eta=3 hours, 44 minutes]
train/CrossEntropyLoss=0.0262
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.824
train/patch_acc=0.5625
train/subpatch_loss=0.8051
train/subpatch_acc=0.6875
train/location_loss=2.104
train/location_acc=0.1875
train/token_losses=26.23
throughput/total_tokens=884,736
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:27:51.872 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=55/600, eta=3 hours, 44 minutes]
train/CrossEntropyLoss=0.0256
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.971
train/patch_acc=0.6250
train/subpatch_loss=0.9569
train/subpatch_acc=0.6250
train/location_loss=2.293
train/location_acc=0.0625
train/token_losses=28.77
throughput/total_tokens=901,120
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:28:16.389 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=56/600, eta=3 hours, 44 minutes]
train/CrossEntropyLoss=0.0121
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.754
train/patch_acc=0.5938
train/subpatch_loss=1.037
train/subpatch_acc=0.6875
train/location_loss=1.989
train/location_acc=0.1875
train/token_losses=26.14
throughput/total_tokens=917,504
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:28:40.946 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=57/600, eta=3 hours, 43 minutes]
train/CrossEntropyLoss=0.0282
train/ZLoss=0.0004
train/Accuracy=0.9939
train/patch_loss=1.549
train/patch_acc=0.5625
train/subpatch_loss=0.7415
train/subpatch_acc=0.6250
train/location_loss=2.097
train/location_acc=0.3125
train/token_losses=23.74
throughput/total_tokens=933,888
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:29:05.482 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=58/600, eta=3 hours, 43 minutes]
train/CrossEntropyLoss=0.0097
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.648
train/patch_acc=0.6875
train/subpatch_loss=0.8569
train/subpatch_acc=0.6875
train/location_loss=2.152
train/location_acc=0.0
train/token_losses=25.22
throughput/total_tokens=950,272
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:29:30.006 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=59/600, eta=3 hours, 42 minutes]
train/CrossEntropyLoss=0.1114
train/ZLoss=0.0007
train/Accuracy=0.9726
train/patch_loss=1.768
train/patch_acc=0.5625
train/subpatch_loss=0.6982
train/subpatch_acc=0.6875
train/location_loss=2.263
train/location_acc=0.0625
train/token_losses=25.99
throughput/total_tokens=966,656
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:29:54.538 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=60/600, eta=3 hours, 42 minutes]
train/CrossEntropyLoss=0.0627
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.609
train/patch_acc=0.5625
train/subpatch_loss=0.6068
train/subpatch_acc=0.7500
train/location_loss=1.872
train/location_acc=0.3125
train/token_losses=22.79
throughput/total_tokens=983,040
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:30:19.077 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=61/600, eta=3 hours, 41 minutes]
train/CrossEntropyLoss=0.1999
train/ZLoss=0.0006
train/Accuracy=0.9543
train/patch_loss=1.820
train/patch_acc=0.6250
train/subpatch_loss=0.6233
train/subpatch_acc=0.6875
train/location_loss=2.193
train/location_acc=0.0625
train/token_losses=25.83
throughput/total_tokens=999,424
throughput/device/tokens_per_second=668.1
throughput/device/batches_per_second=0.0408
2026-07-27 12:30:43.637 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=62/600, eta=3 hours, 41 minutes]
train/CrossEntropyLoss=0.0414
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.615
train/patch_acc=0.5938
train/subpatch_loss=0.6822
train/subpatch_acc=0.6875
train/location_loss=2.172
train/location_acc=0.1250
train/token_losses=24.34
throughput/total_tokens=1,015,808
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 12:31:08.171 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=63/600, eta=3 hours, 41 minutes]
train/CrossEntropyLoss=0.0276
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.926
train/patch_acc=0.5938
train/subpatch_loss=0.6812
train/subpatch_acc=0.8750
train/location_loss=2.034
train/location_acc=0.1875
train/token_losses=26.27
throughput/total_tokens=1,032,192
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 12:31:32.690 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=64/600, eta=3 hours, 40 minutes]
train/CrossEntropyLoss=0.0521
train/ZLoss=0.0002
train/Accuracy=0.9787
train/patch_loss=2.078
train/patch_acc=0.5938
train/subpatch_loss=0.6017
train/subpatch_acc=0.7500
train/location_loss=2.190
train/location_acc=0.2500
train/token_losses=27.79
throughput/total_tokens=1,048,576
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 12:31:57.245 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=65/600, eta=3 hours, 40 minutes]
train/CrossEntropyLoss=0.0282
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.650
train/patch_acc=0.6562
train/subpatch_loss=0.8913
train/subpatch_acc=0.7500
train/location_loss=2.158
train/location_acc=0.1250
train/token_losses=25.40
throughput/total_tokens=1,064,960
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 12:32:21.771 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=66/600, eta=3 hours, 39 minutes]
train/CrossEntropyLoss=0.0329
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.561
train/patch_acc=0.6250
train/subpatch_loss=0.5292
train/subpatch_acc=0.8750
train/location_loss=2.055
train/location_acc=0.0625
train/token_losses=22.82
throughput/total_tokens=1,081,344
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 12:32:46.308 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=67/600, eta=3 hours, 39 minutes]
train/CrossEntropyLoss=0.0409
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.837
train/patch_acc=0.5938
train/subpatch_loss=0.8088
train/subpatch_acc=0.7500
train/location_loss=2.059
train/location_acc=0.1875
train/token_losses=26.16
throughput/total_tokens=1,097,728
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:33:10.827 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=68/600, eta=3 hours, 39 minutes]
train/CrossEntropyLoss=0.1565
train/ZLoss=0.0006
train/Accuracy=0.9573
train/patch_loss=2.061
train/patch_acc=0.5938
train/subpatch_loss=0.9147
train/subpatch_acc=0.5625
train/location_loss=2.236
train/location_acc=0.1250
train/token_losses=29.09
throughput/total_tokens=1,114,112
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:33:35.360 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=69/600, eta=3 hours, 38 minutes]
train/CrossEntropyLoss=0.0007
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=1.928
train/patch_acc=0.6250
train/subpatch_loss=0.5580
train/subpatch_acc=0.8125
train/location_loss=2.060
train/location_acc=0.3125
train/token_losses=25.90
throughput/total_tokens=1,130,496
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:33:59.910 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=70/600, eta=3 hours, 38 minutes]
train/CrossEntropyLoss=0.0874
train/ZLoss=0.0005
train/Accuracy=0.9726
train/patch_loss=1.382
train/patch_acc=0.6250
train/subpatch_loss=0.3766
train/subpatch_acc=0.8750
train/location_loss=1.952
train/location_acc=0.3125
train/token_losses=20.37
throughput/total_tokens=1,146,880
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:34:24.428 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=71/600, eta=3 hours, 37 minutes]
train/CrossEntropyLoss=0.0695
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.619
train/patch_acc=0.6250
train/subpatch_loss=0.7147
train/subpatch_acc=0.6875
train/location_loss=1.833
train/location_acc=0.1875
train/token_losses=23.14
throughput/total_tokens=1,163,264
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:34:48.970 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=72/600, eta=3 hours, 37 minutes]
train/CrossEntropyLoss=0.0145
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.701
train/patch_acc=0.5625
train/subpatch_loss=0.5460
train/subpatch_acc=0.8125
train/location_loss=1.833
train/location_acc=0.2500
train/token_losses=23.12
throughput/total_tokens=1,179,648
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:35:13.523 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=73/600, eta=3 hours, 36 minutes]
train/CrossEntropyLoss=0.0031
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=1.572
train/patch_acc=0.5938
train/subpatch_loss=0.5784
train/subpatch_acc=0.7500
train/location_loss=2.106
train/location_acc=0.1250
train/token_losses=23.31
throughput/total_tokens=1,196,032
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:35:38.034 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=74/600, eta=3 hours, 36 minutes]
train/CrossEntropyLoss=0.0123
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=1.521
train/patch_acc=0.6250
train/subpatch_loss=0.4288
train/subpatch_acc=0.8125
train/location_loss=2.005
train/location_acc=0.2500
train/token_losses=21.91
throughput/total_tokens=1,212,416
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:36:02.576 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=75/600, eta=3 hours, 36 minutes]
train/CrossEntropyLoss=0.0408
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.365
train/patch_acc=0.6562
train/subpatch_loss=0.5951
train/subpatch_acc=0.6875
train/location_loss=2.282
train/location_acc=0.0
train/token_losses=22.43
throughput/total_tokens=1,228,800
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:36:27.093 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=76/600, eta=3 hours, 35 minutes]
train/CrossEntropyLoss=0.0704
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.414
train/patch_acc=0.6875
train/subpatch_loss=0.7394
train/subpatch_acc=0.6875
train/location_loss=1.962
train/location_acc=0.1250
train/token_losses=22.12
throughput/total_tokens=1,245,184
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:36:51.651 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=77/600, eta=3 hours, 35 minutes]
train/CrossEntropyLoss=0.0054
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.185
train/patch_acc=0.6562
train/subpatch_loss=0.5053
train/subpatch_acc=0.8750
train/location_loss=1.873
train/location_acc=0.3750
train/token_losses=18.99
throughput/total_tokens=1,261,568
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:37:16.193 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=78/600, eta=3 hours, 34 minutes]
train/CrossEntropyLoss=0.0450
train/ZLoss=0.0004
train/Accuracy=0.9848
train/patch_loss=1.284
train/patch_acc=0.6562
train/subpatch_loss=0.5920
train/subpatch_acc=0.7500
train/location_loss=2.061
train/location_acc=0.1250
train/token_losses=20.89
throughput/total_tokens=1,277,952
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:37:40.736 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=79/600, eta=3 hours, 34 minutes]
train/CrossEntropyLoss=0.0126
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.665
train/patch_acc=0.6250
train/subpatch_loss=0.7387
train/subpatch_acc=0.7500
train/location_loss=1.927
train/location_acc=0.3750
train/token_losses=23.98
throughput/total_tokens=1,294,336
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:38:05.264 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=80/600, eta=3 hours, 34 minutes]
train/CrossEntropyLoss=0.0672
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=1.334
train/patch_acc=0.6562
train/subpatch_loss=0.4021
train/subpatch_acc=0.8750
train/location_loss=2.202
train/location_acc=0.2500
train/token_losses=21.08
throughput/total_tokens=1,310,720
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:38:29.800 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=81/600, eta=3 hours, 33 minutes]
train/CrossEntropyLoss=0.0382
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.330
train/patch_acc=0.6562
train/subpatch_loss=0.4915
train/subpatch_acc=0.7500
train/location_loss=1.935
train/location_acc=0.3750
train/token_losses=20.35
throughput/total_tokens=1,327,104
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:38:54.347 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=82/600, eta=3 hours, 33 minutes]
train/CrossEntropyLoss=0.0272
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.371
train/patch_acc=0.6875
train/subpatch_loss=0.5538
train/subpatch_acc=0.8125
train/location_loss=2.021
train/location_acc=0.0625
train/token_losses=21.26
throughput/total_tokens=1,343,488
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:39:18.902 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=83/600, eta=3 hours, 32 minutes]
train/CrossEntropyLoss=0.1663
train/ZLoss=0.0006
train/Accuracy=0.9634
train/patch_loss=1.777
train/patch_acc=0.5625
train/subpatch_loss=0.4451
train/subpatch_acc=0.8750
train/location_loss=1.935
train/location_acc=0.3125
train/token_losses=23.74
throughput/total_tokens=1,359,872
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:39:43.416 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=84/600, eta=3 hours, 32 minutes]
train/CrossEntropyLoss=0.0263
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.054
train/patch_acc=0.6875
train/subpatch_loss=0.3436
train/subpatch_acc=0.8125
train/location_loss=1.935
train/location_acc=0.4375
train/token_losses=17.54
throughput/total_tokens=1,376,256
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:40:07.931 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=85/600, eta=3 hours, 31 minutes]
train/CrossEntropyLoss=0.0121
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.605
train/patch_acc=0.5938
train/subpatch_loss=0.7571
train/subpatch_acc=0.6875
train/location_loss=2.133
train/location_acc=0.1250
train/token_losses=24.40
throughput/total_tokens=1,392,640
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:40:32.451 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=86/600, eta=3 hours, 31 minutes]
train/CrossEntropyLoss=0.0221
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.327
train/patch_acc=0.6250
train/subpatch_loss=0.5604
train/subpatch_acc=0.7500
train/location_loss=1.730
train/location_acc=0.4375
train/token_losses=19.78
throughput/total_tokens=1,409,024
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:40:56.982 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=87/600, eta=3 hours, 31 minutes]
train/CrossEntropyLoss=0.0425
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.434
train/patch_acc=0.6875
train/subpatch_loss=0.3956
train/subpatch_acc=0.9375
train/location_loss=1.915
train/location_acc=0.2500
train/token_losses=20.71
throughput/total_tokens=1,425,408
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:41:21.498 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=88/600, eta=3 hours, 30 minutes]
train/CrossEntropyLoss=0.0336
train/ZLoss=0.0003
train/Accuracy=0.9939
train/patch_loss=1.891
train/patch_acc=0.5938
train/subpatch_loss=0.7188
train/subpatch_acc=0.6875
train/location_loss=1.962
train/location_acc=0.1875
train/token_losses=25.85
throughput/total_tokens=1,441,792
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:41:46.026 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=89/600, eta=3 hours, 30 minutes]
train/CrossEntropyLoss=0.0586
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.790
train/patch_acc=0.5625
train/subpatch_loss=0.8238
train/subpatch_acc=0.6250
train/location_loss=2.065
train/location_acc=0.2500
train/token_losses=25.87
throughput/total_tokens=1,458,176
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:42:10.557 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=90/600, eta=3 hours, 29 minutes]
train/CrossEntropyLoss=0.0620
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.413
train/patch_acc=0.6875
train/subpatch_loss=0.3235
train/subpatch_acc=0.9375
train/location_loss=1.841
train/location_acc=0.3750
train/token_losses=19.96
throughput/total_tokens=1,474,560
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:42:35.084 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=91/600, eta=3 hours, 29 minutes]
train/CrossEntropyLoss=0.0473
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.697
train/patch_acc=0.6250
train/subpatch_loss=0.4441
train/subpatch_acc=0.8750
train/location_loss=1.935
train/location_acc=0.1875
train/token_losses=23.10
throughput/total_tokens=1,490,944
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:42:59.633 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=92/600, eta=3 hours, 29 minutes]
train/CrossEntropyLoss=0.0487
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.659
train/patch_acc=0.6250
train/subpatch_loss=0.7066
train/subpatch_acc=0.8125
train/location_loss=1.826
train/location_acc=0.3750
train/token_losses=23.40
throughput/total_tokens=1,507,328
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:43:24.196 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=93/600, eta=3 hours, 28 minutes]
train/CrossEntropyLoss=0.0575
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.521
train/patch_acc=0.6562
train/subpatch_loss=0.6655
train/subpatch_acc=0.6875
train/location_loss=2.134
train/location_acc=0.0
train/token_losses=23.36
throughput/total_tokens=1,523,712
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:43:48.736 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=94/600, eta=3 hours, 28 minutes]
train/CrossEntropyLoss=0.0360
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.240
train/patch_acc=0.7188
train/subpatch_loss=0.5959
train/subpatch_acc=0.7500
train/location_loss=2.001
train/location_acc=0.2500
train/token_losses=20.31
throughput/total_tokens=1,540,096
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:44:13.268 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=95/600, eta=3 hours, 27 minutes]
train/CrossEntropyLoss=0.0269
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.484
train/patch_acc=0.5938
train/subpatch_loss=0.7732
train/subpatch_acc=0.5625
train/location_loss=1.748
train/location_acc=0.3125
train/token_losses=21.96
throughput/total_tokens=1,556,480
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:44:37.818 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=96/600, eta=3 hours, 27 minutes]
train/CrossEntropyLoss=0.0221
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.193
train/patch_acc=0.6875
train/subpatch_loss=0.3718
train/subpatch_acc=0.8750
train/location_loss=2.161
train/location_acc=0.1250
train/token_losses=19.67
throughput/total_tokens=1,572,864
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:45:02.356 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=97/600, eta=3 hours, 26 minutes]
train/CrossEntropyLoss=0.0637
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.708
train/patch_acc=0.5938
train/subpatch_loss=0.3657
train/subpatch_acc=0.8750
train/location_loss=1.853
train/location_acc=0.3750
train/token_losses=22.54
throughput/total_tokens=1,589,248
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:45:26.896 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=98/600, eta=3 hours, 26 minutes]
train/CrossEntropyLoss=0.1550
train/ZLoss=0.0006
train/Accuracy=0.9665
train/patch_loss=1.640
train/patch_acc=0.5938
train/subpatch_loss=0.3962
train/subpatch_acc=0.9375
train/location_loss=1.976
train/location_acc=0.1875
train/token_losses=22.61
throughput/total_tokens=1,605,632
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:45:51.457 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=99/600, eta=3 hours, 26 minutes]
train/CrossEntropyLoss=0.0322
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.305
train/patch_acc=0.6875
train/subpatch_loss=0.7145
train/subpatch_acc=0.7500
train/location_loss=2.007
train/location_acc=0.2500
train/token_losses=21.32
throughput/total_tokens=1,622,016
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:46:15.992 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=100/600, eta=3 hours, 25 minutes]
train/CrossEntropyLoss=0.0467
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=2.040
train/patch_acc=0.5938
train/subpatch_loss=0.6327
train/subpatch_acc=0.8125
train/location_loss=2.074
train/location_acc=0.1875
train/token_losses=27.15
throughput/total_tokens=1,638,400
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:46:15.993 2012bc4f83ae:0 olmo.train.trainer:1415 INFO Saving checkpoint...
2026-07-27 12:46:17.871 2012bc4f83ae:0 olmo.train.trainer:1420 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step100
2026-07-27 12:46:42.857 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=101/600, eta=3 hours, 25 minutes]
train/CrossEntropyLoss=0.0013
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=1.330
train/patch_acc=0.6250
train/subpatch_loss=0.2301
train/subpatch_acc=0.9375
train/location_loss=1.832
train/location_acc=0.3750
train/token_losses=18.89
throughput/total_tokens=1,654,784
throughput/device/tokens_per_second=655.8
throughput/device/batches_per_second=0.0400
2026-07-27 12:47:07.413 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=102/600, eta=3 hours, 25 minutes]
train/CrossEntropyLoss=0.0579
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.655
train/patch_acc=0.5625
train/subpatch_loss=0.5927
train/subpatch_acc=0.8125
train/location_loss=2.163
train/location_acc=0.1250
train/token_losses=24.26
throughput/total_tokens=1,671,168
throughput/device/tokens_per_second=661.5
throughput/device/batches_per_second=0.0404
2026-07-27 12:47:31.938 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=103/600, eta=3 hours, 24 minutes]
train/CrossEntropyLoss=0.0151
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.400
train/patch_acc=0.6875
train/subpatch_loss=0.5329
train/subpatch_acc=0.8750
train/location_loss=1.983
train/location_acc=0.2500
train/token_losses=21.26
throughput/total_tokens=1,687,552
throughput/device/tokens_per_second=663.6
throughput/device/batches_per_second=0.0405
2026-07-27 12:47:56.480 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=104/600, eta=3 hours, 24 minutes]
train/CrossEntropyLoss=0.0365
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.092
train/patch_acc=0.7188
train/subpatch_loss=0.8266
train/subpatch_acc=0.7500
train/location_loss=1.855
train/location_acc=0.2500
train/token_losses=19.46
throughput/total_tokens=1,703,936
throughput/device/tokens_per_second=664.6
throughput/device/batches_per_second=0.0406
2026-07-27 12:48:21.042 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=105/600, eta=3 hours, 23 minutes]
train/CrossEntropyLoss=0.2087
train/ZLoss=0.0007
train/Accuracy=0.9482
train/patch_loss=1.456
train/patch_acc=0.6250
train/subpatch_loss=0.6834
train/subpatch_acc=0.6875
train/location_loss=1.888
train/location_acc=0.2500
train/token_losses=21.94
throughput/total_tokens=1,720,320
throughput/device/tokens_per_second=665.1
throughput/device/batches_per_second=0.0406
2026-07-27 12:48:45.600 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=106/600, eta=3 hours, 23 minutes]
train/CrossEntropyLoss=0.0760
train/ZLoss=0.0001
train/Accuracy=0.9787
train/patch_loss=1.217
train/patch_acc=0.6875
train/subpatch_loss=0.5997
train/subpatch_acc=0.6875
train/location_loss=1.830
train/location_acc=0.2500
train/token_losses=19.45
throughput/total_tokens=1,736,704
throughput/device/tokens_per_second=665.5
throughput/device/batches_per_second=0.0406
2026-07-27 12:49:10.168 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=107/600, eta=3 hours, 23 minutes]
train/CrossEntropyLoss=0.1519
train/ZLoss=0.0011
train/Accuracy=0.9573
train/patch_loss=1.339
train/patch_acc=0.6562
train/subpatch_loss=0.5618
train/subpatch_acc=0.7500
train/location_loss=1.938
train/location_acc=0.3750
train/token_losses=20.71
throughput/total_tokens=1,753,088
throughput/device/tokens_per_second=665.7
throughput/device/batches_per_second=0.0406
2026-07-27 12:49:34.760 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=108/600, eta=3 hours, 22 minutes]
train/CrossEntropyLoss=0.0192
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.366
train/patch_acc=0.6250
train/subpatch_loss=0.4100
train/subpatch_acc=1.0000
train/location_loss=1.883
train/location_acc=0.2500
train/token_losses=20.09
throughput/total_tokens=1,769,472
throughput/device/tokens_per_second=665.7
throughput/device/batches_per_second=0.0406
2026-07-27 12:49:59.342 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=109/600, eta=3 hours, 22 minutes]
train/CrossEntropyLoss=0.0426
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.607
train/patch_acc=0.5625
train/subpatch_loss=0.6100
train/subpatch_acc=0.7500
train/location_loss=1.431
train/location_acc=0.7500
train/token_losses=21.02
throughput/total_tokens=1,785,856
throughput/device/tokens_per_second=665.8
throughput/device/batches_per_second=0.0406
2026-07-27 12:50:23.888 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=110/600, eta=3 hours, 21 minutes]
train/CrossEntropyLoss=0.1849
train/ZLoss=0.0006
train/Accuracy=0.9573
train/patch_loss=1.392
train/patch_acc=0.6250
train/subpatch_loss=0.6008
train/subpatch_acc=0.8125
train/location_loss=2.211
train/location_acc=0.1250
train/token_losses=22.39
throughput/total_tokens=1,802,240
throughput/device/tokens_per_second=666.0
throughput/device/batches_per_second=0.0406
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:50:48.413 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=111/600, eta=3 hours, 21 minutes]
train/CrossEntropyLoss=0.0166
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.419
train/patch_acc=0.5938
train/subpatch_loss=0.3755
train/subpatch_acc=0.9375
train/location_loss=1.551
train/location_acc=0.3125
train/token_losses=19.05
throughput/total_tokens=1,818,624
throughput/device/tokens_per_second=666.2
throughput/device/batches_per_second=0.0407
2026-07-27 12:51:12.958 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=112/600, eta=3 hours, 20 minutes]
train/CrossEntropyLoss=0.0223
train/ZLoss=0.0005
train/Accuracy=0.9909
train/patch_loss=1.411
train/patch_acc=0.6562
train/subpatch_loss=0.4618
train/subpatch_acc=0.8125
train/location_loss=2.008
train/location_acc=0.3750
train/token_losses=21.17
throughput/total_tokens=1,835,008
throughput/device/tokens_per_second=666.3
throughput/device/batches_per_second=0.0407
2026-07-27 12:51:37.516 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=113/600, eta=3 hours, 20 minutes]
train/CrossEntropyLoss=0.0436
train/ZLoss=0.0005
train/Accuracy=0.9787
train/patch_loss=1.517
train/patch_acc=0.5938
train/subpatch_loss=0.2870
train/subpatch_acc=0.9375
train/location_loss=1.583
train/location_acc=0.3125
train/token_losses=19.61
throughput/total_tokens=1,851,392
throughput/device/tokens_per_second=666.3
throughput/device/batches_per_second=0.0407
2026-07-27 12:52:02.061 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=114/600, eta=3 hours, 20 minutes]
train/CrossEntropyLoss=0.0157
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.549
train/patch_acc=0.6250
train/subpatch_loss=0.5199
train/subpatch_acc=0.8125
train/location_loss=2.054
train/location_acc=0.3125
train/token_losses=22.69
throughput/total_tokens=1,867,776
throughput/device/tokens_per_second=666.4
throughput/device/batches_per_second=0.0407
2026-07-27 12:52:26.587 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=115/600, eta=3 hours, 19 minutes]
train/CrossEntropyLoss=0.0375
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.185
train/patch_acc=0.6562
train/subpatch_loss=0.8220
train/subpatch_acc=0.6250
train/location_loss=1.838
train/location_acc=0.1875
train/token_losses=20.12
throughput/total_tokens=1,884,160
throughput/device/tokens_per_second=666.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:52:51.104 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=116/600, eta=3 hours, 19 minutes]
train/CrossEntropyLoss=0.0118
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.379
train/patch_acc=0.6250
train/subpatch_loss=0.3588
train/subpatch_acc=0.8125
train/location_loss=1.447
train/location_acc=0.3750
train/token_losses=18.25
throughput/total_tokens=1,900,544
throughput/device/tokens_per_second=666.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:53:15.636 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=117/600, eta=3 hours, 18 minutes]
train/CrossEntropyLoss=0.0422
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.358
train/patch_acc=0.5938
train/subpatch_loss=0.2071
train/subpatch_acc=1.0000
train/location_loss=1.697
train/location_acc=0.4375
train/token_losses=18.48
throughput/total_tokens=1,916,928
throughput/device/tokens_per_second=666.7
throughput/device/batches_per_second=0.0407
2026-07-27 12:53:40.174 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=118/600, eta=3 hours, 18 minutes]
train/CrossEntropyLoss=0.0236
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.336
train/patch_acc=0.6562
train/subpatch_loss=0.6415
train/subpatch_acc=0.7500
train/location_loss=1.893
train/location_acc=0.1875
train/token_losses=20.82
throughput/total_tokens=1,933,312
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 12:54:04.703 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=119/600, eta=3 hours, 18 minutes]
train/CrossEntropyLoss=0.0440
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=1.170
train/patch_acc=0.7188
train/subpatch_loss=0.4573
train/subpatch_acc=0.8125
train/location_loss=1.937
train/location_acc=0.1250
train/token_losses=18.93
throughput/total_tokens=1,949,696
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 12:54:29.245 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=120/600, eta=3 hours, 17 minutes]
train/CrossEntropyLoss=0.0082
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.337
train/patch_acc=0.5938
train/subpatch_loss=0.3712
train/subpatch_acc=0.9375
train/location_loss=1.791
train/location_acc=0.2500
train/token_losses=19.34
throughput/total_tokens=1,966,080
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:54:53.775 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=121/600, eta=3 hours, 17 minutes]
train/CrossEntropyLoss=0.0336
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.396
train/patch_acc=0.6875
train/subpatch_loss=0.3610
train/subpatch_acc=0.8125
train/location_loss=1.605
train/location_acc=0.5000
train/token_losses=19.03
throughput/total_tokens=1,982,464
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:55:18.316 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=122/600, eta=3 hours, 16 minutes]
train/CrossEntropyLoss=0.1988
train/ZLoss=0.0008
train/Accuracy=0.9512
train/patch_loss=1.206
train/patch_acc=0.6875
train/subpatch_loss=0.3625
train/subpatch_acc=0.8750
train/location_loss=1.861
train/location_acc=0.3750
train/token_losses=18.54
throughput/total_tokens=1,998,848
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:55:42.865 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=123/600, eta=3 hours, 16 minutes]
train/CrossEntropyLoss=0.0182
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9094
train/patch_acc=0.7500
train/subpatch_loss=0.4175
train/subpatch_acc=0.8125
train/location_loss=1.828
train/location_acc=0.4375
train/token_losses=16.26
throughput/total_tokens=2,015,232
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:56:07.401 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=124/600, eta=3 hours, 16 minutes]
train/CrossEntropyLoss=0.0139
train/ZLoss=0.0005
train/Accuracy=0.9939
train/patch_loss=1.441
train/patch_acc=0.5938
train/subpatch_loss=0.2543
train/subpatch_acc=1.0000
train/location_loss=1.819
train/location_acc=0.1875
train/token_losses=19.82
throughput/total_tokens=2,031,616
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:56:31.925 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=125/600, eta=3 hours, 15 minutes]
train/CrossEntropyLoss=0.0238
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.553
train/patch_acc=0.5312
train/subpatch_loss=0.4405
train/subpatch_acc=0.9375
train/location_loss=1.780
train/location_acc=0.2500
train/token_losses=21.31
throughput/total_tokens=2,048,000
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 12:56:56.444 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=126/600, eta=3 hours, 15 minutes]
train/CrossEntropyLoss=0.1086
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=1.395
train/patch_acc=0.7188
train/subpatch_loss=0.5356
train/subpatch_acc=0.8750
train/location_loss=1.939
train/location_acc=0.2500
train/token_losses=21.05
throughput/total_tokens=2,064,384
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:57:21.001 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=127/600, eta=3 hours, 14 minutes]
train/CrossEntropyLoss=0.0365
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.157
train/patch_acc=0.7188
train/subpatch_loss=0.5170
train/subpatch_acc=0.8125
train/location_loss=1.781
train/location_acc=0.3125
train/token_losses=18.44
throughput/total_tokens=2,080,768
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 12:57:45.562 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=128/600, eta=3 hours, 14 minutes]
train/CrossEntropyLoss=0.0266
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8668
train/patch_acc=0.7500
train/subpatch_loss=0.2891
train/subpatch_acc=0.9375
train/location_loss=1.856
train/location_acc=0.1875
train/token_losses=15.51
throughput/total_tokens=2,097,152
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:58:10.086 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=129/600, eta=3 hours, 13 minutes]
train/CrossEntropyLoss=0.0334
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.429
train/patch_acc=0.5938
train/subpatch_loss=0.2467
train/subpatch_acc=0.9375
train/location_loss=1.959
train/location_acc=0.1250
train/token_losses=20.26
throughput/total_tokens=2,113,536
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:58:34.624 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=130/600, eta=3 hours, 13 minutes]
train/CrossEntropyLoss=0.1707
train/ZLoss=0.0006
train/Accuracy=0.9573
train/patch_loss=1.732
train/patch_acc=0.5938
train/subpatch_loss=0.6073
train/subpatch_acc=0.8125
train/location_loss=2.004
train/location_acc=0.1250
train/token_losses=24.30
throughput/total_tokens=2,129,920
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 12:58:59.150 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=131/600, eta=3 hours, 13 minutes]
train/CrossEntropyLoss=0.0336
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.114
train/patch_acc=0.8125
train/subpatch_loss=0.3193
train/subpatch_acc=0.8750
train/location_loss=1.713
train/location_acc=0.3750
train/token_losses=17.04
throughput/total_tokens=2,146,304
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 12:59:23.683 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=132/600, eta=3 hours, 12 minutes]
train/CrossEntropyLoss=0.1020
train/ZLoss=0.0007
train/Accuracy=0.9695
train/patch_loss=1.392
train/patch_acc=0.5938
train/subpatch_loss=0.4300
train/subpatch_acc=0.8125
train/location_loss=1.842
train/location_acc=0.5000
train/token_losses=20.22
throughput/total_tokens=2,162,688
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 12:59:48.202 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=133/600, eta=3 hours, 12 minutes]
train/CrossEntropyLoss=0.0605
train/ZLoss=0.0007
train/Accuracy=0.9848
train/patch_loss=1.475
train/patch_acc=0.6875
train/subpatch_loss=0.3237
train/subpatch_acc=0.8750
train/location_loss=1.845
train/location_acc=0.3125
train/token_losses=20.47
throughput/total_tokens=2,179,072
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:00:12.767 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=134/600, eta=3 hours, 11 minutes]
train/CrossEntropyLoss=0.0575
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.634
train/patch_acc=0.5312
train/subpatch_loss=0.6145
train/subpatch_acc=0.6875
train/location_loss=1.942
train/location_acc=0.1875
train/token_losses=23.30
throughput/total_tokens=2,195,456
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:00:37.283 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=135/600, eta=3 hours, 11 minutes]
train/CrossEntropyLoss=0.0111
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.194
train/patch_acc=0.6562
train/subpatch_loss=0.2538
train/subpatch_acc=0.9375
train/location_loss=1.749
train/location_acc=0.4375
train/token_losses=17.57
throughput/total_tokens=2,211,840
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:01:01.819 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=136/600, eta=3 hours, 11 minutes]
train/CrossEntropyLoss=0.0540
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.416
train/patch_acc=0.6250
train/subpatch_loss=0.4363
train/subpatch_acc=0.8750
train/location_loss=1.720
train/location_acc=0.3750
train/token_losses=19.95
throughput/total_tokens=2,228,224
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:01:26.392 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=137/600, eta=3 hours, 10 minutes]
train/CrossEntropyLoss=0.0908
train/ZLoss=0.0006
train/Accuracy=0.9756
train/patch_loss=1.184
train/patch_acc=0.6875
train/subpatch_loss=0.4609
train/subpatch_acc=0.7500
train/location_loss=1.786
train/location_acc=0.3125
train/token_losses=18.46
throughput/total_tokens=2,244,608
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:01:50.929 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=138/600, eta=3 hours, 10 minutes]
train/CrossEntropyLoss=0.0051
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.359
train/patch_acc=0.6562
train/subpatch_loss=0.3542
train/subpatch_acc=0.8125
train/location_loss=1.872
train/location_acc=0.3125
train/token_losses=19.78
throughput/total_tokens=2,260,992
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:02:15.465 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=139/600, eta=3 hours, 9 minutes]
train/CrossEntropyLoss=0.0216
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.101
train/patch_acc=0.6875
train/subpatch_loss=0.4480
train/subpatch_acc=0.8125
train/location_loss=1.658
train/location_acc=0.3125
train/token_losses=17.24
throughput/total_tokens=2,277,376
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:02:39.993 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=140/600, eta=3 hours, 9 minutes]
train/CrossEntropyLoss=0.0368
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9627
train/patch_acc=0.7188
train/subpatch_loss=0.3435
train/subpatch_acc=0.9375
train/location_loss=1.804
train/location_acc=0.3125
train/token_losses=16.29
throughput/total_tokens=2,293,760
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:03:04.520 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=141/600, eta=3 hours, 9 minutes]
train/CrossEntropyLoss=0.0525
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.332
train/patch_acc=0.6562
train/subpatch_loss=0.4711
train/subpatch_acc=0.8125
train/location_loss=1.731
train/location_acc=0.2500
train/token_losses=19.46
throughput/total_tokens=2,310,144
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:03:29.061 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=142/600, eta=3 hours, 8 minutes]
train/CrossEntropyLoss=0.0642
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.9765
train/patch_acc=0.7188
train/subpatch_loss=0.3014
train/subpatch_acc=0.8750
train/location_loss=1.723
train/location_acc=0.4375
train/token_losses=15.91
throughput/total_tokens=2,326,528
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:03:53.595 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=143/600, eta=3 hours, 8 minutes]
train/CrossEntropyLoss=0.0065
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.298
train/patch_acc=0.6562
train/subpatch_loss=0.3005
train/subpatch_acc=0.9375
train/location_loss=1.505
train/location_acc=0.3125
train/token_losses=17.60
throughput/total_tokens=2,342,912
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:04:18.117 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=144/600, eta=3 hours, 7 minutes]
train/CrossEntropyLoss=0.0146
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.491
train/patch_acc=0.6250
train/subpatch_loss=0.4538
train/subpatch_acc=0.7500
train/location_loss=1.602
train/location_acc=0.3125
train/token_losses=20.15
throughput/total_tokens=2,359,296
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:04:42.628 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=145/600, eta=3 hours, 7 minutes]
train/CrossEntropyLoss=0.0028
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=1.285
train/patch_acc=0.6562
train/subpatch_loss=0.3834
train/subpatch_acc=0.8750
train/location_loss=1.727
train/location_acc=0.1875
train/token_losses=18.72
throughput/total_tokens=2,375,680
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:05:07.162 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=146/600, eta=3 hours, 6 minutes]
train/CrossEntropyLoss=0.0282
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.255
train/patch_acc=0.6562
train/subpatch_loss=0.3107
train/subpatch_acc=0.8750
train/location_loss=1.670
train/location_acc=0.3125
train/token_losses=17.97
throughput/total_tokens=2,392,064
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:05:31.677 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=147/600, eta=3 hours, 6 minutes]
train/CrossEntropyLoss=0.0533
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.565
train/patch_acc=0.5938
train/subpatch_loss=0.3059
train/subpatch_acc=0.9375
train/location_loss=1.772
train/location_acc=0.3125
train/token_losses=20.83
throughput/total_tokens=2,408,448
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:05:56.203 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=148/600, eta=3 hours, 6 minutes]
train/CrossEntropyLoss=0.0157
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.108
train/patch_acc=0.7188
train/subpatch_loss=0.2035
train/subpatch_acc=1.0000
train/location_loss=1.672
train/location_acc=0.5000
train/token_losses=16.37
throughput/total_tokens=2,424,832
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:06:20.727 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=149/600, eta=3 hours, 5 minutes]
train/CrossEntropyLoss=0.0101
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.064
train/patch_acc=0.7188
train/subpatch_loss=0.4900
train/subpatch_acc=0.8125
train/location_loss=1.786
train/location_acc=0.2500
train/token_losses=17.62
throughput/total_tokens=2,441,216
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:06:45.254 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=150/600, eta=3 hours, 5 minutes]
train/CrossEntropyLoss=0.0163
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.165
train/patch_acc=0.7812
train/subpatch_loss=0.2850
train/subpatch_acc=1.0000
train/location_loss=1.599
train/location_acc=0.4375
train/token_losses=16.86
throughput/total_tokens=2,457,600
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:07:09.800 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=151/600, eta=3 hours, 4 minutes]
train/CrossEntropyLoss=0.0474
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=1.170
train/patch_acc=0.7188
train/subpatch_loss=0.4678
train/subpatch_acc=0.8125
train/location_loss=1.663
train/location_acc=0.2500
train/token_losses=17.88
throughput/total_tokens=2,473,984
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:07:34.328 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=152/600, eta=3 hours, 4 minutes]
train/CrossEntropyLoss=0.0191
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.241
train/patch_acc=0.6250
train/subpatch_loss=0.3746
train/subpatch_acc=0.9375
train/location_loss=1.537
train/location_acc=0.3750
train/token_losses=17.58
throughput/total_tokens=2,490,368
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:07:58.863 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=153/600, eta=3 hours, 4 minutes]
train/CrossEntropyLoss=0.0190
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.465
train/patch_acc=0.5938
train/subpatch_loss=0.3709
train/subpatch_acc=0.8750
train/location_loss=1.708
train/location_acc=0.3750
train/token_losses=20.04
throughput/total_tokens=2,506,752
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:08:23.403 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=154/600, eta=3 hours, 3 minutes]
train/CrossEntropyLoss=0.0112
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8831
train/patch_acc=0.7500
train/subpatch_loss=0.3709
train/subpatch_acc=0.8750
train/location_loss=1.668
train/location_acc=0.3125
train/token_losses=15.22
throughput/total_tokens=2,523,136
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:08:47.931 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=155/600, eta=3 hours, 3 minutes]
train/CrossEntropyLoss=0.0064
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.453
train/patch_acc=0.6875
train/subpatch_loss=0.4556
train/subpatch_acc=0.8125
train/location_loss=1.675
train/location_acc=0.4375
train/token_losses=20.15
throughput/total_tokens=2,539,520
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:09:12.452 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=156/600, eta=3 hours, 2 minutes]
train/CrossEntropyLoss=0.0109
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.463
train/patch_acc=0.6250
train/subpatch_loss=0.3136
train/subpatch_acc=0.8750
train/location_loss=1.628
train/location_acc=0.3125
train/token_losses=19.47
throughput/total_tokens=2,555,904
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:09:36.991 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=157/600, eta=3 hours, 2 minutes]
train/CrossEntropyLoss=0.0836
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.431
train/patch_acc=0.6250
train/subpatch_loss=0.4361
train/subpatch_acc=0.8750
train/location_loss=1.859
train/location_acc=0.2500
train/token_losses=20.63
throughput/total_tokens=2,572,288
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:10:01.545 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=158/600, eta=3 hours, 2 minutes]
train/CrossEntropyLoss=0.0269
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.083
train/patch_acc=0.6562
train/subpatch_loss=0.3754
train/subpatch_acc=0.8125
train/location_loss=1.622
train/location_acc=0.3750
train/token_losses=16.65
throughput/total_tokens=2,588,672
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:10:26.079 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=159/600, eta=3 hours, 1 minute]
train/CrossEntropyLoss=0.0153
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.072
train/patch_acc=0.7188
train/subpatch_loss=0.2790
train/subpatch_acc=0.8750
train/location_loss=1.598
train/location_acc=0.4375
train/token_losses=16.08
throughput/total_tokens=2,605,056
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:10:50.618 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=160/600, eta=3 hours, 1 minute]
train/CrossEntropyLoss=0.0260
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.132
train/patch_acc=0.6562
train/subpatch_loss=0.2304
train/subpatch_acc=0.9375
train/location_loss=1.668
train/location_acc=0.5000
train/token_losses=16.65
throughput/total_tokens=2,621,440
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:11:15.129 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=161/600, eta=3 hours]
train/CrossEntropyLoss=0.0129
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.368
train/patch_acc=0.6250
train/subpatch_loss=0.5876
train/subpatch_acc=0.8750
train/location_loss=1.330
train/location_acc=0.5000
train/token_losses=18.62
throughput/total_tokens=2,637,824
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:11:39.657 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=162/600, eta=3 hours]
train/CrossEntropyLoss=0.0313
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.614
train/patch_acc=0.6562
train/subpatch_loss=0.5735
train/subpatch_acc=0.7500
train/location_loss=1.315
train/location_acc=0.5000
train/token_losses=20.47
throughput/total_tokens=2,654,208
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:12:04.193 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=163/600, eta=2 hours, 59 minutes]
train/CrossEntropyLoss=0.0144
train/ZLoss=0.0006
train/Accuracy=0.9878
train/patch_loss=0.9331
train/patch_acc=0.7812
train/subpatch_loss=0.3364
train/subpatch_acc=0.8125
train/location_loss=1.756
train/location_acc=0.3125
train/token_losses=15.84
throughput/total_tokens=2,670,592
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:12:28.734 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=164/600, eta=2 hours, 59 minutes]
train/CrossEntropyLoss=0.0112
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.141
train/patch_acc=0.6875
train/subpatch_loss=0.3451
train/subpatch_acc=0.8750
train/location_loss=1.258
train/location_acc=0.6875
train/token_losses=15.54
throughput/total_tokens=2,686,976
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:12:53.281 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=165/600, eta=2 hours, 59 minutes]
train/CrossEntropyLoss=0.0094
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.152
train/patch_acc=0.6250
train/subpatch_loss=0.1018
train/subpatch_acc=1.0000
train/location_loss=1.549
train/location_acc=0.4375
train/token_losses=15.82
throughput/total_tokens=2,703,360
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:13:17.823 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=166/600, eta=2 hours, 58 minutes]
train/CrossEntropyLoss=0.0410
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.257
train/patch_acc=0.6875
train/subpatch_loss=0.1011
train/subpatch_acc=1.0000
train/location_loss=1.639
train/location_acc=0.3125
train/token_losses=17.01
throughput/total_tokens=2,719,744
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:13:42.371 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=167/600, eta=2 hours, 58 minutes]
train/CrossEntropyLoss=0.0672
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.282
train/patch_acc=0.6875
train/subpatch_loss=0.4091
train/subpatch_acc=0.7500
train/location_loss=1.628
train/location_acc=0.3750
train/token_losses=18.40
throughput/total_tokens=2,736,128
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:14:06.899 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=168/600, eta=2 hours, 57 minutes]
train/CrossEntropyLoss=0.0142
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.163
train/patch_acc=0.7500
train/subpatch_loss=0.3293
train/subpatch_acc=0.8750
train/location_loss=1.592
train/location_acc=0.2500
train/token_losses=16.99
throughput/total_tokens=2,752,512
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:14:31.429 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=169/600, eta=2 hours, 57 minutes]
train/CrossEntropyLoss=0.0060
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.139
train/patch_acc=0.6875
train/subpatch_loss=0.2333
train/subpatch_acc=0.9375
train/location_loss=1.667
train/location_acc=0.3125
train/token_losses=16.72
throughput/total_tokens=2,768,896
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:14:55.983 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=170/600, eta=2 hours, 57 minutes]
train/CrossEntropyLoss=0.0703
train/ZLoss=0.0002
train/Accuracy=0.9787
train/patch_loss=1.083
train/patch_acc=0.5938
train/subpatch_loss=0.2817
train/subpatch_acc=0.9375
train/location_loss=1.695
train/location_acc=0.4375
train/token_losses=16.57
throughput/total_tokens=2,785,280
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:15:20.516 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=171/600, eta=2 hours, 56 minutes]
train/CrossEntropyLoss=0.0451
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.149
train/patch_acc=0.5938
train/subpatch_loss=0.3964
train/subpatch_acc=0.8750
train/location_loss=1.592
train/location_acc=0.3750
train/token_losses=17.15
throughput/total_tokens=2,801,664
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:15:45.042 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=172/600, eta=2 hours, 56 minutes]
train/CrossEntropyLoss=0.0082
train/ZLoss=0.0004
train/Accuracy=0.9939
train/patch_loss=1.276
train/patch_acc=0.5625
train/subpatch_loss=0.3910
train/subpatch_acc=0.8125
train/location_loss=1.544
train/location_acc=0.3750
train/token_losses=17.94
throughput/total_tokens=2,818,048
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:16:09.560 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=173/600, eta=2 hours, 55 minutes]
train/CrossEntropyLoss=0.0532
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6955
train/patch_acc=0.8125
train/subpatch_loss=0.1646
train/subpatch_acc=0.9375
train/location_loss=1.452
train/location_acc=0.4375
train/token_losses=12.03
throughput/total_tokens=2,834,432
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:16:34.090 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=174/600, eta=2 hours, 55 minutes]
train/CrossEntropyLoss=0.0329
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.090
train/patch_acc=0.7188
train/subpatch_loss=0.2271
train/subpatch_acc=0.9375
train/location_loss=1.771
train/location_acc=0.2500
train/token_losses=16.71
throughput/total_tokens=2,850,816
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:16:58.602 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=175/600, eta=2 hours, 55 minutes]
train/CrossEntropyLoss=0.0124
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.091
train/patch_acc=0.6875
train/subpatch_loss=0.2746
train/subpatch_acc=1.0000
train/location_loss=1.659
train/location_acc=0.3750
train/token_losses=16.46
throughput/total_tokens=2,867,200
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:17:23.141 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=176/600, eta=2 hours, 54 minutes]
train/CrossEntropyLoss=0.0240
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.338
train/patch_acc=0.6875
train/subpatch_loss=0.2365
train/subpatch_acc=0.8750
train/location_loss=1.468
train/location_acc=0.5000
train/token_losses=17.52
throughput/total_tokens=2,883,584
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:17:47.667 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=177/600, eta=2 hours, 54 minutes]
train/CrossEntropyLoss=0.0295
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9045
train/patch_acc=0.6875
train/subpatch_loss=0.2576
train/subpatch_acc=0.9375
train/location_loss=1.500
train/location_acc=0.5000
train/token_losses=14.27
throughput/total_tokens=2,899,968
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:18:12.208 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=178/600, eta=2 hours, 53 minutes]
train/CrossEntropyLoss=0.0234
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.113
train/patch_acc=0.7188
train/subpatch_loss=0.3337
train/subpatch_acc=0.8750
train/location_loss=1.221
train/location_acc=0.6250
train/token_losses=15.12
throughput/total_tokens=2,916,352
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:18:36.735 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=179/600, eta=2 hours, 53 minutes]
train/CrossEntropyLoss=0.0345
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.193
train/patch_acc=0.6250
train/subpatch_loss=0.4259
train/subpatch_acc=0.7500
train/location_loss=1.797
train/location_acc=0.3125
train/token_losses=18.44
throughput/total_tokens=2,932,736
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:19:01.278 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=180/600, eta=2 hours, 52 minutes]
train/CrossEntropyLoss=0.0233
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.132
train/patch_acc=0.7188
train/subpatch_loss=0.4378
train/subpatch_acc=0.8125
train/location_loss=1.980
train/location_acc=0.1875
train/token_losses=18.73
throughput/total_tokens=2,949,120
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:19:25.792 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=181/600, eta=2 hours, 52 minutes]
train/CrossEntropyLoss=0.0150
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9730
train/patch_acc=0.7188
train/subpatch_loss=0.3283
train/subpatch_acc=0.9375
train/location_loss=1.438
train/location_acc=0.4375
train/token_losses=14.85
throughput/total_tokens=2,965,504
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:19:50.311 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=182/600, eta=2 hours, 52 minutes]
train/CrossEntropyLoss=0.0263
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.335
train/patch_acc=0.6250
train/subpatch_loss=0.1778
train/subpatch_acc=1.0000
train/location_loss=1.782
train/location_acc=0.3125
train/token_losses=18.52
throughput/total_tokens=2,981,888
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:20:14.838 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=183/600, eta=2 hours, 51 minutes]
train/CrossEntropyLoss=0.0169
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.426
train/patch_acc=0.6562
train/subpatch_loss=0.6100
train/subpatch_acc=0.8125
train/location_loss=1.278
train/location_acc=0.3750
train/token_losses=18.96
throughput/total_tokens=2,998,272
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:20:39.358 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=184/600, eta=2 hours, 51 minutes]
train/CrossEntropyLoss=0.0600
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.185
train/patch_acc=0.6875
train/subpatch_loss=0.3371
train/subpatch_acc=0.9375
train/location_loss=1.671
train/location_acc=0.3750
train/token_losses=17.52
throughput/total_tokens=3,014,656
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:21:03.885 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=185/600, eta=2 hours, 50 minutes]
train/CrossEntropyLoss=0.0675
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.302
train/patch_acc=0.6562
train/subpatch_loss=0.6097
train/subpatch_acc=0.6250
train/location_loss=1.661
train/location_acc=0.4375
train/token_losses=19.50
throughput/total_tokens=3,031,040
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:21:28.413 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=186/600, eta=2 hours, 50 minutes]
train/CrossEntropyLoss=0.0247
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9202
train/patch_acc=0.7188
train/subpatch_loss=0.2108
train/subpatch_acc=0.9375
train/location_loss=1.261
train/location_acc=0.6250
train/token_losses=13.25
throughput/total_tokens=3,047,424
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:21:52.950 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=187/600, eta=2 hours, 50 minutes]
train/CrossEntropyLoss=0.0259
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.9579
train/patch_acc=0.6875
train/subpatch_loss=0.3063
train/subpatch_acc=0.8750
train/location_loss=1.871
train/location_acc=0.1250
train/token_losses=16.37
throughput/total_tokens=3,063,808
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:22:17.463 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=188/600, eta=2 hours, 49 minutes]
train/CrossEntropyLoss=0.0374
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.109
train/patch_acc=0.6875
train/subpatch_loss=0.2732
train/subpatch_acc=0.8750
train/location_loss=1.625
train/location_acc=0.3125
train/token_losses=16.46
throughput/total_tokens=3,080,192
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:22:42.005 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=189/600, eta=2 hours, 49 minutes]
train/CrossEntropyLoss=0.0438
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9955
train/patch_acc=0.7188
train/subpatch_loss=0.2958
train/subpatch_acc=0.8750
train/location_loss=1.362
train/location_acc=0.6250
train/token_losses=14.60
throughput/total_tokens=3,096,576
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:23:06.527 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=190/600, eta=2 hours, 48 minutes]
train/CrossEntropyLoss=0.0068
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.233
train/patch_acc=0.6250
train/subpatch_loss=0.3008
train/subpatch_acc=0.8750
train/location_loss=1.876
train/location_acc=0.3125
train/token_losses=18.58
throughput/total_tokens=3,112,960
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:23:31.109 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=191/600, eta=2 hours, 48 minutes]
train/CrossEntropyLoss=0.0285
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.189
train/patch_acc=0.6562
train/subpatch_loss=0.6557
train/subpatch_acc=0.8125
train/location_loss=1.218
train/location_acc=0.6250
train/token_losses=17.01
throughput/total_tokens=3,129,344
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:23:55.663 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=192/600, eta=2 hours, 48 minutes]
train/CrossEntropyLoss=0.0254
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8855
train/patch_acc=0.7812
train/subpatch_loss=0.2937
train/subpatch_acc=0.9375
train/location_loss=1.316
train/location_acc=0.5625
train/token_losses=13.52
throughput/total_tokens=3,145,728
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:24:20.186 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=193/600, eta=2 hours, 47 minutes]
train/CrossEntropyLoss=0.0634
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9928
train/patch_acc=0.7500
train/subpatch_loss=0.3289
train/subpatch_acc=0.8750
train/location_loss=1.473
train/location_acc=0.3125
train/token_losses=15.15
throughput/total_tokens=3,162,112
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:24:44.698 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=194/600, eta=2 hours, 47 minutes]
train/CrossEntropyLoss=0.0244
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.053
train/patch_acc=0.6562
train/subpatch_loss=0.4282
train/subpatch_acc=0.8125
train/location_loss=1.652
train/location_acc=0.3750
train/token_losses=16.74
throughput/total_tokens=3,178,496
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:25:09.224 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=195/600, eta=2 hours, 46 minutes]
train/CrossEntropyLoss=0.1309
train/ZLoss=0.0005
train/Accuracy=0.9695
train/patch_loss=1.219
train/patch_acc=0.6562
train/subpatch_loss=0.3268
train/subpatch_acc=0.8750
train/location_loss=1.750
train/location_acc=0.3750
train/token_losses=18.06
throughput/total_tokens=3,194,880
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:25:33.777 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=196/600, eta=2 hours, 46 minutes]
train/CrossEntropyLoss=0.0285
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.229
train/patch_acc=0.6562
train/subpatch_loss=0.2189
train/subpatch_acc=0.9375
train/location_loss=1.428
train/location_acc=0.3750
train/token_losses=16.42
throughput/total_tokens=3,211,264
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:25:58.309 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=197/600, eta=2 hours, 45 minutes]
train/CrossEntropyLoss=0.0283
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.8187
train/patch_acc=0.8125
train/subpatch_loss=0.2523
train/subpatch_acc=0.9375
train/location_loss=1.410
train/location_acc=0.6875
train/token_losses=13.20
throughput/total_tokens=3,227,648
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:26:22.846 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=198/600, eta=2 hours, 45 minutes]
train/CrossEntropyLoss=0.0526
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.059
train/patch_acc=0.6875
train/subpatch_loss=0.2388
train/subpatch_acc=1.0000
train/location_loss=1.254
train/location_acc=0.5625
train/token_losses=14.44
throughput/total_tokens=3,244,032
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:26:47.394 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=199/600, eta=2 hours, 45 minutes]
train/CrossEntropyLoss=0.0289
train/ZLoss=0.0005
train/Accuracy=0.9909
train/patch_loss=1.007
train/patch_acc=0.7188
train/subpatch_loss=0.2643
train/subpatch_acc=0.9375
train/location_loss=1.346
train/location_acc=0.6250
train/token_losses=14.50
throughput/total_tokens=3,260,416
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:27:11.948 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=200/600, eta=2 hours, 44 minutes]
train/CrossEntropyLoss=0.0600
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=1.159
train/patch_acc=0.6562
train/subpatch_loss=0.2863
train/subpatch_acc=0.8125
train/location_loss=1.838
train/location_acc=0.2500
train/token_losses=17.77
throughput/total_tokens=3,276,800
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:27:11.949 2012bc4f83ae:0 olmo.train.trainer:1415 INFO Saving checkpoint...
2026-07-27 13:27:14.021 2012bc4f83ae:0 olmo.train.trainer:1420 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step200
2026-07-27 13:27:38.825 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=201/600, eta=2 hours, 44 minutes]
train/CrossEntropyLoss=0.0293
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.051
train/patch_acc=0.6875
train/subpatch_loss=0.3679
train/subpatch_acc=0.8125
train/location_loss=1.443
train/location_acc=0.3750
train/token_losses=15.65
throughput/total_tokens=3,293,184
throughput/device/tokens_per_second=660.6
throughput/device/batches_per_second=0.0403
2026-07-27 13:28:03.333 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=202/600, eta=2 hours, 44 minutes]
train/CrossEntropyLoss=0.0306
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.054
train/patch_acc=0.7500
train/subpatch_loss=0.2129
train/subpatch_acc=0.8750
train/location_loss=1.518
train/location_acc=0.3125
train/token_losses=15.35
throughput/total_tokens=3,309,568
throughput/device/tokens_per_second=664.5
throughput/device/batches_per_second=0.0406
2026-07-27 13:28:27.845 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=203/600, eta=2 hours, 43 minutes]
train/CrossEntropyLoss=0.0073
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8504
train/patch_acc=0.7812
train/subpatch_loss=0.4100
train/subpatch_acc=0.8125
train/location_loss=1.256
train/location_acc=0.6250
train/token_losses=13.47
throughput/total_tokens=3,325,952
throughput/device/tokens_per_second=665.8
throughput/device/batches_per_second=0.0406
2026-07-27 13:28:52.367 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=204/600, eta=2 hours, 43 minutes]
train/CrossEntropyLoss=0.0616
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.9215
train/patch_acc=0.7500
train/subpatch_loss=0.3042
train/subpatch_acc=0.9375
train/location_loss=1.487
train/location_acc=0.5625
train/token_losses=14.54
throughput/total_tokens=3,342,336
throughput/device/tokens_per_second=666.4
throughput/device/batches_per_second=0.0407
2026-07-27 13:29:16.910 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=205/600, eta=2 hours, 42 minutes]
train/CrossEntropyLoss=0.0230
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.298
train/patch_acc=0.6250
train/subpatch_loss=0.0624
train/subpatch_acc=1.0000
train/location_loss=1.345
train/location_acc=0.5000
train/token_losses=16.02
throughput/total_tokens=3,358,720
throughput/device/tokens_per_second=666.6
throughput/device/batches_per_second=0.0407
2026-07-27 13:29:41.457 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=206/600, eta=2 hours, 42 minutes]
train/CrossEntropyLoss=0.0925
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.8903
train/patch_acc=0.7188
train/subpatch_loss=0.1460
train/subpatch_acc=1.0000
train/location_loss=1.410
train/location_acc=0.6250
train/token_losses=13.35
throughput/total_tokens=3,375,104
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 13:30:06.044 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=207/600, eta=2 hours, 41 minutes]
train/CrossEntropyLoss=0.0411
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8739
train/patch_acc=0.7188
train/subpatch_loss=0.3182
train/subpatch_acc=0.9375
train/location_loss=1.522
train/location_acc=0.5000
train/token_losses=14.35
throughput/total_tokens=3,391,488
throughput/device/tokens_per_second=666.7
throughput/device/batches_per_second=0.0407
2026-07-27 13:30:30.612 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=208/600, eta=2 hours, 41 minutes]
train/CrossEntropyLoss=0.0214
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.8943
train/patch_acc=0.7500
train/subpatch_loss=0.2751
train/subpatch_acc=0.8750
train/location_loss=1.418
train/location_acc=0.4375
train/token_losses=13.93
throughput/total_tokens=3,407,872
throughput/device/tokens_per_second=666.7
throughput/device/batches_per_second=0.0407
2026-07-27 13:30:55.163 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=209/600, eta=2 hours, 41 minutes]
train/CrossEntropyLoss=0.0188
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.078
train/patch_acc=0.6562
train/subpatch_loss=0.2879
train/subpatch_acc=0.9375
train/location_loss=1.392
train/location_acc=0.5625
train/token_losses=15.34
throughput/total_tokens=3,424,256
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 13:31:19.711 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=210/600, eta=2 hours, 40 minutes]
train/CrossEntropyLoss=0.0155
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.275
train/patch_acc=0.5312
train/subpatch_loss=0.3120
train/subpatch_acc=0.8750
train/location_loss=1.386
train/location_acc=0.5000
train/token_losses=17.00
throughput/total_tokens=3,440,640
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:31:44.270 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=211/600, eta=2 hours, 40 minutes]
train/CrossEntropyLoss=0.0205
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.103
train/patch_acc=0.7500
train/subpatch_loss=0.2393
train/subpatch_acc=0.9375
train/location_loss=1.606
train/location_acc=0.5000
train/token_losses=16.21
throughput/total_tokens=3,457,024
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 13:32:08.832 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=212/600, eta=2 hours, 39 minutes]
train/CrossEntropyLoss=0.0853
train/ZLoss=0.0007
train/Accuracy=0.9756
train/patch_loss=0.8682
train/patch_acc=0.7812
train/subpatch_loss=0.1177
train/subpatch_acc=1.0000
train/location_loss=1.465
train/location_acc=0.6250
train/token_losses=13.28
throughput/total_tokens=3,473,408
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 13:32:33.365 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=213/600, eta=2 hours, 39 minutes]
train/CrossEntropyLoss=0.0334
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.273
train/patch_acc=0.6875
train/subpatch_loss=0.1824
train/subpatch_acc=0.9375
train/location_loss=1.257
train/location_acc=0.5625
train/token_losses=15.94
throughput/total_tokens=3,489,792
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 13:32:57.926 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=214/600, eta=2 hours, 39 minutes]
train/CrossEntropyLoss=0.1748
train/ZLoss=0.0008
train/Accuracy=0.9573
train/patch_loss=1.301
train/patch_acc=0.6875
train/subpatch_loss=0.5103
train/subpatch_acc=0.7500
train/location_loss=1.428
train/location_acc=0.4375
train/token_losses=18.16
throughput/total_tokens=3,506,176
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 13:33:22.468 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=215/600, eta=2 hours, 38 minutes]
train/CrossEntropyLoss=0.0172
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9626
train/patch_acc=0.7188
train/subpatch_loss=0.2434
train/subpatch_acc=0.9375
train/location_loss=1.455
train/location_acc=0.3750
train/token_losses=14.49
throughput/total_tokens=3,522,560
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 13:33:47.011 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=216/600, eta=2 hours, 38 minutes]
train/CrossEntropyLoss=0.0399
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.217
train/patch_acc=0.6562
train/subpatch_loss=0.2705
train/subpatch_acc=0.8750
train/location_loss=1.464
train/location_acc=0.5000
train/token_losses=16.68
throughput/total_tokens=3,538,944
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 13:34:11.551 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=217/600, eta=2 hours, 37 minutes]
train/CrossEntropyLoss=0.0432
train/ZLoss=0.0008
train/Accuracy=0.9878
train/patch_loss=0.9462
train/patch_acc=0.7188
train/subpatch_loss=0.2386
train/subpatch_acc=0.8750
train/location_loss=1.495
train/location_acc=0.2500
train/token_losses=14.50
throughput/total_tokens=3,555,328
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 13:34:36.078 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=218/600, eta=2 hours, 37 minutes]
train/CrossEntropyLoss=0.0701
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.220
train/patch_acc=0.7188
train/subpatch_loss=0.1752
train/subpatch_acc=1.0000
train/location_loss=1.483
train/location_acc=0.5000
train/token_losses=16.39
throughput/total_tokens=3,571,712
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 13:35:00.625 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=219/600, eta=2 hours, 37 minutes]
train/CrossEntropyLoss=0.1799
train/ZLoss=0.0006
train/Accuracy=0.9512
train/patch_loss=1.607
train/patch_acc=0.5938
train/subpatch_loss=0.4187
train/subpatch_acc=0.9375
train/location_loss=1.806
train/location_acc=0.2500
train/token_losses=21.75
throughput/total_tokens=3,588,096
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 13:35:25.146 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=220/600, eta=2 hours, 36 minutes]
train/CrossEntropyLoss=0.0537
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7372
train/patch_acc=0.8125
train/subpatch_loss=0.3014
train/subpatch_acc=0.9375
train/location_loss=1.344
train/location_acc=0.4375
train/token_losses=12.48
throughput/total_tokens=3,604,480
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:35:49.675 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=221/600, eta=2 hours, 36 minutes]
train/CrossEntropyLoss=0.0350
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.025
train/patch_acc=0.6875
train/subpatch_loss=0.2243
train/subpatch_acc=1.0000
train/location_loss=1.018
train/location_acc=0.6875
train/token_losses=13.17
throughput/total_tokens=3,620,864
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 13:36:14.204 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=222/600, eta=2 hours, 35 minutes]
train/CrossEntropyLoss=0.0978
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.8802
train/patch_acc=0.7812
train/subpatch_loss=0.1338
train/subpatch_acc=1.0000
train/location_loss=1.189
train/location_acc=0.3750
train/token_losses=12.33
throughput/total_tokens=3,637,248
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 13:36:38.754 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=223/600, eta=2 hours, 35 minutes]
train/CrossEntropyLoss=0.0557
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.046
train/patch_acc=0.7500
train/subpatch_loss=0.2319
train/subpatch_acc=0.9375
train/location_loss=1.626
train/location_acc=0.4375
train/token_losses=15.80
throughput/total_tokens=3,653,632
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 13:37:03.264 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=224/600, eta=2 hours, 34 minutes]
train/CrossEntropyLoss=0.0472
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.042
train/patch_acc=0.6250
train/subpatch_loss=0.2576
train/subpatch_acc=0.9375
train/location_loss=1.410
train/location_acc=0.4375
train/token_losses=15.00
throughput/total_tokens=3,670,016
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 13:37:27.795 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=225/600, eta=2 hours, 34 minutes]
train/CrossEntropyLoss=0.0534
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.149
train/patch_acc=0.6250
train/subpatch_loss=0.2417
train/subpatch_acc=0.9375
train/location_loss=1.706
train/location_acc=0.3125
train/token_losses=16.98
throughput/total_tokens=3,686,400
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 13:37:52.329 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=226/600, eta=2 hours, 34 minutes]
train/CrossEntropyLoss=0.0523
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.309
train/patch_acc=0.5938
train/subpatch_loss=0.2771
train/subpatch_acc=0.9375
train/location_loss=1.254
train/location_acc=0.5000
train/token_losses=16.59
throughput/total_tokens=3,702,784
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 13:38:16.830 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=227/600, eta=2 hours, 33 minutes]
train/CrossEntropyLoss=0.0206
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.9175
train/patch_acc=0.7812
train/subpatch_loss=0.2195
train/subpatch_acc=0.9375
train/location_loss=1.121
train/location_acc=0.5625
train/token_losses=12.70
throughput/total_tokens=3,719,168
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:38:41.343 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=228/600, eta=2 hours, 33 minutes]
train/CrossEntropyLoss=0.1867
train/ZLoss=0.0007
train/Accuracy=0.9573
train/patch_loss=1.037
train/patch_acc=0.7188
train/subpatch_loss=0.2476
train/subpatch_acc=0.9375
train/location_loss=1.236
train/location_acc=0.6875
train/token_losses=14.23
throughput/total_tokens=3,735,552
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:39:05.891 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=229/600, eta=2 hours, 32 minutes]
train/CrossEntropyLoss=0.0551
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.5599
train/patch_acc=0.8750
train/subpatch_loss=0.1258
train/subpatch_acc=1.0000
train/location_loss=1.237
train/location_acc=0.4375
train/token_losses=9.929
throughput/total_tokens=3,751,936
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:39:30.417 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=230/600, eta=2 hours, 32 minutes]
train/CrossEntropyLoss=0.0035
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.8505
train/patch_acc=0.7500
train/subpatch_loss=0.1754
train/subpatch_acc=0.8750
train/location_loss=1.046
train/location_acc=0.7500
train/token_losses=11.69
throughput/total_tokens=3,768,320
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:39:54.960 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=231/600, eta=2 hours, 32 minutes]
train/CrossEntropyLoss=0.0323
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.8545
train/patch_acc=0.7812
train/subpatch_loss=0.3340
train/subpatch_acc=0.8750
train/location_loss=1.460
train/location_acc=0.6250
train/token_losses=14.01
throughput/total_tokens=3,784,704
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:40:19.488 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=232/600, eta=2 hours, 31 minutes]
train/CrossEntropyLoss=0.0479
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.033
train/patch_acc=0.7188
train/subpatch_loss=0.1189
train/subpatch_acc=0.9375
train/location_loss=1.682
train/location_acc=0.3125
train/token_losses=15.47
throughput/total_tokens=3,801,088
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:40:43.992 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=233/600, eta=2 hours, 31 minutes]
train/CrossEntropyLoss=0.0547
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.8717
train/patch_acc=0.7500
train/subpatch_loss=0.2355
train/subpatch_acc=0.9375
train/location_loss=1.269
train/location_acc=0.6875
train/token_losses=12.99
throughput/total_tokens=3,817,472
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:41:08.517 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=234/600, eta=2 hours, 30 minutes]
train/CrossEntropyLoss=0.0361
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.9063
train/patch_acc=0.7500
train/subpatch_loss=0.0744
train/subpatch_acc=1.0000
train/location_loss=1.626
train/location_acc=0.3125
train/token_losses=14.05
throughput/total_tokens=3,833,856
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:41:33.040 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=235/600, eta=2 hours, 30 minutes]
train/CrossEntropyLoss=0.0025
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=1.220
train/patch_acc=0.6875
train/subpatch_loss=0.1315
train/subpatch_acc=1.0000
train/location_loss=1.569
train/location_acc=0.4375
train/token_losses=16.56
throughput/total_tokens=3,850,240
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:41:57.567 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=236/600, eta=2 hours, 30 minutes]
train/CrossEntropyLoss=0.0628
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8635
train/patch_acc=0.7500
train/subpatch_loss=0.1229
train/subpatch_acc=1.0000
train/location_loss=1.443
train/location_acc=0.4375
train/token_losses=13.17
throughput/total_tokens=3,866,624
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:42:22.114 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=237/600, eta=2 hours, 29 minutes]
train/CrossEntropyLoss=0.0222
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.098
train/patch_acc=0.7500
train/subpatch_loss=0.2108
train/subpatch_acc=1.0000
train/location_loss=1.544
train/location_acc=0.4375
train/token_losses=15.80
throughput/total_tokens=3,883,008
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:42:46.651 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=238/600, eta=2 hours, 29 minutes]
train/CrossEntropyLoss=0.0095
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8421
train/patch_acc=0.7812
train/subpatch_loss=0.2148
train/subpatch_acc=0.9375
train/location_loss=1.535
train/location_acc=0.2500
train/token_losses=13.74
throughput/total_tokens=3,899,392
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:43:11.211 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=239/600, eta=2 hours, 28 minutes]
train/CrossEntropyLoss=0.3109
train/ZLoss=0.0011
train/Accuracy=0.9177
train/patch_loss=1.624
train/patch_acc=0.5312
train/subpatch_loss=0.3211
train/subpatch_acc=0.8750
train/location_loss=1.556
train/location_acc=0.4375
train/token_losses=20.50
throughput/total_tokens=3,915,776
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:43:35.734 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=240/600, eta=2 hours, 28 minutes]
train/CrossEntropyLoss=0.0193
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9187
train/patch_acc=0.8125
train/subpatch_loss=0.1558
train/subpatch_acc=1.0000
train/location_loss=1.304
train/location_acc=0.4375
train/token_losses=13.19
throughput/total_tokens=3,932,160
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:44:00.252 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=241/600, eta=2 hours, 28 minutes]
train/CrossEntropyLoss=0.0043
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.9915
train/patch_acc=0.6875
train/subpatch_loss=0.2114
train/subpatch_acc=0.9375
train/location_loss=1.313
train/location_acc=0.5000
train/token_losses=14.03
throughput/total_tokens=3,948,544
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:44:24.793 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=242/600, eta=2 hours, 27 minutes]
train/CrossEntropyLoss=0.0011
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.8858
train/patch_acc=0.7812
train/subpatch_loss=0.2204
train/subpatch_acc=0.9375
train/location_loss=1.245
train/location_acc=0.6250
train/token_losses=12.95
throughput/total_tokens=3,964,928
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:44:49.321 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=243/600, eta=2 hours, 27 minutes]
train/CrossEntropyLoss=0.0215
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9519
train/patch_acc=0.7188
train/subpatch_loss=0.1854
train/subpatch_acc=0.9375
train/location_loss=0.8762
train/location_acc=0.8125
train/token_losses=11.86
throughput/total_tokens=3,981,312
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 13:45:13.869 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=244/600, eta=2 hours, 26 minutes]
train/CrossEntropyLoss=0.1967
train/ZLoss=0.0007
train/Accuracy=0.9573
train/patch_loss=1.119
train/patch_acc=0.6875
train/subpatch_loss=0.2741
train/subpatch_acc=0.8750
train/location_loss=1.164
train/location_acc=0.5625
train/token_losses=14.70
throughput/total_tokens=3,997,696
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:45:38.386 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=245/600, eta=2 hours, 26 minutes]
train/CrossEntropyLoss=0.0808
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=1.004
train/patch_acc=0.7812
train/subpatch_loss=0.2528
train/subpatch_acc=0.8750
train/location_loss=1.253
train/location_acc=0.5625
train/token_losses=14.06
throughput/total_tokens=4,014,080
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:46:02.943 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=246/600, eta=2 hours, 25 minutes]
train/CrossEntropyLoss=0.0040
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.8777
train/patch_acc=0.7812
train/subpatch_loss=0.0873
train/subpatch_acc=1.0000
train/location_loss=1.353
train/location_acc=0.5000
train/token_losses=12.78
throughput/total_tokens=4,030,464
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:46:27.483 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=247/600, eta=2 hours, 25 minutes]
train/CrossEntropyLoss=0.0429
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.8240
train/patch_acc=0.7812
train/subpatch_loss=0.3038
train/subpatch_acc=0.8750
train/location_loss=1.192
train/location_acc=0.6250
train/token_losses=12.57
throughput/total_tokens=4,046,848
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:46:52.034 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=248/600, eta=2 hours, 25 minutes]
train/CrossEntropyLoss=0.0131
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.345
train/patch_acc=0.6875
train/subpatch_loss=0.5328
train/subpatch_acc=0.8125
train/location_loss=1.558
train/location_acc=0.2500
train/token_losses=19.12
throughput/total_tokens=4,063,232
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:47:16.564 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=249/600, eta=2 hours, 24 minutes]
train/CrossEntropyLoss=0.0241
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7069
train/patch_acc=0.8125
train/subpatch_loss=0.2217
train/subpatch_acc=0.9375
train/location_loss=1.622
train/location_acc=0.5000
train/token_losses=13.03
throughput/total_tokens=4,079,616
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:47:41.106 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=250/600, eta=2 hours, 24 minutes]
train/CrossEntropyLoss=0.2137
train/ZLoss=0.0005
train/Accuracy=0.9543
train/patch_loss=1.249
train/patch_acc=0.6875
train/subpatch_loss=0.3959
train/subpatch_acc=0.8750
train/location_loss=1.353
train/location_acc=0.6250
train/token_losses=16.99
throughput/total_tokens=4,096,000
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:48:05.634 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=251/600, eta=2 hours, 23 minutes]
train/CrossEntropyLoss=0.0515
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7189
train/patch_acc=0.9062
train/subpatch_loss=0.2479
train/subpatch_acc=0.8750
train/location_loss=1.613
train/location_acc=0.4375
train/token_losses=13.20
throughput/total_tokens=4,112,384
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:48:30.161 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=252/600, eta=2 hours, 23 minutes]
train/CrossEntropyLoss=0.1297
train/ZLoss=0.0008
train/Accuracy=0.9634
train/patch_loss=0.7759
train/patch_acc=0.7812
train/subpatch_loss=0.1923
train/subpatch_acc=0.8750
train/location_loss=1.050
train/location_acc=0.7500
train/token_losses=11.18
throughput/total_tokens=4,128,768
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:48:54.711 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=253/600, eta=2 hours, 23 minutes]
train/CrossEntropyLoss=0.0535
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.9858
train/patch_acc=0.7812
train/subpatch_loss=0.2582
train/subpatch_acc=0.8750
train/location_loss=1.177
train/location_acc=0.5625
train/token_losses=13.63
throughput/total_tokens=4,145,152
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:49:19.246 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=254/600, eta=2 hours, 22 minutes]
train/CrossEntropyLoss=0.0135
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.034
train/patch_acc=0.7188
train/subpatch_loss=0.2770
train/subpatch_acc=0.9375
train/location_loss=1.378
train/location_acc=0.3750
train/token_losses=14.89
throughput/total_tokens=4,161,536
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 13:49:43.753 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=255/600, eta=2 hours, 22 minutes]
train/CrossEntropyLoss=0.0803
train/ZLoss=0.0008
train/Accuracy=0.9787
train/patch_loss=0.9407
train/patch_acc=0.7500
train/subpatch_loss=0.1897
train/subpatch_acc=0.9375
train/location_loss=1.147
train/location_acc=0.6250
train/token_losses=12.87
throughput/total_tokens=4,177,920
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:50:08.264 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=256/600, eta=2 hours, 21 minutes]
train/CrossEntropyLoss=0.0515
train/ZLoss=0.0004
train/Accuracy=0.9878
train/patch_loss=0.6302
train/patch_acc=0.8125
train/subpatch_loss=0.0882
train/subpatch_acc=1.0000
train/location_loss=1.074
train/location_acc=0.6875
train/token_losses=9.691
throughput/total_tokens=4,194,304
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:50:32.811 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=257/600, eta=2 hours, 21 minutes]
train/CrossEntropyLoss=0.0159
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7382
train/patch_acc=0.8125
train/subpatch_loss=0.3982
train/subpatch_acc=0.8750
train/location_loss=1.150
train/location_acc=0.5000
train/token_losses=12.10
throughput/total_tokens=4,210,688
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:50:57.356 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=258/600, eta=2 hours, 21 minutes]
train/CrossEntropyLoss=0.1636
train/ZLoss=0.0007
train/Accuracy=0.9573
train/patch_loss=1.554
train/patch_acc=0.5625
train/subpatch_loss=0.4832
train/subpatch_acc=0.8750
train/location_loss=1.421
train/location_acc=0.6250
train/token_losses=20.05
throughput/total_tokens=4,227,072
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:51:21.882 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=259/600, eta=2 hours, 20 minutes]
train/CrossEntropyLoss=0.0355
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.8291
train/patch_acc=0.7188
train/subpatch_loss=0.1790
train/subpatch_acc=1.0000
train/location_loss=1.386
train/location_acc=0.5625
train/token_losses=12.89
throughput/total_tokens=4,243,456
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:51:46.398 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=260/600, eta=2 hours, 20 minutes]
train/CrossEntropyLoss=0.0893
train/ZLoss=0.0006
train/Accuracy=0.9787
train/patch_loss=1.039
train/patch_acc=0.6250
train/subpatch_loss=0.1693
train/subpatch_acc=1.0000
train/location_loss=1.249
train/location_acc=0.6875
train/token_losses=13.98
throughput/total_tokens=4,259,840
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:52:10.935 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=261/600, eta=2 hours, 19 minutes]
train/CrossEntropyLoss=0.0612
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.090
train/patch_acc=0.6562
train/subpatch_loss=0.2390
train/subpatch_acc=0.9375
train/location_loss=1.391
train/location_acc=0.5000
train/token_losses=15.24
throughput/total_tokens=4,276,224
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:52:35.477 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=262/600, eta=2 hours, 19 minutes]
train/CrossEntropyLoss=0.0375
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8209
train/patch_acc=0.8125
train/subpatch_loss=0.3192
train/subpatch_acc=0.9375
train/location_loss=1.142
train/location_acc=0.5625
train/token_losses=12.41
throughput/total_tokens=4,292,608
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:53:00.022 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=263/600, eta=2 hours, 18 minutes]
train/CrossEntropyLoss=0.0042
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=1.010
train/patch_acc=0.7500
train/subpatch_loss=0.1934
train/subpatch_acc=0.9375
train/location_loss=1.286
train/location_acc=0.6250
train/token_losses=13.99
throughput/total_tokens=4,308,992
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:53:24.533 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=264/600, eta=2 hours, 18 minutes]
train/CrossEntropyLoss=0.0446
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.082
train/patch_acc=0.7188
train/subpatch_loss=0.2024
train/subpatch_acc=1.0000
train/location_loss=1.399
train/location_acc=0.3125
train/token_losses=15.06
throughput/total_tokens=4,325,376
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:53:49.071 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=265/600, eta=2 hours, 18 minutes]
train/CrossEntropyLoss=0.0089
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.150
train/patch_acc=0.6875
train/subpatch_loss=0.0925
train/subpatch_acc=1.0000
train/location_loss=1.067
train/location_acc=0.7500
train/token_losses=13.84
throughput/total_tokens=4,341,760
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:54:13.580 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=266/600, eta=2 hours, 17 minutes]
train/CrossEntropyLoss=0.0018
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=1.131
train/patch_acc=0.6250
train/subpatch_loss=0.1726
train/subpatch_acc=0.9375
train/location_loss=1.333
train/location_acc=0.5625
train/token_losses=15.07
throughput/total_tokens=4,358,144
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:54:38.118 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=267/600, eta=2 hours, 17 minutes]
train/CrossEntropyLoss=0.0014
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.5914
train/patch_acc=0.8125
train/subpatch_loss=0.2046
train/subpatch_acc=0.9375
train/location_loss=1.494
train/location_acc=0.4375
train/token_losses=11.53
throughput/total_tokens=4,374,528
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:55:02.665 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=268/600, eta=2 hours, 16 minutes]
train/CrossEntropyLoss=0.0215
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7138
train/patch_acc=0.8438
train/subpatch_loss=0.1758
train/subpatch_acc=1.0000
train/location_loss=1.276
train/location_acc=0.4375
train/token_losses=11.52
throughput/total_tokens=4,390,912
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:55:27.210 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=269/600, eta=2 hours, 16 minutes]
train/CrossEntropyLoss=0.0554
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.9182
train/patch_acc=0.7500
train/subpatch_loss=0.2269
train/subpatch_acc=0.9375
train/location_loss=1.532
train/location_acc=0.5000
train/token_losses=14.38
throughput/total_tokens=4,407,296
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:55:51.754 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=270/600, eta=2 hours, 16 minutes]
train/CrossEntropyLoss=0.1530
train/ZLoss=0.0006
train/Accuracy=0.9665
train/patch_loss=0.9809
train/patch_acc=0.7812
train/subpatch_loss=0.1028
train/subpatch_acc=1.0000
train/location_loss=1.162
train/location_acc=0.6250
train/token_losses=12.91
throughput/total_tokens=4,423,680
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 13:56:16.293 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=271/600, eta=2 hours, 15 minutes]
train/CrossEntropyLoss=0.0847
train/ZLoss=0.0004
train/Accuracy=0.9726
train/patch_loss=0.6748
train/patch_acc=0.8125
train/subpatch_loss=0.1608
train/subpatch_acc=0.9375
train/location_loss=1.132
train/location_acc=0.6250
train/token_losses=10.57
throughput/total_tokens=4,440,064
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:56:40.823 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=272/600, eta=2 hours, 15 minutes]
train/CrossEntropyLoss=0.0452
train/ZLoss=0.0001
train/Accuracy=0.9787
train/patch_loss=0.8922
train/patch_acc=0.7500
train/subpatch_loss=0.1380
train/subpatch_acc=1.0000
train/location_loss=1.225
train/location_acc=0.5625
train/token_losses=12.59
throughput/total_tokens=4,456,448
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:57:05.345 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=273/600, eta=2 hours, 14 minutes]
train/CrossEntropyLoss=0.0259
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.9467
train/patch_acc=0.7188
train/subpatch_loss=0.0916
train/subpatch_acc=1.0000
train/location_loss=1.033
train/location_acc=0.6250
train/token_losses=12.07
throughput/total_tokens=4,472,832
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:57:29.883 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=274/600, eta=2 hours, 14 minutes]
train/CrossEntropyLoss=0.0201
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9665
train/patch_acc=0.7500
train/subpatch_loss=0.1336
train/subpatch_acc=1.0000
train/location_loss=1.290
train/location_acc=0.4375
train/token_losses=13.43
throughput/total_tokens=4,489,216
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:57:54.406 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=275/600, eta=2 hours, 14 minutes]
train/CrossEntropyLoss=0.0213
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.241
train/patch_acc=0.6562
train/subpatch_loss=0.1724
train/subpatch_acc=0.9375
train/location_loss=1.397
train/location_acc=0.5000
train/token_losses=16.21
throughput/total_tokens=4,505,600
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:58:18.925 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=276/600, eta=2 hours, 13 minutes]
train/CrossEntropyLoss=0.0753
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9129
train/patch_acc=0.7500
train/subpatch_loss=0.3990
train/subpatch_acc=0.8125
train/location_loss=1.445
train/location_acc=0.5000
train/token_losses=14.68
throughput/total_tokens=4,521,984
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:58:43.466 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=277/600, eta=2 hours, 13 minutes]
train/CrossEntropyLoss=0.0272
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9636
train/patch_acc=0.6562
train/subpatch_loss=0.2264
train/subpatch_acc=0.9375
train/location_loss=1.244
train/location_acc=0.5000
train/token_losses=13.59
throughput/total_tokens=4,538,368
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 13:59:08.019 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=278/600, eta=2 hours, 12 minutes]
train/CrossEntropyLoss=0.0171
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.8862
train/patch_acc=0.8125
train/subpatch_loss=0.1262
train/subpatch_acc=1.0000
train/location_loss=1.399
train/location_acc=0.5000
train/token_losses=13.19
throughput/total_tokens=4,554,752
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:59:32.548 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=279/600, eta=2 hours, 12 minutes]
train/CrossEntropyLoss=0.0354
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9921
train/patch_acc=0.7500
train/subpatch_loss=0.1984
train/subpatch_acc=0.9375
train/location_loss=1.725
train/location_acc=0.2500
train/token_losses=15.63
throughput/total_tokens=4,571,136
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 13:59:57.062 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=280/600, eta=2 hours, 12 minutes]
train/CrossEntropyLoss=0.0543
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.6057
train/patch_acc=0.8125
train/subpatch_loss=0.2687
train/subpatch_acc=0.9375
train/location_loss=1.150
train/location_acc=0.5625
train/token_losses=10.52
throughput/total_tokens=4,587,520
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:00:21.605 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=281/600, eta=2 hours, 11 minutes]
train/CrossEntropyLoss=0.1121
train/ZLoss=0.0002
train/Accuracy=0.9756
train/patch_loss=0.6370
train/patch_acc=0.8125
train/subpatch_loss=0.1307
train/subpatch_acc=0.9375
train/location_loss=1.144
train/location_acc=0.5625
train/token_losses=10.19
throughput/total_tokens=4,603,904
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:00:46.163 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=282/600, eta=2 hours, 11 minutes]
train/CrossEntropyLoss=0.0452
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.069
train/patch_acc=0.6562
train/subpatch_loss=0.3253
train/subpatch_acc=0.9375
train/location_loss=1.467
train/location_acc=0.5000
train/token_losses=15.72
throughput/total_tokens=4,620,288
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:01:10.712 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=283/600, eta=2 hours, 10 minutes]
train/CrossEntropyLoss=0.0306
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9368
train/patch_acc=0.7812
train/subpatch_loss=0.1957
train/subpatch_acc=0.8750
train/location_loss=1.143
train/location_acc=0.6250
train/token_losses=12.85
throughput/total_tokens=4,636,672
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:01:35.247 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=284/600, eta=2 hours, 10 minutes]
train/CrossEntropyLoss=0.1712
train/ZLoss=0.0007
train/Accuracy=0.9573
train/patch_loss=1.285
train/patch_acc=0.6562
train/subpatch_loss=0.1342
train/subpatch_acc=1.0000
train/location_loss=1.081
train/location_acc=0.5625
train/token_losses=15.14
throughput/total_tokens=4,653,056
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:01:59.803 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=285/600, eta=2 hours, 9 minutes]
train/CrossEntropyLoss=0.0157
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.9416
train/patch_acc=0.7812
train/subpatch_loss=0.1288
train/subpatch_acc=1.0000
train/location_loss=1.321
train/location_acc=0.3750
train/token_losses=13.33
throughput/total_tokens=4,669,440
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 14:02:24.340 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=286/600, eta=2 hours, 9 minutes]
train/CrossEntropyLoss=0.0110
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.7429
train/patch_acc=0.7812
train/subpatch_loss=0.1221
train/subpatch_acc=1.0000
train/location_loss=1.037
train/location_acc=0.5000
train/token_losses=10.58
throughput/total_tokens=4,685,824
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 14:02:48.893 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=287/600, eta=2 hours, 9 minutes]
train/CrossEntropyLoss=0.0421
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.9527
train/patch_acc=0.7812
train/subpatch_loss=0.1770
train/subpatch_acc=1.0000
train/location_loss=1.216
train/location_acc=0.6875
train/token_losses=13.19
throughput/total_tokens=4,702,208
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 14:03:13.433 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=288/600, eta=2 hours, 8 minutes]
train/CrossEntropyLoss=0.1889
train/ZLoss=0.0006
train/Accuracy=0.9543
train/patch_loss=1.077
train/patch_acc=0.7500
train/subpatch_loss=0.1086
train/subpatch_acc=1.0000
train/location_loss=1.142
train/location_acc=0.6875
train/token_losses=13.61
throughput/total_tokens=4,718,592
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 14:03:37.950 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=289/600, eta=2 hours, 8 minutes]
train/CrossEntropyLoss=0.0659
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.4705
train/patch_acc=0.9375
train/subpatch_loss=0.1327
train/subpatch_acc=1.0000
train/location_loss=0.7408
train/location_acc=0.8750
train/token_losses=7.258
throughput/total_tokens=4,734,976
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 14:04:02.473 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=290/600, eta=2 hours, 7 minutes]
train/CrossEntropyLoss=0.0125
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.9708
train/patch_acc=0.7188
train/subpatch_loss=0.0878
train/subpatch_acc=1.0000
train/location_loss=1.168
train/location_acc=0.6250
train/token_losses=12.79
throughput/total_tokens=4,751,360
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:04:26.986 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=291/600, eta=2 hours, 7 minutes]
train/CrossEntropyLoss=0.0445
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9665
train/patch_acc=0.7500
train/subpatch_loss=0.2025
train/subpatch_acc=0.9375
train/location_loss=1.010
train/location_acc=0.7500
train/token_losses=12.58
throughput/total_tokens=4,767,744
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:04:51.527 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=292/600, eta=2 hours, 7 minutes]
train/CrossEntropyLoss=0.1540
train/ZLoss=0.0007
train/Accuracy=0.9665
train/patch_loss=1.121
train/patch_acc=0.7500
train/subpatch_loss=0.4523
train/subpatch_acc=0.6875
train/location_loss=1.193
train/location_acc=0.6250
train/token_losses=15.55
throughput/total_tokens=4,784,128
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:05:16.061 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=293/600, eta=2 hours, 6 minutes]
train/CrossEntropyLoss=0.0396
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.199
train/patch_acc=0.7500
train/subpatch_loss=0.1894
train/subpatch_acc=1.0000
train/location_loss=1.013
train/location_acc=0.6875
train/token_losses=14.41
throughput/total_tokens=4,800,512
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:05:40.596 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=294/600, eta=2 hours, 6 minutes]
train/CrossEntropyLoss=0.0357
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.132
train/patch_acc=0.7188
train/subpatch_loss=0.1881
train/subpatch_acc=1.0000
train/location_loss=1.282
train/location_acc=0.6250
train/token_losses=14.94
throughput/total_tokens=4,816,896
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:06:05.113 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=295/600, eta=2 hours, 5 minutes]
train/CrossEntropyLoss=0.0176
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8736
train/patch_acc=0.7812
train/subpatch_loss=0.2572
train/subpatch_acc=0.9375
train/location_loss=1.279
train/location_acc=0.4375
train/token_losses=13.13
throughput/total_tokens=4,833,280
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:06:29.637 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=296/600, eta=2 hours, 5 minutes]
train/CrossEntropyLoss=0.0205
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.8626
train/patch_acc=0.7188
train/subpatch_loss=0.0957
train/subpatch_acc=1.0000
train/location_loss=0.9775
train/location_acc=0.6875
train/token_losses=11.19
throughput/total_tokens=4,849,664
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:06:54.179 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=297/600, eta=2 hours, 5 minutes]
train/CrossEntropyLoss=0.0339
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7433
train/patch_acc=0.7812
train/subpatch_loss=0.2801
train/subpatch_acc=0.8750
train/location_loss=1.049
train/location_acc=0.6875
train/token_losses=11.26
throughput/total_tokens=4,866,048
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:07:18.715 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=298/600, eta=2 hours, 4 minutes]
train/CrossEntropyLoss=0.0331
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.294
train/patch_acc=0.6875
train/subpatch_loss=0.1719
train/subpatch_acc=1.0000
train/location_loss=1.285
train/location_acc=0.4375
train/token_losses=16.18
throughput/total_tokens=4,882,432
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:07:43.250 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=299/600, eta=2 hours, 4 minutes]
train/CrossEntropyLoss=0.0247
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.000
train/patch_acc=0.7500
train/subpatch_loss=0.1110
train/subpatch_acc=1.0000
train/location_loss=0.8691
train/location_acc=0.6875
train/token_losses=11.92
throughput/total_tokens=4,898,816
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:08:07.783 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=300/600, eta=2 hours, 3 minutes]
train/CrossEntropyLoss=0.0511
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.8662
train/patch_acc=0.8125
train/subpatch_loss=0.0519
train/subpatch_acc=1.0000
train/location_loss=1.091
train/location_acc=0.6875
train/token_losses=11.50
throughput/total_tokens=4,915,200
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:08:07.785 2012bc4f83ae:0 olmo.train.trainer:1415 INFO Saving checkpoint...
2026-07-27 14:08:09.925 2012bc4f83ae:0 olmo.train.trainer:1420 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step300
2026-07-27 14:08:35.097 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=301/600, eta=2 hours, 3 minutes]
train/CrossEntropyLoss=0.1504
train/ZLoss=0.0007
train/Accuracy=0.9543
train/patch_loss=0.6194
train/patch_acc=0.8125
train/subpatch_loss=0.1394
train/subpatch_acc=1.0000
train/location_loss=1.380
train/location_acc=0.5000
train/token_losses=11.03
throughput/total_tokens=4,931,584
throughput/device/tokens_per_second=651.0
throughput/device/batches_per_second=0.0397
2026-07-27 14:08:59.610 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=302/600, eta=2 hours, 3 minutes]
train/CrossEntropyLoss=0.0087
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.7440
train/patch_acc=0.8750
train/subpatch_loss=0.1336
train/subpatch_acc=1.0000
train/location_loss=1.096
train/location_acc=0.5000
train/token_losses=10.87
throughput/total_tokens=4,947,968
throughput/device/tokens_per_second=659.5
throughput/device/batches_per_second=0.0403
2026-07-27 14:09:24.135 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=303/600, eta=2 hours, 2 minutes]
train/CrossEntropyLoss=0.0422
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8593
train/patch_acc=0.7188
train/subpatch_loss=0.0955
train/subpatch_acc=0.9375
train/location_loss=0.8936
train/location_acc=0.8125
train/token_losses=10.83
throughput/total_tokens=4,964,352
throughput/device/tokens_per_second=662.4
throughput/device/batches_per_second=0.0404
2026-07-27 14:09:48.664 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=304/600, eta=2 hours, 2 minutes]
train/CrossEntropyLoss=0.0928
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.7629
train/patch_acc=0.8438
train/subpatch_loss=0.2155
train/subpatch_acc=0.9375
train/location_loss=0.9941
train/location_acc=0.7500
train/token_losses=10.94
throughput/total_tokens=4,980,736
throughput/device/tokens_per_second=663.7
throughput/device/batches_per_second=0.0405
2026-07-27 14:10:13.177 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=305/600, eta=2 hours, 1 minute]
train/CrossEntropyLoss=0.0449
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.9480
train/patch_acc=0.7188
train/subpatch_loss=0.1819
train/subpatch_acc=0.9375
train/location_loss=1.075
train/location_acc=0.5625
train/token_losses=12.61
throughput/total_tokens=4,997,120
throughput/device/tokens_per_second=664.7
throughput/device/batches_per_second=0.0406
2026-07-27 14:10:37.724 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=306/600, eta=2 hours, 1 minute]
train/CrossEntropyLoss=0.0392
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.013
train/patch_acc=0.7188
train/subpatch_loss=0.2021
train/subpatch_acc=0.8750
train/location_loss=1.253
train/location_acc=0.7500
train/token_losses=13.92
throughput/total_tokens=5,013,504
throughput/device/tokens_per_second=665.1
throughput/device/batches_per_second=0.0406
2026-07-27 14:11:02.262 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=307/600, eta=2 hours]
train/CrossEntropyLoss=0.0413
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8773
train/patch_acc=0.7812
train/subpatch_loss=0.3322
train/subpatch_acc=0.8750
train/location_loss=1.295
train/location_acc=0.6250
train/token_losses=13.53
throughput/total_tokens=5,029,888
throughput/device/tokens_per_second=665.5
throughput/device/batches_per_second=0.0406
2026-07-27 14:11:26.781 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=308/600, eta=2 hours]
train/CrossEntropyLoss=0.0496
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.8507
train/patch_acc=0.7812
train/subpatch_loss=0.1386
train/subpatch_acc=1.0000
train/location_loss=1.331
train/location_acc=0.5000
train/token_losses=12.68
throughput/total_tokens=5,046,272
throughput/device/tokens_per_second=665.8
throughput/device/batches_per_second=0.0406
2026-07-27 14:11:51.328 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=309/600, eta=2 hours]
train/CrossEntropyLoss=0.0294
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8443
train/patch_acc=0.6875
train/subpatch_loss=0.1552
train/subpatch_acc=1.0000
train/location_loss=1.149
train/location_acc=0.5000
train/token_losses=11.97
throughput/total_tokens=5,062,656
throughput/device/tokens_per_second=666.0
throughput/device/batches_per_second=0.0407
2026-07-27 14:12:15.850 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=310/600, eta=1 hour, 59 minutes]
train/CrossEntropyLoss=0.0439
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.191
train/patch_acc=0.6250
train/subpatch_loss=0.2023
train/subpatch_acc=0.9375
train/location_loss=1.482
train/location_acc=0.6250
train/token_losses=16.26
throughput/total_tokens=5,079,040
throughput/device/tokens_per_second=666.2
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:12:40.372 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=311/600, eta=1 hour, 59 minutes]
train/CrossEntropyLoss=0.0412
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.7569
train/patch_acc=0.7812
train/subpatch_loss=0.1726
train/subpatch_acc=0.9375
train/location_loss=1.239
train/location_acc=0.6250
train/token_losses=11.70
throughput/total_tokens=5,095,424
throughput/device/tokens_per_second=666.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:13:04.935 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=312/600, eta=1 hour, 58 minutes]
train/CrossEntropyLoss=0.0651
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9967
train/patch_acc=0.6875
train/subpatch_loss=0.2212
train/subpatch_acc=0.9375
train/location_loss=1.318
train/location_acc=0.3750
train/token_losses=14.13
throughput/total_tokens=5,111,808
throughput/device/tokens_per_second=666.5
throughput/device/batches_per_second=0.0407
2026-07-27 14:13:29.460 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=313/600, eta=1 hour, 58 minutes]
train/CrossEntropyLoss=0.0235
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.113
train/patch_acc=0.7188
train/subpatch_loss=0.1366
train/subpatch_acc=1.0000
train/location_loss=1.191
train/location_acc=0.6250
train/token_losses=14.22
throughput/total_tokens=5,128,192
throughput/device/tokens_per_second=666.6
throughput/device/batches_per_second=0.0407
2026-07-27 14:13:53.984 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=314/600, eta=1 hour, 58 minutes]
train/CrossEntropyLoss=0.1291
train/ZLoss=0.0006
train/Accuracy=0.9604
train/patch_loss=1.010
train/patch_acc=0.7500
train/subpatch_loss=0.1287
train/subpatch_acc=1.0000
train/location_loss=1.311
train/location_acc=0.5625
train/token_losses=13.84
throughput/total_tokens=5,144,576
throughput/device/tokens_per_second=666.7
throughput/device/batches_per_second=0.0407
2026-07-27 14:14:18.499 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=315/600, eta=1 hour, 57 minutes]
train/CrossEntropyLoss=0.0371
train/ZLoss=0.0005
train/Accuracy=0.9878
train/patch_loss=1.146
train/patch_acc=0.5938
train/subpatch_loss=0.0987
train/subpatch_acc=1.0000
train/location_loss=1.247
train/location_acc=0.5625
train/token_losses=14.55
throughput/total_tokens=5,160,960
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 14:14:43.043 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=316/600, eta=1 hour, 57 minutes]
train/CrossEntropyLoss=0.0219
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.017
train/patch_acc=0.7188
train/subpatch_loss=0.1595
train/subpatch_acc=0.9375
train/location_loss=1.458
train/location_acc=0.3750
train/token_losses=14.60
throughput/total_tokens=5,177,344
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 14:15:07.594 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=317/600, eta=1 hour, 56 minutes]
train/CrossEntropyLoss=0.0340
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7337
train/patch_acc=0.7812
train/subpatch_loss=0.2857
train/subpatch_acc=0.8750
train/location_loss=0.9099
train/location_acc=0.8125
train/token_losses=10.65
throughput/total_tokens=5,193,728
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 14:15:32.138 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=318/600, eta=1 hour, 56 minutes]
train/CrossEntropyLoss=0.0675
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9511
train/patch_acc=0.7500
train/subpatch_loss=0.2293
train/subpatch_acc=0.8750
train/location_loss=1.194
train/location_acc=0.5625
train/token_losses=13.30
throughput/total_tokens=5,210,112
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 14:15:56.675 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=319/600, eta=1 hour, 56 minutes]
train/CrossEntropyLoss=0.0099
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.083
train/patch_acc=0.6250
train/subpatch_loss=0.3430
train/subpatch_acc=0.8750
train/location_loss=1.618
train/location_acc=0.4375
train/token_losses=16.51
throughput/total_tokens=5,226,496
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 14:16:21.190 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=320/600, eta=1 hour, 55 minutes]
train/CrossEntropyLoss=0.0384
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.004
train/patch_acc=0.7188
train/subpatch_loss=0.2516
train/subpatch_acc=0.8750
train/location_loss=1.039
train/location_acc=0.7500
train/token_losses=13.20
throughput/total_tokens=5,242,880
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:16:45.692 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=321/600, eta=1 hour, 55 minutes]
train/CrossEntropyLoss=0.2115
train/ZLoss=0.0006
train/Accuracy=0.9451
train/patch_loss=0.8418
train/patch_acc=0.7188
train/subpatch_loss=0.2313
train/subpatch_acc=0.9375
train/location_loss=1.369
train/location_acc=0.3750
train/token_losses=13.13
throughput/total_tokens=5,259,264
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:17:10.202 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=322/600, eta=1 hour, 54 minutes]
train/CrossEntropyLoss=0.0184
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8040
train/patch_acc=0.7500
train/subpatch_loss=0.0753
train/subpatch_acc=1.0000
train/location_loss=0.9972
train/location_acc=0.8750
train/token_losses=10.72
throughput/total_tokens=5,275,648
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:17:34.714 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=323/600, eta=1 hour, 54 minutes]
train/CrossEntropyLoss=0.0421
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.8985
train/patch_acc=0.8125
train/subpatch_loss=0.1278
train/subpatch_acc=1.0000
train/location_loss=1.097
train/location_acc=0.6250
train/token_losses=12.09
throughput/total_tokens=5,292,032
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:17:59.224 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=324/600, eta=1 hour, 54 minutes]
train/CrossEntropyLoss=0.0032
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=0.9348
train/patch_acc=0.8125
train/subpatch_loss=0.0931
train/subpatch_acc=1.0000
train/location_loss=1.279
train/location_acc=0.6250
train/token_losses=12.97
throughput/total_tokens=5,308,416
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:18:23.737 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=325/600, eta=1 hour, 53 minutes]
train/CrossEntropyLoss=0.0425
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6744
train/patch_acc=0.7188
train/subpatch_loss=0.1788
train/subpatch_acc=1.0000
train/location_loss=1.152
train/location_acc=0.6250
train/token_losses=10.72
throughput/total_tokens=5,324,800
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:18:48.333 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=326/600, eta=1 hour, 53 minutes]
train/CrossEntropyLoss=0.0442
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.039
train/patch_acc=0.7188
train/subpatch_loss=0.0777
train/subpatch_acc=1.0000
train/location_loss=1.150
train/location_acc=0.6250
train/token_losses=13.22
throughput/total_tokens=5,341,184
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:19:12.845 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=327/600, eta=1 hour, 52 minutes]
train/CrossEntropyLoss=0.0584
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.6000
train/patch_acc=0.8750
train/subpatch_loss=0.1293
train/subpatch_acc=0.9375
train/location_loss=1.172
train/location_acc=0.5000
train/token_losses=10.01
throughput/total_tokens=5,357,568
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:19:37.348 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=328/600, eta=1 hour, 52 minutes]
train/CrossEntropyLoss=0.0263
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7558
train/patch_acc=0.7188
train/subpatch_loss=0.1653
train/subpatch_acc=1.0000
train/location_loss=1.130
train/location_acc=0.5000
train/token_losses=11.23
throughput/total_tokens=5,373,952
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:20:01.869 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=329/600, eta=1 hour, 51 minutes]
train/CrossEntropyLoss=0.1208
train/ZLoss=0.0008
train/Accuracy=0.9726
train/patch_loss=1.302
train/patch_acc=0.6562
train/subpatch_loss=0.1052
train/subpatch_acc=1.0000
train/location_loss=1.282
train/location_acc=0.5000
train/token_losses=15.96
throughput/total_tokens=5,390,336
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:20:26.413 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=330/600, eta=1 hour, 51 minutes]
train/CrossEntropyLoss=0.1679
train/ZLoss=0.0011
train/Accuracy=0.9543
train/patch_loss=1.168
train/patch_acc=0.6562
train/subpatch_loss=0.0813
train/subpatch_acc=1.0000
train/location_loss=1.107
train/location_acc=0.6875
train/token_losses=14.10
throughput/total_tokens=5,406,720
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:20:50.940 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=331/600, eta=1 hour, 51 minutes]
train/CrossEntropyLoss=0.0432
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.080
train/patch_acc=0.6562
train/subpatch_loss=0.1471
train/subpatch_acc=0.9375
train/location_loss=1.119
train/location_acc=0.5625
train/token_losses=13.70
throughput/total_tokens=5,423,104
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:21:15.464 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=332/600, eta=1 hour, 50 minutes]
train/CrossEntropyLoss=0.0202
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6699
train/patch_acc=0.8438
train/subpatch_loss=0.0482
train/subpatch_acc=1.0000
train/location_loss=1.089
train/location_acc=0.6250
train/token_losses=9.906
throughput/total_tokens=5,439,488
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:21:39.997 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=333/600, eta=1 hour, 50 minutes]
train/CrossEntropyLoss=0.0213
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8133
train/patch_acc=0.8125
train/subpatch_loss=0.1321
train/subpatch_acc=1.0000
train/location_loss=0.9430
train/location_acc=0.8125
train/token_losses=10.81
throughput/total_tokens=5,455,872
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:22:04.539 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=334/600, eta=1 hour, 49 minutes]
train/CrossEntropyLoss=0.0507
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.8311
train/patch_acc=0.7188
train/subpatch_loss=0.1533
train/subpatch_acc=0.9375
train/location_loss=0.9267
train/location_acc=0.7500
train/token_losses=10.97
throughput/total_tokens=5,472,256
throughput/device/tokens_per_second=668.0
throughput/device/batches_per_second=0.0408
2026-07-27 14:22:29.094 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=335/600, eta=1 hour, 49 minutes]
train/CrossEntropyLoss=0.0103
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9921
train/patch_acc=0.7188
train/subpatch_loss=0.1703
train/subpatch_acc=1.0000
train/location_loss=1.372
train/location_acc=0.4375
train/token_losses=14.10
throughput/total_tokens=5,488,640
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:22:53.634 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=336/600, eta=1 hour, 49 minutes]
train/CrossEntropyLoss=0.0315
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9215
train/patch_acc=0.7188
train/subpatch_loss=0.1838
train/subpatch_acc=1.0000
train/location_loss=1.044
train/location_acc=0.6250
train/token_losses=12.28
throughput/total_tokens=5,505,024
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:23:18.182 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=337/600, eta=1 hour, 48 minutes]
train/CrossEntropyLoss=0.0424
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.7049
train/patch_acc=0.8125
train/subpatch_loss=0.1373
train/subpatch_acc=0.9375
train/location_loss=1.119
train/location_acc=0.5625
train/token_losses=10.66
throughput/total_tokens=5,521,408
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:23:42.741 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=338/600, eta=1 hour, 48 minutes]
train/CrossEntropyLoss=0.0230
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.016
train/patch_acc=0.7500
train/subpatch_loss=0.2663
train/subpatch_acc=0.8750
train/location_loss=1.031
train/location_acc=0.6250
train/token_losses=13.32
throughput/total_tokens=5,537,792
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:24:07.295 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=339/600, eta=1 hour, 47 minutes]
train/CrossEntropyLoss=0.1185
train/ZLoss=0.0008
train/Accuracy=0.9756
train/patch_loss=0.8718
train/patch_acc=0.7500
train/subpatch_loss=0.2158
train/subpatch_acc=0.9375
train/location_loss=1.139
train/location_acc=0.5000
train/token_losses=12.39
throughput/total_tokens=5,554,176
throughput/device/tokens_per_second=667.9
throughput/device/batches_per_second=0.0408
2026-07-27 14:24:31.857 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=340/600, eta=1 hour, 47 minutes]
train/CrossEntropyLoss=0.0297
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=1.053
train/patch_acc=0.6875
train/subpatch_loss=0.1734
train/subpatch_acc=0.9375
train/location_loss=1.026
train/location_acc=0.7500
train/token_losses=13.22
throughput/total_tokens=5,570,560
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:24:56.406 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=341/600, eta=1 hour, 47 minutes]
train/CrossEntropyLoss=0.0302
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.7152
train/patch_acc=0.7812
train/subpatch_loss=0.1408
train/subpatch_acc=0.9375
train/location_loss=1.084
train/location_acc=0.8125
train/token_losses=10.62
throughput/total_tokens=5,586,944
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 14:25:20.962 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=342/600, eta=1 hour, 46 minutes]
train/CrossEntropyLoss=0.0382
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.8378
train/patch_acc=0.7812
train/subpatch_loss=0.1011
train/subpatch_acc=1.0000
train/location_loss=0.8535
train/location_acc=0.8125
train/token_losses=10.52
throughput/total_tokens=5,603,328
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 14:25:45.535 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=343/600, eta=1 hour, 46 minutes]
train/CrossEntropyLoss=0.0304
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.3953
train/patch_acc=0.9375
train/subpatch_loss=0.0955
train/subpatch_acc=1.0000
train/location_loss=0.9114
train/location_acc=0.6250
train/token_losses=7.190
throughput/total_tokens=5,619,712
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 14:26:10.075 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=344/600, eta=1 hour, 45 minutes]
train/CrossEntropyLoss=0.0957
train/ZLoss=0.0005
train/Accuracy=0.9848
train/patch_loss=1.036
train/patch_acc=0.7500
train/subpatch_loss=0.2708
train/subpatch_acc=0.8750
train/location_loss=1.162
train/location_acc=0.6875
train/token_losses=14.02
throughput/total_tokens=5,636,096
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 14:26:34.626 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=345/600, eta=1 hour, 45 minutes]
train/CrossEntropyLoss=0.0157
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8330
train/patch_acc=0.7188
train/subpatch_loss=0.1904
train/subpatch_acc=0.9375
train/location_loss=0.7190
train/location_acc=0.8125
train/token_losses=10.30
throughput/total_tokens=5,652,480
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 14:26:59.200 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=346/600, eta=1 hour, 45 minutes]
train/CrossEntropyLoss=0.0287
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.021
train/patch_acc=0.7812
train/subpatch_loss=0.1808
train/subpatch_acc=0.9375
train/location_loss=1.150
train/location_acc=0.5000
train/token_losses=13.49
throughput/total_tokens=5,668,864
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 14:27:23.772 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=347/600, eta=1 hour, 44 minutes]
train/CrossEntropyLoss=0.0432
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.046
train/patch_acc=0.7188
train/subpatch_loss=0.1321
train/subpatch_acc=0.9375
train/location_loss=1.291
train/location_acc=0.5625
train/token_losses=14.06
throughput/total_tokens=5,685,248
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 14:27:48.339 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=348/600, eta=1 hour, 44 minutes]
train/CrossEntropyLoss=0.0080
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6053
train/patch_acc=0.8125
train/subpatch_loss=0.0901
train/subpatch_acc=1.0000
train/location_loss=0.8419
train/location_acc=0.6875
train/token_losses=8.571
throughput/total_tokens=5,701,632
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:28:12.886 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=349/600, eta=1 hour, 43 minutes]
train/CrossEntropyLoss=0.0220
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7056
train/patch_acc=0.8125
train/subpatch_loss=0.1371
train/subpatch_acc=1.0000
train/location_loss=0.8736
train/location_acc=0.6875
train/token_losses=9.688
throughput/total_tokens=5,718,016
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:28:37.427 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=350/600, eta=1 hour, 43 minutes]
train/CrossEntropyLoss=0.0242
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.4931
train/patch_acc=0.9062
train/subpatch_loss=0.0503
train/subpatch_acc=1.0000
train/location_loss=0.9375
train/location_acc=0.5000
train/token_losses=7.897
throughput/total_tokens=5,734,400
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:29:01.997 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=351/600, eta=1 hour, 42 minutes]
train/CrossEntropyLoss=0.0692
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.8682
train/patch_acc=0.7500
train/subpatch_loss=0.0847
train/subpatch_acc=1.0000
train/location_loss=1.074
train/location_acc=0.6250
train/token_losses=11.58
throughput/total_tokens=5,750,784
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:29:26.550 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=352/600, eta=1 hour, 42 minutes]
train/CrossEntropyLoss=0.0858
train/ZLoss=0.0005
train/Accuracy=0.9726
train/patch_loss=0.9692
train/patch_acc=0.7188
train/subpatch_loss=0.2187
train/subpatch_acc=0.9375
train/location_loss=1.129
train/location_acc=0.6875
train/token_losses=13.14
throughput/total_tokens=5,767,168
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:29:51.085 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=353/600, eta=1 hour, 42 minutes]
train/CrossEntropyLoss=0.0006
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.7114
train/patch_acc=0.8438
train/subpatch_loss=0.1272
train/subpatch_acc=0.9375
train/location_loss=1.154
train/location_acc=0.5000
train/token_losses=10.81
throughput/total_tokens=5,783,552
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:30:15.709 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=354/600, eta=1 hour, 41 minutes]
train/CrossEntropyLoss=0.0413
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.8765
train/patch_acc=0.7812
train/subpatch_loss=0.1639
train/subpatch_acc=0.9375
train/location_loss=1.178
train/location_acc=0.6875
train/token_losses=12.38
throughput/total_tokens=5,799,936
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:30:40.283 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=355/600, eta=1 hour, 41 minutes]
train/CrossEntropyLoss=0.0175
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.4612
train/patch_acc=0.8750
train/subpatch_loss=0.0488
train/subpatch_acc=1.0000
train/location_loss=0.7984
train/location_acc=0.8750
train/token_losses=7.078
throughput/total_tokens=5,816,320
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:31:04.841 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=356/600, eta=1 hour, 40 minutes]
train/CrossEntropyLoss=0.0334
train/ZLoss=0.0006
train/Accuracy=0.9939
train/patch_loss=1.035
train/patch_acc=0.8125
train/subpatch_loss=0.0741
train/subpatch_acc=1.0000
train/location_loss=1.339
train/location_acc=0.5625
train/token_losses=13.93
throughput/total_tokens=5,832,704
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:31:29.386 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=357/600, eta=1 hour, 40 minutes]
train/CrossEntropyLoss=0.0587
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.7686
train/patch_acc=0.8125
train/subpatch_loss=0.2499
train/subpatch_acc=0.9375
train/location_loss=1.157
train/location_acc=0.6250
train/token_losses=11.78
throughput/total_tokens=5,849,088
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:31:53.920 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=358/600, eta=1 hour, 40 minutes]
train/CrossEntropyLoss=0.0291
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.8134
train/patch_acc=0.7500
train/subpatch_loss=0.2464
train/subpatch_acc=0.8125
train/location_loss=1.003
train/location_acc=0.6250
train/token_losses=11.50
throughput/total_tokens=5,865,472
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:32:18.456 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=359/600, eta=1 hour, 39 minutes]
train/CrossEntropyLoss=0.0082
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9491
train/patch_acc=0.8125
train/subpatch_loss=0.2010
train/subpatch_acc=1.0000
train/location_loss=1.122
train/location_acc=0.6250
train/token_losses=12.88
throughput/total_tokens=5,881,856
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:32:43.016 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=360/600, eta=1 hour, 39 minutes]
train/CrossEntropyLoss=0.0099
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9552
train/patch_acc=0.7188
train/subpatch_loss=0.0609
train/subpatch_acc=1.0000
train/location_loss=1.086
train/location_acc=0.7500
train/token_losses=12.23
throughput/total_tokens=5,898,240
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:33:07.582 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=361/600, eta=1 hour, 38 minutes]
train/CrossEntropyLoss=0.1003
train/ZLoss=0.0007
train/Accuracy=0.9756
train/patch_loss=0.8210
train/patch_acc=0.7500
train/subpatch_loss=0.1032
train/subpatch_acc=1.0000
train/location_loss=1.145
train/location_acc=0.5625
train/token_losses=11.56
throughput/total_tokens=5,914,624
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:33:32.135 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=362/600, eta=1 hour, 38 minutes]
train/CrossEntropyLoss=0.0785
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.8524
train/patch_acc=0.7812
train/subpatch_loss=0.1155
train/subpatch_acc=1.0000
train/location_loss=1.034
train/location_acc=0.5625
train/token_losses=11.42
throughput/total_tokens=5,931,008
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:33:56.676 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=363/600, eta=1 hour, 38 minutes]
train/CrossEntropyLoss=0.0799
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.9092
train/patch_acc=0.7188
train/subpatch_loss=0.1551
train/subpatch_acc=1.0000
train/location_loss=0.8875
train/location_acc=0.8125
train/token_losses=11.44
throughput/total_tokens=5,947,392
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:34:21.225 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=364/600, eta=1 hour, 37 minutes]
train/CrossEntropyLoss=0.0323
train/ZLoss=0.0011
train/Accuracy=0.9909
train/patch_loss=1.083
train/patch_acc=0.7188
train/subpatch_loss=0.1099
train/subpatch_acc=1.0000
train/location_loss=1.051
train/location_acc=0.5625
train/token_losses=13.31
throughput/total_tokens=5,963,776
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:34:45.780 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=365/600, eta=1 hour, 37 minutes]
train/CrossEntropyLoss=0.0045
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=0.6102
train/patch_acc=0.8750
train/subpatch_loss=0.0445
train/subpatch_acc=1.0000
train/location_loss=0.7487
train/location_acc=0.8125
train/token_losses=8.054
throughput/total_tokens=5,980,160
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:35:10.354 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=366/600, eta=1 hour, 36 minutes]
train/CrossEntropyLoss=0.0191
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8128
train/patch_acc=0.7812
train/subpatch_loss=0.1747
train/subpatch_acc=0.9375
train/location_loss=0.8191
train/location_acc=0.8125
train/token_losses=10.48
throughput/total_tokens=5,996,544
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:35:34.893 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=367/600, eta=1 hour, 36 minutes]
train/CrossEntropyLoss=0.2460
train/ZLoss=0.0006
train/Accuracy=0.9421
train/patch_loss=0.8669
train/patch_acc=0.7812
train/subpatch_loss=0.0899
train/subpatch_acc=1.0000
train/location_loss=1.272
train/location_acc=0.6250
train/token_losses=12.38
throughput/total_tokens=6,012,928
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:35:59.441 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=368/600, eta=1 hour, 35 minutes]
train/CrossEntropyLoss=0.0058
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=1.071
train/patch_acc=0.6250
train/subpatch_loss=0.2343
train/subpatch_acc=0.8750
train/location_loss=1.131
train/location_acc=0.5625
train/token_losses=14.03
throughput/total_tokens=6,029,312
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:36:23.990 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=369/600, eta=1 hour, 35 minutes]
train/CrossEntropyLoss=0.0079
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6570
train/patch_acc=0.7812
train/subpatch_loss=0.1164
train/subpatch_acc=1.0000
train/location_loss=0.8039
train/location_acc=0.7500
train/token_losses=8.937
throughput/total_tokens=6,045,696
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:36:48.530 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=370/600, eta=1 hour, 35 minutes]
train/CrossEntropyLoss=0.0521
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.6311
train/patch_acc=0.8125
train/subpatch_loss=0.0938
train/subpatch_acc=1.0000
train/location_loss=0.9289
train/location_acc=0.6875
train/token_losses=9.140
throughput/total_tokens=6,062,080
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:37:13.082 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=371/600, eta=1 hour, 34 minutes]
train/CrossEntropyLoss=0.0089
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=1.172
train/patch_acc=0.6250
train/subpatch_loss=0.0535
train/subpatch_acc=1.0000
train/location_loss=0.8906
train/location_acc=0.6875
train/token_losses=13.15
throughput/total_tokens=6,078,464
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:37:37.620 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=372/600, eta=1 hour, 34 minutes]
train/CrossEntropyLoss=0.0307
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=1.297
train/patch_acc=0.6250
train/subpatch_loss=0.1250
train/subpatch_acc=1.0000
train/location_loss=0.9459
train/location_acc=0.6875
train/token_losses=14.66
throughput/total_tokens=6,094,848
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:38:02.167 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=373/600, eta=1 hour, 33 minutes]
train/CrossEntropyLoss=0.0202
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=1.170
train/patch_acc=0.6875
train/subpatch_loss=0.1222
train/subpatch_acc=1.0000
train/location_loss=1.263
train/location_acc=0.6250
train/token_losses=14.90
throughput/total_tokens=6,111,232
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:38:26.731 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=374/600, eta=1 hour, 33 minutes]
train/CrossEntropyLoss=0.0156
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.3230
train/patch_acc=0.9688
train/subpatch_loss=0.0938
train/subpatch_acc=1.0000
train/location_loss=0.5464
train/location_acc=0.9375
train/token_losses=5.145
throughput/total_tokens=6,127,616
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:38:51.309 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=375/600, eta=1 hour, 33 minutes]
train/CrossEntropyLoss=0.0127
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.6190
train/patch_acc=0.8438
train/subpatch_loss=0.0396
train/subpatch_acc=1.0000
train/location_loss=1.168
train/location_acc=0.6250
train/token_losses=9.784
throughput/total_tokens=6,144,000
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:39:15.851 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=376/600, eta=1 hour, 32 minutes]
train/CrossEntropyLoss=0.0171
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=1.079
train/patch_acc=0.6562
train/subpatch_loss=0.1995
train/subpatch_acc=1.0000
train/location_loss=0.9650
train/location_acc=0.7500
train/token_losses=13.29
throughput/total_tokens=6,160,384
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:39:40.402 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=377/600, eta=1 hour, 32 minutes]
train/CrossEntropyLoss=0.0324
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.6221
train/patch_acc=0.8438
train/subpatch_loss=0.2021
train/subpatch_acc=0.9375
train/location_loss=0.7669
train/location_acc=0.8125
train/token_losses=8.852
throughput/total_tokens=6,176,768
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:40:04.952 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=378/600, eta=1 hour, 31 minutes]
train/CrossEntropyLoss=0.0394
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6855
train/patch_acc=0.8125
train/subpatch_loss=0.0569
train/subpatch_acc=1.0000
train/location_loss=1.085
train/location_acc=0.5625
train/token_losses=10.05
throughput/total_tokens=6,193,152
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:40:29.484 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=379/600, eta=1 hour, 31 minutes]
train/CrossEntropyLoss=0.0175
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9403
train/patch_acc=0.7500
train/subpatch_loss=0.1711
train/subpatch_acc=0.8750
train/location_loss=0.7769
train/location_acc=0.8750
train/token_losses=11.31
throughput/total_tokens=6,209,536
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:40:54.044 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=380/600, eta=1 hour, 31 minutes]
train/CrossEntropyLoss=0.0206
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.8908
train/patch_acc=0.8438
train/subpatch_loss=0.1397
train/subpatch_acc=1.0000
train/location_loss=1.211
train/location_acc=0.6250
train/token_losses=12.53
throughput/total_tokens=6,225,920
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:41:18.612 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=381/600, eta=1 hour, 30 minutes]
train/CrossEntropyLoss=0.0586
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.6052
train/patch_acc=0.8438
train/subpatch_loss=0.1185
train/subpatch_acc=1.0000
train/location_loss=0.7060
train/location_acc=0.8125
train/token_losses=8.140
throughput/total_tokens=6,242,304
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:41:43.157 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=382/600, eta=1 hour, 30 minutes]
train/CrossEntropyLoss=0.0495
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9669
train/patch_acc=0.7500
train/subpatch_loss=0.2232
train/subpatch_acc=0.8750
train/location_loss=0.8818
train/location_acc=0.6875
train/token_losses=12.15
throughput/total_tokens=6,258,688
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:42:07.728 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=383/600, eta=1 hour, 29 minutes]
train/CrossEntropyLoss=0.0271
train/ZLoss=0.0004
train/Accuracy=0.9878
train/patch_loss=0.8754
train/patch_acc=0.7500
train/subpatch_loss=0.1482
train/subpatch_acc=1.0000
train/location_loss=1.057
train/location_acc=0.6250
train/token_losses=11.82
throughput/total_tokens=6,275,072
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:42:32.284 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=384/600, eta=1 hour, 29 minutes]
train/CrossEntropyLoss=0.0421
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7511
train/patch_acc=0.8125
train/subpatch_loss=0.0557
train/subpatch_acc=1.0000
train/location_loss=1.036
train/location_acc=0.6875
train/token_losses=10.38
throughput/total_tokens=6,291,456
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:42:56.821 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=385/600, eta=1 hour, 29 minutes]
train/CrossEntropyLoss=0.0333
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.200
train/patch_acc=0.6250
train/subpatch_loss=0.0318
train/subpatch_acc=1.0000
train/location_loss=0.8447
train/location_acc=0.7500
train/token_losses=13.11
throughput/total_tokens=6,307,840
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:43:21.376 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=386/600, eta=1 hour, 28 minutes]
train/CrossEntropyLoss=0.0198
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.5353
train/patch_acc=0.8750
train/subpatch_loss=0.1601
train/subpatch_acc=0.9375
train/location_loss=0.7699
train/location_acc=0.8125
train/token_losses=8.002
throughput/total_tokens=6,324,224
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:43:45.903 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=387/600, eta=1 hour, 28 minutes]
train/CrossEntropyLoss=0.1494
train/ZLoss=0.0006
train/Accuracy=0.9665
train/patch_loss=0.9262
train/patch_acc=0.7188
train/subpatch_loss=0.1992
train/subpatch_acc=0.9375
train/location_loss=1.056
train/location_acc=0.6250
train/token_losses=12.43
throughput/total_tokens=6,340,608
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 14:44:10.478 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=388/600, eta=1 hour, 27 minutes]
train/CrossEntropyLoss=0.0312
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.9358
train/patch_acc=0.7188
train/subpatch_loss=0.0860
train/subpatch_acc=1.0000
train/location_loss=0.7661
train/location_acc=0.7500
train/token_losses=10.89
throughput/total_tokens=6,356,992
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:44:35.020 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=389/600, eta=1 hour, 27 minutes]
train/CrossEntropyLoss=0.0321
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7853
train/patch_acc=0.7812
train/subpatch_loss=0.1053
train/subpatch_acc=1.0000
train/location_loss=0.7358
train/location_acc=0.8125
train/token_losses=9.647
throughput/total_tokens=6,373,376
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:44:59.598 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=390/600, eta=1 hour, 26 minutes]
train/CrossEntropyLoss=0.0126
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.9140
train/patch_acc=0.7500
train/subpatch_loss=0.0871
train/subpatch_acc=1.0000
train/location_loss=0.9279
train/location_acc=0.6250
train/token_losses=11.37
throughput/total_tokens=6,389,760
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:45:24.157 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=391/600, eta=1 hour, 26 minutes]
train/CrossEntropyLoss=0.0403
train/ZLoss=0.0007
train/Accuracy=0.9878
train/patch_loss=0.9248
train/patch_acc=0.6562
train/subpatch_loss=0.1124
train/subpatch_acc=1.0000
train/location_loss=1.133
train/location_acc=0.5625
train/token_losses=12.38
throughput/total_tokens=6,406,144
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:45:48.723 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=392/600, eta=1 hour, 26 minutes]
train/CrossEntropyLoss=0.0047
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=0.8147
train/patch_acc=0.6875
train/subpatch_loss=0.0479
train/subpatch_acc=1.0000
train/location_loss=0.5358
train/location_acc=0.8125
train/token_losses=8.853
throughput/total_tokens=6,422,528
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:46:13.280 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=393/600, eta=1 hour, 25 minutes]
train/CrossEntropyLoss=0.0145
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.7660
train/patch_acc=0.8438
train/subpatch_loss=0.1249
train/subpatch_acc=1.0000
train/location_loss=1.378
train/location_acc=0.3750
train/token_losses=12.14
throughput/total_tokens=6,438,912
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:46:37.854 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=394/600, eta=1 hour, 25 minutes]
train/CrossEntropyLoss=0.0291
train/ZLoss=0.0003
train/Accuracy=0.9939
train/patch_loss=0.7523
train/patch_acc=0.7188
train/subpatch_loss=0.0809
train/subpatch_acc=1.0000
train/location_loss=1.061
train/location_acc=0.7500
train/token_losses=10.58
throughput/total_tokens=6,455,296
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:47:02.396 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=395/600, eta=1 hour, 24 minutes]
train/CrossEntropyLoss=0.0064
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.8084
train/patch_acc=0.8125
train/subpatch_loss=0.0668
train/subpatch_acc=1.0000
train/location_loss=1.005
train/location_acc=0.6250
train/token_losses=10.75
throughput/total_tokens=6,471,680
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:47:26.919 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=396/600, eta=1 hour, 24 minutes]
train/CrossEntropyLoss=0.0467
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7176
train/patch_acc=0.7188
train/subpatch_loss=0.1693
train/subpatch_acc=1.0000
train/location_loss=1.308
train/location_acc=0.5000
train/token_losses=11.65
throughput/total_tokens=6,488,064
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:47:51.473 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=397/600, eta=1 hour, 24 minutes]
train/CrossEntropyLoss=0.0231
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8129
train/patch_acc=0.7500
train/subpatch_loss=0.1048
train/subpatch_acc=1.0000
train/location_loss=1.216
train/location_acc=0.5000
train/token_losses=11.79
throughput/total_tokens=6,504,448
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:48:16.017 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=398/600, eta=1 hour, 23 minutes]
train/CrossEntropyLoss=0.0447
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.7317
train/patch_acc=0.7500
train/subpatch_loss=0.1439
train/subpatch_acc=1.0000
train/location_loss=1.335
train/location_acc=0.5000
train/token_losses=11.77
throughput/total_tokens=6,520,832
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:48:40.558 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=399/600, eta=1 hour, 23 minutes]
train/CrossEntropyLoss=0.0524
train/ZLoss=0.0002
train/Accuracy=0.9787
train/patch_loss=0.6636
train/patch_acc=0.7500
train/subpatch_loss=0.1996
train/subpatch_acc=0.8750
train/location_loss=1.300
train/location_acc=0.5625
train/token_losses=11.31
throughput/total_tokens=6,537,216
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 14:49:05.103 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=400/600, eta=1 hour, 22 minutes]
train/CrossEntropyLoss=0.0489
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=1.086
train/patch_acc=0.7188
train/subpatch_loss=0.0300
train/subpatch_acc=1.0000
train/location_loss=0.8495
train/location_acc=0.6875
train/token_losses=12.20
throughput/total_tokens=6,553,600
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:49:05.104 2012bc4f83ae:0 olmo.train.trainer:1415 INFO Saving checkpoint...
2026-07-27 14:49:12.852 2012bc4f83ae:0 olmo.train.trainer:1420 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step400
2026-07-27 14:49:38.494 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=401/600, eta=1 hour, 22 minutes]
train/CrossEntropyLoss=0.0069
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.7918
train/patch_acc=0.6875
train/subpatch_loss=0.2056
train/subpatch_acc=1.0000
train/location_loss=1.037
train/location_acc=0.6875
train/token_losses=11.31
throughput/total_tokens=6,569,984
throughput/device/tokens_per_second=639.1
throughput/device/batches_per_second=0.0390
2026-07-27 14:50:03.031 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=402/600, eta=1 hour, 22 minutes]
train/CrossEntropyLoss=0.0435
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7456
train/patch_acc=0.7812
train/subpatch_loss=0.0785
train/subpatch_acc=1.0000
train/location_loss=0.7108
train/location_acc=0.8125
train/token_losses=9.122
throughput/total_tokens=6,586,368
throughput/device/tokens_per_second=653.1
throughput/device/batches_per_second=0.0399
2026-07-27 14:50:27.582 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=403/600, eta=1 hour, 21 minutes]
train/CrossEntropyLoss=0.0216
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6839
train/patch_acc=0.8125
train/subpatch_loss=0.1257
train/subpatch_acc=1.0000
train/location_loss=0.9212
train/location_acc=0.6250
train/token_losses=9.658
throughput/total_tokens=6,602,752
throughput/device/tokens_per_second=657.8
throughput/device/batches_per_second=0.0401
2026-07-27 14:50:52.146 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=404/600, eta=1 hour, 21 minutes]
train/CrossEntropyLoss=0.0219
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.7496
train/patch_acc=0.8125
train/subpatch_loss=0.0437
train/subpatch_acc=1.0000
train/location_loss=0.6998
train/location_acc=0.8125
train/token_losses=8.971
throughput/total_tokens=6,619,136
throughput/device/tokens_per_second=660.1
throughput/device/batches_per_second=0.0403
2026-07-27 14:51:16.686 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=405/600, eta=1 hour, 20 minutes]
train/CrossEntropyLoss=0.0156
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9175
train/patch_acc=0.7188
train/subpatch_loss=0.0572
train/subpatch_acc=1.0000
train/location_loss=1.011
train/location_acc=0.6875
train/token_losses=11.61
throughput/total_tokens=6,635,520
throughput/device/tokens_per_second=661.6
throughput/device/batches_per_second=0.0404
2026-07-27 14:51:41.239 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=406/600, eta=1 hour, 20 minutes]
train/CrossEntropyLoss=0.0722
train/ZLoss=0.0007
train/Accuracy=0.9817
train/patch_loss=0.9934
train/patch_acc=0.7188
train/subpatch_loss=0.1907
train/subpatch_acc=0.8750
train/location_loss=0.7134
train/location_acc=0.8750
train/token_losses=11.56
throughput/total_tokens=6,651,904
throughput/device/tokens_per_second=662.5
throughput/device/batches_per_second=0.0404
2026-07-27 14:52:05.792 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=407/600, eta=1 hour, 20 minutes]
train/CrossEntropyLoss=0.0190
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9561
train/patch_acc=0.7188
train/subpatch_loss=0.0896
train/subpatch_acc=0.9375
train/location_loss=0.8081
train/location_acc=0.7500
train/token_losses=11.24
throughput/total_tokens=6,668,288
throughput/device/tokens_per_second=663.2
throughput/device/batches_per_second=0.0405
2026-07-27 14:52:30.334 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=408/600, eta=1 hour, 19 minutes]
train/CrossEntropyLoss=0.0848
train/ZLoss=0.0001
train/Accuracy=0.9787
train/patch_loss=0.6812
train/patch_acc=0.8125
train/subpatch_loss=0.1229
train/subpatch_acc=1.0000
train/location_loss=0.9424
train/location_acc=0.6250
train/token_losses=9.710
throughput/total_tokens=6,684,672
throughput/device/tokens_per_second=663.7
throughput/device/batches_per_second=0.0405
2026-07-27 14:52:54.896 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=409/600, eta=1 hour, 19 minutes]
train/CrossEntropyLoss=0.1862
train/ZLoss=0.0007
train/Accuracy=0.9482
train/patch_loss=1.079
train/patch_acc=0.7188
train/subpatch_loss=0.1028
train/subpatch_acc=1.0000
train/location_loss=0.9627
train/location_acc=0.6250
train/token_losses=12.89
throughput/total_tokens=6,701,056
throughput/device/tokens_per_second=664.1
throughput/device/batches_per_second=0.0405
2026-07-27 14:53:19.456 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=410/600, eta=1 hour, 18 minutes]
train/CrossEntropyLoss=0.0375
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8700
train/patch_acc=0.7188
train/subpatch_loss=0.1005
train/subpatch_acc=1.0000
train/location_loss=0.8973
train/location_acc=0.6875
train/token_losses=10.95
throughput/total_tokens=6,717,440
throughput/device/tokens_per_second=664.4
throughput/device/batches_per_second=0.0406
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:53:44.035 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=411/600, eta=1 hour, 18 minutes]
train/CrossEntropyLoss=0.0341
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9256
train/patch_acc=0.6562
train/subpatch_loss=0.1408
train/subpatch_acc=0.9375
train/location_loss=1.077
train/location_acc=0.5625
train/token_losses=12.28
throughput/total_tokens=6,733,824
throughput/device/tokens_per_second=664.6
throughput/device/batches_per_second=0.0406
2026-07-27 14:54:08.622 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=412/600, eta=1 hour, 18 minutes]
train/CrossEntropyLoss=0.0505
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.7329
train/patch_acc=0.7188
train/subpatch_loss=0.0981
train/subpatch_acc=1.0000
train/location_loss=0.8506
train/location_acc=0.7500
train/token_losses=9.658
throughput/total_tokens=6,750,208
throughput/device/tokens_per_second=664.7
throughput/device/batches_per_second=0.0406
2026-07-27 14:54:33.190 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=413/600, eta=1 hour, 17 minutes]
train/CrossEntropyLoss=0.0158
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6177
train/patch_acc=0.7812
train/subpatch_loss=0.0502
train/subpatch_acc=1.0000
train/location_loss=0.8859
train/location_acc=0.6875
train/token_losses=8.686
throughput/total_tokens=6,766,592
throughput/device/tokens_per_second=664.9
throughput/device/batches_per_second=0.0406
2026-07-27 14:54:57.723 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=414/600, eta=1 hour, 17 minutes]
train/CrossEntropyLoss=0.2676
train/ZLoss=0.0006
train/Accuracy=0.9390
train/patch_loss=1.181
train/patch_acc=0.6875
train/subpatch_loss=0.2297
train/subpatch_acc=0.8750
train/location_loss=1.111
train/location_acc=0.5625
train/token_losses=14.81
throughput/total_tokens=6,782,976
throughput/device/tokens_per_second=665.1
throughput/device/batches_per_second=0.0406
2026-07-27 14:55:22.287 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=415/600, eta=1 hour, 16 minutes]
train/CrossEntropyLoss=0.0099
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9563
train/patch_acc=0.7188
train/subpatch_loss=0.0995
train/subpatch_acc=1.0000
train/location_loss=0.7696
train/location_acc=0.7500
train/token_losses=11.13
throughput/total_tokens=6,799,360
throughput/device/tokens_per_second=665.2
throughput/device/batches_per_second=0.0406
2026-07-27 14:55:46.845 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=416/600, eta=1 hour, 16 minutes]
train/CrossEntropyLoss=0.0063
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6717
train/patch_acc=0.8125
train/subpatch_loss=0.0924
train/subpatch_acc=1.0000
train/location_loss=0.7291
train/location_acc=0.8750
train/token_losses=8.660
throughput/total_tokens=6,815,744
throughput/device/tokens_per_second=665.4
throughput/device/batches_per_second=0.0406
2026-07-27 14:56:11.410 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=417/600, eta=1 hour, 15 minutes]
train/CrossEntropyLoss=0.0382
train/ZLoss=0.0006
train/Accuracy=0.9909
train/patch_loss=0.6628
train/patch_acc=0.8438
train/subpatch_loss=0.2013
train/subpatch_acc=0.9375
train/location_loss=1.129
train/location_acc=0.6250
train/token_losses=10.62
throughput/total_tokens=6,832,128
throughput/device/tokens_per_second=665.5
throughput/device/batches_per_second=0.0406
2026-07-27 14:56:35.974 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=418/600, eta=1 hour, 15 minutes]
train/CrossEntropyLoss=0.0131
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9146
train/patch_acc=0.7500
train/subpatch_loss=0.0661
train/subpatch_acc=1.0000
train/location_loss=0.9992
train/location_acc=0.6250
train/token_losses=11.58
throughput/total_tokens=6,848,512
throughput/device/tokens_per_second=665.5
throughput/device/batches_per_second=0.0406
2026-07-27 14:57:00.543 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=419/600, eta=1 hour, 15 minutes]
train/CrossEntropyLoss=0.0223
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7061
train/patch_acc=0.8125
train/subpatch_loss=0.1429
train/subpatch_acc=0.9375
train/location_loss=0.8645
train/location_acc=0.7500
train/token_losses=9.678
throughput/total_tokens=6,864,896
throughput/device/tokens_per_second=665.6
throughput/device/batches_per_second=0.0406
2026-07-27 14:57:25.103 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=420/600, eta=1 hour, 14 minutes]
train/CrossEntropyLoss=0.0543
train/ZLoss=0.0006
train/Accuracy=0.9817
train/patch_loss=0.8363
train/patch_acc=0.7812
train/subpatch_loss=0.0411
train/subpatch_acc=1.0000
train/location_loss=0.7516
train/location_acc=0.8750
train/token_losses=9.861
throughput/total_tokens=6,881,280
throughput/device/tokens_per_second=665.7
throughput/device/batches_per_second=0.0406
System/Peak GPU Memory (MB)=41,776
2026-07-27 14:57:49.657 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=421/600, eta=1 hour, 14 minutes]
train/CrossEntropyLoss=0.0480
train/ZLoss=0.0008
train/Accuracy=0.9848
train/patch_loss=0.8301
train/patch_acc=0.7812
train/subpatch_loss=0.0961
train/subpatch_acc=0.9375
train/location_loss=0.9265
train/location_acc=0.7500
train/token_losses=10.73
throughput/total_tokens=6,897,664
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:58:14.191 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=422/600, eta=1 hour, 13 minutes]
train/CrossEntropyLoss=0.0568
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.5608
train/patch_acc=0.7812
train/subpatch_loss=0.1055
train/subpatch_acc=1.0000
train/location_loss=0.9370
train/location_acc=0.6875
train/token_losses=8.657
throughput/total_tokens=6,914,048
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 14:58:38.759 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=423/600, eta=1 hour, 13 minutes]
train/CrossEntropyLoss=0.0256
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7156
train/patch_acc=0.7812
train/subpatch_loss=0.3009
train/subpatch_acc=0.8750
train/location_loss=1.002
train/location_acc=0.6875
train/token_losses=10.94
throughput/total_tokens=6,930,432
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:59:03.348 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=424/600, eta=1 hour, 13 minutes]
train/CrossEntropyLoss=0.1453
train/ZLoss=0.0008
train/Accuracy=0.9573
train/patch_loss=0.8683
train/patch_acc=0.7812
train/subpatch_loss=0.1113
train/subpatch_acc=1.0000
train/location_loss=0.8833
train/location_acc=0.6875
train/token_losses=10.93
throughput/total_tokens=6,946,816
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 14:59:27.943 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=425/600, eta=1 hour, 12 minutes]
train/CrossEntropyLoss=0.0115
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.7304
train/patch_acc=0.8125
train/subpatch_loss=0.0412
train/subpatch_acc=1.0000
train/location_loss=0.9847
train/location_acc=0.6250
train/token_losses=9.947
throughput/total_tokens=6,963,200
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 14:59:52.519 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=426/600, eta=1 hour, 12 minutes]
train/CrossEntropyLoss=0.0019
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.7911
train/patch_acc=0.7188
train/subpatch_loss=0.0665
train/subpatch_acc=1.0000
train/location_loss=0.7590
train/location_acc=0.6875
train/token_losses=9.631
throughput/total_tokens=6,979,584
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:00:17.090 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=427/600, eta=1 hour, 11 minutes]
train/CrossEntropyLoss=0.0374
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6778
train/patch_acc=0.7812
train/subpatch_loss=0.1775
train/subpatch_acc=0.8750
train/location_loss=1.057
train/location_acc=0.6250
train/token_losses=10.36
throughput/total_tokens=6,995,968
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:00:41.718 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=428/600, eta=1 hour, 11 minutes]
train/CrossEntropyLoss=0.0578
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9269
train/patch_acc=0.7500
train/subpatch_loss=0.1018
train/subpatch_acc=1.0000
train/location_loss=0.8423
train/location_acc=0.8125
train/token_losses=11.19
throughput/total_tokens=7,012,352
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:01:06.294 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=429/600, eta=1 hour, 11 minutes]
train/CrossEntropyLoss=0.0347
train/ZLoss=0.0005
train/Accuracy=0.9878
train/patch_loss=0.6049
train/patch_acc=0.8125
train/subpatch_loss=0.0722
train/subpatch_acc=1.0000
train/location_loss=0.7160
train/location_acc=0.9375
train/token_losses=7.992
throughput/total_tokens=7,028,736
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 15:01:30.882 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=430/600, eta=1 hour, 10 minutes]
train/CrossEntropyLoss=0.0219
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.6645
train/patch_acc=0.7812
train/subpatch_loss=0.0737
train/subpatch_acc=1.0000
train/location_loss=0.8201
train/location_acc=0.8750
train/token_losses=8.891
throughput/total_tokens=7,045,120
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:01:55.444 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=431/600, eta=1 hour, 10 minutes]
train/CrossEntropyLoss=0.0564
train/ZLoss=0.0005
train/Accuracy=0.9878
train/patch_loss=0.7289
train/patch_acc=0.8125
train/subpatch_loss=0.1728
train/subpatch_acc=0.9375
train/location_loss=1.459
train/location_acc=0.3750
train/token_losses=12.36
throughput/total_tokens=7,061,504
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 15:02:19.989 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=432/600, eta=1 hour, 9 minutes]
train/CrossEntropyLoss=0.0257
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.000
train/patch_acc=0.7500
train/subpatch_loss=0.1077
train/subpatch_acc=1.0000
train/location_loss=1.053
train/location_acc=0.6875
train/token_losses=12.65
throughput/total_tokens=7,077,888
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:02:44.536 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=433/600, eta=1 hour, 9 minutes]
train/CrossEntropyLoss=0.0047
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8599
train/patch_acc=0.6875
train/subpatch_loss=0.1220
train/subpatch_acc=0.9375
train/location_loss=1.061
train/location_acc=0.5625
train/token_losses=11.61
throughput/total_tokens=7,094,272
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:03:09.126 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=434/600, eta=1 hour, 9 minutes]
train/CrossEntropyLoss=0.0697
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.6423
train/patch_acc=0.7812
train/subpatch_loss=0.1098
train/subpatch_acc=0.9375
train/location_loss=1.083
train/location_acc=0.5000
train/token_losses=9.910
throughput/total_tokens=7,110,656
throughput/device/tokens_per_second=666.8
throughput/device/batches_per_second=0.0407
2026-07-27 15:03:33.663 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=435/600, eta=1 hour, 8 minutes]
train/CrossEntropyLoss=0.0068
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.9622
train/patch_acc=0.7188
train/subpatch_loss=0.2504
train/subpatch_acc=0.9375
train/location_loss=1.088
train/location_acc=0.5625
train/token_losses=13.05
throughput/total_tokens=7,127,040
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:03:58.198 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=436/600, eta=1 hour, 8 minutes]
train/CrossEntropyLoss=0.0082
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8694
train/patch_acc=0.8125
train/subpatch_loss=0.0515
train/subpatch_acc=1.0000
train/location_loss=0.9557
train/location_acc=0.6250
train/token_losses=10.98
throughput/total_tokens=7,143,424
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:04:22.740 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=437/600, eta=1 hour, 7 minutes]
train/CrossEntropyLoss=0.0663
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.5832
train/patch_acc=0.7500
train/subpatch_loss=0.1038
train/subpatch_acc=1.0000
train/location_loss=0.8387
train/location_acc=0.6875
train/token_losses=8.436
throughput/total_tokens=7,159,808
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:04:47.289 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=438/600, eta=1 hour, 7 minutes]
train/CrossEntropyLoss=0.0127
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.9841
train/patch_acc=0.6875
train/subpatch_loss=0.1270
train/subpatch_acc=1.0000
train/location_loss=1.049
train/location_acc=0.6250
train/token_losses=12.58
throughput/total_tokens=7,176,192
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:05:11.835 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=439/600, eta=1 hour, 6 minutes]
train/CrossEntropyLoss=0.0056
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6356
train/patch_acc=0.9062
train/subpatch_loss=0.0818
train/subpatch_acc=1.0000
train/location_loss=0.7644
train/location_acc=0.8750
train/token_losses=8.470
throughput/total_tokens=7,192,576
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:05:36.400 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=440/600, eta=1 hour, 6 minutes]
train/CrossEntropyLoss=0.3321
train/ZLoss=0.0010
train/Accuracy=0.9085
train/patch_loss=0.8960
train/patch_acc=0.6875
train/subpatch_loss=0.1883
train/subpatch_acc=0.9375
train/location_loss=1.058
train/location_acc=0.5625
train/token_losses=12.15
throughput/total_tokens=7,208,960
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:06:00.959 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=441/600, eta=1 hour, 6 minutes]
train/CrossEntropyLoss=0.0229
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.6184
train/patch_acc=0.8750
train/subpatch_loss=0.0747
train/subpatch_acc=1.0000
train/location_loss=1.031
train/location_acc=0.6250
train/token_losses=9.369
throughput/total_tokens=7,225,344
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:06:25.506 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=442/600, eta=1 hour, 5 minutes]
train/CrossEntropyLoss=0.0602
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.6613
train/patch_acc=0.8750
train/subpatch_loss=0.2190
train/subpatch_acc=0.9375
train/location_loss=0.9787
train/location_acc=0.8125
train/token_losses=10.08
throughput/total_tokens=7,241,728
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:06:50.048 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=443/600, eta=1 hour, 5 minutes]
train/CrossEntropyLoss=0.0790
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.8992
train/patch_acc=0.6875
train/subpatch_loss=0.0985
train/subpatch_acc=1.0000
train/location_loss=0.7601
train/location_acc=0.7500
train/token_losses=10.63
throughput/total_tokens=7,258,112
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:07:14.579 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=444/600, eta=1 hour, 4 minutes]
train/CrossEntropyLoss=0.0375
train/ZLoss=0.0001
train/Accuracy=0.9787
train/patch_loss=0.8333
train/patch_acc=0.7188
train/subpatch_loss=0.0927
train/subpatch_acc=1.0000
train/location_loss=1.112
train/location_acc=0.5625
train/token_losses=11.48
throughput/total_tokens=7,274,496
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:07:39.157 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=445/600, eta=1 hour, 4 minutes]
train/CrossEntropyLoss=0.1293
train/ZLoss=0.0006
train/Accuracy=0.9695
train/patch_loss=0.8254
train/patch_acc=0.8125
train/subpatch_loss=0.2270
train/subpatch_acc=0.9375
train/location_loss=1.336
train/location_acc=0.5625
train/token_losses=12.85
throughput/total_tokens=7,290,880
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:08:03.695 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=446/600, eta=1 hour, 4 minutes]
train/CrossEntropyLoss=0.0347
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.4632
train/patch_acc=0.8750
train/subpatch_loss=0.0674
train/subpatch_acc=1.0000
train/location_loss=0.5818
train/location_acc=0.9375
train/token_losses=6.303
throughput/total_tokens=7,307,264
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:08:28.237 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=447/600, eta=1 hour, 3 minutes]
train/CrossEntropyLoss=0.0176
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.7406
train/patch_acc=0.7812
train/subpatch_loss=0.1186
train/subpatch_acc=0.9375
train/location_loss=0.8447
train/location_acc=0.8125
train/token_losses=9.778
throughput/total_tokens=7,323,648
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:08:52.769 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=448/600, eta=1 hour, 3 minutes]
train/CrossEntropyLoss=0.0294
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.6940
train/patch_acc=0.7812
train/subpatch_loss=0.1023
train/subpatch_acc=1.0000
train/location_loss=0.9211
train/location_acc=0.6250
train/token_losses=9.646
throughput/total_tokens=7,340,032
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:09:17.300 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=449/600, eta=1 hour, 2 minutes]
train/CrossEntropyLoss=0.0614
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.5965
train/patch_acc=0.8125
train/subpatch_loss=0.1180
train/subpatch_acc=1.0000
train/location_loss=1.155
train/location_acc=0.4375
train/token_losses=9.865
throughput/total_tokens=7,356,416
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:09:41.860 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=450/600, eta=1 hour, 2 minutes]
train/CrossEntropyLoss=0.0306
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.7588
train/patch_acc=0.7812
train/subpatch_loss=0.1281
train/subpatch_acc=1.0000
train/location_loss=0.9021
train/location_acc=0.7500
train/token_losses=10.19
throughput/total_tokens=7,372,800
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:10:06.433 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=451/600, eta=1 hour, 2 minutes]
train/CrossEntropyLoss=0.0345
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.8339
train/patch_acc=0.6875
train/subpatch_loss=0.1496
train/subpatch_acc=0.9375
train/location_loss=1.033
train/location_acc=0.8125
train/token_losses=11.40
throughput/total_tokens=7,389,184
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:10:30.990 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=452/600, eta=1 hour, 1 minute]
train/CrossEntropyLoss=0.0242
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.8769
train/patch_acc=0.7500
train/subpatch_loss=0.0576
train/subpatch_acc=1.0000
train/location_loss=0.9490
train/location_acc=0.6875
train/token_losses=11.04
throughput/total_tokens=7,405,568
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:10:55.542 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=453/600, eta=1 hour, 1 minute]
train/CrossEntropyLoss=0.0409
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7026
train/patch_acc=0.8125
train/subpatch_loss=0.1432
train/subpatch_acc=0.9375
train/location_loss=0.8377
train/location_acc=0.6875
train/token_losses=9.545
throughput/total_tokens=7,421,952
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:11:20.085 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=454/600, eta=1 hour]
train/CrossEntropyLoss=0.0327
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7268
train/patch_acc=0.7500
train/subpatch_loss=0.0772
train/subpatch_acc=1.0000
train/location_loss=0.8477
train/location_acc=0.8125
train/token_losses=9.514
throughput/total_tokens=7,438,336
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:11:44.623 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=455/600, eta=1 hour]
train/CrossEntropyLoss=0.0580
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.8193
train/patch_acc=0.7812
train/subpatch_loss=0.1486
train/subpatch_acc=0.9375
train/location_loss=0.7605
train/location_acc=0.6875
train/token_losses=10.19
throughput/total_tokens=7,454,720
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:12:09.187 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=456/600, eta=1 hour]
train/CrossEntropyLoss=0.0027
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.9614
train/patch_acc=0.7812
train/subpatch_loss=0.0983
train/subpatch_acc=1.0000
train/location_loss=0.8565
train/location_acc=0.7500
train/token_losses=11.51
throughput/total_tokens=7,471,104
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:12:33.788 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=457/600, eta=59 minutes]
train/CrossEntropyLoss=0.0166
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=1.001
train/patch_acc=0.6562
train/subpatch_loss=0.1108
train/subpatch_acc=1.0000
train/location_loss=1.045
train/location_acc=0.6250
train/token_losses=12.64
throughput/total_tokens=7,487,488
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:12:58.361 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=458/600, eta=59 minutes]
train/CrossEntropyLoss=0.0049
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.7869
train/patch_acc=0.6875
train/subpatch_loss=0.0946
train/subpatch_acc=1.0000
train/location_loss=0.9444
train/location_acc=0.6250
train/token_losses=10.45
throughput/total_tokens=7,503,872
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:13:22.926 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=459/600, eta=58 minutes]
train/CrossEntropyLoss=0.0038
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.4307
train/patch_acc=0.8750
train/subpatch_loss=0.0656
train/subpatch_acc=1.0000
train/location_loss=0.7051
train/location_acc=0.8125
train/token_losses=6.528
throughput/total_tokens=7,520,256
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:13:47.489 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=460/600, eta=58 minutes]
train/CrossEntropyLoss=0.1728
train/ZLoss=0.0007
train/Accuracy=0.9604
train/patch_loss=0.9205
train/patch_acc=0.7500
train/subpatch_loss=0.1514
train/subpatch_acc=0.9375
train/location_loss=1.077
train/location_acc=0.7500
train/token_losses=12.28
throughput/total_tokens=7,536,640
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:14:12.049 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=461/600, eta=57 minutes]
train/CrossEntropyLoss=0.0371
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.038
train/patch_acc=0.6250
train/subpatch_loss=0.1419
train/subpatch_acc=0.9375
train/location_loss=0.9045
train/location_acc=0.6250
train/token_losses=12.49
throughput/total_tokens=7,553,024
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:14:36.598 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=462/600, eta=57 minutes]
train/CrossEntropyLoss=0.0276
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8621
train/patch_acc=0.7500
train/subpatch_loss=0.1333
train/subpatch_acc=0.9375
train/location_loss=0.9416
train/location_acc=0.6250
train/token_losses=11.20
throughput/total_tokens=7,569,408
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:15:01.158 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=463/600, eta=57 minutes]
train/CrossEntropyLoss=0.0478
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.7947
train/patch_acc=0.7812
train/subpatch_loss=0.0904
train/subpatch_acc=1.0000
train/location_loss=0.8288
train/location_acc=0.7500
train/token_losses=10.03
throughput/total_tokens=7,585,792
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:15:25.736 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=464/600, eta=56 minutes]
train/CrossEntropyLoss=0.0226
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.8882
train/patch_acc=0.7500
train/subpatch_loss=0.1084
train/subpatch_acc=1.0000
train/location_loss=0.8556
train/location_acc=0.8125
train/token_losses=10.96
throughput/total_tokens=7,602,176
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:15:50.291 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=465/600, eta=56 minutes]
train/CrossEntropyLoss=0.0402
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6907
train/patch_acc=0.7812
train/subpatch_loss=0.0962
train/subpatch_acc=1.0000
train/location_loss=0.8174
train/location_acc=0.7500
train/token_losses=9.180
throughput/total_tokens=7,618,560
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:16:14.854 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=466/600, eta=55 minutes]
train/CrossEntropyLoss=0.1558
train/ZLoss=0.0007
train/Accuracy=0.9604
train/patch_loss=0.7580
train/patch_acc=0.7188
train/subpatch_loss=0.1060
train/subpatch_acc=1.0000
train/location_loss=0.9698
train/location_acc=0.6875
train/token_losses=10.37
throughput/total_tokens=7,634,944
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:16:39.415 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=467/600, eta=55 minutes]
train/CrossEntropyLoss=0.0583
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.5193
train/patch_acc=0.8750
train/subpatch_loss=0.0745
train/subpatch_acc=1.0000
train/location_loss=0.9777
train/location_acc=0.6875
train/token_losses=8.363
throughput/total_tokens=7,651,328
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:17:03.973 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=468/600, eta=55 minutes]
train/CrossEntropyLoss=0.0461
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.9686
train/patch_acc=0.7188
train/subpatch_loss=0.0884
train/subpatch_acc=1.0000
train/location_loss=0.7868
train/location_acc=0.7500
train/token_losses=11.25
throughput/total_tokens=7,667,712
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:17:28.540 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=469/600, eta=54 minutes]
train/CrossEntropyLoss=0.0054
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.5542
train/patch_acc=0.8750
train/subpatch_loss=0.0796
train/subpatch_acc=1.0000
train/location_loss=0.5957
train/location_acc=0.8750
train/token_losses=7.134
throughput/total_tokens=7,684,096
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:17:53.090 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=470/600, eta=54 minutes]
train/CrossEntropyLoss=0.0435
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.7890
train/patch_acc=0.8125
train/subpatch_loss=0.1305
train/subpatch_acc=1.0000
train/location_loss=1.140
train/location_acc=0.5000
train/token_losses=11.39
throughput/total_tokens=7,700,480
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:18:17.642 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=471/600, eta=53 minutes]
train/CrossEntropyLoss=0.0325
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.5993
train/patch_acc=0.9375
train/subpatch_loss=0.0540
train/subpatch_acc=1.0000
train/location_loss=1.203
train/location_acc=0.6250
train/token_losses=9.821
throughput/total_tokens=7,716,864
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:18:42.219 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=472/600, eta=53 minutes]
train/CrossEntropyLoss=0.0212
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7179
train/patch_acc=0.8125
train/subpatch_loss=0.0299
train/subpatch_acc=1.0000
train/location_loss=0.8080
train/location_acc=0.6875
train/token_losses=9.095
throughput/total_tokens=7,733,248
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:19:06.792 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=473/600, eta=53 minutes]
train/CrossEntropyLoss=0.0136
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=1.002
train/patch_acc=0.7188
train/subpatch_loss=0.0897
train/subpatch_acc=1.0000
train/location_loss=0.8574
train/location_acc=0.6250
train/token_losses=11.80
throughput/total_tokens=7,749,632
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:19:31.365 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=474/600, eta=52 minutes]
train/CrossEntropyLoss=0.0164
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.7197
train/patch_acc=0.7500
train/subpatch_loss=0.0503
train/subpatch_acc=1.0000
train/location_loss=0.9600
train/location_acc=0.6250
train/token_losses=9.799
throughput/total_tokens=7,766,016
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:19:55.951 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=475/600, eta=52 minutes]
train/CrossEntropyLoss=0.0589
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7920
train/patch_acc=0.8438
train/subpatch_loss=0.0775
train/subpatch_acc=1.0000
train/location_loss=0.8149
train/location_acc=0.7500
train/token_losses=9.906
throughput/total_tokens=7,782,400
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:20:20.534 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=476/600, eta=51 minutes]
train/CrossEntropyLoss=0.0092
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6888
train/patch_acc=0.7812
train/subpatch_loss=0.1061
train/subpatch_acc=1.0000
train/location_loss=0.6831
train/location_acc=0.8750
train/token_losses=8.667
throughput/total_tokens=7,798,784
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:20:45.099 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=477/600, eta=51 minutes]
train/CrossEntropyLoss=0.0660
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.5519
train/patch_acc=0.8438
train/subpatch_loss=0.1082
train/subpatch_acc=1.0000
train/location_loss=0.6813
train/location_acc=0.8750
train/token_losses=7.573
throughput/total_tokens=7,815,168
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:21:09.653 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=478/600, eta=50 minutes]
train/CrossEntropyLoss=0.0583
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.6171
train/patch_acc=0.8750
train/subpatch_loss=0.1035
train/subpatch_acc=1.0000
train/location_loss=0.9645
train/location_acc=0.7500
train/token_losses=9.209
throughput/total_tokens=7,831,552
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:21:34.222 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=479/600, eta=50 minutes]
train/CrossEntropyLoss=0.1175
train/ZLoss=0.0008
train/Accuracy=0.9604
train/patch_loss=0.6898
train/patch_acc=0.7812
train/subpatch_loss=0.0759
train/subpatch_acc=0.9375
train/location_loss=0.9610
train/location_acc=0.7500
train/token_losses=9.666
throughput/total_tokens=7,847,936
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:21:58.791 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=480/600, eta=50 minutes]
train/CrossEntropyLoss=0.0368
train/ZLoss=0.0006
train/Accuracy=0.9909
train/patch_loss=0.6523
train/patch_acc=0.8438
train/subpatch_loss=0.0869
train/subpatch_acc=1.0000
train/location_loss=0.9042
train/location_acc=0.7500
train/token_losses=9.183
throughput/total_tokens=7,864,320
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:22:23.354 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=481/600, eta=49 minutes]
train/CrossEntropyLoss=0.0485
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.6479
train/patch_acc=0.7812
train/subpatch_loss=0.1167
train/subpatch_acc=0.9375
train/location_loss=0.9887
train/location_acc=0.6250
train/token_losses=9.605
throughput/total_tokens=7,880,704
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:22:47.911 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=482/600, eta=49 minutes]
train/CrossEntropyLoss=0.0410
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.6231
train/patch_acc=0.8125
train/subpatch_loss=0.0521
train/subpatch_acc=1.0000
train/location_loss=0.9455
train/location_acc=0.6250
train/token_losses=8.975
throughput/total_tokens=7,897,088
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:23:12.516 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=483/600, eta=48 minutes]
train/CrossEntropyLoss=0.0401
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.6249
train/patch_acc=0.7812
train/subpatch_loss=0.1628
train/subpatch_acc=0.9375
train/location_loss=0.7639
train/location_acc=0.6875
train/token_losses=8.706
throughput/total_tokens=7,913,472
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:23:37.071 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=484/600, eta=48 minutes]
train/CrossEntropyLoss=0.0573
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.9664
train/patch_acc=0.7500
train/subpatch_loss=0.0913
train/subpatch_acc=1.0000
train/location_loss=1.073
train/location_acc=0.5000
train/token_losses=12.39
throughput/total_tokens=7,929,856
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:24:01.623 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=485/600, eta=48 minutes]
train/CrossEntropyLoss=0.1525
train/ZLoss=0.0006
train/Accuracy=0.9634
train/patch_loss=0.9878
train/patch_acc=0.7500
train/subpatch_loss=0.1693
train/subpatch_acc=0.9375
train/location_loss=1.109
train/location_acc=0.6250
train/token_losses=13.02
throughput/total_tokens=7,946,240
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:24:26.173 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=486/600, eta=47 minutes]
train/CrossEntropyLoss=0.0086
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8732
train/patch_acc=0.7812
train/subpatch_loss=0.1046
train/subpatch_acc=1.0000
train/location_loss=1.093
train/location_acc=0.4375
train/token_losses=11.78
throughput/total_tokens=7,962,624
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:24:50.742 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=487/600, eta=47 minutes]
train/CrossEntropyLoss=0.0226
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.5989
train/patch_acc=0.9375
train/subpatch_loss=0.0600
train/subpatch_acc=1.0000
train/location_loss=0.8121
train/location_acc=0.7500
train/token_losses=8.280
throughput/total_tokens=7,979,008
throughput/device/tokens_per_second=666.9
throughput/device/batches_per_second=0.0407
2026-07-27 15:25:15.273 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=488/600, eta=46 minutes]
train/CrossEntropyLoss=0.0228
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.4839
train/patch_acc=0.8750
train/subpatch_loss=0.0589
train/subpatch_acc=1.0000
train/location_loss=0.5779
train/location_acc=0.7500
train/token_losses=6.418
throughput/total_tokens=7,995,392
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:25:39.817 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=489/600, eta=46 minutes]
train/CrossEntropyLoss=0.0292
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.001
train/patch_acc=0.6875
train/subpatch_loss=0.0956
train/subpatch_acc=1.0000
train/location_loss=1.050
train/location_acc=0.6250
train/token_losses=12.59
throughput/total_tokens=8,011,776
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:26:04.370 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=490/600, eta=46 minutes]
train/CrossEntropyLoss=0.0770
train/ZLoss=0.0001
train/Accuracy=0.9817
train/patch_loss=0.6076
train/patch_acc=0.8750
train/subpatch_loss=0.1037
train/subpatch_acc=1.0000
train/location_loss=0.5755
train/location_acc=0.9375
train/token_losses=7.577
throughput/total_tokens=8,028,160
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:26:28.918 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=491/600, eta=45 minutes]
train/CrossEntropyLoss=0.0381
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.8913
train/patch_acc=0.7812
train/subpatch_loss=0.1071
train/subpatch_acc=1.0000
train/location_loss=1.145
train/location_acc=0.5625
train/token_losses=12.14
throughput/total_tokens=8,044,544
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:26:53.460 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=492/600, eta=45 minutes]
train/CrossEntropyLoss=0.0198
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9596
train/patch_acc=0.7500
train/subpatch_loss=0.0964
train/subpatch_acc=1.0000
train/location_loss=0.9559
train/location_acc=0.6875
train/token_losses=11.89
throughput/total_tokens=8,060,928
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:27:18.003 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=493/600, eta=44 minutes]
train/CrossEntropyLoss=0.0295
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9356
train/patch_acc=0.7812
train/subpatch_loss=0.0889
train/subpatch_acc=1.0000
train/location_loss=0.8212
train/location_acc=0.6875
train/token_losses=11.13
throughput/total_tokens=8,077,312
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:27:42.590 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=494/600, eta=44 minutes]
train/CrossEntropyLoss=0.0315
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.6014
train/patch_acc=0.8125
train/subpatch_loss=0.1227
train/subpatch_acc=1.0000
train/location_loss=0.6589
train/location_acc=0.9375
train/token_losses=7.937
throughput/total_tokens=8,093,696
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:28:07.126 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=495/600, eta=44 minutes]
train/CrossEntropyLoss=0.0489
train/ZLoss=0.0005
train/Accuracy=0.9909
train/patch_loss=0.9568
train/patch_acc=0.7500
train/subpatch_loss=0.1692
train/subpatch_acc=0.8750
train/location_loss=1.085
train/location_acc=0.5625
train/token_losses=12.67
throughput/total_tokens=8,110,080
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:28:31.673 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=496/600, eta=43 minutes]
train/CrossEntropyLoss=0.0311
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.9590
train/patch_acc=0.7188
train/subpatch_loss=0.0493
train/subpatch_acc=1.0000
train/location_loss=1.021
train/location_acc=0.7500
train/token_losses=11.95
throughput/total_tokens=8,126,464
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:28:56.225 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=497/600, eta=43 minutes]
train/CrossEntropyLoss=0.0148
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6332
train/patch_acc=0.8125
train/subpatch_loss=0.1208
train/subpatch_acc=0.9375
train/location_loss=0.6475
train/location_acc=0.8750
train/token_losses=8.139
throughput/total_tokens=8,142,848
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:29:20.793 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=498/600, eta=42 minutes]
train/CrossEntropyLoss=0.0575
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6437
train/patch_acc=0.8125
train/subpatch_loss=0.0533
train/subpatch_acc=1.0000
train/location_loss=0.7760
train/location_acc=0.8125
train/token_losses=8.467
throughput/total_tokens=8,159,232
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:29:45.347 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=499/600, eta=42 minutes]
train/CrossEntropyLoss=0.0286
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6729
train/patch_acc=0.9062
train/subpatch_loss=0.0751
train/subpatch_acc=1.0000
train/location_loss=1.006
train/location_acc=0.6875
train/token_losses=9.706
throughput/total_tokens=8,175,616
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:30:09.905 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=500/600, eta=41 minutes]
train/CrossEntropyLoss=0.0048
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.8884
train/patch_acc=0.7812
train/subpatch_loss=0.0320
train/subpatch_acc=1.0000
train/location_loss=1.028
train/location_acc=0.3750
train/token_losses=11.35
throughput/total_tokens=8,192,000
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:30:09.907 2012bc4f83ae:0 olmo.train.trainer:1415 INFO Saving checkpoint...
2026-07-27 15:30:13.606 2012bc4f83ae:0 olmo.train.trainer:1420 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step500
2026-07-27 15:30:39.852 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=501/600, eta=41 minutes]
train/CrossEntropyLoss=0.0298
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.5618
train/patch_acc=0.8438
train/subpatch_loss=0.1336
train/subpatch_acc=0.9375
train/location_loss=0.7197
train/location_acc=0.8750
train/token_losses=7.907
throughput/total_tokens=8,208,384
throughput/device/tokens_per_second=624.4
throughput/device/batches_per_second=0.0381
2026-07-27 15:31:04.373 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=502/600, eta=41 minutes]
train/CrossEntropyLoss=0.0265
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.6853
train/patch_acc=0.8125
train/subpatch_loss=0.1202
train/subpatch_acc=1.0000
train/location_loss=0.6040
train/location_acc=1.0000
train/token_losses=8.379
throughput/total_tokens=8,224,768
throughput/device/tokens_per_second=645.5
throughput/device/batches_per_second=0.0394
2026-07-27 15:31:28.900 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=503/600, eta=40 minutes]
train/CrossEntropyLoss=0.0302
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7491
train/patch_acc=0.7188
train/subpatch_loss=0.1454
train/subpatch_acc=0.9375
train/location_loss=0.8142
train/location_acc=0.8125
train/token_losses=9.831
throughput/total_tokens=8,241,152
throughput/device/tokens_per_second=652.8
throughput/device/batches_per_second=0.0398
2026-07-27 15:31:53.432 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=504/600, eta=40 minutes]
train/CrossEntropyLoss=0.0172
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.6600
train/patch_acc=0.7812
train/subpatch_loss=0.0967
train/subpatch_acc=1.0000
train/location_loss=0.8456
train/location_acc=0.8125
train/token_losses=9.049
throughput/total_tokens=8,257,536
throughput/device/tokens_per_second=656.5
throughput/device/batches_per_second=0.0401
2026-07-27 15:32:18.006 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=505/600, eta=39 minutes]
train/CrossEntropyLoss=0.0473
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.7885
train/patch_acc=0.7812
train/subpatch_loss=0.0548
train/subpatch_acc=1.0000
train/location_loss=0.6937
train/location_acc=0.8125
train/token_losses=9.302
throughput/total_tokens=8,273,920
throughput/device/tokens_per_second=658.5
throughput/device/batches_per_second=0.0402
2026-07-27 15:32:42.561 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=506/600, eta=39 minutes]
train/CrossEntropyLoss=0.0341
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.5657
train/patch_acc=0.8125
train/subpatch_loss=0.0714
train/subpatch_acc=1.0000
train/location_loss=0.6653
train/location_acc=0.8750
train/token_losses=7.472
throughput/total_tokens=8,290,304
throughput/device/tokens_per_second=660.0
throughput/device/batches_per_second=0.0403
2026-07-27 15:33:07.125 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=507/600, eta=39 minutes]
train/CrossEntropyLoss=0.0285
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.7445
train/patch_acc=0.8125
train/subpatch_loss=0.1115
train/subpatch_acc=1.0000
train/location_loss=0.6832
train/location_acc=0.7500
train/token_losses=9.134
throughput/total_tokens=8,306,688
throughput/device/tokens_per_second=661.0
throughput/device/batches_per_second=0.0403
2026-07-27 15:33:31.667 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=508/600, eta=38 minutes]
train/CrossEntropyLoss=0.0214
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.6137
train/patch_acc=0.7500
train/subpatch_loss=0.0342
train/subpatch_acc=1.0000
train/location_loss=0.9023
train/location_acc=0.6875
train/token_losses=8.655
throughput/total_tokens=8,323,072
throughput/device/tokens_per_second=661.8
throughput/device/batches_per_second=0.0404
2026-07-27 15:33:56.214 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=509/600, eta=38 minutes]
train/CrossEntropyLoss=0.0349
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.8265
train/patch_acc=0.7500
train/subpatch_loss=0.1254
train/subpatch_acc=0.9375
train/location_loss=0.9228
train/location_acc=0.7500
train/token_losses=10.81
throughput/total_tokens=8,339,456
throughput/device/tokens_per_second=662.4
throughput/device/batches_per_second=0.0404
2026-07-27 15:34:20.772 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=510/600, eta=37 minutes]
train/CrossEntropyLoss=0.0092
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.7083
train/patch_acc=0.7812
train/subpatch_loss=0.0436
train/subpatch_acc=1.0000
train/location_loss=0.9967
train/location_acc=0.5625
train/token_losses=9.828
throughput/total_tokens=8,355,840
throughput/device/tokens_per_second=662.9
throughput/device/batches_per_second=0.0405
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:34:45.331 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=511/600, eta=37 minutes]
train/CrossEntropyLoss=0.0293
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.8079
train/patch_acc=0.8125
train/subpatch_loss=0.1013
train/subpatch_acc=1.0000
train/location_loss=1.227
train/location_acc=0.4375
train/token_losses=11.77
throughput/total_tokens=8,372,224
throughput/device/tokens_per_second=663.3
throughput/device/batches_per_second=0.0405
2026-07-27 15:35:09.889 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=512/600, eta=37 minutes]
train/CrossEntropyLoss=0.0482
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.6447
train/patch_acc=0.8438
train/subpatch_loss=0.1109
train/subpatch_acc=1.0000
train/location_loss=0.8628
train/location_acc=0.7500
train/token_losses=9.052
throughput/total_tokens=8,388,608
throughput/device/tokens_per_second=663.6
throughput/device/batches_per_second=0.0405
2026-07-27 15:35:34.446 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=513/600, eta=36 minutes]
train/CrossEntropyLoss=0.0108
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.7362
train/patch_acc=0.8125
train/subpatch_loss=0.0396
train/subpatch_acc=1.0000
train/location_loss=0.7168
train/location_acc=0.8125
train/token_losses=8.916
throughput/total_tokens=8,404,992
throughput/device/tokens_per_second=663.9
throughput/device/batches_per_second=0.0405
2026-07-27 15:35:58.993 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=514/600, eta=36 minutes]
train/CrossEntropyLoss=0.0511
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.9167
train/patch_acc=0.6875
train/subpatch_loss=0.0528
train/subpatch_acc=1.0000
train/location_loss=0.8715
train/location_acc=0.6250
train/token_losses=11.03
throughput/total_tokens=8,421,376
throughput/device/tokens_per_second=664.1
throughput/device/batches_per_second=0.0405
2026-07-27 15:36:23.534 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=515/600, eta=35 minutes]
train/CrossEntropyLoss=0.0449
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.6894
train/patch_acc=0.7500
train/subpatch_loss=0.1657
train/subpatch_acc=0.8750
train/location_loss=0.8861
train/location_acc=0.6250
train/token_losses=9.722
throughput/total_tokens=8,437,760
throughput/device/tokens_per_second=664.4
throughput/device/batches_per_second=0.0405
2026-07-27 15:36:48.078 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=516/600, eta=35 minutes]
train/CrossEntropyLoss=0.0473
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.4726
train/patch_acc=0.9688
train/subpatch_loss=0.0858
train/subpatch_acc=1.0000
train/location_loss=0.4710
train/location_acc=0.8750
train/token_losses=6.008
throughput/total_tokens=8,454,144
throughput/device/tokens_per_second=664.6
throughput/device/batches_per_second=0.0406
2026-07-27 15:37:12.638 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=517/600, eta=35 minutes]
train/CrossEntropyLoss=0.0652
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.6961
train/patch_acc=0.8125
train/subpatch_loss=0.0855
train/subpatch_acc=1.0000
train/location_loss=0.7024
train/location_acc=0.7500
train/token_losses=8.721
throughput/total_tokens=8,470,528
throughput/device/tokens_per_second=664.7
throughput/device/batches_per_second=0.0406
2026-07-27 15:37:37.252 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=518/600, eta=34 minutes]
train/CrossEntropyLoss=0.0173
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.9318
train/patch_acc=0.7812
train/subpatch_loss=0.1435
train/subpatch_acc=0.9375
train/location_loss=0.7206
train/location_acc=0.8750
train/token_losses=10.91
throughput/total_tokens=8,486,912
throughput/device/tokens_per_second=664.8
throughput/device/batches_per_second=0.0406
2026-07-27 15:38:01.833 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=519/600, eta=34 minutes]
train/CrossEntropyLoss=0.2052
train/ZLoss=0.0006
train/Accuracy=0.9421
train/patch_loss=0.8302
train/patch_acc=0.7812
train/subpatch_loss=0.1161
train/subpatch_acc=1.0000
train/location_loss=0.9584
train/location_acc=0.7500
train/token_losses=10.94
throughput/total_tokens=8,503,296
throughput/device/tokens_per_second=664.8
throughput/device/batches_per_second=0.0406
2026-07-27 15:38:26.424 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=520/600, eta=33 minutes]
train/CrossEntropyLoss=0.0277
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6004
train/patch_acc=0.8125
train/subpatch_loss=0.1963
train/subpatch_acc=1.0000
train/location_loss=0.7482
train/location_acc=0.8125
train/token_losses=8.581
throughput/total_tokens=8,519,680
throughput/device/tokens_per_second=664.9
throughput/device/batches_per_second=0.0406
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:38:51.026 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=521/600, eta=33 minutes]
train/CrossEntropyLoss=0.0379
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.8750
train/patch_acc=0.7500
train/subpatch_loss=0.1209
train/subpatch_acc=1.0000
train/location_loss=0.8024
train/location_acc=0.7500
train/token_losses=10.69
throughput/total_tokens=8,536,064
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:39:15.585 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=522/600, eta=32 minutes]
train/CrossEntropyLoss=0.0699
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.4185
train/patch_acc=0.9375
train/subpatch_loss=0.0486
train/subpatch_acc=1.0000
train/location_loss=0.8511
train/location_acc=0.7500
train/token_losses=6.947
throughput/total_tokens=8,552,448
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:39:40.138 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=523/600, eta=32 minutes]
train/CrossEntropyLoss=0.0463
train/ZLoss=0.0006
train/Accuracy=0.9848
train/patch_loss=0.6065
train/patch_acc=0.8438
train/subpatch_loss=0.0800
train/subpatch_acc=0.9375
train/location_loss=0.9782
train/location_acc=0.8125
train/token_losses=9.085
throughput/total_tokens=8,568,832
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:40:04.691 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=524/600, eta=32 minutes]
train/CrossEntropyLoss=0.0742
train/ZLoss=0.0002
train/Accuracy=0.9787
train/patch_loss=1.012
train/patch_acc=0.6875
train/subpatch_loss=0.1102
train/subpatch_acc=0.9375
train/location_loss=1.106
train/location_acc=0.5000
train/token_losses=12.96
throughput/total_tokens=8,585,216
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:40:29.235 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=525/600, eta=31 minutes]
train/CrossEntropyLoss=0.0066
train/ZLoss=0.0004
train/Accuracy=0.9970
train/patch_loss=0.9182
train/patch_acc=0.7500
train/subpatch_loss=0.0637
train/subpatch_acc=1.0000
train/location_loss=1.071
train/location_acc=0.6875
train/token_losses=11.89
throughput/total_tokens=8,601,600
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:40:53.794 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=526/600, eta=31 minutes]
train/CrossEntropyLoss=0.0230
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.6879
train/patch_acc=0.8438
train/subpatch_loss=0.1540
train/subpatch_acc=0.9375
train/location_loss=0.7884
train/location_acc=0.8750
train/token_losses=9.272
throughput/total_tokens=8,617,984
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:41:18.344 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=527/600, eta=30 minutes]
train/CrossEntropyLoss=0.0093
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6435
train/patch_acc=0.8125
train/subpatch_loss=0.0718
train/subpatch_acc=1.0000
train/location_loss=0.8904
train/location_acc=0.6875
train/token_losses=8.997
throughput/total_tokens=8,634,368
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:41:42.925 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=528/600, eta=30 minutes]
train/CrossEntropyLoss=0.0170
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.8956
train/patch_acc=0.7188
train/subpatch_loss=0.0724
train/subpatch_acc=1.0000
train/location_loss=0.5983
train/location_acc=0.8750
train/token_losses=9.848
throughput/total_tokens=8,650,752
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:42:07.480 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=529/600, eta=30 minutes]
train/CrossEntropyLoss=0.0269
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.8112
train/patch_acc=0.7500
train/subpatch_loss=0.1180
train/subpatch_acc=1.0000
train/location_loss=0.7718
train/location_acc=0.9375
train/token_losses=10.05
throughput/total_tokens=8,667,136
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:42:32.034 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=530/600, eta=29 minutes]
train/CrossEntropyLoss=0.0049
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.7034
train/patch_acc=0.7812
train/subpatch_loss=0.0238
train/subpatch_acc=1.0000
train/location_loss=0.6707
train/location_acc=0.8750
train/token_losses=8.406
throughput/total_tokens=8,683,520
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:42:56.608 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=531/600, eta=29 minutes]
train/CrossEntropyLoss=0.0219
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.7132
train/patch_acc=0.7812
train/subpatch_loss=0.1099
train/subpatch_acc=1.0000
train/location_loss=1.006
train/location_acc=0.6250
train/token_losses=10.17
throughput/total_tokens=8,699,904
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:43:21.149 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=532/600, eta=28 minutes]
train/CrossEntropyLoss=0.0385
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.5749
train/patch_acc=0.8438
train/subpatch_loss=0.0549
train/subpatch_acc=1.0000
train/location_loss=0.5667
train/location_acc=0.8750
train/token_losses=7.086
throughput/total_tokens=8,716,288
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:43:45.703 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=533/600, eta=28 minutes]
train/CrossEntropyLoss=0.0185
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.5597
train/patch_acc=0.8438
train/subpatch_loss=0.0853
train/subpatch_acc=1.0000
train/location_loss=0.5794
train/location_acc=0.8750
train/token_losses=7.137
throughput/total_tokens=8,732,672
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:44:10.262 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=534/600, eta=28 minutes]
train/CrossEntropyLoss=0.0991
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.7860
train/patch_acc=0.7188
train/subpatch_loss=0.1378
train/subpatch_acc=0.9375
train/location_loss=0.5340
train/location_acc=0.8750
train/token_losses=8.975
throughput/total_tokens=8,749,056
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:44:34.809 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=535/600, eta=27 minutes]
train/CrossEntropyLoss=0.2089
train/ZLoss=0.0007
train/Accuracy=0.9512
train/patch_loss=0.8731
train/patch_acc=0.7188
train/subpatch_loss=0.1104
train/subpatch_acc=0.9375
train/location_loss=0.8998
train/location_acc=0.6875
train/token_losses=11.03
throughput/total_tokens=8,765,440
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:44:59.357 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=536/600, eta=27 minutes]
train/CrossEntropyLoss=0.0233
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.4925
train/patch_acc=0.9375
train/subpatch_loss=0.0872
train/subpatch_acc=1.0000
train/location_loss=0.5886
train/location_acc=0.8750
train/token_losses=6.643
throughput/total_tokens=8,781,824
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:45:23.903 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=537/600, eta=26 minutes]
train/CrossEntropyLoss=0.0520
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6404
train/patch_acc=0.7812
train/subpatch_loss=0.1417
train/subpatch_acc=0.9375
train/location_loss=0.5521
train/location_acc=0.9375
train/token_losses=7.898
throughput/total_tokens=8,798,208
throughput/device/tokens_per_second=667.0
throughput/device/batches_per_second=0.0407
2026-07-27 15:45:48.446 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=538/600, eta=26 minutes]
train/CrossEntropyLoss=0.0360
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.9087
train/patch_acc=0.7812
train/subpatch_loss=0.0584
train/subpatch_acc=1.0000
train/location_loss=0.8110
train/location_acc=0.6875
train/token_losses=10.75
throughput/total_tokens=8,814,592
throughput/device/tokens_per_second=667.1
throughput/device/batches_per_second=0.0407
2026-07-27 15:46:13.006 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=539/600, eta=26 minutes]
train/CrossEntropyLoss=0.0079
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.5850
train/patch_acc=0.8438
train/subpatch_loss=0.0461
train/subpatch_acc=1.0000
train/location_loss=0.7996
train/location_acc=0.7500
train/token_losses=8.063
throughput/total_tokens=8,830,976
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
2026-07-27 15:46:37.571 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=540/600, eta=25 minutes]
train/CrossEntropyLoss=0.0304
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.8531
train/patch_acc=0.7812
train/subpatch_loss=0.1258
train/subpatch_acc=1.0000
train/location_loss=1.198
train/location_acc=0.6875
train/token_losses=12.12
throughput/total_tokens=8,847,360
throughput/device/tokens_per_second=667.2
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:47:02.105 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=541/600, eta=25 minutes]
train/CrossEntropyLoss=0.0232
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.8486
train/patch_acc=0.7500
train/subpatch_loss=0.0609
train/subpatch_acc=1.0000
train/location_loss=1.041
train/location_acc=0.6875
train/token_losses=11.20
throughput/total_tokens=8,863,744
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:47:26.642 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=542/600, eta=24 minutes]
train/CrossEntropyLoss=0.0258
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7667
train/patch_acc=0.7188
train/subpatch_loss=0.2075
train/subpatch_acc=0.8750
train/location_loss=0.7128
train/location_acc=0.8750
train/token_losses=9.814
throughput/total_tokens=8,880,128
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:47:51.194 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=543/600, eta=24 minutes]
train/CrossEntropyLoss=0.0230
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.4858
train/patch_acc=0.8438
train/subpatch_loss=0.0441
train/subpatch_acc=1.0000
train/location_loss=1.049
train/location_acc=0.6875
train/token_losses=8.257
throughput/total_tokens=8,896,512
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:48:15.748 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=544/600, eta=23 minutes]
train/CrossEntropyLoss=0.0715
train/ZLoss=0.0001
train/Accuracy=0.9787
train/patch_loss=0.9265
train/patch_acc=0.7500
train/subpatch_loss=0.1066
train/subpatch_acc=1.0000
train/location_loss=1.026
train/location_acc=0.6875
train/token_losses=11.94
throughput/total_tokens=8,912,896
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:48:40.307 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=545/600, eta=23 minutes]
train/CrossEntropyLoss=0.0186
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8085
train/patch_acc=0.6875
train/subpatch_loss=0.0943
train/subpatch_acc=1.0000
train/location_loss=1.000
train/location_acc=0.5625
train/token_losses=10.85
throughput/total_tokens=8,929,280
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:49:04.846 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=546/600, eta=23 minutes]
train/CrossEntropyLoss=0.0400
train/ZLoss=0.0003
train/Accuracy=0.9848
train/patch_loss=0.7937
train/patch_acc=0.7812
train/subpatch_loss=0.0446
train/subpatch_acc=1.0000
train/location_loss=1.002
train/location_acc=0.5000
train/token_losses=10.53
throughput/total_tokens=8,945,664
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:49:29.392 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=547/600, eta=22 minutes]
train/CrossEntropyLoss=0.0389
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6598
train/patch_acc=0.7812
train/subpatch_loss=0.0309
train/subpatch_acc=1.0000
train/location_loss=0.7700
train/location_acc=0.6875
train/token_losses=8.482
throughput/total_tokens=8,962,048
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:49:53.948 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=548/600, eta=22 minutes]
train/CrossEntropyLoss=0.0172
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=1.040
train/patch_acc=0.7188
train/subpatch_loss=0.0616
train/subpatch_acc=1.0000
train/location_loss=0.9460
train/location_acc=0.7500
train/token_losses=12.35
throughput/total_tokens=8,978,432
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:50:18.510 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=549/600, eta=21 minutes]
train/CrossEntropyLoss=0.1276
train/ZLoss=0.0006
train/Accuracy=0.9695
train/patch_loss=0.8785
train/patch_acc=0.6875
train/subpatch_loss=0.1738
train/subpatch_acc=0.9375
train/location_loss=1.105
train/location_acc=0.6250
train/token_losses=12.14
throughput/total_tokens=8,994,816
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:50:43.066 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=550/600, eta=21 minutes]
train/CrossEntropyLoss=0.0485
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.5384
train/patch_acc=0.8750
train/subpatch_loss=0.0819
train/subpatch_acc=1.0000
train/location_loss=0.5901
train/location_acc=0.7500
train/token_losses=6.996
throughput/total_tokens=9,011,200
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:51:07.602 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=551/600, eta=21 minutes]
train/CrossEntropyLoss=0.0102
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.8669
train/patch_acc=0.6875
train/subpatch_loss=0.1996
train/subpatch_acc=0.9375
train/location_loss=0.6256
train/location_acc=0.9375
train/token_losses=10.24
throughput/total_tokens=9,027,584
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:51:32.152 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=552/600, eta=20 minutes]
train/CrossEntropyLoss=0.0313
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.6038
train/patch_acc=0.8125
train/subpatch_loss=0.0851
train/subpatch_acc=1.0000
train/location_loss=0.9766
train/location_acc=0.8125
train/token_losses=9.077
throughput/total_tokens=9,043,968
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:51:56.704 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=553/600, eta=20 minutes]
train/CrossEntropyLoss=0.0161
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.7408
train/patch_acc=0.8125
train/subpatch_loss=0.0368
train/subpatch_acc=1.0000
train/location_loss=0.9783
train/location_acc=0.6250
train/token_losses=9.986
throughput/total_tokens=9,060,352
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:52:21.251 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=554/600, eta=19 minutes]
train/CrossEntropyLoss=0.0414
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.5605
train/patch_acc=0.8750
train/subpatch_loss=0.0869
train/subpatch_acc=1.0000
train/location_loss=0.5817
train/location_acc=0.8125
train/token_losses=7.158
throughput/total_tokens=9,076,736
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:52:45.817 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=555/600, eta=19 minutes]
train/CrossEntropyLoss=0.0559
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.7241
train/patch_acc=0.7812
train/subpatch_loss=0.1820
train/subpatch_acc=0.9375
train/location_loss=0.8879
train/location_acc=0.7500
train/token_losses=10.07
throughput/total_tokens=9,093,120
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:53:10.380 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=556/600, eta=19 minutes]
train/CrossEntropyLoss=0.0513
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.5740
train/patch_acc=0.8438
train/subpatch_loss=0.0392
train/subpatch_acc=1.0000
train/location_loss=0.9152
train/location_acc=0.5625
train/token_losses=8.410
throughput/total_tokens=9,109,504
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:53:34.913 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=557/600, eta=18 minutes]
train/CrossEntropyLoss=0.0185
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.6256
train/patch_acc=0.8438
train/subpatch_loss=0.0332
train/subpatch_acc=1.0000
train/location_loss=0.5626
train/location_acc=0.7500
train/token_losses=7.388
throughput/total_tokens=9,125,888
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:53:59.473 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=558/600, eta=18 minutes]
train/CrossEntropyLoss=0.2341
train/ZLoss=0.0013
train/Accuracy=0.9390
train/patch_loss=0.9278
train/patch_acc=0.7500
train/subpatch_loss=0.1044
train/subpatch_acc=1.0000
train/location_loss=1.050
train/location_acc=0.6875
train/token_losses=12.04
throughput/total_tokens=9,142,272
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:54:23.994 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=559/600, eta=17 minutes]
train/CrossEntropyLoss=0.0201
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.9587
train/patch_acc=0.7188
train/subpatch_loss=0.0693
train/subpatch_acc=1.0000
train/location_loss=0.7721
train/location_acc=0.6875
train/token_losses=11.03
throughput/total_tokens=9,158,656
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:54:48.530 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=560/600, eta=17 minutes]
train/CrossEntropyLoss=0.0111
train/ZLoss=0.0004
train/Accuracy=0.9970
train/patch_loss=0.6091
train/patch_acc=0.8125
train/subpatch_loss=0.0782
train/subpatch_acc=1.0000
train/location_loss=0.5067
train/location_acc=0.9375
train/token_losses=7.213
throughput/total_tokens=9,175,040
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:55:13.074 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=561/600, eta=16 minutes]
train/CrossEntropyLoss=0.0431
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.5746
train/patch_acc=0.8125
train/subpatch_loss=0.1241
train/subpatch_acc=0.9375
train/location_loss=1.086
train/location_acc=0.6250
train/token_losses=9.438
throughput/total_tokens=9,191,424
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:55:37.608 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=562/600, eta=16 minutes]
train/CrossEntropyLoss=0.0114
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6448
train/patch_acc=0.8125
train/subpatch_loss=0.0700
train/subpatch_acc=1.0000
train/location_loss=0.6725
train/location_acc=0.8750
train/token_losses=8.128
throughput/total_tokens=9,207,808
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:56:02.168 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=563/600, eta=16 minutes]
train/CrossEntropyLoss=0.0124
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.7340
train/patch_acc=0.7188
train/subpatch_loss=0.0249
train/subpatch_acc=1.0000
train/location_loss=0.9777
train/location_acc=0.7500
train/token_losses=9.883
throughput/total_tokens=9,224,192
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:56:26.728 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=564/600, eta=15 minutes]
train/CrossEntropyLoss=0.0139
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.7291
train/patch_acc=0.7500
train/subpatch_loss=0.1106
train/subpatch_acc=0.9375
train/location_loss=0.9497
train/location_acc=0.7500
train/token_losses=10.07
throughput/total_tokens=9,240,576
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:56:51.323 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=565/600, eta=15 minutes]
train/CrossEntropyLoss=0.0303
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6144
train/patch_acc=0.8125
train/subpatch_loss=0.0730
train/subpatch_acc=1.0000
train/location_loss=0.7526
train/location_acc=0.8750
train/token_losses=8.218
throughput/total_tokens=9,256,960
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:57:15.875 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=566/600, eta=14 minutes]
train/CrossEntropyLoss=0.0125
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.6551
train/patch_acc=0.8750
train/subpatch_loss=0.0719
train/subpatch_acc=1.0000
train/location_loss=1.097
train/location_acc=0.5625
train/token_losses=9.916
throughput/total_tokens=9,273,344
throughput/device/tokens_per_second=667.3
throughput/device/batches_per_second=0.0407
2026-07-27 15:57:40.413 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=567/600, eta=14 minutes]
train/CrossEntropyLoss=0.0094
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6342
train/patch_acc=0.8125
train/subpatch_loss=0.0443
train/subpatch_acc=1.0000
train/location_loss=0.7294
train/location_acc=0.8125
train/token_losses=8.168
throughput/total_tokens=9,289,728
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:58:04.967 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=568/600, eta=14 minutes]
train/CrossEntropyLoss=0.0437
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.5158
train/patch_acc=0.8438
train/subpatch_loss=0.1227
train/subpatch_acc=0.9375
train/location_loss=0.9515
train/location_acc=0.6250
train/token_losses=8.423
throughput/total_tokens=9,306,112
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:58:29.517 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=569/600, eta=13 minutes]
train/CrossEntropyLoss=0.0131
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6919
train/patch_acc=0.8125
train/subpatch_loss=0.0859
train/subpatch_acc=1.0000
train/location_loss=0.6673
train/location_acc=0.6875
train/token_losses=8.549
throughput/total_tokens=9,322,496
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:58:54.034 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=570/600, eta=13 minutes]
train/CrossEntropyLoss=0.0376
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.6314
train/patch_acc=0.7812
train/subpatch_loss=0.0833
train/subpatch_acc=1.0000
train/location_loss=1.076
train/location_acc=0.5625
train/token_losses=9.690
throughput/total_tokens=9,338,880
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 15:59:18.565 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=571/600, eta=12 minutes]
train/CrossEntropyLoss=0.0107
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.7059
train/patch_acc=0.7188
train/subpatch_loss=0.0933
train/subpatch_acc=1.0000
train/location_loss=0.6769
train/location_acc=0.8750
train/token_losses=8.728
throughput/total_tokens=9,355,264
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 15:59:43.098 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=572/600, eta=12 minutes]
train/CrossEntropyLoss=0.0263
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6615
train/patch_acc=0.7812
train/subpatch_loss=0.1552
train/subpatch_acc=0.9375
train/location_loss=0.7474
train/location_acc=0.8125
train/token_losses=8.903
throughput/total_tokens=9,371,648
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 16:00:07.649 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=573/600, eta=12 minutes]
train/CrossEntropyLoss=0.0329
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8871
train/patch_acc=0.7500
train/subpatch_loss=0.0523
train/subpatch_acc=1.0000
train/location_loss=0.7794
train/location_acc=0.6875
train/token_losses=10.42
throughput/total_tokens=9,388,032
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 16:00:32.253 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=574/600, eta=11 minutes]
train/CrossEntropyLoss=0.2209
train/ZLoss=0.0005
train/Accuracy=0.9451
train/patch_loss=1.032
train/patch_acc=0.6875
train/subpatch_loss=0.1516
train/subpatch_acc=0.9375
train/location_loss=1.317
train/location_acc=0.5000
train/token_losses=14.13
throughput/total_tokens=9,404,416
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:00:56.809 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=575/600, eta=11 minutes]
train/CrossEntropyLoss=0.0305
train/ZLoss=0.0001
train/Accuracy=0.9878
train/patch_loss=0.5358
train/patch_acc=0.8125
train/subpatch_loss=0.1999
train/subpatch_acc=0.8750
train/location_loss=0.6366
train/location_acc=0.8125
train/token_losses=7.632
throughput/total_tokens=9,420,800
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:01:21.357 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=576/600, eta=10 minutes]
train/CrossEntropyLoss=0.0699
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.5291
train/patch_acc=0.8438
train/subpatch_loss=0.0760
train/subpatch_acc=1.0000
train/location_loss=0.7021
train/location_acc=0.8125
train/token_losses=7.346
throughput/total_tokens=9,437,184
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:01:45.894 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=577/600, eta=10 minutes]
train/CrossEntropyLoss=0.0245
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.6863
train/patch_acc=0.8438
train/subpatch_loss=0.0595
train/subpatch_acc=1.0000
train/location_loss=0.7199
train/location_acc=0.7500
train/token_losses=8.608
throughput/total_tokens=9,453,568
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:02:10.439 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=578/600, eta=10 minutes]
train/CrossEntropyLoss=0.0511
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.7662
train/patch_acc=0.7500
train/subpatch_loss=0.1177
train/subpatch_acc=0.9375
train/location_loss=0.7206
train/location_acc=0.8125
train/token_losses=9.483
throughput/total_tokens=9,469,952
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:02:34.968 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=579/600, eta=9 minutes]
train/CrossEntropyLoss=0.0278
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.7493
train/patch_acc=0.8125
train/subpatch_loss=0.0450
train/subpatch_acc=1.0000
train/location_loss=0.6891
train/location_acc=0.8750
train/token_losses=8.930
throughput/total_tokens=9,486,336
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:02:59.503 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=580/600, eta=9 minutes]
train/CrossEntropyLoss=0.0535
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.8401
train/patch_acc=0.7500
train/subpatch_loss=0.0901
train/subpatch_acc=1.0000
train/location_loss=0.8209
train/location_acc=0.6250
train/token_losses=10.36
throughput/total_tokens=9,502,720
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 16:03:24.048 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=581/600, eta=8 minutes]
train/CrossEntropyLoss=0.0136
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6458
train/patch_acc=0.7812
train/subpatch_loss=0.0680
train/subpatch_acc=0.9375
train/location_loss=0.6202
train/location_acc=0.8750
train/token_losses=7.920
throughput/total_tokens=9,519,104
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:03:48.589 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=582/600, eta=8 minutes]
train/CrossEntropyLoss=0.0057
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.4736
train/patch_acc=0.8438
train/subpatch_loss=0.0811
train/subpatch_acc=1.0000
train/location_loss=1.103
train/location_acc=0.7500
train/token_losses=8.524
throughput/total_tokens=9,535,488
throughput/device/tokens_per_second=667.4
throughput/device/batches_per_second=0.0407
2026-07-27 16:04:13.113 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=583/600, eta=7 minutes]
train/CrossEntropyLoss=0.0562
train/ZLoss=0.0002
train/Accuracy=0.9909
train/patch_loss=0.7869
train/patch_acc=0.7812
train/subpatch_loss=0.1270
train/subpatch_acc=1.0000
train/location_loss=0.8574
train/location_acc=0.8750
train/token_losses=10.23
throughput/total_tokens=9,551,872
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 16:04:37.646 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=584/600, eta=7 minutes]
train/CrossEntropyLoss=0.0424
train/ZLoss=0.0001
train/Accuracy=0.9848
train/patch_loss=0.8771
train/patch_acc=0.8125
train/subpatch_loss=0.1662
train/subpatch_acc=0.9375
train/location_loss=0.8377
train/location_acc=0.8125
train/token_losses=11.03
throughput/total_tokens=9,568,256
throughput/device/tokens_per_second=667.5
throughput/device/batches_per_second=0.0407
2026-07-27 16:05:02.191 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=585/600, eta=7 minutes]
train/CrossEntropyLoss=0.0029
train/ZLoss=0.0002
train/Accuracy=1.0000
train/patch_loss=0.9436
train/patch_acc=0.7188
train/subpatch_loss=0.0308
train/subpatch_acc=1.0000
train/location_loss=0.6829
train/location_acc=0.6875
train/token_losses=10.40
throughput/total_tokens=9,584,640
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 16:05:26.730 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=586/600, eta=6 minutes]
train/CrossEntropyLoss=0.0900
train/ZLoss=0.0007
train/Accuracy=0.9787
train/patch_loss=1.047
train/patch_acc=0.6875
train/subpatch_loss=0.0768
train/subpatch_acc=1.0000
train/location_loss=0.8193
train/location_acc=0.6875
train/token_losses=11.96
throughput/total_tokens=9,601,024
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 16:05:51.237 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=587/600, eta=6 minutes]
train/CrossEntropyLoss=0.0292
train/ZLoss=0.0002
train/Accuracy=0.9848
train/patch_loss=0.6220
train/patch_acc=0.7500
train/subpatch_loss=0.0600
train/subpatch_acc=1.0000
train/location_loss=0.5098
train/location_acc=0.8750
train/token_losses=7.255
throughput/total_tokens=9,617,408
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 16:06:15.769 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=588/600, eta=5 minutes]
train/CrossEntropyLoss=0.0447
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.5397
train/patch_acc=0.7812
train/subpatch_loss=0.0422
train/subpatch_acc=1.0000
train/location_loss=0.6746
train/location_acc=0.8125
train/token_losses=7.185
throughput/total_tokens=9,633,792
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 16:06:40.303 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=589/600, eta=5 minutes]
train/CrossEntropyLoss=0.0112
train/ZLoss=0.0002
train/Accuracy=0.9970
train/patch_loss=0.6568
train/patch_acc=0.7812
train/subpatch_loss=0.0598
train/subpatch_acc=1.0000
train/location_loss=0.9357
train/location_acc=0.7500
train/token_losses=9.236
throughput/total_tokens=9,650,176
throughput/device/tokens_per_second=667.7
throughput/device/batches_per_second=0.0408
2026-07-27 16:07:04.870 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=590/600, eta=5 minutes]
train/CrossEntropyLoss=0.0628
train/ZLoss=0.0005
train/Accuracy=0.9787
train/patch_loss=0.8171
train/patch_acc=0.7188
train/subpatch_loss=0.0551
train/subpatch_acc=1.0000
train/location_loss=0.9442
train/location_acc=0.6250
train/token_losses=10.53
throughput/total_tokens=9,666,560
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
System/Peak GPU Memory (MB)=41,776
2026-07-27 16:07:29.408 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=591/600, eta=4 minutes]
train/CrossEntropyLoss=0.0064
train/ZLoss=0.0001
train/Accuracy=1.0000
train/patch_loss=0.7360
train/patch_acc=0.7812
train/subpatch_loss=0.0522
train/subpatch_acc=1.0000
train/location_loss=0.8306
train/location_acc=0.6875
train/token_losses=9.419
throughput/total_tokens=9,682,944
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 16:07:53.936 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=592/600, eta=4 minutes]
train/CrossEntropyLoss=0.0175
train/ZLoss=0.0001
train/Accuracy=0.9970
train/patch_loss=0.8587
train/patch_acc=0.7188
train/subpatch_loss=0.1135
train/subpatch_acc=1.0000
train/location_loss=0.9184
train/location_acc=0.7500
train/token_losses=11.00
throughput/total_tokens=9,699,328
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 16:08:18.455 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=593/600, eta=3 minutes]
train/CrossEntropyLoss=0.0527
train/ZLoss=0.0001
train/Accuracy=0.9787
train/patch_loss=0.9136
train/patch_acc=0.7812
train/subpatch_loss=0.1482
train/subpatch_acc=0.9375
train/location_loss=0.7207
train/location_acc=0.7500
train/token_losses=10.78
throughput/total_tokens=9,715,712
throughput/device/tokens_per_second=667.6
throughput/device/batches_per_second=0.0407
2026-07-27 16:08:42.982 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=594/600, eta=3 minutes]
train/CrossEntropyLoss=0.0495
train/ZLoss=0.0006
train/Accuracy=0.9848
train/patch_loss=0.5771
train/patch_acc=0.8750
train/subpatch_loss=0.0726
train/subpatch_acc=1.0000
train/location_loss=0.9124
train/location_acc=0.7500
train/token_losses=8.557
throughput/total_tokens=9,732,096
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 16:09:07.520 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=595/600, eta=3 minutes]
train/CrossEntropyLoss=0.0206
train/ZLoss=0.0002
train/Accuracy=0.9939
train/patch_loss=0.5541
train/patch_acc=0.9062
train/subpatch_loss=0.0654
train/subpatch_acc=1.0000
train/location_loss=0.5786
train/location_acc=0.8125
train/token_losses=7.009
throughput/total_tokens=9,748,480
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 16:09:32.068 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=596/600, eta=2 minutes]
train/CrossEntropyLoss=0.0385
train/ZLoss=0.0002
train/Accuracy=0.9878
train/patch_loss=0.7315
train/patch_acc=0.7812
train/subpatch_loss=0.0235
train/subpatch_acc=1.0000
train/location_loss=0.6193
train/location_acc=0.9375
train/token_losses=8.424
throughput/total_tokens=9,764,864
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 16:09:56.621 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=597/600, eta=2 minutes]
train/CrossEntropyLoss=0.0530
train/ZLoss=0.0001
train/Accuracy=0.9909
train/patch_loss=0.6641
train/patch_acc=0.8438
train/subpatch_loss=0.0924
train/subpatch_acc=1.0000
train/location_loss=0.6889
train/location_acc=0.7500
train/token_losses=8.438
throughput/total_tokens=9,781,248
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 16:10:21.153 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=598/600, eta=1 minute]
train/CrossEntropyLoss=0.2034
train/ZLoss=0.0007
train/Accuracy=0.9482
train/patch_loss=0.6522
train/patch_acc=0.7812
train/subpatch_loss=0.0989
train/subpatch_acc=1.0000
train/location_loss=0.9145
train/location_acc=0.8125
train/token_losses=9.271
throughput/total_tokens=9,797,632
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 16:10:45.690 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=599/600, eta=1 minute]
train/CrossEntropyLoss=0.0442
train/ZLoss=0.0002
train/Accuracy=0.9817
train/patch_loss=0.5183
train/patch_acc=0.8438
train/subpatch_loss=0.0702
train/subpatch_acc=1.0000
train/location_loss=0.4058
train/location_acc=1.0000
train/token_losses=6.050
throughput/total_tokens=9,814,016
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
2026-07-27 16:11:10.206 2012bc4f83ae:0 olmo.train.trainer:1050 INFO [step=600/600, eta=1 minute]
train/CrossEntropyLoss=0.0098
train/ZLoss=0.0001
train/Accuracy=0.9939
train/patch_loss=0.9509
train/patch_acc=0.7188
train/subpatch_loss=0.0828
train/subpatch_acc=1.0000
train/location_loss=0.8065
train/location_acc=0.7500
train/token_losses=11.16
throughput/total_tokens=9,830,400
throughput/device/tokens_per_second=667.8
throughput/device/batches_per_second=0.0408
System/Peak GPU Memory (MB)=41,776
2026-07-27 16:11:10.209 2012bc4f83ae:0 olmo.train.trainer:1415 INFO Saving checkpoint...
2026-07-27 16:11:11.445 2012bc4f83ae:0 olmo.train.trainer:1420 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step600
2026-07-27 16:11:13.064 2012bc4f83ae:0 olmo.train.trainer:1516 INFO Saving final checkpoint...
2026-07-27 16:11:13.796 2012bc4f83ae:0 olmo.train.trainer:1519 INFO Checkpoint saved to /runpod-volume/firewarning-pointing/checkpoints/molmopoint-strong-curated-ground-v1/step600
2026-07-27 16:11:13.797 2012bc4f83ae:0 train:412 INFO Training complete
[rank0]:[W727 16:11:14.561122367 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())