| ## Base model training | |
| timestamp: 2025-11-03 09:00:42 | |
| - run: fal | |
| - device_type: | |
| - depth: 20 | |
| - max_seq_len: 2048 | |
| - num_iterations: -1 | |
| - target_flops: -1.0000 | |
| - target_param_data_ratio: 20 | |
| - device_batch_size: 32 | |
| - total_batch_size: 524,288 | |
| - embedding_lr: 0.2000 | |
| - unembedding_lr: 0.0040 | |
| - weight_decay: 0.0000 | |
| - matrix_lr: 0.0200 | |
| - grad_clip: 1.0000 | |
| - warmup_ratio: 0.0000 | |
| - warmdown_ratio: 0.2000 | |
| - final_lr_frac: 0.0000 | |
| - eval_every: 250 | |
| - eval_tokens: 10,485,760 | |
| - core_metric_every: 2000 | |
| - core_metric_max_per_task: 500 | |
| - sample_every: 2000 | |
| - model_tag: | |
| - Number of parameters: 560,988,160 | |
| - Number of FLOPs per token: 3.491758e+09 | |
| - Calculated number of iterations: 21,400 | |
| - Number of training tokens: 11,219,763,200 | |
| - Tokens : Params ratio: 20.0000 | |
| - DDP world size: 8 | |
| - warmup_ratio: 0.0000 | |
| - warmdown_ratio: 0.2000 | |
| - final_lr_frac: 0.0000 | |
| - Minimum validation bpb: 0.8118 | |
| - Final validation bpb: 0.8118 | |
| - CORE metric estimate: 0.2236 | |
| - MFU %: 48.44% | |
| - Total training flops: 3.917670e+19 | |
| - Total training time: 172.84m | |
| - Peak memory usage: 75422.02MiB | |