| 2026-04-03 03:31:37 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 03:31:37 - ReXMoE - INFO - ReXMoE Training Log - 0304_033137 |
| 2026-04-03 03:31:37 - ReXMoE - INFO - Log file: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/logs/rexmoe_training_0304_033137.log |
| 2026-04-03 03:31:37 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 03:31:37 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 03:31:37 - ReXMoE - INFO - ReXMoE Cross-Layer Expert Reuse Training |
| 2026-04-03 03:31:37 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 03:31:37 - ReXMoE - INFO - MET enabled: False |
| 2026-04-03 03:31:37 - ReXMoE - INFO - |
| Configuration: |
| Model: microsoft/Phi-mini-MoE-instruct |
| Dataset: ../dataset/alpaca_data_cleaned.json |
| Dataset mode: IF_2 |
| Reuse Scale (R): 3 |
| Prune Ratio (MET): N/A |
| Epochs: 1 |
| Num of samples: 20000 |
| Batch Size: 4 |
| Sequence Length: 1024 |
| Learning Rate: 2e-05 |
| PSR Enabled: True |
| LR Scheduler: True |
| Save Path: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3 |
| Gradient Checkpointing: False |
| LoRA Rank: 16 (Full LoRA: True) |
| LoRA Alpha: 32 |
| MET Enabled: False (Mask Ratio: 0.1, Warmup: 0.5) |
| Log File: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/logs/rexmoe_training_0304_033137.log |
| Aux loss weight: 0.05 |
|
|
| 2026-04-03 03:31:37 - ReXMoE - INFO - 💻 Using device: cuda) |
| 2026-04-03 03:31:37 - ReXMoE - INFO - GPU: NVIDIA RTX A6000, Memory: 47.53 GB |
| 2026-04-03 03:31:43 - ReXMoE - INFO - [5/7] Setting up optimizer and dataset... |
| 2026-04-03 03:31:43 - ReXMoE - INFO - Using 8-bit AdamW optimizer |
| 2026-04-03 03:31:43 - ReXMoE - INFO - LR Scheduler: CosineAnnealingLR (2e-05 → 2.0000000000000003e-06) |
| 2026-04-03 03:31:51 - ReXMoE - INFO - |
| First batch statistics: |
| 2026-04-03 03:31:51 - ReXMoE - INFO - LM Loss: 1.0094 |
| 2026-04-03 03:31:51 - ReXMoE - INFO - Aux Loss: 0.092773 |
| 2026-04-03 03:31:51 - ReXMoE - INFO - Total Loss: 1.1022 |
| 2026-04-03 03:31:51 - ReXMoE - INFO - Current R: 2 |
| 2026-04-03 03:31:51 - ReXMoE - INFO - Active experts per layer: 32 |
| 2026-04-03 03:31:51 - ReXMoE - INFO - Gradient norm: 1.0000 |
| 2026-04-03 03:31:51 - ReXMoE - INFO - |
| |
| 2026-04-03 03:35:09 - ReXMoE - INFO - [50/5000] loss=1.1939 aux=0.062988 R=2 |
| 2026-04-03 03:38:21 - ReXMoE - INFO - [100/5000] loss=1.1803 aux=0.040039 R=2 |
| 2026-04-03 03:41:36 - ReXMoE - INFO - [150/5000] loss=1.2968 aux=0.036621 R=2 |
| 2026-04-03 03:44:50 - ReXMoE - INFO - [200/5000] loss=1.2447 aux=0.028198 R=2 |
| 2026-04-03 03:48:01 - ReXMoE - INFO - [250/5000] loss=1.1971 aux=0.034180 R=2 |
| 2026-04-03 03:51:10 - ReXMoE - INFO - [300/5000] loss=2.1766 aux=0.024658 R=2 |
| 2026-04-03 03:54:19 - ReXMoE - INFO - [350/5000] loss=1.1092 aux=0.017578 R=2 |
| 2026-04-03 03:57:29 - ReXMoE - INFO - [400/5000] loss=0.9343 aux=0.024414 R=2 |
| 2026-04-03 04:00:40 - ReXMoE - INFO - [450/5000] loss=1.2180 aux=0.045410 R=2 |
| 2026-04-03 04:03:47 - ReXMoE - INFO - Warmup completed at step 500. Enabling FULL QLoRA with r = 16 and alpha = 32 on experts and updating optimizer... |
| 2026-04-03 04:03:51 - ReXMoE - INFO - Trainable params (routers + LoRA): 144179200 (1.8509%) |
| 2026-04-03 04:03:51 - ReXMoE - INFO - Sample trainable params after QLoRA: ['base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight', 'base_model.model.model.layers.0.block_sparse_moe.gate.weight', 'base_model.model.model.layers.0.block_sparse_moe.experts.0.w1.lora_A.default.weight'] |
| 2026-04-03 04:03:58 - ReXMoE - INFO - [500/5000] loss=1.0733 aux=0.036621 R=2 |
| 2026-04-03 04:09:15 - ReXMoE - INFO - [550/5000] loss=0.6253 aux=0.014709 R=2 |
| 2026-04-03 04:14:28 - ReXMoE - INFO - [600/5000] loss=1.5688 aux=0.010986 R=2 |
| 2026-04-03 04:19:39 - ReXMoE - INFO - [650/5000] loss=0.7864 aux=0.016357 R=2 |
| 2026-04-03 04:24:52 - ReXMoE - INFO - [700/5000] loss=1.5303 aux=0.010681 R=2 |
| 2026-04-03 04:30:02 - ReXMoE - INFO - [750/5000] loss=1.0098 aux=0.007812 R=2 |
| 2026-04-03 04:35:13 - ReXMoE - INFO - [800/5000] loss=1.0523 aux=0.014282 R=2 |
| 2026-04-03 04:40:24 - ReXMoE - INFO - [850/5000] loss=0.6447 aux=0.009094 R=2 |
| 2026-04-03 04:45:37 - ReXMoE - INFO - [900/5000] loss=0.7665 aux=0.004822 R=2 |
| 2026-04-03 04:50:50 - ReXMoE - INFO - [950/5000] loss=0.7762 aux=0.005737 R=2 |
| 2026-04-03 04:56:03 - ReXMoE - INFO - [1000/5000] loss=1.0254 aux=0.003571 R=2 |
| 2026-04-03 05:01:16 - ReXMoE - INFO - [1050/5000] loss=1.1320 aux=0.005737 R=2 |
| 2026-04-03 05:06:28 - ReXMoE - INFO - [1100/5000] loss=0.7519 aux=0.004974 R=2 |
| 2026-04-03 05:11:40 - ReXMoE - INFO - [1150/5000] loss=0.8246 aux=0.003204 R=2 |
| 2026-04-03 05:16:55 - ReXMoE - INFO - [1200/5000] loss=1.0041 aux=0.006042 R=2 |
| 2026-04-03 05:22:09 - ReXMoE - INFO - [1250/5000] loss=0.6804 aux=0.005859 R=2 |
| 2026-04-03 05:27:21 - ReXMoE - INFO - [1300/5000] loss=0.9695 aux=0.011108 R=2 |
| 2026-04-03 05:32:33 - ReXMoE - INFO - [1350/5000] loss=1.0448 aux=0.012634 R=2 |
| 2026-04-03 05:37:45 - ReXMoE - INFO - [1400/5000] loss=0.7468 aux=0.002136 R=2 |
| 2026-04-03 05:42:58 - ReXMoE - INFO - [1450/5000] loss=1.6307 aux=0.003510 R=2 |
| 2026-04-03 05:48:10 - ReXMoE - INFO - [1500/5000] loss=1.1833 aux=0.002625 R=2 |
| 2026-04-03 05:53:21 - ReXMoE - INFO - [1550/5000] loss=0.9216 aux=0.002991 R=2 |
| 2026-04-03 05:58:33 - ReXMoE - INFO - [1600/5000] loss=0.5969 aux=0.003708 R=2 |
| 2026-04-03 06:03:46 - ReXMoE - INFO - [1650/5000] loss=0.5240 aux=0.002518 R=2 |
| 2026-04-03 06:08:58 - ReXMoE - INFO - [1700/5000] loss=0.7681 aux=0.001785 R=2 |
| 2026-04-03 06:14:09 - ReXMoE - INFO - [1750/5000] loss=1.0812 aux=0.002899 R=2 |
| 2026-04-03 06:19:21 - ReXMoE - INFO - [1800/5000] loss=0.8171 aux=0.010986 R=2 |
| 2026-04-03 06:24:34 - ReXMoE - INFO - [1850/5000] loss=0.9029 aux=0.005371 R=2 |
| 2026-04-03 06:29:46 - ReXMoE - INFO - [1900/5000] loss=1.0440 aux=0.001839 R=2 |
| 2026-04-03 06:35:00 - ReXMoE - INFO - [1950/5000] loss=1.2026 aux=0.005096 R=2 |
| 2026-04-03 06:40:13 - ReXMoE - INFO - [2000/5000] loss=0.7174 aux=0.003372 R=2 |
| 2026-04-03 06:45:25 - ReXMoE - INFO - [2050/5000] loss=1.5737 aux=0.003571 R=2 |
| 2026-04-03 06:50:37 - ReXMoE - INFO - [2100/5000] loss=0.8508 aux=0.003403 R=2 |
| 2026-04-03 06:55:51 - ReXMoE - INFO - [2150/5000] loss=0.7965 aux=0.001656 R=2 |
| 2026-04-03 07:01:02 - ReXMoE - INFO - [2200/5000] loss=1.3079 aux=0.002747 R=2 |
| 2026-04-03 07:06:14 - ReXMoE - INFO - [2250/5000] loss=0.9750 aux=0.002228 R=2 |
| 2026-04-03 07:11:28 - ReXMoE - INFO - [2300/5000] loss=0.9549 aux=0.002228 R=2 |
| 2026-04-03 07:16:40 - ReXMoE - INFO - [2350/5000] loss=1.2216 aux=0.004089 R=2 |
| 2026-04-03 07:21:53 - ReXMoE - INFO - [2400/5000] loss=0.9801 aux=0.002289 R=2 |
| 2026-04-03 07:27:07 - ReXMoE - INFO - [2450/5000] loss=1.6587 aux=0.001602 R=2 |
| 2026-04-03 07:32:23 - ReXMoE - INFO - [2500/5000] loss=1.7420 aux=0.014648 R=3 |
| 2026-04-03 07:39:14 - ReXMoE - INFO - [2550/5000] loss=1.0498 aux=0.001801 R=3 |
| 2026-04-03 07:46:08 - ReXMoE - INFO - [2600/5000] loss=0.7848 aux=0.002792 R=3 |
| 2026-04-03 07:53:01 - ReXMoE - INFO - [2650/5000] loss=0.6119 aux=0.000992 R=3 |
| 2026-04-03 07:59:55 - ReXMoE - INFO - [2700/5000] loss=1.0871 aux=0.002014 R=3 |
| 2026-04-03 08:06:48 - ReXMoE - INFO - [2750/5000] loss=1.0422 aux=0.001411 R=3 |
| 2026-04-03 08:13:45 - ReXMoE - INFO - [2800/5000] loss=1.0147 aux=0.002762 R=3 |
| 2026-04-03 08:20:37 - ReXMoE - INFO - [2850/5000] loss=0.6756 aux=0.001953 R=3 |
| 2026-04-03 08:27:28 - ReXMoE - INFO - [2900/5000] loss=0.6243 aux=0.001671 R=3 |
| 2026-04-03 08:34:22 - ReXMoE - INFO - [2950/5000] loss=0.8838 aux=0.004974 R=3 |
| 2026-04-03 08:41:13 - ReXMoE - INFO - [3000/5000] loss=0.7627 aux=0.002060 R=3 |
| 2026-04-03 08:48:05 - ReXMoE - INFO - [3050/5000] loss=0.8120 aux=0.000668 R=3 |
| 2026-04-03 08:54:56 - ReXMoE - INFO - [3100/5000] loss=0.9701 aux=0.002121 R=3 |
| 2026-04-03 09:01:47 - ReXMoE - INFO - [3150/5000] loss=0.8151 aux=0.001289 R=3 |
| 2026-04-03 09:08:39 - ReXMoE - INFO - [3200/5000] loss=0.6943 aux=0.002777 R=3 |
| 2026-04-03 09:15:30 - ReXMoE - INFO - [3250/5000] loss=0.9401 aux=0.002350 R=3 |
| 2026-04-03 09:22:20 - ReXMoE - INFO - [3300/5000] loss=0.7034 aux=0.007935 R=3 |
| 2026-04-03 09:29:11 - ReXMoE - INFO - [3350/5000] loss=1.1980 aux=0.003006 R=3 |
| 2026-04-03 09:36:04 - ReXMoE - INFO - [3400/5000] loss=0.6413 aux=0.002045 R=3 |
| 2026-04-03 09:43:01 - ReXMoE - INFO - [3450/5000] loss=1.1729 aux=0.001686 R=3 |
| 2026-04-03 09:49:52 - ReXMoE - INFO - [3500/5000] loss=1.1667 aux=0.002045 R=3 |
| 2026-04-03 09:56:42 - ReXMoE - INFO - [3550/5000] loss=0.3543 aux=0.007324 R=3 |
| 2026-04-03 10:03:29 - ReXMoE - INFO - [3600/5000] loss=1.0002 aux=0.002792 R=3 |
| 2026-04-03 10:10:20 - ReXMoE - INFO - [3650/5000] loss=0.8748 aux=0.001503 R=3 |
| 2026-04-03 10:17:12 - ReXMoE - INFO - [3700/5000] loss=0.9026 aux=0.021118 R=3 |
| 2026-04-03 10:24:05 - ReXMoE - INFO - [3750/5000] loss=0.3710 aux=0.002182 R=3 |
| 2026-04-03 10:30:57 - ReXMoE - INFO - [3800/5000] loss=1.2199 aux=0.001564 R=3 |
| 2026-04-03 10:37:48 - ReXMoE - INFO - [3850/5000] loss=0.4812 aux=0.008057 R=3 |
| 2026-04-03 10:44:38 - ReXMoE - INFO - [3900/5000] loss=0.9683 aux=0.002487 R=3 |
| 2026-04-03 10:51:31 - ReXMoE - INFO - [3950/5000] loss=0.7649 aux=0.001732 R=3 |
| 2026-04-03 10:58:23 - ReXMoE - INFO - [4000/5000] loss=0.7234 aux=0.001839 R=3 |
| 2026-04-03 11:05:13 - ReXMoE - INFO - [4050/5000] loss=0.7793 aux=0.001289 R=3 |
| 2026-04-03 11:12:02 - ReXMoE - INFO - [4100/5000] loss=1.2237 aux=0.001968 R=3 |
| 2026-04-03 11:18:51 - ReXMoE - INFO - [4150/5000] loss=1.0040 aux=0.002701 R=3 |
| 2026-04-03 11:25:38 - ReXMoE - INFO - [4200/5000] loss=0.4700 aux=0.001945 R=3 |
| 2026-04-03 11:32:25 - ReXMoE - INFO - [4250/5000] loss=0.6833 aux=0.004486 R=3 |
| 2026-04-03 11:39:11 - ReXMoE - INFO - [4300/5000] loss=0.8191 aux=0.003754 R=3 |
| 2026-04-03 11:45:56 - ReXMoE - INFO - [4350/5000] loss=0.3914 aux=0.001312 R=3 |
| 2026-04-03 11:52:41 - ReXMoE - INFO - [4400/5000] loss=0.9623 aux=0.001854 R=3 |
| 2026-04-03 11:59:28 - ReXMoE - INFO - [4450/5000] loss=0.6550 aux=0.005615 R=3 |
| 2026-04-03 12:06:15 - ReXMoE - INFO - [4500/5000] loss=0.9616 aux=0.002777 R=3 |
| 2026-04-03 12:13:01 - ReXMoE - INFO - [4550/5000] loss=0.5557 aux=0.008789 R=3 |
| 2026-04-03 12:19:46 - ReXMoE - INFO - [4600/5000] loss=0.6275 aux=0.018555 R=3 |
| 2026-04-03 12:26:33 - ReXMoE - INFO - [4650/5000] loss=1.2395 aux=0.001549 R=3 |
| 2026-04-03 12:33:18 - ReXMoE - INFO - [4700/5000] loss=0.6769 aux=0.002060 R=3 |
| 2026-04-03 12:40:05 - ReXMoE - INFO - [4750/5000] loss=1.1499 aux=0.006348 R=3 |
| 2026-04-03 12:46:50 - ReXMoE - INFO - [4800/5000] loss=0.7449 aux=0.001022 R=3 |
| 2026-04-03 12:53:34 - ReXMoE - INFO - [4850/5000] loss=0.8246 aux=0.001823 R=3 |
| 2026-04-03 13:00:22 - ReXMoE - INFO - [4900/5000] loss=0.9550 aux=0.002029 R=3 |
| 2026-04-03 13:07:10 - ReXMoE - INFO - [4950/5000] loss=1.2535 aux=0.001610 R=3 |
| 2026-04-03 13:13:50 - ReXMoE - INFO - |
| [Step 5000/5000] Running evaluation at eval_steps... |
| 2026-04-03 13:13:50 - ReXMoE - INFO - |
| Evaluating model with 3 sample prompts... |
| 2026-04-03 13:13:52 - ReXMoE - INFO - |
| --- Prompt 1/3 --- |
| 2026-04-03 13:13:52 - ReXMoE - INFO - Instruction: What is the capital of France? |
| 2026-04-03 13:13:52 - ReXMoE - INFO - Input: None |
| 2026-04-03 13:13:52 - ReXMoE - INFO - Generated completion (len 9): The capital of France is Paris. |
| 2026-04-03 13:14:11 - ReXMoE - INFO - |
| --- Prompt 2/3 --- |
| 2026-04-03 13:14:11 - ReXMoE - INFO - Instruction: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense. What will most likely result if a high-pressure system remains in an area for a long period of time? |
| A. fog |
| B. rain |
| C. drought |
| D. tornado |
| Answer: |
| 2026-04-03 13:14:11 - ReXMoE - INFO - Input: None |
| 2026-04-03 13:14:11 - ReXMoE - INFO - Generated completion (len 77): A. fog |
| |
| High-pressure systems often lead to fog formation because they can hold moisture and prevent it from evaporating. This can occur when the high-pressure system remains in an area for a long period of time. Fog forms when moist air cools and condenses into water droplets near the surface of the Earth. |
| 2026-04-03 13:14:13 - ReXMoE - INFO - |
| --- Prompt 3/3 --- |
| 2026-04-03 13:14:13 - ReXMoE - INFO - Instruction: Given the fact: predators eat prey |
| Question: Predators eat |
| A. lions |
| B. humans |
| C. bunnies |
| D. grass |
| Answer: |
| 2026-04-03 13:14:13 - ReXMoE - INFO - Input: None |
| 2026-04-03 13:14:13 - ReXMoE - INFO - Generated completion (len 7): C. bunnies |
| 2026-04-03 13:14:13 - ReXMoE - INFO - Evaluation of all 3 prompts complete. |
| 2026-04-03 13:14:13 - ReXMoE - INFO - |
| [Step 5000] Analyzing routing patterns at eval_steps... |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| Analyzing ACTUAL routing patterns from 10 batches (15,294 tokens) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Current reuse scale: R=3 |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| [IG-MET Pruning Report]: |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Global: 0/0 UNIQUE experts pruned (0.0%) | threshold=-1.000000 |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Cross-Layer Routing Distribution (ACTUAL selections): |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Same layer (i): 781,056 ( 29.8%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Previous layer (i-1): 965,741 ( 36.8%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Next layer (i+1): 815,206 ( 31.1%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Distant layers: 59,437 ( 2.3%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Sample Layer-Specific Routing Patterns: |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| Layer 8: |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 7 from layer 9 ( L9): 5,937 times ( 38.8%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 7 from layer 7 ( L7): 5,895 times ( 38.5%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 14 from layer 9 ( L9): 5,822 times ( 38.1%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 14 from layer 7 ( L7): 5,618 times ( 36.7%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 2 from layer 7 ( L7): 4,200 times ( 27.5%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| Layer 16: |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 8 from layer 15 ( L15): 8,999 times ( 58.8%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 8 from layer 17 ( L17): 7,847 times ( 51.3%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 10 from layer 15 ( L15): 6,002 times ( 39.2%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 10 from layer 17 ( L17): 5,879 times ( 38.4%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 15 from layer 15 ( L15): 4,030 times ( 26.4%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| Layer 24: |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 8 from layer 23 ( L23): 9,213 times ( 60.2%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 9 from layer 23 ( L23): 7,912 times ( 51.7%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 8 from layer 25 ( L25): 6,819 times ( 44.6%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 9 from layer 25 ( L25): 6,403 times ( 41.9%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Expert 8 from layer 24 (same): 4,350 times ( 28.4%) |
| 2026-04-03 13:14:37 - ReXMoE - INFO - ✅ Cross-layer expert reuse detected: 70.2% of routing uses adjacent layers |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| [Step 5000] Saving checkpoint at eval_steps to ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3... |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.0.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.0.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.1.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.1.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.2.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.2.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.3.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.3.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.4.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.4.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.5.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.5.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.6.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.6.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.7.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.7.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.8.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.8.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.9.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.9.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.10.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.10.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.11.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.11.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.12.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.12.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.13.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.13.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.14.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.14.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.15.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.15.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.16.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.16.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.17.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.17.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.18.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.18.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.19.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.19.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.20.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.20.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.21.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.21.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.22.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.22.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.23.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.23.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.24.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.24.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.25.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.25.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.26.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.26.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.27.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.27.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.28.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.28.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.29.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.29.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.30.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.30.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.31.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.31.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:14:37 - ReXMoE - INFO - ✓ Saved trained router weights: 96 parameters |
| 2026-04-03 13:14:37 - ReXMoE - INFO - File: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/rexmoe_routers.pt |
| 2026-04-03 13:14:37 - ReXMoE - INFO - Size: 12.03 MB |
| 2026-04-03 13:14:37 - ReXMoE - INFO - |
| Also saving full model with ReXMoE architecture... |
| 2026-04-03 13:14:39 - ReXMoE - INFO - |
| Merging LoRA adapters into base weights and saving to: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/merged |
| 2026-04-03 13:15:00 - ReXMoE - INFO - ✓ Saved merged full model (base+routers+LoRA) for one-step loading |
| 2026-04-03 13:15:00 - ReXMoE - INFO - |
| ============================================================ |
| 2026-04-03 13:15:00 - ReXMoE - INFO - Epoch 1 Summary: |
| 2026-04-03 13:15:00 - ReXMoE - INFO - Average LM Loss: 0.9498 |
| 2026-04-03 13:15:00 - ReXMoE - INFO - Average Aux Loss: 0.008843 |
| 2026-04-03 13:15:00 - ReXMoE - INFO - Average Total Loss: 0.9586 |
| 2026-04-03 13:15:00 - ReXMoE - INFO - Final R: 3 |
| 2026-04-03 13:15:00 - ReXMoE - INFO - |
| Evaluating model with 3 sample prompts... |
| 2026-04-03 13:15:02 - ReXMoE - INFO - |
| --- Prompt 1/3 --- |
| 2026-04-03 13:15:02 - ReXMoE - INFO - Instruction: What is the capital of France? |
| 2026-04-03 13:15:02 - ReXMoE - INFO - Input: None |
| 2026-04-03 13:15:02 - ReXMoE - INFO - Generated completion (len 9): The capital of France is Paris. |
| 2026-04-03 13:15:04 - ReXMoE - INFO - |
| --- Prompt 2/3 --- |
| 2026-04-03 13:15:04 - ReXMoE - INFO - Instruction: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense. What will most likely result if a high-pressure system remains in an area for a long period of time? |
| A. fog |
| B. rain |
| C. drought |
| D. tornado |
| Answer: |
| 2026-04-03 13:15:04 - ReXMoE - INFO - Input: None |
| 2026-04-03 13:15:04 - ReXMoE - INFO - Generated completion (len 5): A. fog |
| 2026-04-03 13:15:05 - ReXMoE - INFO - |
| --- Prompt 3/3 --- |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Instruction: Given the fact: predators eat prey |
| Question: Predators eat |
| A. lions |
| B. humans |
| C. bunnies |
| D. grass |
| Answer: |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Input: None |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Generated completion (len 7): C. bunnies |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Evaluation of all 3 prompts complete. |
| 2026-04-03 13:15:05 - ReXMoE - INFO - New best epoch 1 with avg LM loss 0.9498 — saving checkpoint to ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3 |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.0.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.0.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.1.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.1.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.2.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.2.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.3.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.3.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.4.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.4.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.5.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.5.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.6.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.6.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.7.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.7.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.8.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.8.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.9.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.9.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.10.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.10.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.11.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.11.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.12.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.12.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.13.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.13.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.14.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.14.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.15.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.15.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.16.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.16.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.17.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.17.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.18.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.18.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.19.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.19.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.20.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.20.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.21.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.21.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.22.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.22.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.23.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.23.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.24.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.24.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.25.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.25.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.26.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.26.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.27.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.27.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.28.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.28.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.29.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.29.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.30.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.30.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.31.block_sparse_moe.router.ema_utilization with shape torch.Size([48]) for pruning evaluation |
| 2026-04-03 13:15:05 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.31.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-03 13:15:06 - ReXMoE - INFO - ✓ Saved trained router weights: 96 parameters |
| 2026-04-03 13:15:06 - ReXMoE - INFO - File: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/rexmoe_routers.pt |
| 2026-04-03 13:15:06 - ReXMoE - INFO - Size: 12.03 MB |
| 2026-04-03 13:15:06 - ReXMoE - INFO - |
| Also saving full model with ReXMoE architecture... |
| 2026-04-03 13:15:06 - ReXMoE - INFO - |
| Merging LoRA adapters into base weights and saving to: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/merged |
| 2026-04-03 13:15:44 - ReXMoE - INFO - ✓ Saved merged full model (base+routers+LoRA) for one-step loading |
| 2026-04-03 13:15:44 - ReXMoE - INFO - |
| 📊 Convergence Metrics: |
| 2026-04-03 13:15:44 - ReXMoE - INFO - Convergence Metrics: |
| 2026-04-03 13:15:44 - ReXMoE - INFO - Avg Router Grad Norm: 0.084278 |
| 2026-04-03 13:15:44 - ReXMoE - INFO - Current Learning Rate: 2.00e-05 |
| 2026-04-03 13:15:44 - ReXMoE - INFO - ℹ️ Collecting convergence data (need 5 epochs minimum)... |
| 2026-04-03 13:15:44 - ReXMoE - INFO - Routing Pattern Analysis (Epoch 1): |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| Analyzing ACTUAL routing patterns from 10 batches (17,341 tokens) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Current reuse scale: R=3 |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| [IG-MET Pruning Report]: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Global: 0/0 UNIQUE experts pruned (0.0%) | threshold=-1.000000 |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Cross-Layer Routing Distribution (ACTUAL selections): |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Same layer (i): 869,591 ( 33.2%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Previous layer (i-1): 896,913 ( 34.2%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Next layer (i+1): 797,210 ( 30.4%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Distant layers: 57,726 ( 2.2%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Sample Layer-Specific Routing Patterns: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| Layer 8: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 7 from layer 9 ( L9): 6,917 times ( 39.9%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 14 from layer 9 ( L9): 6,553 times ( 37.8%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 7 from layer 7 ( L7): 6,305 times ( 36.4%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 14 from layer 7 ( L7): 5,503 times ( 31.7%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 2 from layer 7 ( L7): 3,988 times ( 23.0%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| Layer 16: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 8 from layer 15 ( L15): 8,873 times ( 51.2%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 8 from layer 17 ( L17): 8,226 times ( 47.4%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 10 from layer 15 ( L15): 5,752 times ( 33.2%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 10 from layer 17 ( L17): 4,996 times ( 28.8%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 9 from layer 16 (same): 3,718 times ( 21.4%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| Layer 24: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 8 from layer 23 ( L23): 9,676 times ( 55.8%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 8 from layer 25 ( L25): 7,087 times ( 40.9%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 9 from layer 23 ( L23): 6,982 times ( 40.3%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 9 from layer 25 ( L25): 4,908 times ( 28.3%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Expert 4 from layer 24 (same): 3,902 times ( 22.5%) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - ✅ Cross-layer expert reuse detected: 66.8% of routing uses adjacent layers |
| 2026-04-03 13:15:59 - ReXMoE - INFO - LR stepped to: 2.00e-05 |
| 2026-04-03 13:15:59 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Training Convergence Summary |
| 2026-04-03 13:15:59 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Router Gradient Norms Over Epochs: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Epoch 1: 0.084278 |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Auxiliary Loss Over Epochs: |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Epoch 1: 0.008843 |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Convergence Status: Insufficient data (< 5 epochs) |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| Saving trained router weights only... |
| 2026-04-03 13:15:59 - ReXMoE - INFO - ✓ Saved trained router weights: 96 parameters |
| 2026-04-03 13:15:59 - ReXMoE - INFO - File: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/rexmoe_routers.pt |
| 2026-04-03 13:15:59 - ReXMoE - INFO - Size: 12.03 MB |
| 2026-04-03 13:15:59 - ReXMoE - INFO - |
| Also saving full model with ReXMoE architecture... |
| 2026-04-03 13:16:00 - ReXMoE - INFO - |
| Merging LoRA adapters into base weights and saving to: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3/merged |
| 2026-04-03 13:16:32 - ReXMoE - INFO - ✓ Saved merged full model (base+routers+LoRA) for one-step loading |
| 2026-04-03 13:16:32 - ReXMoE - INFO - ================================================================================ |
| 2026-04-03 13:16:32 - ReXMoE - INFO - ✓ Training complete. Two checkpoint formats saved: |
| 2026-04-03 13:16:32 - ReXMoE - INFO - 1. Router weights only: rexmoe_routers.pt (portable) |
| 2026-04-03 13:16:32 - ReXMoE - INFO - 2. Full model: pytorch_model.bin (requires rexmoe_architecture.py) |
| 2026-04-03 13:16:32 - ReXMoE - INFO - |
| Checkpoint directory: ./0304_033137_10_rexmoe_natural_phi_mini_moe_R3 |
| 2026-04-03 13:16:32 - ReXMoE - INFO - Full model size: 0.00 GB |
| 2026-04-03 13:16:32 - ReXMoE - INFO - ================================================================================ |
| |
