| 2026-04-02 04:16:28 - ReXMoE - INFO - ================================================================================ |
| 2026-04-02 04:16:28 - ReXMoE - INFO - ReXMoE Training Log - 0204_041628 |
| 2026-04-02 04:16:28 - ReXMoE - INFO - Log file: ./0204_041628_10_rexmoe_natural_phi_mini_moe_R2/logs/rexmoe_training_0204_041628.log |
| 2026-04-02 04:16:28 - ReXMoE - INFO - ================================================================================ |
| 2026-04-02 04:16:28 - ReXMoE - INFO - ================================================================================ |
| 2026-04-02 04:16:28 - ReXMoE - INFO - ReXMoE Cross-Layer Expert Reuse Training |
| 2026-04-02 04:16:28 - ReXMoE - INFO - ================================================================================ |
| 2026-04-02 04:16:28 - ReXMoE - INFO - MET enabled: False |
| 2026-04-02 04:16:28 - ReXMoE - INFO - |
| Configuration: |
| Model: microsoft/Phi-mini-MoE-instruct |
| Dataset: ../dataset/alpaca_data_cleaned.json |
| Dataset mode: IF_2 |
| Reuse Scale (R): 2 |
| Prune Ratio (MET): N/A |
| Epochs: 2 |
| Num of samples: 20000 |
| Batch Size: 2 |
| Sequence Length: 1024 |
| Learning Rate: 5e-05 |
| PSR Enabled: True |
| LR Scheduler: True |
| Save Path: ./0204_041628_10_rexmoe_natural_phi_mini_moe_R2 |
| Gradient Checkpointing: False |
| LoRA Rank: 16 (Full LoRA: True) |
| LoRA Alpha: 32 |
| MET Enabled: False (Mask Ratio: 0.1, Warmup: 0.5) |
| Log File: ./0204_041628_10_rexmoe_natural_phi_mini_moe_R2/logs/rexmoe_training_0204_041628.log |
| Aux loss weight: 0.05 |
|
|
| 2026-04-02 04:16:28 - ReXMoE - INFO - 💻 Using device: cuda) |
| 2026-04-02 04:16:28 - ReXMoE - INFO - GPU: NVIDIA RTX A6000, Memory: 47.53 GB |
| 2026-04-02 04:16:37 - ReXMoE - INFO - [5/7] Setting up optimizer and dataset... |
| 2026-04-02 04:16:37 - ReXMoE - INFO - Using 8-bit AdamW optimizer |
| 2026-04-02 04:16:37 - ReXMoE - INFO - LR Scheduler: CosineAnnealingLR (5e-05 → 5e-06) |
| 2026-04-02 04:16:45 - ReXMoE - INFO - |
| First batch statistics: |
| 2026-04-02 04:16:45 - ReXMoE - INFO - LM Loss: 1.0013 |
| 2026-04-02 04:16:45 - ReXMoE - INFO - Aux Loss: 0.108887 |
| 2026-04-02 04:16:45 - ReXMoE - INFO - Total Loss: 1.1102 |
| 2026-04-02 04:16:45 - ReXMoE - INFO - Current R: 2 |
| 2026-04-02 04:16:45 - ReXMoE - INFO - Active experts per layer: 32 |
| 2026-04-02 04:16:45 - ReXMoE - INFO - Gradient norm: 1.0000 |
| 2026-04-02 04:16:45 - ReXMoE - INFO - |
| |
| 2026-04-02 04:21:32 - ReXMoE - INFO - [100/10000] loss=2.2108 aux=0.046143 R=2 |
| 2026-04-02 04:26:17 - ReXMoE - INFO - [200/10000] loss=0.7772 aux=0.018188 R=2 |
| 2026-04-02 04:31:00 - ReXMoE - INFO - [300/10000] loss=0.9490 aux=0.021973 R=2 |
| 2026-04-02 04:35:41 - ReXMoE - INFO - [400/10000] loss=0.6892 aux=0.051025 R=2 |
| 2026-04-02 04:40:24 - ReXMoE - INFO - [500/10000] loss=1.5216 aux=0.014099 R=2 |
| 2026-04-02 04:45:08 - ReXMoE - INFO - [600/10000] loss=1.6429 aux=0.006897 R=2 |
| 2026-04-02 04:49:52 - ReXMoE - INFO - [700/10000] loss=1.0995 aux=0.008118 R=2 |
| 2026-04-02 04:54:33 - ReXMoE - INFO - [800/10000] loss=1.4248 aux=0.006561 R=2 |
| 2026-04-02 04:59:14 - ReXMoE - INFO - [900/10000] loss=1.7497 aux=0.008850 R=2 |
| 2026-04-02 05:03:55 - ReXMoE - INFO - Warmup completed at step 1000. Enabling FULL QLoRA with r = 16 and alpha = 32 on experts and updating optimizer... |
| 2026-04-02 05:04:09 - ReXMoE - INFO - Trainable params (routers + LoRA): 142082048 (1.8245%) |
| 2026-04-02 05:04:09 - ReXMoE - INFO - Sample trainable params after QLoRA: ['base_model.model.model.layers.0.self_attn.q_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.q_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.k_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.v_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.v_proj.lora_B.default.weight', 'base_model.model.model.layers.0.self_attn.o_proj.lora_A.default.weight', 'base_model.model.model.layers.0.self_attn.o_proj.lora_B.default.weight', 'base_model.model.model.layers.0.block_sparse_moe.gate.weight', 'base_model.model.model.layers.0.block_sparse_moe.experts.0.w1.lora_A.default.weight'] |
| 2026-04-02 05:04:16 - ReXMoE - INFO - [1000/10000] loss=0.7002 aux=0.005341 R=2 |
| 2026-04-02 05:13:07 - ReXMoE - INFO - [1100/10000] loss=0.6426 aux=0.007233 R=2 |
| 2026-04-02 05:21:43 - ReXMoE - INFO - [1200/10000] loss=0.5568 aux=0.007019 R=2 |
| 2026-04-02 05:30:24 - ReXMoE - INFO - [1300/10000] loss=0.8270 aux=0.006531 R=2 |
| 2026-04-02 05:39:34 - ReXMoE - INFO - [1400/10000] loss=0.8769 aux=0.005249 R=2 |
| 2026-04-02 05:48:47 - ReXMoE - INFO - [1500/10000] loss=0.6700 aux=0.010498 R=2 |
| 2026-04-02 05:57:41 - ReXMoE - INFO - [1600/10000] loss=0.6573 aux=0.004242 R=2 |
| 2026-04-02 06:06:26 - ReXMoE - INFO - [1700/10000] loss=0.8331 aux=0.002289 R=2 |
| 2026-04-02 06:15:13 - ReXMoE - INFO - [1800/10000] loss=1.0478 aux=0.010437 R=2 |
| 2026-04-02 06:24:01 - ReXMoE - INFO - [1900/10000] loss=1.7566 aux=0.002350 R=2 |
| 2026-04-02 06:32:51 - ReXMoE - INFO - [2000/10000] loss=0.9031 aux=0.001572 R=2 |
| 2026-04-02 06:41:39 - ReXMoE - INFO - [2100/10000] loss=0.8975 aux=0.005035 R=2 |
| 2026-04-02 06:50:29 - ReXMoE - INFO - [2200/10000] loss=1.4439 aux=0.021606 R=2 |
| 2026-04-02 06:59:22 - ReXMoE - INFO - [2300/10000] loss=1.1146 aux=0.003876 R=2 |
| 2026-04-02 07:08:13 - ReXMoE - INFO - [2400/10000] loss=1.2936 aux=0.003204 R=2 |
| 2026-04-02 07:17:11 - ReXMoE - INFO - [2500/10000] loss=0.5071 aux=0.002731 R=2 |
| 2026-04-02 07:26:02 - ReXMoE - INFO - [2600/10000] loss=0.6228 aux=0.004730 R=2 |
| 2026-04-02 07:34:42 - ReXMoE - INFO - [2700/10000] loss=0.6085 aux=0.099609 R=2 |
| 2026-04-02 07:43:17 - ReXMoE - INFO - [2800/10000] loss=0.6629 aux=0.003265 R=2 |
| 2026-04-02 07:51:52 - ReXMoE - INFO - [2900/10000] loss=0.4027 aux=0.002472 R=2 |
| 2026-04-02 08:00:24 - ReXMoE - INFO - [3000/10000] loss=0.5227 aux=0.002045 R=2 |
| 2026-04-02 08:09:03 - ReXMoE - INFO - [3100/10000] loss=1.5149 aux=0.001709 R=2 |
| 2026-04-02 08:17:32 - ReXMoE - INFO - [3200/10000] loss=0.8226 aux=0.006104 R=2 |
| 2026-04-02 08:26:01 - ReXMoE - INFO - [3300/10000] loss=1.2210 aux=0.001472 R=2 |
| 2026-04-02 08:34:30 - ReXMoE - INFO - [3400/10000] loss=0.8192 aux=0.008179 R=2 |
| 2026-04-02 08:43:04 - ReXMoE - INFO - [3500/10000] loss=1.1357 aux=0.001709 R=2 |
| 2026-04-02 08:51:39 - ReXMoE - INFO - [3600/10000] loss=2.2026 aux=0.003387 R=2 |
| 2026-04-02 09:00:07 - ReXMoE - INFO - [3700/10000] loss=1.1631 aux=0.010376 R=2 |
| 2026-04-02 09:08:41 - ReXMoE - INFO - [3800/10000] loss=1.9066 aux=0.001495 R=2 |
| 2026-04-02 09:17:11 - ReXMoE - INFO - [3900/10000] loss=0.4675 aux=0.002136 R=2 |
| 2026-04-02 09:25:43 - ReXMoE - INFO - [4000/10000] loss=0.6906 aux=0.002838 R=2 |
| 2026-04-02 09:34:13 - ReXMoE - INFO - [4100/10000] loss=1.1627 aux=0.003693 R=2 |
| 2026-04-02 09:42:49 - ReXMoE - INFO - [4200/10000] loss=1.0517 aux=0.001595 R=2 |
| 2026-04-02 09:51:25 - ReXMoE - INFO - [4300/10000] loss=0.4299 aux=0.003799 R=2 |
| 2026-04-02 09:59:56 - ReXMoE - INFO - [4400/10000] loss=0.6066 aux=0.002121 R=2 |
| 2026-04-02 10:08:25 - ReXMoE - INFO - [4500/10000] loss=1.0687 aux=0.027466 R=2 |
| 2026-04-02 10:16:55 - ReXMoE - INFO - [4600/10000] loss=0.6690 aux=0.004272 R=2 |
| 2026-04-02 10:25:25 - ReXMoE - INFO - [4700/10000] loss=0.7443 aux=0.001610 R=2 |
| 2026-04-02 10:33:55 - ReXMoE - INFO - [4800/10000] loss=0.5073 aux=0.001488 R=2 |
| 2026-04-02 10:42:24 - ReXMoE - INFO - [4900/10000] loss=0.8397 aux=0.004517 R=2 |
| 2026-04-02 10:50:56 - ReXMoE - INFO - [5000/10000] loss=0.4906 aux=0.001007 R=2 |
| 2026-04-02 10:59:27 - ReXMoE - INFO - [5100/10000] loss=0.6861 aux=0.002716 R=2 |
| 2026-04-02 11:07:57 - ReXMoE - INFO - [5200/10000] loss=0.4963 aux=0.002823 R=2 |
| 2026-04-02 11:16:24 - ReXMoE - INFO - [5300/10000] loss=0.7556 aux=0.002533 R=2 |
| 2026-04-02 11:24:50 - ReXMoE - INFO - [5400/10000] loss=0.6053 aux=0.020874 R=2 |
| 2026-04-02 11:33:16 - ReXMoE - INFO - [5500/10000] loss=2.0345 aux=0.000778 R=2 |
| 2026-04-02 11:41:41 - ReXMoE - INFO - [5600/10000] loss=0.7234 aux=0.025269 R=2 |
| 2026-04-02 11:50:07 - ReXMoE - INFO - [5700/10000] loss=0.3542 aux=0.000467 R=2 |
| 2026-04-02 11:58:34 - ReXMoE - INFO - [5800/10000] loss=0.8516 aux=0.002274 R=2 |
| 2026-04-02 12:07:01 - ReXMoE - INFO - [5900/10000] loss=0.5901 aux=0.001198 R=2 |
| 2026-04-02 12:15:26 - ReXMoE - INFO - [6000/10000] loss=1.3560 aux=0.001457 R=2 |
| 2026-04-02 12:23:51 - ReXMoE - INFO - [6100/10000] loss=1.2492 aux=0.000854 R=2 |
| 2026-04-02 12:32:18 - ReXMoE - INFO - [6200/10000] loss=0.3890 aux=0.001007 R=2 |
| 2026-04-02 12:40:43 - ReXMoE - INFO - [6300/10000] loss=0.5883 aux=0.003555 R=2 |
| 2026-04-02 12:49:08 - ReXMoE - INFO - [6400/10000] loss=0.5396 aux=0.001968 R=2 |
| 2026-04-02 12:57:35 - ReXMoE - INFO - [6500/10000] loss=1.0615 aux=0.000652 R=2 |
| 2026-04-02 13:06:04 - ReXMoE - INFO - [6600/10000] loss=0.8583 aux=0.003906 R=2 |
| 2026-04-02 13:14:33 - ReXMoE - INFO - [6700/10000] loss=0.4472 aux=0.001663 R=2 |
| 2026-04-02 13:23:03 - ReXMoE - INFO - [6800/10000] loss=1.3548 aux=0.001114 R=2 |
| 2026-04-02 13:31:35 - ReXMoE - INFO - [6900/10000] loss=1.0129 aux=0.001389 R=2 |
| 2026-04-02 13:40:04 - ReXMoE - INFO - [7000/10000] loss=1.3876 aux=0.000656 R=2 |
| 2026-04-02 13:48:35 - ReXMoE - INFO - [7100/10000] loss=1.0568 aux=0.002686 R=2 |
| 2026-04-02 13:57:05 - ReXMoE - INFO - [7200/10000] loss=1.7856 aux=0.001724 R=2 |
| 2026-04-02 14:05:34 - ReXMoE - INFO - [7300/10000] loss=0.9223 aux=0.000748 R=2 |
| 2026-04-02 14:14:03 - ReXMoE - INFO - [7400/10000] loss=0.3757 aux=0.021851 R=2 |
| 2026-04-02 14:22:34 - ReXMoE - INFO - [7500/10000] loss=0.8600 aux=0.010559 R=2 |
| 2026-04-02 14:31:04 - ReXMoE - INFO - [7600/10000] loss=0.8164 aux=0.002304 R=2 |
| 2026-04-02 14:39:35 - ReXMoE - INFO - [7700/10000] loss=1.2134 aux=0.001442 R=2 |
| 2026-04-02 14:48:04 - ReXMoE - INFO - [7800/10000] loss=0.8161 aux=0.005493 R=2 |
| 2026-04-02 14:56:35 - ReXMoE - INFO - [7900/10000] loss=1.6585 aux=0.001915 R=2 |
| 2026-04-02 15:05:04 - ReXMoE - INFO - [8000/10000] loss=1.2704 aux=0.000740 R=2 |
| 2026-04-02 15:13:35 - ReXMoE - INFO - [8100/10000] loss=0.5500 aux=0.001045 R=2 |
| 2026-04-02 15:22:06 - ReXMoE - INFO - [8200/10000] loss=1.5001 aux=0.001602 R=2 |
| 2026-04-02 15:30:37 - ReXMoE - INFO - [8300/10000] loss=0.5917 aux=0.000664 R=2 |
| 2026-04-02 15:39:07 - ReXMoE - INFO - [8400/10000] loss=0.1938 aux=0.001656 R=2 |
| 2026-04-02 15:47:35 - ReXMoE - INFO - [8500/10000] loss=0.6856 aux=0.006897 R=2 |
| 2026-04-02 15:56:03 - ReXMoE - INFO - [8600/10000] loss=1.3575 aux=0.001816 R=2 |
| 2026-04-02 16:04:31 - ReXMoE - INFO - [8700/10000] loss=0.4273 aux=0.003571 R=2 |
| 2026-04-02 16:13:00 - ReXMoE - INFO - [8800/10000] loss=0.7848 aux=0.001518 R=2 |
| 2026-04-02 16:21:28 - ReXMoE - INFO - [8900/10000] loss=0.4080 aux=0.001038 R=2 |
| 2026-04-02 16:29:58 - ReXMoE - INFO - [9000/10000] loss=0.5492 aux=0.005341 R=2 |
| 2026-04-02 16:38:27 - ReXMoE - INFO - [9100/10000] loss=1.4560 aux=0.000690 R=2 |
| 2026-04-02 16:46:57 - ReXMoE - INFO - [9200/10000] loss=0.8392 aux=0.000916 R=2 |
| 2026-04-02 16:55:25 - ReXMoE - INFO - [9300/10000] loss=1.3378 aux=0.002823 R=2 |
| 2026-04-02 17:03:56 - ReXMoE - INFO - [9400/10000] loss=1.1250 aux=0.001312 R=2 |
| 2026-04-02 17:12:27 - ReXMoE - INFO - [9500/10000] loss=0.9452 aux=0.000587 R=2 |
| 2026-04-02 17:20:56 - ReXMoE - INFO - [9600/10000] loss=0.3425 aux=0.015869 R=2 |
| 2026-04-02 17:29:25 - ReXMoE - INFO - [9700/10000] loss=1.1245 aux=0.001831 R=2 |
| 2026-04-02 17:37:52 - ReXMoE - INFO - [9800/10000] loss=0.8069 aux=0.001640 R=2 |
| 2026-04-02 17:46:19 - ReXMoE - INFO - [9900/10000] loss=1.7475 aux=0.002563 R=2 |
| 2026-04-02 17:54:44 - ReXMoE - INFO - |
| [Step 10000/20000] Running evaluation at eval_steps... |
| 2026-04-02 17:54:44 - ReXMoE - INFO - |
| Evaluating model with 3 sample prompts... |
| 2026-04-02 17:54:47 - ReXMoE - INFO - |
| --- Prompt 1/3 --- |
| 2026-04-02 17:54:47 - ReXMoE - INFO - Instruction: What is the capital of France? |
| 2026-04-02 17:54:47 - ReXMoE - INFO - Input: None |
| 2026-04-02 17:54:47 - ReXMoE - INFO - Generated completion (len 9): The capital of France is Paris. |
| 2026-04-02 17:54:49 - ReXMoE - INFO - |
| --- Prompt 2/3 --- |
| 2026-04-02 17:54:49 - ReXMoE - INFO - Instruction: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense. What will most likely result if a high-pressure system remains in an area for a long period of time? |
| A. fog |
| B. rain |
| C. drought |
| D. tornado |
| Answer: |
| 2026-04-02 17:54:49 - ReXMoE - INFO - Input: None |
| 2026-04-02 17:54:49 - ReXMoE - INFO - Generated completion (len 6): C. drought |
| 2026-04-02 17:54:50 - ReXMoE - INFO - |
| --- Prompt 3/3 --- |
| 2026-04-02 17:54:50 - ReXMoE - INFO - Instruction: Given the fact: predators eat prey |
| Question: Predators eat |
| A. lions |
| B. humans |
| C. bunnies |
| D. grass |
| Answer: |
| 2026-04-02 17:54:50 - ReXMoE - INFO - Input: None |
| 2026-04-02 17:54:50 - ReXMoE - INFO - Generated completion (len 7): C. bunnies |
| 2026-04-02 17:54:50 - ReXMoE - INFO - Evaluation of all 3 prompts complete. |
| 2026-04-02 17:54:50 - ReXMoE - INFO - |
| [Step 10000] Analyzing routing patterns at eval_steps... |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| Analyzing ACTUAL routing patterns from 10 batches (7,610 tokens) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Current reuse scale: R=2 |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| [IG-MET Pruning Report]: |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Global: 0/0 UNIQUE experts pruned (0.0%) | threshold=-1.000000 |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Cross-Layer Routing Distribution (ACTUAL selections): |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Same layer (i): 639,745 ( 48.8%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Previous layer (i-1): 18,093 ( 1.4%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Next layer (i+1): 652,882 ( 49.8%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Sample Layer-Specific Routing Patterns: |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| Layer 8: |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 14 from layer 9 ( L9): 5,114 times ( 67.2%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 2 from layer 9 ( L9): 3,743 times ( 49.2%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 7 from layer 9 ( L9): 3,668 times ( 48.2%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 11 from layer 9 ( L9): 2,931 times ( 38.5%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 13 from layer 9 ( L9): 2,786 times ( 36.6%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| Layer 16: |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 8 from layer 17 ( L17): 6,835 times ( 89.8%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 8 from layer 16 (same): 5,658 times ( 74.3%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 10 from layer 17 ( L17): 3,268 times ( 42.9%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 1 from layer 17 ( L17): 2,518 times ( 33.1%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 9 from layer 16 (same): 1,967 times ( 25.8%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| Layer 24: |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 8 from layer 25 ( L25): 6,353 times ( 83.5%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 8 from layer 24 (same): 5,423 times ( 71.3%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 9 from layer 25 ( L25): 3,424 times ( 45.0%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 10 from layer 24 (same): 3,301 times ( 43.4%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Expert 4 from layer 24 (same): 1,978 times ( 26.0%) |
| 2026-04-02 17:55:07 - ReXMoE - INFO - ✅ Cross-layer expert reuse detected: 51.2% of routing uses adjacent layers |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| [Step 10000] Saving checkpoint at eval_steps to ./0204_041628_10_rexmoe_natural_phi_mini_moe_R2... |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.0.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.0.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.1.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.1.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.2.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.2.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.3.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.3.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.4.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.4.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.5.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.5.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.6.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.6.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.7.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.7.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.8.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.8.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.9.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.9.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.10.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.10.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.11.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.11.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.12.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.12.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.13.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.13.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.14.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.14.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.15.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.15.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.16.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.16.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.17.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.17.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.18.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.18.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.19.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.19.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.20.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.20.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.21.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.21.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.22.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.22.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.23.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.23.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.24.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.24.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.25.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.25.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.26.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.26.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.27.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.27.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.28.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.28.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.29.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.29.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.30.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.30.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.31.block_sparse_moe.router.ema_utilization with shape torch.Size([32]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Saving buffer base_model.model.model.layers.31.block_sparse_moe.router.mask_threshold with shape torch.Size([]) for pruning evaluation |
| 2026-04-02 17:55:07 - ReXMoE - INFO - ✓ Saved trained router weights: 96 parameters |
| 2026-04-02 17:55:07 - ReXMoE - INFO - File: ./0204_041628_10_rexmoe_natural_phi_mini_moe_R2/rexmoe_routers.pt |
| 2026-04-02 17:55:07 - ReXMoE - INFO - Size: 8.03 MB |
| 2026-04-02 17:55:07 - ReXMoE - INFO - |
| Also saving full model with ReXMoE architecture... |
| 2026-04-02 17:55:08 - ReXMoE - INFO - |
| Merging LoRA adapters into base weights and saving to: ./0204_041628_10_rexmoe_natural_phi_mini_moe_R2/merged |
| 2026-04-02 17:55:25 - ReXMoE - INFO - ✓ Saved merged full model (base+routers+LoRA) for one-step loading |
| 2026-04-02 17:55:25 - ReXMoE - INFO - |
| ============================================================ |
| 2026-04-02 17:55:25 - ReXMoE - INFO - Epoch 1 Summary: |
| 2026-04-02 17:55:25 - ReXMoE - INFO - Average LM Loss: nan |
| 2026-04-02 17:55:25 - ReXMoE - INFO - Average Aux Loss: 0.007334 |
| 2026-04-02 17:55:25 - ReXMoE - INFO - Average Total Loss: nan |
| 2026-04-02 17:55:25 - ReXMoE - INFO - Final R: 2 |
| 2026-04-02 17:55:25 - ReXMoE - INFO - |
| Evaluating model with 3 sample prompts... |
| 2026-04-02 17:55:26 - ReXMoE - INFO - |
| --- Prompt 1/3 --- |
| 2026-04-02 17:55:26 - ReXMoE - INFO - Instruction: What is the capital of France? |
| 2026-04-02 17:55:26 - ReXMoE - INFO - Input: None |
| 2026-04-02 17:55:26 - ReXMoE - INFO - Generated completion (len 9): The capital of France is Paris. |
| 2026-04-02 17:55:28 - ReXMoE - INFO - |
| --- Prompt 2/3 --- |
| 2026-04-02 17:55:28 - ReXMoE - INFO - Instruction: High-pressure systems stop air from rising into the colder regions of the atmosphere where water can condense. What will most likely result if a high-pressure system remains in an area for a long period of time? |
| A. fog |
| B. rain |
| C. drought |
| D. tornado |
| Answer: |
| 2026-04-02 17:55:28 - ReXMoE - INFO - Input: None |
| 2026-04-02 17:55:28 - ReXMoE - INFO - Generated completion (len 6): C. drought |
| 2026-04-02 17:55:29 - ReXMoE - INFO - |
| --- Prompt 3/3 --- |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Instruction: Given the fact: predators eat prey |
| Question: Predators eat |
| A. lions |
| B. humans |
| C. bunnies |
| D. grass |
| Answer: |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Input: None |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Generated completion (len 7): C. bunnies |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Evaluation of all 3 prompts complete. |
| 2026-04-02 17:55:29 - ReXMoE - INFO - |
| 📊 Convergence Metrics: |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Convergence Metrics: |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Avg Router Grad Norm: 0.100342 |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Current Learning Rate: 5.00e-05 |
| 2026-04-02 17:55:29 - ReXMoE - INFO - ℹ️ Collecting convergence data (need 5 epochs minimum)... |
| 2026-04-02 17:55:29 - ReXMoE - INFO - Routing Pattern Analysis (Epoch 1): |
| 2026-04-02 17:55:39 - ReXMoE - INFO - |
| Analyzing ACTUAL routing patterns from 10 batches (7,965 tokens) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Current reuse scale: R=2 |
| 2026-04-02 17:55:39 - ReXMoE - INFO - |
| [IG-MET Pruning Report]: |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Global: 0/0 UNIQUE experts pruned (0.0%) | threshold=-1.000000 |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Cross-Layer Routing Distribution (ACTUAL selections): |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Same layer (i): 636,430 ( 48.6%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Previous layer (i-1): 19,584 ( 1.5%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Next layer (i+1): 654,706 ( 50.0%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Sample Layer-Specific Routing Patterns: |
| 2026-04-02 17:55:39 - ReXMoE - INFO - |
| Layer 8: |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 14 from layer 9 ( L9): 4,872 times ( 61.2%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 13 from layer 9 ( L9): 3,980 times ( 50.0%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 7 from layer 9 ( L9): 3,882 times ( 48.7%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 2 from layer 9 ( L9): 3,156 times ( 39.6%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 11 from layer 9 ( L9): 3,019 times ( 37.9%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - |
| Layer 16: |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 8 from layer 17 ( L17): 6,786 times ( 85.2%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 8 from layer 16 (same): 5,551 times ( 69.7%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 10 from layer 17 ( L17): 3,275 times ( 41.1%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 15 from layer 17 ( L17): 2,236 times ( 28.1%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 10 from layer 16 (same): 2,116 times ( 26.6%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - |
| Layer 24: |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 8 from layer 25 ( L25): 6,017 times ( 75.5%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 8 from layer 24 (same): 4,472 times ( 56.1%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 9 from layer 25 ( L25): 4,132 times ( 51.9%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 10 from layer 24 (same): 3,410 times ( 42.8%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - Expert 4 from layer 24 (same): 2,419 times ( 30.4%) |
| 2026-04-02 17:55:39 - ReXMoE - INFO - ✅ Cross-layer expert reuse detected: 51.4% of routing uses adjacent layers |
| 2026-04-02 17:55:39 - ReXMoE - INFO - LR stepped to: 5.00e-05 |
| 2026-04-02 18:00:28 - ReXMoE - INFO - [100/10000] loss=0.5745 aux=0.001076 R=2 |
| |
