/usr/local/lib/python3.11/dist-packages/torch/cuda/__init__.py:58: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/usr/local/lib/python3.11/dist-packages/torch/cuda/__init__.py:58: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]
/workspace/FLAME-MoE/.venv/lib/python3.11/site-packages/transformer_engine/pytorch/attention.py:108: UserWarning: To use flash-attn v3, please use the following commands to install: 
(1) pip install "git+https://github.com/Dao-AILab/flash-attention.git#egg=flashattn-hopper&subdirectory=hopper" 
(2) python_path=`python -c "import site; print(site.getsitepackages()[0])"` 
(3) mkdir -p $python_path/flashattn_hopper 
(4) wget -P $python_path/flashattn_hopper https://raw.githubusercontent.com/Dao-AILab/flash-attention/main/hopper/flash_attn_interface.py
  warnings.warn(
/usr/local/lib/python3.11/dist-packages/torch/distributed/checkpoint/state_dict_loader.py:140: UserWarning: torch.distributed is unavailable or uninitialized, assuming the intent is to load in a single process.
  warnings.warn(
[ok] dense_local: 18 rows (0 routed)
[ok] g1_moe: 1554 rows (1024 routed)
[ok] g1_temporal: 1554 rows (1024 routed)
[ok] g3_moe: 4626 rows (3072 routed)
[ok] g3_temporal: 4626 rows (3072 routed)
[ok] dense_1e19: 28 rows (0 routed)
[ok] moe_coarse_1e19: 2524 rows (1664 routed)
[ok] temporal_coarse_1e19: 2524 rows (1664 routed)
[ok] temporal_fine_1e19: 7516 rows (4992 routed)
[write] /workspace/FLAME-MoE/results/ablations/stability_weights.csv: 24970 rows
SANITY1 randn excess kurtosis = +0.0008  (|.|<0.05 -> True)
SANITY2 component row counts: {'dense_ffn': 60, 'routed': 16512, 'router_row': 8256, 'shared': 142}
SANITY3 g1_moe: routed 1024 == 8L x 64E x 2M -> True
SANITY3 g1_temporal: routed 1024 == 8L x 64E x 2M -> True
SANITY3 g3_moe: routed 3072 == 8L x 192E x 2M -> True
SANITY3 g3_temporal: routed 3072 == 8L x 192E x 2M -> True
SANITY3 moe_coarse_1e19: routed 1664 == 13L x 64E x 2M -> True
SANITY3 temporal_coarse_1e19: routed 1664 == 13L x 64E x 2M -> True
SANITY3 temporal_fine_1e19: routed 4992 == 13L x 192E x 2M -> True