diff --git "a/summary.md" "b/summary.md" new file mode 100644--- /dev/null +++ "b/summary.md" @@ -0,0 +1,1104 @@ +# Xplain Task Vector Analysis Summary + +Base/checkpoint groups: + +- `granite-3.3-8b-base`: `ibm-granite/granite-3.3-8b-base` + - `xplain1.4-7base-32k-step1743`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-7base/32k/checkpoint/ckpt_step_1743` +- `granite-3.1-8b-base`: `ibm-granite/granite-3.1-8b-base` + - `xplain1.4-6base-64k-step1743`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-64k/checkpoint/ckpt_step_1743` + - `xplain1.4-6base-32k-step1743`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-32k/checkpoint/ckpt_step_1743` + - `xplain1.4-6base-16k-step2319`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-16k/checkpoint/ckpt_step_2319` + - `xplain1.4-6base-8k-step2319`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-8k/checkpoint/ckpt_step_2319` + - `xplain1.3-6base-64k-step1647`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-64k/checkpoint/ckpt_step_1647` + - `xplain1.3-6base-32k-step1647`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-32k/checkpoint/ckpt_step_1647` + - `xplain1.3-6base-16k-step2199`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-16k/checkpoint/ckpt_step_2199` + - `xplain1.3-6base-8k-step2199`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-8k/checkpoint/ckpt_step_2199` +Base model: `multiple-base-models` + +Finetuned models: + +- `xplain1.4-7base-32k-step1743 vs granite-3.3-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-7base/32k/checkpoint/ckpt_step_1743` +- `xplain1.4-6base-64k-step1743 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-64k/checkpoint/ckpt_step_1743` +- `xplain1.4-6base-32k-step1743 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-32k/checkpoint/ckpt_step_1743` +- `xplain1.4-6base-16k-step2319 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-16k/checkpoint/ckpt_step_2319` +- `xplain1.4-6base-8k-step2319 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.4-6base-8k/checkpoint/ckpt_step_2319` +- `xplain1.3-6base-64k-step1647 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-64k/checkpoint/ckpt_step_1647` +- `xplain1.3-6base-32k-step1647 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-32k/checkpoint/ckpt_step_1647` +- `xplain1.3-6base-16k-step2199 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-16k/checkpoint/ckpt_step_2199` +- `xplain1.3-6base-8k-step2199 vs granite-3.1-8b-base`: `/wca4z-pvc-ckpt/shared/model_delivery_experiments/xplain-long/xplain1.3-6base-8k/checkpoint/ckpt_step_2199` + +Statistic mode: `signed` + +Sublayer granularity: `module` + +Include non-layer parameters: `True` + +Device: `cuda:0` + +Compute dtype: `torch.float32` + +Summary top-k per model: `50` + +## Output files + +- `parameter_stats.csv` +- `sublayer_stats.csv` +- `layerwise_stats.csv` +- `module_type_stats.csv` +- `sublayer_std.pdf` +- `layerwise_std.pdf` +- `module_type_std.pdf` + +## Compared parameter tensors + +- `xplain1.3-6base-16k-step2199`: 362 parameter tensors +- `xplain1.3-6base-32k-step1647`: 362 parameter tensors +- `xplain1.3-6base-64k-step1647`: 362 parameter tensors +- `xplain1.3-6base-8k-step2199`: 362 parameter tensors +- `xplain1.4-6base-16k-step2319`: 362 parameter tensors +- `xplain1.4-6base-32k-step1743`: 362 parameter tensors +- `xplain1.4-6base-64k-step1743`: 362 parameter tensors +- `xplain1.4-6base-8k-step2319`: 362 parameter tensors +- `xplain1.4-7base-32k-step1743`: 362 parameter tensors + +## Top sublayers by std + +### `xplain1.3-6base-16k-step2199` + + model sublayer block_label module_type count std stat_mode +xplain1.3-6base-16k-step2199 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.003985 signed +xplain1.3-6base-16k-step2199 Final Norm | model.norm.weight Final Norm final_norm 4096 0.003860 signed +xplain1.3-6base-16k-step2199 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.003845 signed +xplain1.3-6base-16k-step2199 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.003739 signed +xplain1.3-6base-16k-step2199 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.002777 signed +xplain1.3-6base-16k-step2199 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.002399 signed +xplain1.3-6base-16k-step2199 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.001991 signed +xplain1.3-6base-16k-step2199 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.001628 signed +xplain1.3-6base-16k-step2199 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.001616 signed +xplain1.3-6base-16k-step2199 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001615 signed +xplain1.3-6base-16k-step2199 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001575 signed +xplain1.3-6base-16k-step2199 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001559 signed +xplain1.3-6base-16k-step2199 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.001556 signed +xplain1.3-6base-16k-step2199 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001552 signed +xplain1.3-6base-16k-step2199 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001551 signed +xplain1.3-6base-16k-step2199 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001546 signed +xplain1.3-6base-16k-step2199 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.001496 signed +xplain1.3-6base-16k-step2199 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001495 signed +xplain1.3-6base-16k-step2199 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.001493 signed +xplain1.3-6base-16k-step2199 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.001488 signed +xplain1.3-6base-16k-step2199 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001483 signed +xplain1.3-6base-16k-step2199 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001481 signed +xplain1.3-6base-16k-step2199 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001466 signed +xplain1.3-6base-16k-step2199 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.001441 signed +xplain1.3-6base-16k-step2199 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001440 signed +xplain1.3-6base-16k-step2199 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001434 signed +xplain1.3-6base-16k-step2199 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001433 signed +xplain1.3-6base-16k-step2199 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001420 signed +xplain1.3-6base-16k-step2199 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.001418 signed +xplain1.3-6base-16k-step2199 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001414 signed +xplain1.3-6base-16k-step2199 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001409 signed +xplain1.3-6base-16k-step2199 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001401 signed +xplain1.3-6base-16k-step2199 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001398 signed +xplain1.3-6base-16k-step2199 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001396 signed +xplain1.3-6base-16k-step2199 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001388 signed +xplain1.3-6base-16k-step2199 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001375 signed +xplain1.3-6base-16k-step2199 Layer 35 | mlp.down_proj Layer 35 mlp.down_proj 52428800 0.001369 signed +xplain1.3-6base-16k-step2199 Layer 38 | mlp.down_proj Layer 38 mlp.down_proj 52428800 0.001368 signed +xplain1.3-6base-16k-step2199 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.001366 signed +xplain1.3-6base-16k-step2199 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001363 signed +xplain1.3-6base-16k-step2199 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001358 signed +xplain1.3-6base-16k-step2199 Layer 28 | mlp.gate_proj Layer 28 mlp.gate_proj 52428800 0.001356 signed +xplain1.3-6base-16k-step2199 Layer 21 | self_attn.q_proj Layer 21 self_attn.q_proj 16777216 0.001356 signed +xplain1.3-6base-16k-step2199 Layer 27 | mlp.gate_proj Layer 27 mlp.gate_proj 52428800 0.001351 signed +xplain1.3-6base-16k-step2199 Layer 36 | self_attn.k_proj Layer 36 self_attn.k_proj 4194304 0.001351 signed +xplain1.3-6base-16k-step2199 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001349 signed +xplain1.3-6base-16k-step2199 Layer 29 | mlp.down_proj Layer 29 mlp.down_proj 52428800 0.001347 signed +xplain1.3-6base-16k-step2199 Layer 31 | mlp.down_proj Layer 31 mlp.down_proj 52428800 0.001344 signed +xplain1.3-6base-16k-step2199 Layer 31 | mlp.up_proj Layer 31 mlp.up_proj 52428800 0.001343 signed +xplain1.3-6base-16k-step2199 Layer 26 | mlp.gate_proj Layer 26 mlp.gate_proj 52428800 0.001343 signed + +### `xplain1.3-6base-32k-step1647` + + model sublayer block_label module_type count std stat_mode +xplain1.3-6base-32k-step1647 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.005447 signed +xplain1.3-6base-32k-step1647 Final Norm | model.norm.weight Final Norm final_norm 4096 0.005347 signed +xplain1.3-6base-32k-step1647 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.005222 signed +xplain1.3-6base-32k-step1647 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.005059 signed +xplain1.3-6base-32k-step1647 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.003657 signed +xplain1.3-6base-32k-step1647 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.003223 signed +xplain1.3-6base-32k-step1647 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.002641 signed +xplain1.3-6base-32k-step1647 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.001945 signed +xplain1.3-6base-32k-step1647 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.001918 signed +xplain1.3-6base-32k-step1647 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.001915 signed +xplain1.3-6base-32k-step1647 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.001854 signed +xplain1.3-6base-32k-step1647 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.001846 signed +xplain1.3-6base-32k-step1647 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.001841 signed +xplain1.3-6base-32k-step1647 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001839 signed +xplain1.3-6base-32k-step1647 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.001798 signed +xplain1.3-6base-32k-step1647 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001796 signed +xplain1.3-6base-32k-step1647 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001786 signed +xplain1.3-6base-32k-step1647 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001780 signed +xplain1.3-6base-32k-step1647 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001771 signed +xplain1.3-6base-32k-step1647 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001767 signed +xplain1.3-6base-32k-step1647 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.001762 signed +xplain1.3-6base-32k-step1647 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001721 signed +xplain1.3-6base-32k-step1647 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001705 signed +xplain1.3-6base-32k-step1647 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001701 signed +xplain1.3-6base-32k-step1647 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001669 signed +xplain1.3-6base-32k-step1647 Layer 30 | input_layernorm Layer 30 input_layernorm 4096 0.001663 signed +xplain1.3-6base-32k-step1647 Layer 31 | input_layernorm Layer 31 input_layernorm 4096 0.001660 signed +xplain1.3-6base-32k-step1647 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001646 signed +xplain1.3-6base-32k-step1647 Layer 19 | input_layernorm Layer 19 input_layernorm 4096 0.001646 signed +xplain1.3-6base-32k-step1647 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001643 signed +xplain1.3-6base-32k-step1647 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001639 signed +xplain1.3-6base-32k-step1647 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001634 signed +xplain1.3-6base-32k-step1647 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001634 signed +xplain1.3-6base-32k-step1647 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001630 signed +xplain1.3-6base-32k-step1647 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001624 signed +xplain1.3-6base-32k-step1647 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001623 signed +xplain1.3-6base-32k-step1647 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001612 signed +xplain1.3-6base-32k-step1647 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001598 signed +xplain1.3-6base-32k-step1647 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001596 signed +xplain1.3-6base-32k-step1647 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001596 signed +xplain1.3-6base-32k-step1647 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001587 signed +xplain1.3-6base-32k-step1647 Layer 29 | mlp.down_proj Layer 29 mlp.down_proj 52428800 0.001586 signed +xplain1.3-6base-32k-step1647 Layer 28 | mlp.gate_proj Layer 28 mlp.gate_proj 52428800 0.001582 signed +xplain1.3-6base-32k-step1647 Layer 31 | mlp.down_proj Layer 31 mlp.down_proj 52428800 0.001582 signed +xplain1.3-6base-32k-step1647 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001581 signed +xplain1.3-6base-32k-step1647 Layer 35 | mlp.down_proj Layer 35 mlp.down_proj 52428800 0.001580 signed +xplain1.3-6base-32k-step1647 Layer 30 | mlp.down_proj Layer 30 mlp.down_proj 52428800 0.001580 signed +xplain1.3-6base-32k-step1647 Layer 31 | mlp.up_proj Layer 31 mlp.up_proj 52428800 0.001579 signed +xplain1.3-6base-32k-step1647 Layer 28 | mlp.down_proj Layer 28 mlp.down_proj 52428800 0.001577 signed +xplain1.3-6base-32k-step1647 Layer 27 | mlp.gate_proj Layer 27 mlp.gate_proj 52428800 0.001575 signed + +### `xplain1.3-6base-64k-step1647` + + model sublayer block_label module_type count std stat_mode +xplain1.3-6base-64k-step1647 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.006970 signed +xplain1.3-6base-64k-step1647 Final Norm | model.norm.weight Final Norm final_norm 4096 0.006869 signed +xplain1.3-6base-64k-step1647 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.006613 signed +xplain1.3-6base-64k-step1647 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.006396 signed +xplain1.3-6base-64k-step1647 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.004488 signed +xplain1.3-6base-64k-step1647 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.004040 signed +xplain1.3-6base-64k-step1647 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.003279 signed +xplain1.3-6base-64k-step1647 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.002411 signed +xplain1.3-6base-64k-step1647 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.002349 signed +xplain1.3-6base-64k-step1647 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.002266 signed +xplain1.3-6base-64k-step1647 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.002246 signed +xplain1.3-6base-64k-step1647 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.002107 signed +xplain1.3-6base-64k-step1647 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.002046 signed +xplain1.3-6base-64k-step1647 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.002033 signed +xplain1.3-6base-64k-step1647 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.002031 signed +xplain1.3-6base-64k-step1647 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.002014 signed +xplain1.3-6base-64k-step1647 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001987 signed +xplain1.3-6base-64k-step1647 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001982 signed +xplain1.3-6base-64k-step1647 Layer 19 | input_layernorm Layer 19 input_layernorm 4096 0.001978 signed +xplain1.3-6base-64k-step1647 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001977 signed +xplain1.3-6base-64k-step1647 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001965 signed +xplain1.3-6base-64k-step1647 Layer 30 | input_layernorm Layer 30 input_layernorm 4096 0.001959 signed +xplain1.3-6base-64k-step1647 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001954 signed +xplain1.3-6base-64k-step1647 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001919 signed +xplain1.3-6base-64k-step1647 Layer 31 | input_layernorm Layer 31 input_layernorm 4096 0.001907 signed +xplain1.3-6base-64k-step1647 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001898 signed +xplain1.3-6base-64k-step1647 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001897 signed +xplain1.3-6base-64k-step1647 Layer 14 | input_layernorm Layer 14 input_layernorm 4096 0.001861 signed +xplain1.3-6base-64k-step1647 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001850 signed +xplain1.3-6base-64k-step1647 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001844 signed +xplain1.3-6base-64k-step1647 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001831 signed +xplain1.3-6base-64k-step1647 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001824 signed +xplain1.3-6base-64k-step1647 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001822 signed +xplain1.3-6base-64k-step1647 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001817 signed +xplain1.3-6base-64k-step1647 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001816 signed +xplain1.3-6base-64k-step1647 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001813 signed +xplain1.3-6base-64k-step1647 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001804 signed +xplain1.3-6base-64k-step1647 Layer 32 | input_layernorm Layer 32 input_layernorm 4096 0.001804 signed +xplain1.3-6base-64k-step1647 Layer 33 | input_layernorm Layer 33 input_layernorm 4096 0.001803 signed +xplain1.3-6base-64k-step1647 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001798 signed +xplain1.3-6base-64k-step1647 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001794 signed +xplain1.3-6base-64k-step1647 Layer 29 | mlp.down_proj Layer 29 mlp.down_proj 52428800 0.001790 signed +xplain1.3-6base-64k-step1647 Layer 31 | mlp.down_proj Layer 31 mlp.down_proj 52428800 0.001785 signed +xplain1.3-6base-64k-step1647 Layer 30 | mlp.down_proj Layer 30 mlp.down_proj 52428800 0.001785 signed +xplain1.3-6base-64k-step1647 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001784 signed +xplain1.3-6base-64k-step1647 Layer 15 | input_layernorm Layer 15 input_layernorm 4096 0.001783 signed +xplain1.3-6base-64k-step1647 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001782 signed +xplain1.3-6base-64k-step1647 Layer 31 | mlp.up_proj Layer 31 mlp.up_proj 52428800 0.001781 signed +xplain1.3-6base-64k-step1647 Layer 28 | mlp.down_proj Layer 28 mlp.down_proj 52428800 0.001779 signed +xplain1.3-6base-64k-step1647 Layer 28 | mlp.gate_proj Layer 28 mlp.gate_proj 52428800 0.001775 signed + +### `xplain1.3-6base-8k-step2199` + + model sublayer block_label module_type count std stat_mode +xplain1.3-6base-8k-step2199 Final Norm | model.norm.weight Final Norm final_norm 4096 0.002388 signed +xplain1.3-6base-8k-step2199 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.002092 signed +xplain1.3-6base-8k-step2199 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.002092 signed +xplain1.3-6base-8k-step2199 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.002019 signed +xplain1.3-6base-8k-step2199 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.001531 signed +xplain1.3-6base-8k-step2199 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.001351 signed +xplain1.3-6base-8k-step2199 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.001253 signed +xplain1.3-6base-8k-step2199 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.001244 signed +xplain1.3-6base-8k-step2199 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001238 signed +xplain1.3-6base-8k-step2199 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001196 signed +xplain1.3-6base-8k-step2199 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001177 signed +xplain1.3-6base-8k-step2199 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001174 signed +xplain1.3-6base-8k-step2199 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001173 signed +xplain1.3-6base-8k-step2199 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001170 signed +xplain1.3-6base-8k-step2199 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.001120 signed +xplain1.3-6base-8k-step2199 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001119 signed +xplain1.3-6base-8k-step2199 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001116 signed +xplain1.3-6base-8k-step2199 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001115 signed +xplain1.3-6base-8k-step2199 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001107 signed +xplain1.3-6base-8k-step2199 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001101 signed +xplain1.3-6base-8k-step2199 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001095 signed +xplain1.3-6base-8k-step2199 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.001087 signed +xplain1.3-6base-8k-step2199 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001075 signed +xplain1.3-6base-8k-step2199 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001072 signed +xplain1.3-6base-8k-step2199 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001070 signed +xplain1.3-6base-8k-step2199 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001064 signed +xplain1.3-6base-8k-step2199 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001064 signed +xplain1.3-6base-8k-step2199 Layer 38 | mlp.down_proj Layer 38 mlp.down_proj 52428800 0.001052 signed +xplain1.3-6base-8k-step2199 Layer 36 | self_attn.k_proj Layer 36 self_attn.k_proj 4194304 0.001038 signed +xplain1.3-6base-8k-step2199 Layer 35 | mlp.down_proj Layer 35 mlp.down_proj 52428800 0.001028 signed +xplain1.3-6base-8k-step2199 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.001027 signed +xplain1.3-6base-8k-step2199 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001019 signed +xplain1.3-6base-8k-step2199 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001015 signed +xplain1.3-6base-8k-step2199 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.001007 signed +xplain1.3-6base-8k-step2199 Layer 39 | self_attn.k_proj Layer 39 self_attn.k_proj 4194304 0.001005 signed +xplain1.3-6base-8k-step2199 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.000999 signed +xplain1.3-6base-8k-step2199 Layer 35 | self_attn.k_proj Layer 35 self_attn.k_proj 4194304 0.000989 signed +xplain1.3-6base-8k-step2199 Layer 38 | self_attn.k_proj Layer 38 self_attn.k_proj 4194304 0.000988 signed +xplain1.3-6base-8k-step2199 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.000984 signed +xplain1.3-6base-8k-step2199 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.000981 signed +xplain1.3-6base-8k-step2199 Layer 21 | self_attn.q_proj Layer 21 self_attn.q_proj 16777216 0.000980 signed +xplain1.3-6base-8k-step2199 Layer 17 | self_attn.q_proj Layer 17 self_attn.q_proj 16777216 0.000976 signed +xplain1.3-6base-8k-step2199 Layer 28 | mlp.gate_proj Layer 28 mlp.gate_proj 52428800 0.000975 signed +xplain1.3-6base-8k-step2199 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.000974 signed +xplain1.3-6base-8k-step2199 Layer 17 | mlp.gate_proj Layer 17 mlp.gate_proj 52428800 0.000974 signed +xplain1.3-6base-8k-step2199 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.000974 signed +xplain1.3-6base-8k-step2199 Layer 27 | mlp.gate_proj Layer 27 mlp.gate_proj 52428800 0.000974 signed +xplain1.3-6base-8k-step2199 Layer 18 | mlp.gate_proj Layer 18 mlp.gate_proj 52428800 0.000973 signed +xplain1.3-6base-8k-step2199 Layer 19 | self_attn.q_proj Layer 19 self_attn.q_proj 16777216 0.000971 signed +xplain1.3-6base-8k-step2199 Layer 19 | mlp.gate_proj Layer 19 mlp.gate_proj 52428800 0.000970 signed + +### `xplain1.4-6base-16k-step2319` + + model sublayer block_label module_type count std stat_mode +xplain1.4-6base-16k-step2319 Final Norm | model.norm.weight Final Norm final_norm 4096 0.005594 signed +xplain1.4-6base-16k-step2319 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.004097 signed +xplain1.4-6base-16k-step2319 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.004061 signed +xplain1.4-6base-16k-step2319 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.003967 signed +xplain1.4-6base-16k-step2319 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.003069 signed +xplain1.4-6base-16k-step2319 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.002584 signed +xplain1.4-6base-16k-step2319 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.002278 signed +xplain1.4-6base-16k-step2319 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.001990 signed +xplain1.4-6base-16k-step2319 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.001911 signed +xplain1.4-6base-16k-step2319 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001874 signed +xplain1.4-6base-16k-step2319 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001832 signed +xplain1.4-6base-16k-step2319 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001829 signed +xplain1.4-6base-16k-step2319 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001743 signed +xplain1.4-6base-16k-step2319 Layer 39 | post_attention_layernorm Layer 39 post_attention_layernorm 4096 0.001732 signed +xplain1.4-6base-16k-step2319 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001715 signed +xplain1.4-6base-16k-step2319 Layer 38 | mlp.down_proj Layer 38 mlp.down_proj 52428800 0.001702 signed +xplain1.4-6base-16k-step2319 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.001679 signed +xplain1.4-6base-16k-step2319 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.001678 signed +xplain1.4-6base-16k-step2319 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.001660 signed +xplain1.4-6base-16k-step2319 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001632 signed +xplain1.4-6base-16k-step2319 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001611 signed +xplain1.4-6base-16k-step2319 Layer 37 | post_attention_layernorm Layer 37 post_attention_layernorm 4096 0.001591 signed +xplain1.4-6base-16k-step2319 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001560 signed +xplain1.4-6base-16k-step2319 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001555 signed +xplain1.4-6base-16k-step2319 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001549 signed +xplain1.4-6base-16k-step2319 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.001547 signed +xplain1.4-6base-16k-step2319 Layer 39 | self_attn.k_proj Layer 39 self_attn.k_proj 4194304 0.001546 signed +xplain1.4-6base-16k-step2319 Layer 38 | post_attention_layernorm Layer 38 post_attention_layernorm 4096 0.001541 signed +xplain1.4-6base-16k-step2319 Layer 38 | self_attn.k_proj Layer 38 self_attn.k_proj 4194304 0.001533 signed +xplain1.4-6base-16k-step2319 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.001527 signed +xplain1.4-6base-16k-step2319 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001513 signed +xplain1.4-6base-16k-step2319 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.001508 signed +xplain1.4-6base-16k-step2319 Layer 30 | input_layernorm Layer 30 input_layernorm 4096 0.001490 signed +xplain1.4-6base-16k-step2319 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.001487 signed +xplain1.4-6base-16k-step2319 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001487 signed +xplain1.4-6base-16k-step2319 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001458 signed +xplain1.4-6base-16k-step2319 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001450 signed +xplain1.4-6base-16k-step2319 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001444 signed +xplain1.4-6base-16k-step2319 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001431 signed +xplain1.4-6base-16k-step2319 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001430 signed +xplain1.4-6base-16k-step2319 Layer 39 | input_layernorm Layer 39 input_layernorm 4096 0.001430 signed +xplain1.4-6base-16k-step2319 Layer 31 | input_layernorm Layer 31 input_layernorm 4096 0.001421 signed +xplain1.4-6base-16k-step2319 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001418 signed +xplain1.4-6base-16k-step2319 Layer 37 | self_attn.k_proj Layer 37 self_attn.k_proj 4194304 0.001413 signed +xplain1.4-6base-16k-step2319 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001409 signed +xplain1.4-6base-16k-step2319 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001392 signed +xplain1.4-6base-16k-step2319 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001389 signed +xplain1.4-6base-16k-step2319 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001387 signed +xplain1.4-6base-16k-step2319 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001387 signed +xplain1.4-6base-16k-step2319 Layer 35 | mlp.down_proj Layer 35 mlp.down_proj 52428800 0.001381 signed + +### `xplain1.4-6base-32k-step1743` + + model sublayer block_label module_type count std stat_mode +xplain1.4-6base-32k-step1743 Final Norm | model.norm.weight Final Norm final_norm 4096 0.006660 signed +xplain1.4-6base-32k-step1743 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.005665 signed +xplain1.4-6base-32k-step1743 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.005453 signed +xplain1.4-6base-32k-step1743 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.005436 signed +xplain1.4-6base-32k-step1743 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.004031 signed +xplain1.4-6base-32k-step1743 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.003478 signed +xplain1.4-6base-32k-step1743 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.002953 signed +xplain1.4-6base-32k-step1743 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.002214 signed +xplain1.4-6base-32k-step1743 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.002141 signed +xplain1.4-6base-32k-step1743 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.002105 signed +xplain1.4-6base-32k-step1743 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.002073 signed +xplain1.4-6base-32k-step1743 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.002072 signed +xplain1.4-6base-32k-step1743 Layer 39 | post_attention_layernorm Layer 39 post_attention_layernorm 4096 0.002072 signed +xplain1.4-6base-32k-step1743 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.002064 signed +xplain1.4-6base-32k-step1743 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.002055 signed +xplain1.4-6base-32k-step1743 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.002037 signed +xplain1.4-6base-32k-step1743 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.002007 signed +xplain1.4-6base-32k-step1743 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001931 signed +xplain1.4-6base-32k-step1743 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001905 signed +xplain1.4-6base-32k-step1743 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.001901 signed +xplain1.4-6base-32k-step1743 Layer 38 | mlp.down_proj Layer 38 mlp.down_proj 52428800 0.001889 signed +xplain1.4-6base-32k-step1743 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001865 signed +xplain1.4-6base-32k-step1743 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.001864 signed +xplain1.4-6base-32k-step1743 Layer 37 | post_attention_layernorm Layer 37 post_attention_layernorm 4096 0.001838 signed +xplain1.4-6base-32k-step1743 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.001836 signed +xplain1.4-6base-32k-step1743 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001827 signed +xplain1.4-6base-32k-step1743 Layer 30 | input_layernorm Layer 30 input_layernorm 4096 0.001813 signed +xplain1.4-6base-32k-step1743 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001812 signed +xplain1.4-6base-32k-step1743 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001800 signed +xplain1.4-6base-32k-step1743 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001769 signed +xplain1.4-6base-32k-step1743 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001763 signed +xplain1.4-6base-32k-step1743 Layer 38 | post_attention_layernorm Layer 38 post_attention_layernorm 4096 0.001744 signed +xplain1.4-6base-32k-step1743 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001735 signed +xplain1.4-6base-32k-step1743 Layer 39 | input_layernorm Layer 39 input_layernorm 4096 0.001728 signed +xplain1.4-6base-32k-step1743 Layer 31 | input_layernorm Layer 31 input_layernorm 4096 0.001726 signed +xplain1.4-6base-32k-step1743 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001711 signed +xplain1.4-6base-32k-step1743 Layer 39 | self_attn.k_proj Layer 39 self_attn.k_proj 4194304 0.001703 signed +xplain1.4-6base-32k-step1743 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001699 signed +xplain1.4-6base-32k-step1743 Layer 38 | self_attn.k_proj Layer 38 self_attn.k_proj 4194304 0.001686 signed +xplain1.4-6base-32k-step1743 Layer 19 | input_layernorm Layer 19 input_layernorm 4096 0.001684 signed +xplain1.4-6base-32k-step1743 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001682 signed +xplain1.4-6base-32k-step1743 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001675 signed +xplain1.4-6base-32k-step1743 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001671 signed +xplain1.4-6base-32k-step1743 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001668 signed +xplain1.4-6base-32k-step1743 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001652 signed +xplain1.4-6base-32k-step1743 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001644 signed +xplain1.4-6base-32k-step1743 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001643 signed +xplain1.4-6base-32k-step1743 Layer 32 | input_layernorm Layer 32 input_layernorm 4096 0.001638 signed +xplain1.4-6base-32k-step1743 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001634 signed +xplain1.4-6base-32k-step1743 Layer 31 | mlp.up_proj Layer 31 mlp.up_proj 52428800 0.001630 signed + +### `xplain1.4-6base-64k-step1743` + + model sublayer block_label module_type count std stat_mode +xplain1.4-6base-64k-step1743 Final Norm | model.norm.weight Final Norm final_norm 4096 0.007889 signed +xplain1.4-6base-64k-step1743 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.007021 signed +xplain1.4-6base-64k-step1743 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.006637 signed +xplain1.4-6base-64k-step1743 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.006622 signed +xplain1.4-6base-64k-step1743 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.004975 signed +xplain1.4-6base-64k-step1743 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.004198 signed +xplain1.4-6base-64k-step1743 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.003582 signed +xplain1.4-6base-64k-step1743 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.002491 signed +xplain1.4-6base-64k-step1743 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.002454 signed +xplain1.4-6base-64k-step1743 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.002429 signed +xplain1.4-6base-64k-step1743 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.002357 signed +xplain1.4-6base-64k-step1743 Layer 39 | post_attention_layernorm Layer 39 post_attention_layernorm 4096 0.002316 signed +xplain1.4-6base-64k-step1743 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.002259 signed +xplain1.4-6base-64k-step1743 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.002255 signed +xplain1.4-6base-64k-step1743 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.002247 signed +xplain1.4-6base-64k-step1743 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.002206 signed +xplain1.4-6base-64k-step1743 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.002186 signed +xplain1.4-6base-64k-step1743 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.002164 signed +xplain1.4-6base-64k-step1743 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.002126 signed +xplain1.4-6base-64k-step1743 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.002050 signed +xplain1.4-6base-64k-step1743 Layer 30 | input_layernorm Layer 30 input_layernorm 4096 0.002045 signed +xplain1.4-6base-64k-step1743 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.002030 signed +xplain1.4-6base-64k-step1743 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.002021 signed +xplain1.4-6base-64k-step1743 Layer 38 | mlp.down_proj Layer 38 mlp.down_proj 52428800 0.002011 signed +xplain1.4-6base-64k-step1743 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001999 signed +xplain1.4-6base-64k-step1743 Layer 19 | input_layernorm Layer 19 input_layernorm 4096 0.001980 signed +xplain1.4-6base-64k-step1743 Layer 39 | input_layernorm Layer 39 input_layernorm 4096 0.001972 signed +xplain1.4-6base-64k-step1743 Layer 31 | input_layernorm Layer 31 input_layernorm 4096 0.001948 signed +xplain1.4-6base-64k-step1743 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001947 signed +xplain1.4-6base-64k-step1743 Layer 37 | post_attention_layernorm Layer 37 post_attention_layernorm 4096 0.001946 signed +xplain1.4-6base-64k-step1743 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001936 signed +xplain1.4-6base-64k-step1743 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001930 signed +xplain1.4-6base-64k-step1743 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001911 signed +xplain1.4-6base-64k-step1743 Layer 33 | input_layernorm Layer 33 input_layernorm 4096 0.001875 signed +xplain1.4-6base-64k-step1743 Layer 38 | post_attention_layernorm Layer 38 post_attention_layernorm 4096 0.001872 signed +xplain1.4-6base-64k-step1743 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001868 signed +xplain1.4-6base-64k-step1743 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001866 signed +xplain1.4-6base-64k-step1743 Layer 14 | input_layernorm Layer 14 input_layernorm 4096 0.001866 signed +xplain1.4-6base-64k-step1743 Layer 32 | input_layernorm Layer 32 input_layernorm 4096 0.001862 signed +xplain1.4-6base-64k-step1743 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001846 signed +xplain1.4-6base-64k-step1743 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001843 signed +xplain1.4-6base-64k-step1743 Layer 15 | input_layernorm Layer 15 input_layernorm 4096 0.001836 signed +xplain1.4-6base-64k-step1743 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001823 signed +xplain1.4-6base-64k-step1743 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001822 signed +xplain1.4-6base-64k-step1743 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001816 signed +xplain1.4-6base-64k-step1743 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001804 signed +xplain1.4-6base-64k-step1743 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001799 signed +xplain1.4-6base-64k-step1743 Layer 39 | self_attn.k_proj Layer 39 self_attn.k_proj 4194304 0.001799 signed +xplain1.4-6base-64k-step1743 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001790 signed +xplain1.4-6base-64k-step1743 Layer 28 | mlp.gate_proj Layer 28 mlp.gate_proj 52428800 0.001780 signed + +### `xplain1.4-6base-8k-step2319` + + model sublayer block_label module_type count std stat_mode +xplain1.4-6base-8k-step2319 Final Norm | model.norm.weight Final Norm final_norm 4096 0.003269 signed +xplain1.4-6base-8k-step2319 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.002199 signed +xplain1.4-6base-8k-step2319 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.002092 signed +xplain1.4-6base-8k-step2319 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.002085 signed +xplain1.4-6base-8k-step2319 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.001681 signed +xplain1.4-6base-8k-step2319 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.001586 signed +xplain1.4-6base-8k-step2319 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.001515 signed +xplain1.4-6base-8k-step2319 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001469 signed +xplain1.4-6base-8k-step2319 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001428 signed +xplain1.4-6base-8k-step2319 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001405 signed +xplain1.4-6base-8k-step2319 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001402 signed +xplain1.4-6base-8k-step2319 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.001373 signed +xplain1.4-6base-8k-step2319 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001353 signed +xplain1.4-6base-8k-step2319 Layer 38 | mlp.down_proj Layer 38 mlp.down_proj 52428800 0.001327 signed +xplain1.4-6base-8k-step2319 Layer 37 | mlp.down_proj Layer 37 mlp.down_proj 52428800 0.001265 signed +xplain1.4-6base-8k-step2319 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.001259 signed +xplain1.4-6base-8k-step2319 Layer 38 | self_attn.k_proj Layer 38 self_attn.k_proj 4194304 0.001250 signed +xplain1.4-6base-8k-step2319 Layer 39 | self_attn.k_proj Layer 39 self_attn.k_proj 4194304 0.001244 signed +xplain1.4-6base-8k-step2319 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.001241 signed +xplain1.4-6base-8k-step2319 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001226 signed +xplain1.4-6base-8k-step2319 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001212 signed +xplain1.4-6base-8k-step2319 Layer 38 | post_attention_layernorm Layer 38 post_attention_layernorm 4096 0.001187 signed +xplain1.4-6base-8k-step2319 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.001185 signed +xplain1.4-6base-8k-step2319 Layer 37 | post_attention_layernorm Layer 37 post_attention_layernorm 4096 0.001183 signed +xplain1.4-6base-8k-step2319 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001175 signed +xplain1.4-6base-8k-step2319 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001160 signed +xplain1.4-6base-8k-step2319 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001139 signed +xplain1.4-6base-8k-step2319 Layer 23 | input_layernorm Layer 23 input_layernorm 4096 0.001125 signed +xplain1.4-6base-8k-step2319 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001116 signed +xplain1.4-6base-8k-step2319 Layer 37 | self_attn.k_proj Layer 37 self_attn.k_proj 4194304 0.001100 signed +xplain1.4-6base-8k-step2319 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001088 signed +xplain1.4-6base-8k-step2319 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001080 signed +xplain1.4-6base-8k-step2319 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001072 signed +xplain1.4-6base-8k-step2319 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001070 signed +xplain1.4-6base-8k-step2319 Layer 26 | input_layernorm Layer 26 input_layernorm 4096 0.001063 signed +xplain1.4-6base-8k-step2319 Layer 36 | mlp.down_proj Layer 36 mlp.down_proj 52428800 0.001055 signed +xplain1.4-6base-8k-step2319 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001048 signed +xplain1.4-6base-8k-step2319 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001041 signed +xplain1.4-6base-8k-step2319 Layer 18 | mlp.gate_proj Layer 18 mlp.gate_proj 52428800 0.001035 signed +xplain1.4-6base-8k-step2319 Layer 19 | mlp.gate_proj Layer 19 mlp.gate_proj 52428800 0.001032 signed +xplain1.4-6base-8k-step2319 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001030 signed +xplain1.4-6base-8k-step2319 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001028 signed +xplain1.4-6base-8k-step2319 Layer 17 | mlp.gate_proj Layer 17 mlp.gate_proj 52428800 0.001027 signed +xplain1.4-6base-8k-step2319 Layer 39 | post_attention_layernorm Layer 39 post_attention_layernorm 4096 0.001026 signed +xplain1.4-6base-8k-step2319 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001026 signed +xplain1.4-6base-8k-step2319 Layer 35 | mlp.down_proj Layer 35 mlp.down_proj 52428800 0.001025 signed +xplain1.4-6base-8k-step2319 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001024 signed +xplain1.4-6base-8k-step2319 Layer 28 | mlp.gate_proj Layer 28 mlp.gate_proj 52428800 0.001023 signed +xplain1.4-6base-8k-step2319 Layer 27 | mlp.gate_proj Layer 27 mlp.gate_proj 52428800 0.001022 signed +xplain1.4-6base-8k-step2319 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001021 signed + +### `xplain1.4-7base-32k-step1743` + + model sublayer block_label module_type count std stat_mode +xplain1.4-7base-32k-step1743 Final Norm | model.norm.weight Final Norm final_norm 4096 0.006105 signed +xplain1.4-7base-32k-step1743 Layer 22 | input_layernorm Layer 22 input_layernorm 4096 0.004702 signed +xplain1.4-7base-32k-step1743 Layer 28 | input_layernorm Layer 28 input_layernorm 4096 0.004587 signed +xplain1.4-7base-32k-step1743 Layer 25 | input_layernorm Layer 25 input_layernorm 4096 0.004584 signed +xplain1.4-7base-32k-step1743 Layer 18 | input_layernorm Layer 18 input_layernorm 4096 0.004540 signed +xplain1.4-7base-32k-step1743 Layer 21 | input_layernorm Layer 21 input_layernorm 4096 0.002993 signed +xplain1.4-7base-32k-step1743 Layer 38 | input_layernorm Layer 38 input_layernorm 4096 0.002686 signed +xplain1.4-7base-32k-step1743 Layer 17 | input_layernorm Layer 17 input_layernorm 4096 0.002019 signed +xplain1.4-7base-32k-step1743 Layer 36 | input_layernorm Layer 36 input_layernorm 4096 0.001978 signed +xplain1.4-7base-32k-step1743 Layer 37 | input_layernorm Layer 37 input_layernorm 4096 0.001967 signed +xplain1.4-7base-32k-step1743 Layer 16 | input_layernorm Layer 16 input_layernorm 4096 0.001832 signed +xplain1.4-7base-32k-step1743 Layer 14 | input_layernorm Layer 14 input_layernorm 4096 0.001804 signed +xplain1.4-7base-32k-step1743 Layer 15 | input_layernorm Layer 15 input_layernorm 4096 0.001766 signed +xplain1.4-7base-32k-step1743 Layer 38 | mlp.gate_proj Layer 38 mlp.gate_proj 52428800 0.001752 signed +xplain1.4-7base-32k-step1743 Layer 37 | mlp.gate_proj Layer 37 mlp.gate_proj 52428800 0.001746 signed +xplain1.4-7base-32k-step1743 Layer 36 | mlp.gate_proj Layer 36 mlp.gate_proj 52428800 0.001724 signed +xplain1.4-7base-32k-step1743 Embedding | model.embed_tokens.weight Embedding embed_tokens 201326592 0.001716 signed +xplain1.4-7base-32k-step1743 Layer 37 | mlp.up_proj Layer 37 mlp.up_proj 52428800 0.001685 signed +xplain1.4-7base-32k-step1743 Layer 38 | mlp.up_proj Layer 38 mlp.up_proj 52428800 0.001680 signed +xplain1.4-7base-32k-step1743 Layer 36 | mlp.up_proj Layer 36 mlp.up_proj 52428800 0.001668 signed +xplain1.4-7base-32k-step1743 Layer 39 | post_attention_layernorm Layer 39 post_attention_layernorm 4096 0.001665 signed +xplain1.4-7base-32k-step1743 Layer 39 | mlp.gate_proj Layer 39 mlp.gate_proj 52428800 0.001665 signed +xplain1.4-7base-32k-step1743 Layer 35 | mlp.gate_proj Layer 35 mlp.gate_proj 52428800 0.001662 signed +xplain1.4-7base-32k-step1743 Layer 34 | mlp.gate_proj Layer 34 mlp.gate_proj 52428800 0.001657 signed +xplain1.4-7base-32k-step1743 Layer 19 | input_layernorm Layer 19 input_layernorm 4096 0.001644 signed +xplain1.4-7base-32k-step1743 Layer 13 | input_layernorm Layer 13 input_layernorm 4096 0.001611 signed +xplain1.4-7base-32k-step1743 Layer 00 | input_layernorm Layer 00 input_layernorm 4096 0.001606 signed +xplain1.4-7base-32k-step1743 Layer 35 | mlp.up_proj Layer 35 mlp.up_proj 52428800 0.001599 signed +xplain1.4-7base-32k-step1743 Layer 34 | mlp.up_proj Layer 34 mlp.up_proj 52428800 0.001599 signed +xplain1.4-7base-32k-step1743 Layer 33 | mlp.gate_proj Layer 33 mlp.gate_proj 52428800 0.001592 signed +xplain1.4-7base-32k-step1743 Layer 34 | self_attn.q_proj Layer 34 self_attn.q_proj 16777216 0.001577 signed +xplain1.4-7base-32k-step1743 Layer 36 | self_attn.q_proj Layer 36 self_attn.q_proj 16777216 0.001575 signed +xplain1.4-7base-32k-step1743 Layer 32 | mlp.gate_proj Layer 32 mlp.gate_proj 52428800 0.001565 signed +xplain1.4-7base-32k-step1743 Layer 31 | mlp.gate_proj Layer 31 mlp.gate_proj 52428800 0.001558 signed +xplain1.4-7base-32k-step1743 Layer 35 | self_attn.q_proj Layer 35 self_attn.q_proj 16777216 0.001557 signed +xplain1.4-7base-32k-step1743 Layer 38 | self_attn.q_proj Layer 38 self_attn.q_proj 16777216 0.001556 signed +xplain1.4-7base-32k-step1743 Layer 39 | mlp.up_proj Layer 39 mlp.up_proj 52428800 0.001542 signed +xplain1.4-7base-32k-step1743 Layer 30 | input_layernorm Layer 30 input_layernorm 4096 0.001540 signed +xplain1.4-7base-32k-step1743 Layer 30 | mlp.gate_proj Layer 30 mlp.gate_proj 52428800 0.001533 signed +xplain1.4-7base-32k-step1743 Layer 33 | mlp.up_proj Layer 33 mlp.up_proj 52428800 0.001533 signed +xplain1.4-7base-32k-step1743 Layer 31 | mlp.down_proj Layer 31 mlp.down_proj 52428800 0.001528 signed +xplain1.4-7base-32k-step1743 Layer 30 | mlp.down_proj Layer 30 mlp.down_proj 52428800 0.001525 signed +xplain1.4-7base-32k-step1743 Layer 35 | input_layernorm Layer 35 input_layernorm 4096 0.001522 signed +xplain1.4-7base-32k-step1743 Layer 29 | mlp.down_proj Layer 29 mlp.down_proj 52428800 0.001521 signed +xplain1.4-7base-32k-step1743 Layer 31 | mlp.up_proj Layer 31 mlp.up_proj 52428800 0.001520 signed +xplain1.4-7base-32k-step1743 Layer 29 | mlp.gate_proj Layer 29 mlp.gate_proj 52428800 0.001518 signed +xplain1.4-7base-32k-step1743 Layer 39 | self_attn.q_proj Layer 39 self_attn.q_proj 16777216 0.001518 signed +xplain1.4-7base-32k-step1743 Layer 34 | mlp.down_proj Layer 34 mlp.down_proj 52428800 0.001517 signed +xplain1.4-7base-32k-step1743 Layer 32 | mlp.up_proj Layer 32 mlp.up_proj 52428800 0.001515 signed +xplain1.4-7base-32k-step1743 Layer 37 | self_attn.q_proj Layer 37 self_attn.q_proj 16777216 0.001511 signed + +## Top layers or global blocks by std + +### `xplain1.3-6base-16k-step2199` + + model block_label count std stat_mode +xplain1.3-6base-16k-step2199 Final Norm 4096 0.003860 signed +xplain1.3-6base-16k-step2199 Embedding 201326592 0.001496 signed +xplain1.3-6base-16k-step2199 Layer 36 199237632 0.001477 signed +xplain1.3-6base-16k-step2199 Layer 37 199237632 0.001473 signed +xplain1.3-6base-16k-step2199 Layer 38 199237632 0.001456 signed +xplain1.3-6base-16k-step2199 Layer 34 199237632 0.001435 signed +xplain1.3-6base-16k-step2199 Layer 35 199237632 0.001420 signed +xplain1.3-6base-16k-step2199 Layer 31 199237632 0.001318 signed +xplain1.3-6base-16k-step2199 Layer 33 199237632 0.001307 signed +xplain1.3-6base-16k-step2199 Layer 30 199237632 0.001303 signed +xplain1.3-6base-16k-step2199 Layer 32 199237632 0.001296 signed +xplain1.3-6base-16k-step2199 Layer 28 199237632 0.001292 signed +xplain1.3-6base-16k-step2199 Layer 25 199237632 0.001272 signed +xplain1.3-6base-16k-step2199 Layer 39 199237632 0.001269 signed +xplain1.3-6base-16k-step2199 Layer 22 199237632 0.001259 signed +xplain1.3-6base-16k-step2199 Layer 21 199237632 0.001256 signed +xplain1.3-6base-16k-step2199 Layer 19 199237632 0.001233 signed +xplain1.3-6base-16k-step2199 Layer 29 199237632 0.001223 signed +xplain1.3-6base-16k-step2199 Layer 18 199237632 0.001220 signed +xplain1.3-6base-16k-step2199 Layer 27 199237632 0.001213 signed +xplain1.3-6base-16k-step2199 Layer 17 199237632 0.001212 signed +xplain1.3-6base-16k-step2199 Layer 26 199237632 0.001204 signed +xplain1.3-6base-16k-step2199 Layer 16 199237632 0.001190 signed +xplain1.3-6base-16k-step2199 Layer 24 199237632 0.001186 signed +xplain1.3-6base-16k-step2199 Layer 15 199237632 0.001182 signed +xplain1.3-6base-16k-step2199 Layer 23 199237632 0.001177 signed +xplain1.3-6base-16k-step2199 Layer 20 199237632 0.001158 signed +xplain1.3-6base-16k-step2199 Layer 14 199237632 0.001155 signed +xplain1.3-6base-16k-step2199 Layer 13 199237632 0.001132 signed +xplain1.3-6base-16k-step2199 Layer 12 199237632 0.001125 signed +xplain1.3-6base-16k-step2199 Layer 11 199237632 0.001106 signed +xplain1.3-6base-16k-step2199 Layer 10 199237632 0.001089 signed +xplain1.3-6base-16k-step2199 Layer 09 199237632 0.001072 signed +xplain1.3-6base-16k-step2199 Layer 08 199237632 0.001057 signed +xplain1.3-6base-16k-step2199 Layer 06 199237632 0.001048 signed +xplain1.3-6base-16k-step2199 Layer 07 199237632 0.001047 signed +xplain1.3-6base-16k-step2199 Layer 05 199237632 0.001040 signed +xplain1.3-6base-16k-step2199 Layer 00 199237632 0.001025 signed +xplain1.3-6base-16k-step2199 Layer 04 199237632 0.001025 signed +xplain1.3-6base-16k-step2199 Layer 01 199237632 0.001012 signed +xplain1.3-6base-16k-step2199 Layer 03 199237632 0.000982 signed +xplain1.3-6base-16k-step2199 Layer 02 199237632 0.000980 signed + +### `xplain1.3-6base-32k-step1647` + + model block_label count std stat_mode +xplain1.3-6base-32k-step1647 Final Norm 4096 0.005347 signed +xplain1.3-6base-32k-step1647 Embedding 201326592 0.001762 signed +xplain1.3-6base-32k-step1647 Layer 36 199237632 0.001691 signed +xplain1.3-6base-32k-step1647 Layer 37 199237632 0.001681 signed +xplain1.3-6base-32k-step1647 Layer 38 199237632 0.001660 signed +xplain1.3-6base-32k-step1647 Layer 34 199237632 0.001654 signed +xplain1.3-6base-32k-step1647 Layer 35 199237632 0.001632 signed +xplain1.3-6base-32k-step1647 Layer 31 199237632 0.001549 signed +xplain1.3-6base-32k-step1647 Layer 30 199237632 0.001533 signed +xplain1.3-6base-32k-step1647 Layer 33 199237632 0.001532 signed +xplain1.3-6base-32k-step1647 Layer 32 199237632 0.001524 signed +xplain1.3-6base-32k-step1647 Layer 28 199237632 0.001520 signed +xplain1.3-6base-32k-step1647 Layer 25 199237632 0.001491 signed +xplain1.3-6base-32k-step1647 Layer 22 199237632 0.001467 signed +xplain1.3-6base-32k-step1647 Layer 39 199237632 0.001464 signed +xplain1.3-6base-32k-step1647 Layer 21 199237632 0.001460 signed +xplain1.3-6base-32k-step1647 Layer 29 199237632 0.001432 signed +xplain1.3-6base-32k-step1647 Layer 19 199237632 0.001427 signed +xplain1.3-6base-32k-step1647 Layer 27 199237632 0.001417 signed +xplain1.3-6base-32k-step1647 Layer 18 199237632 0.001410 signed +xplain1.3-6base-32k-step1647 Layer 26 199237632 0.001406 signed +xplain1.3-6base-32k-step1647 Layer 17 199237632 0.001398 signed +xplain1.3-6base-32k-step1647 Layer 24 199237632 0.001384 signed +xplain1.3-6base-32k-step1647 Layer 16 199237632 0.001372 signed +xplain1.3-6base-32k-step1647 Layer 23 199237632 0.001371 signed +xplain1.3-6base-32k-step1647 Layer 15 199237632 0.001361 signed +xplain1.3-6base-32k-step1647 Layer 20 199237632 0.001342 signed +xplain1.3-6base-32k-step1647 Layer 14 199237632 0.001329 signed +xplain1.3-6base-32k-step1647 Layer 13 199237632 0.001302 signed +xplain1.3-6base-32k-step1647 Layer 12 199237632 0.001293 signed +xplain1.3-6base-32k-step1647 Layer 11 199237632 0.001271 signed +xplain1.3-6base-32k-step1647 Layer 10 199237632 0.001250 signed +xplain1.3-6base-32k-step1647 Layer 09 199237632 0.001230 signed +xplain1.3-6base-32k-step1647 Layer 08 199237632 0.001212 signed +xplain1.3-6base-32k-step1647 Layer 07 199237632 0.001192 signed +xplain1.3-6base-32k-step1647 Layer 06 199237632 0.001188 signed +xplain1.3-6base-32k-step1647 Layer 05 199237632 0.001171 signed +xplain1.3-6base-32k-step1647 Layer 04 199237632 0.001147 signed +xplain1.3-6base-32k-step1647 Layer 00 199237632 0.001121 signed +xplain1.3-6base-32k-step1647 Layer 01 199237632 0.001114 signed +xplain1.3-6base-32k-step1647 Layer 03 199237632 0.001100 signed +xplain1.3-6base-32k-step1647 Layer 02 199237632 0.001088 signed + +### `xplain1.3-6base-64k-step1647` + + model block_label count std stat_mode +xplain1.3-6base-64k-step1647 Final Norm 4096 0.006869 signed +xplain1.3-6base-64k-step1647 Embedding 201326592 0.002014 signed +xplain1.3-6base-64k-step1647 Layer 36 199237632 0.001877 signed +xplain1.3-6base-64k-step1647 Layer 37 199237632 0.001862 signed +xplain1.3-6base-64k-step1647 Layer 34 199237632 0.001847 signed +xplain1.3-6base-64k-step1647 Layer 38 199237632 0.001836 signed +xplain1.3-6base-64k-step1647 Layer 35 199237632 0.001819 signed +xplain1.3-6base-64k-step1647 Layer 31 199237632 0.001748 signed +xplain1.3-6base-64k-step1647 Layer 30 199237632 0.001732 signed +xplain1.3-6base-64k-step1647 Layer 33 199237632 0.001730 signed +xplain1.3-6base-64k-step1647 Layer 32 199237632 0.001722 signed +xplain1.3-6base-64k-step1647 Layer 28 199237632 0.001717 signed +xplain1.3-6base-64k-step1647 Layer 25 199237632 0.001685 signed +xplain1.3-6base-64k-step1647 Layer 22 199237632 0.001650 signed +xplain1.3-6base-64k-step1647 Layer 21 199237632 0.001639 signed +xplain1.3-6base-64k-step1647 Layer 39 199237632 0.001636 signed +xplain1.3-6base-64k-step1647 Layer 29 199237632 0.001614 signed +xplain1.3-6base-64k-step1647 Layer 19 199237632 0.001598 signed +xplain1.3-6base-64k-step1647 Layer 27 199237632 0.001595 signed +xplain1.3-6base-64k-step1647 Layer 26 199237632 0.001583 signed +xplain1.3-6base-64k-step1647 Layer 18 199237632 0.001580 signed +xplain1.3-6base-64k-step1647 Layer 17 199237632 0.001566 signed +xplain1.3-6base-64k-step1647 Layer 24 199237632 0.001556 signed +xplain1.3-6base-64k-step1647 Layer 23 199237632 0.001542 signed +xplain1.3-6base-64k-step1647 Layer 16 199237632 0.001539 signed +xplain1.3-6base-64k-step1647 Layer 15 199237632 0.001527 signed +xplain1.3-6base-64k-step1647 Layer 20 199237632 0.001506 signed +xplain1.3-6base-64k-step1647 Layer 14 199237632 0.001493 signed +xplain1.3-6base-64k-step1647 Layer 13 199237632 0.001467 signed +xplain1.3-6base-64k-step1647 Layer 12 199237632 0.001456 signed +xplain1.3-6base-64k-step1647 Layer 11 199237632 0.001433 signed +xplain1.3-6base-64k-step1647 Layer 10 199237632 0.001413 signed +xplain1.3-6base-64k-step1647 Layer 09 199237632 0.001394 signed +xplain1.3-6base-64k-step1647 Layer 08 199237632 0.001377 signed +xplain1.3-6base-64k-step1647 Layer 00 199237632 0.001373 signed +xplain1.3-6base-64k-step1647 Layer 01 199237632 0.001365 signed +xplain1.3-6base-64k-step1647 Layer 07 199237632 0.001354 signed +xplain1.3-6base-64k-step1647 Layer 06 199237632 0.001354 signed +xplain1.3-6base-64k-step1647 Layer 04 199237632 0.001353 signed +xplain1.3-6base-64k-step1647 Layer 05 199237632 0.001339 signed +xplain1.3-6base-64k-step1647 Layer 02 199237632 0.001336 signed +xplain1.3-6base-64k-step1647 Layer 03 199237632 0.001333 signed + +### `xplain1.3-6base-8k-step2199` + + model block_label count std stat_mode +xplain1.3-6base-8k-step2199 Final Norm 4096 0.002388 signed +xplain1.3-6base-8k-step2199 Layer 37 199237632 0.001120 signed +xplain1.3-6base-8k-step2199 Layer 36 199237632 0.001117 signed +xplain1.3-6base-8k-step2199 Layer 38 199237632 0.001110 signed +xplain1.3-6base-8k-step2199 Layer 34 199237632 0.001072 signed +xplain1.3-6base-8k-step2199 Layer 35 199237632 0.001063 signed +xplain1.3-6base-8k-step2199 Embedding 201326592 0.001027 signed +xplain1.3-6base-8k-step2199 Layer 39 199237632 0.000948 signed +xplain1.3-6base-8k-step2199 Layer 31 199237632 0.000937 signed +xplain1.3-6base-8k-step2199 Layer 33 199237632 0.000935 signed +xplain1.3-6base-8k-step2199 Layer 30 199237632 0.000921 signed +xplain1.3-6base-8k-step2199 Layer 32 199237632 0.000918 signed +xplain1.3-6base-8k-step2199 Layer 28 199237632 0.000914 signed +xplain1.3-6base-8k-step2199 Layer 21 199237632 0.000904 signed +xplain1.3-6base-8k-step2199 Layer 22 199237632 0.000904 signed +xplain1.3-6base-8k-step2199 Layer 25 199237632 0.000902 signed +xplain1.3-6base-8k-step2199 Layer 19 199237632 0.000894 signed +xplain1.3-6base-8k-step2199 Layer 18 199237632 0.000884 signed +xplain1.3-6base-8k-step2199 Layer 17 199237632 0.000881 signed +xplain1.3-6base-8k-step2199 Layer 29 199237632 0.000868 signed +xplain1.3-6base-8k-step2199 Layer 27 199237632 0.000862 signed +xplain1.3-6base-8k-step2199 Layer 16 199237632 0.000860 signed +xplain1.3-6base-8k-step2199 Layer 26 199237632 0.000857 signed +xplain1.3-6base-8k-step2199 Layer 15 199237632 0.000855 signed +xplain1.3-6base-8k-step2199 Layer 24 199237632 0.000845 signed +xplain1.3-6base-8k-step2199 Layer 23 199237632 0.000838 signed +xplain1.3-6base-8k-step2199 Layer 14 199237632 0.000833 signed +xplain1.3-6base-8k-step2199 Layer 20 199237632 0.000832 signed +xplain1.3-6base-8k-step2199 Layer 13 199237632 0.000808 signed +xplain1.3-6base-8k-step2199 Layer 12 199237632 0.000802 signed +xplain1.3-6base-8k-step2199 Layer 11 199237632 0.000783 signed +xplain1.3-6base-8k-step2199 Layer 10 199237632 0.000761 signed +xplain1.3-6base-8k-step2199 Layer 09 199237632 0.000736 signed +xplain1.3-6base-8k-step2199 Layer 08 199237632 0.000710 signed +xplain1.3-6base-8k-step2199 Layer 07 199237632 0.000691 signed +xplain1.3-6base-8k-step2199 Layer 06 199237632 0.000684 signed +xplain1.3-6base-8k-step2199 Layer 05 199237632 0.000665 signed +xplain1.3-6base-8k-step2199 Layer 04 199237632 0.000648 signed +xplain1.3-6base-8k-step2199 Layer 03 199237632 0.000563 signed +xplain1.3-6base-8k-step2199 Layer 02 199237632 0.000544 signed +xplain1.3-6base-8k-step2199 Layer 00 199237632 0.000541 signed +xplain1.3-6base-8k-step2199 Layer 01 199237632 0.000535 signed + +### `xplain1.4-6base-16k-step2319` + + model block_label count std stat_mode +xplain1.4-6base-16k-step2319 Final Norm 4096 0.005594 signed +xplain1.4-6base-16k-step2319 Layer 38 199237632 0.001784 signed +xplain1.4-6base-16k-step2319 Layer 37 199237632 0.001714 signed +xplain1.4-6base-16k-step2319 Embedding 201326592 0.001679 signed +xplain1.4-6base-16k-step2319 Layer 39 199237632 0.001580 signed +xplain1.4-6base-16k-step2319 Layer 36 199237632 0.001482 signed +xplain1.4-6base-16k-step2319 Layer 35 199237632 0.001424 signed +xplain1.4-6base-16k-step2319 Layer 34 199237632 0.001416 signed +xplain1.4-6base-16k-step2319 Layer 31 199237632 0.001349 signed +xplain1.4-6base-16k-step2319 Layer 32 199237632 0.001348 signed +xplain1.4-6base-16k-step2319 Layer 33 199237632 0.001346 signed +xplain1.4-6base-16k-step2319 Layer 30 199237632 0.001339 signed +xplain1.4-6base-16k-step2319 Layer 28 199237632 0.001325 signed +xplain1.4-6base-16k-step2319 Layer 25 199237632 0.001304 signed +xplain1.4-6base-16k-step2319 Layer 22 199237632 0.001278 signed +xplain1.4-6base-16k-step2319 Layer 21 199237632 0.001266 signed +xplain1.4-6base-16k-step2319 Layer 19 199237632 0.001251 signed +xplain1.4-6base-16k-step2319 Layer 29 199237632 0.001246 signed +xplain1.4-6base-16k-step2319 Layer 18 199237632 0.001232 signed +xplain1.4-6base-16k-step2319 Layer 27 199237632 0.001229 signed +xplain1.4-6base-16k-step2319 Layer 17 199237632 0.001220 signed +xplain1.4-6base-16k-step2319 Layer 26 199237632 0.001218 signed +xplain1.4-6base-16k-step2319 Layer 24 199237632 0.001200 signed +xplain1.4-6base-16k-step2319 Layer 16 199237632 0.001199 signed +xplain1.4-6base-16k-step2319 Layer 23 199237632 0.001190 signed +xplain1.4-6base-16k-step2319 Layer 15 199237632 0.001180 signed +xplain1.4-6base-16k-step2319 Layer 20 199237632 0.001160 signed +xplain1.4-6base-16k-step2319 Layer 14 199237632 0.001151 signed +xplain1.4-6base-16k-step2319 Layer 13 199237632 0.001126 signed +xplain1.4-6base-16k-step2319 Layer 12 199237632 0.001113 signed +xplain1.4-6base-16k-step2319 Layer 11 199237632 0.001093 signed +xplain1.4-6base-16k-step2319 Layer 10 199237632 0.001075 signed +xplain1.4-6base-16k-step2319 Layer 09 199237632 0.001057 signed +xplain1.4-6base-16k-step2319 Layer 08 199237632 0.001039 signed +xplain1.4-6base-16k-step2319 Layer 00 199237632 0.001036 signed +xplain1.4-6base-16k-step2319 Layer 04 199237632 0.001036 signed +xplain1.4-6base-16k-step2319 Layer 07 199237632 0.001030 signed +xplain1.4-6base-16k-step2319 Layer 06 199237632 0.001030 signed +xplain1.4-6base-16k-step2319 Layer 05 199237632 0.001024 signed +xplain1.4-6base-16k-step2319 Layer 01 199237632 0.001006 signed +xplain1.4-6base-16k-step2319 Layer 02 199237632 0.001004 signed +xplain1.4-6base-16k-step2319 Layer 03 199237632 0.001000 signed + +### `xplain1.4-6base-32k-step1743` + + model block_label count std stat_mode +xplain1.4-6base-32k-step1743 Final Norm 4096 0.006660 signed +xplain1.4-6base-32k-step1743 Embedding 201326592 0.002007 signed +xplain1.4-6base-32k-step1743 Layer 38 199237632 0.001992 signed +xplain1.4-6base-32k-step1743 Layer 37 199237632 0.001929 signed +xplain1.4-6base-32k-step1743 Layer 39 199237632 0.001778 signed +xplain1.4-6base-32k-step1743 Layer 36 199237632 0.001722 signed +xplain1.4-6base-32k-step1743 Layer 35 199237632 0.001661 signed +xplain1.4-6base-32k-step1743 Layer 34 199237632 0.001658 signed +xplain1.4-6base-32k-step1743 Layer 31 199237632 0.001601 signed +xplain1.4-6base-32k-step1743 Layer 32 199237632 0.001594 signed +xplain1.4-6base-32k-step1743 Layer 30 199237632 0.001592 signed +xplain1.4-6base-32k-step1743 Layer 33 199237632 0.001590 signed +xplain1.4-6base-32k-step1743 Layer 28 199237632 0.001575 signed +xplain1.4-6base-32k-step1743 Layer 25 199237632 0.001549 signed +xplain1.4-6base-32k-step1743 Layer 22 199237632 0.001512 signed +xplain1.4-6base-32k-step1743 Layer 21 199237632 0.001498 signed +xplain1.4-6base-32k-step1743 Layer 29 199237632 0.001482 signed +xplain1.4-6base-32k-step1743 Layer 19 199237632 0.001474 signed +xplain1.4-6base-32k-step1743 Layer 27 199237632 0.001462 signed +xplain1.4-6base-32k-step1743 Layer 18 199237632 0.001452 signed +xplain1.4-6base-32k-step1743 Layer 26 199237632 0.001448 signed +xplain1.4-6base-32k-step1743 Layer 17 199237632 0.001438 signed +xplain1.4-6base-32k-step1743 Layer 24 199237632 0.001426 signed +xplain1.4-6base-32k-step1743 Layer 16 199237632 0.001417 signed +xplain1.4-6base-32k-step1743 Layer 23 199237632 0.001414 signed +xplain1.4-6base-32k-step1743 Layer 15 199237632 0.001397 signed +xplain1.4-6base-32k-step1743 Layer 20 199237632 0.001376 signed +xplain1.4-6base-32k-step1743 Layer 14 199237632 0.001367 signed +xplain1.4-6base-32k-step1743 Layer 13 199237632 0.001344 signed +xplain1.4-6base-32k-step1743 Layer 00 199237632 0.001340 signed +xplain1.4-6base-32k-step1743 Layer 12 199237632 0.001331 signed +xplain1.4-6base-32k-step1743 Layer 01 199237632 0.001320 signed +xplain1.4-6base-32k-step1743 Layer 11 199237632 0.001314 signed +xplain1.4-6base-32k-step1743 Layer 10 199237632 0.001301 signed +xplain1.4-6base-32k-step1743 Layer 04 199237632 0.001290 signed +xplain1.4-6base-32k-step1743 Layer 09 199237632 0.001289 signed +xplain1.4-6base-32k-step1743 Layer 02 199237632 0.001284 signed +xplain1.4-6base-32k-step1743 Layer 08 199237632 0.001276 signed +xplain1.4-6base-32k-step1743 Layer 03 199237632 0.001275 signed +xplain1.4-6base-32k-step1743 Layer 05 199237632 0.001270 signed +xplain1.4-6base-32k-step1743 Layer 06 199237632 0.001269 signed +xplain1.4-6base-32k-step1743 Layer 07 199237632 0.001267 signed + +### `xplain1.4-6base-64k-step1743` + + model block_label count std stat_mode +xplain1.4-6base-64k-step1743 Final Norm 4096 0.007889 signed +xplain1.4-6base-64k-step1743 Embedding 201326592 0.002247 signed +xplain1.4-6base-64k-step1743 Layer 38 199237632 0.002120 signed +xplain1.4-6base-64k-step1743 Layer 37 199237632 0.002040 signed +xplain1.4-6base-64k-step1743 Layer 39 199237632 0.001909 signed +xplain1.4-6base-64k-step1743 Layer 36 199237632 0.001846 signed +xplain1.4-6base-64k-step1743 Layer 34 199237632 0.001789 signed +xplain1.4-6base-64k-step1743 Layer 35 199237632 0.001785 signed +xplain1.4-6base-64k-step1743 Layer 31 199237632 0.001745 signed +xplain1.4-6base-64k-step1743 Layer 30 199237632 0.001735 signed +xplain1.4-6base-64k-step1743 Layer 32 199237632 0.001732 signed +xplain1.4-6base-64k-step1743 Layer 33 199237632 0.001722 signed +xplain1.4-6base-64k-step1743 Layer 28 199237632 0.001721 signed +xplain1.4-6base-64k-step1743 Layer 25 199237632 0.001697 signed +xplain1.4-6base-64k-step1743 Layer 22 199237632 0.001658 signed +xplain1.4-6base-64k-step1743 Layer 21 199237632 0.001643 signed +xplain1.4-6base-64k-step1743 Layer 29 199237632 0.001619 signed +xplain1.4-6base-64k-step1743 Layer 19 199237632 0.001614 signed +xplain1.4-6base-64k-step1743 Layer 27 199237632 0.001600 signed +xplain1.4-6base-64k-step1743 Layer 18 199237632 0.001592 signed +xplain1.4-6base-64k-step1743 Layer 26 199237632 0.001587 signed +xplain1.4-6base-64k-step1743 Layer 17 199237632 0.001577 signed +xplain1.4-6base-64k-step1743 Layer 24 199237632 0.001565 signed +xplain1.4-6base-64k-step1743 Layer 16 199237632 0.001554 signed +xplain1.4-6base-64k-step1743 Layer 23 199237632 0.001552 signed +xplain1.4-6base-64k-step1743 Layer 15 199237632 0.001533 signed +xplain1.4-6base-64k-step1743 Layer 20 199237632 0.001510 signed +xplain1.4-6base-64k-step1743 Layer 14 199237632 0.001501 signed +xplain1.4-6base-64k-step1743 Layer 13 199237632 0.001479 signed +xplain1.4-6base-64k-step1743 Layer 12 199237632 0.001466 signed +xplain1.4-6base-64k-step1743 Layer 11 199237632 0.001449 signed +xplain1.4-6base-64k-step1743 Layer 10 199237632 0.001436 signed +xplain1.4-6base-64k-step1743 Layer 09 199237632 0.001425 signed +xplain1.4-6base-64k-step1743 Layer 08 199237632 0.001414 signed +xplain1.4-6base-64k-step1743 Layer 00 199237632 0.001412 signed +xplain1.4-6base-64k-step1743 Layer 07 199237632 0.001401 signed +xplain1.4-6base-64k-step1743 Layer 01 199237632 0.001401 signed +xplain1.4-6base-64k-step1743 Layer 06 199237632 0.001400 signed +xplain1.4-6base-64k-step1743 Layer 05 199237632 0.001388 signed +xplain1.4-6base-64k-step1743 Layer 04 199237632 0.001370 signed +xplain1.4-6base-64k-step1743 Layer 02 199237632 0.001358 signed +xplain1.4-6base-64k-step1743 Layer 03 199237632 0.001353 signed + +### `xplain1.4-6base-8k-step2319` + + model block_label count std stat_mode +xplain1.4-6base-8k-step2319 Final Norm 4096 0.003269 signed +xplain1.4-6base-8k-step2319 Layer 38 199237632 0.001407 signed +xplain1.4-6base-8k-step2319 Layer 37 199237632 0.001340 signed +xplain1.4-6base-8k-step2319 Layer 39 199237632 0.001194 signed +xplain1.4-6base-8k-step2319 Embedding 201326592 0.001185 signed +xplain1.4-6base-8k-step2319 Layer 36 199237632 0.001110 signed +xplain1.4-6base-8k-step2319 Layer 35 199237632 0.001067 signed +xplain1.4-6base-8k-step2319 Layer 34 199237632 0.001059 signed +xplain1.4-6base-8k-step2319 Layer 33 199237632 0.000993 signed +xplain1.4-6base-8k-step2319 Layer 32 199237632 0.000990 signed +xplain1.4-6base-8k-step2319 Layer 31 199237632 0.000988 signed +xplain1.4-6base-8k-step2319 Layer 30 199237632 0.000982 signed +xplain1.4-6base-8k-step2319 Layer 28 199237632 0.000975 signed +xplain1.4-6base-8k-step2319 Layer 25 199237632 0.000966 signed +xplain1.4-6base-8k-step2319 Layer 22 199237632 0.000954 signed +xplain1.4-6base-8k-step2319 Layer 19 199237632 0.000948 signed +xplain1.4-6base-8k-step2319 Layer 21 199237632 0.000948 signed +xplain1.4-6base-8k-step2319 Layer 18 199237632 0.000933 signed +xplain1.4-6base-8k-step2319 Layer 17 199237632 0.000926 signed +xplain1.4-6base-8k-step2319 Layer 29 199237632 0.000920 signed +xplain1.4-6base-8k-step2319 Layer 27 199237632 0.000912 signed +xplain1.4-6base-8k-step2319 Layer 16 199237632 0.000910 signed +xplain1.4-6base-8k-step2319 Layer 26 199237632 0.000907 signed +xplain1.4-6base-8k-step2319 Layer 15 199237632 0.000896 signed +xplain1.4-6base-8k-step2319 Layer 24 199237632 0.000896 signed +xplain1.4-6base-8k-step2319 Layer 23 199237632 0.000890 signed +xplain1.4-6base-8k-step2319 Layer 14 199237632 0.000873 signed +xplain1.4-6base-8k-step2319 Layer 20 199237632 0.000871 signed +xplain1.4-6base-8k-step2319 Layer 13 199237632 0.000852 signed +xplain1.4-6base-8k-step2319 Layer 12 199237632 0.000840 signed +xplain1.4-6base-8k-step2319 Layer 11 199237632 0.000826 signed +xplain1.4-6base-8k-step2319 Layer 10 199237632 0.000811 signed +xplain1.4-6base-8k-step2319 Layer 09 199237632 0.000794 signed +xplain1.4-6base-8k-step2319 Layer 04 199237632 0.000781 signed +xplain1.4-6base-8k-step2319 Layer 08 199237632 0.000777 signed +xplain1.4-6base-8k-step2319 Layer 07 199237632 0.000770 signed +xplain1.4-6base-8k-step2319 Layer 06 199237632 0.000769 signed +xplain1.4-6base-8k-step2319 Layer 05 199237632 0.000766 signed +xplain1.4-6base-8k-step2319 Layer 00 199237632 0.000746 signed +xplain1.4-6base-8k-step2319 Layer 02 199237632 0.000734 signed +xplain1.4-6base-8k-step2319 Layer 03 199237632 0.000734 signed +xplain1.4-6base-8k-step2319 Layer 01 199237632 0.000724 signed + +### `xplain1.4-7base-32k-step1743` + + model block_label count std stat_mode +xplain1.4-7base-32k-step1743 Final Norm 4096 0.006105 signed +xplain1.4-7base-32k-step1743 Embedding 201326592 0.001716 signed +xplain1.4-7base-32k-step1743 Layer 36 199237632 0.001582 signed +xplain1.4-7base-32k-step1743 Layer 38 199237632 0.001579 signed +xplain1.4-7base-32k-step1743 Layer 37 199237632 0.001578 signed +xplain1.4-7base-32k-step1743 Layer 34 199237632 0.001548 signed +xplain1.4-7base-32k-step1743 Layer 35 199237632 0.001533 signed +xplain1.4-7base-32k-step1743 Layer 31 199237632 0.001490 signed +xplain1.4-7base-32k-step1743 Layer 32 199237632 0.001485 signed +xplain1.4-7base-32k-step1743 Layer 33 199237632 0.001478 signed +xplain1.4-7base-32k-step1743 Layer 30 199237632 0.001478 signed +xplain1.4-7base-32k-step1743 Layer 39 199237632 0.001470 signed +xplain1.4-7base-32k-step1743 Layer 28 199237632 0.001458 signed +xplain1.4-7base-32k-step1743 Layer 25 199237632 0.001429 signed +xplain1.4-7base-32k-step1743 Layer 22 199237632 0.001393 signed +xplain1.4-7base-32k-step1743 Layer 21 199237632 0.001378 signed +xplain1.4-7base-32k-step1743 Layer 29 199237632 0.001366 signed +xplain1.4-7base-32k-step1743 Layer 19 199237632 0.001364 signed +xplain1.4-7base-32k-step1743 Layer 27 199237632 0.001347 signed +xplain1.4-7base-32k-step1743 Layer 18 199237632 0.001343 signed +xplain1.4-7base-32k-step1743 Layer 17 199237632 0.001337 signed +xplain1.4-7base-32k-step1743 Layer 26 199237632 0.001335 signed +xplain1.4-7base-32k-step1743 Layer 16 199237632 0.001315 signed +xplain1.4-7base-32k-step1743 Layer 24 199237632 0.001310 signed +xplain1.4-7base-32k-step1743 Layer 15 199237632 0.001303 signed +xplain1.4-7base-32k-step1743 Layer 23 199237632 0.001299 signed +xplain1.4-7base-32k-step1743 Layer 14 199237632 0.001274 signed +xplain1.4-7base-32k-step1743 Layer 20 199237632 0.001263 signed +xplain1.4-7base-32k-step1743 Layer 13 199237632 0.001257 signed +xplain1.4-7base-32k-step1743 Layer 12 199237632 0.001241 signed +xplain1.4-7base-32k-step1743 Layer 11 199237632 0.001209 signed +xplain1.4-7base-32k-step1743 Layer 10 199237632 0.001181 signed +xplain1.4-7base-32k-step1743 Layer 09 199237632 0.001158 signed +xplain1.4-7base-32k-step1743 Layer 08 199237632 0.001131 signed +xplain1.4-7base-32k-step1743 Layer 06 199237632 0.001112 signed +xplain1.4-7base-32k-step1743 Layer 07 199237632 0.001107 signed +xplain1.4-7base-32k-step1743 Layer 05 199237632 0.001093 signed +xplain1.4-7base-32k-step1743 Layer 04 199237632 0.001086 signed +xplain1.4-7base-32k-step1743 Layer 03 199237632 0.001056 signed +xplain1.4-7base-32k-step1743 Layer 02 199237632 0.001010 signed +xplain1.4-7base-32k-step1743 Layer 01 199237632 0.000961 signed +xplain1.4-7base-32k-step1743 Layer 00 199237632 0.000917 signed + +## Module types by std + +### `xplain1.3-6base-16k-step2199` + + model module_type count std stat_mode +xplain1.3-6base-16k-step2199 final_norm 4096 0.003860 signed +xplain1.3-6base-16k-step2199 input_layernorm 163840 0.002186 signed +xplain1.3-6base-16k-step2199 embed_tokens 201326592 0.001496 signed +xplain1.3-6base-16k-step2199 mlp.gate_proj 2097152000 0.001318 signed +xplain1.3-6base-16k-step2199 post_attention_layernorm 163840 0.001232 signed +xplain1.3-6base-16k-step2199 mlp.up_proj 2097152000 0.001229 signed +xplain1.3-6base-16k-step2199 mlp.down_proj 2097152000 0.001185 signed +xplain1.3-6base-16k-step2199 self_attn.q_proj 671088640 0.001164 signed +xplain1.3-6base-16k-step2199 self_attn.k_proj 167772160 0.001097 signed +xplain1.3-6base-16k-step2199 self_attn.o_proj 671088640 0.000932 signed +xplain1.3-6base-16k-step2199 self_attn.v_proj 167772160 0.000902 signed + +### `xplain1.3-6base-32k-step1647` + + model module_type count std stat_mode +xplain1.3-6base-32k-step1647 final_norm 4096 0.005347 signed +xplain1.3-6base-32k-step1647 input_layernorm 163840 0.002933 signed +xplain1.3-6base-32k-step1647 embed_tokens 201326592 0.001762 signed +xplain1.3-6base-32k-step1647 mlp.gate_proj 2097152000 0.001524 signed +xplain1.3-6base-32k-step1647 post_attention_layernorm 163840 0.001465 signed +xplain1.3-6base-32k-step1647 mlp.up_proj 2097152000 0.001415 signed +xplain1.3-6base-32k-step1647 mlp.down_proj 2097152000 0.001370 signed +xplain1.3-6base-32k-step1647 self_attn.q_proj 671088640 0.001339 signed +xplain1.3-6base-32k-step1647 self_attn.k_proj 167772160 0.001255 signed +xplain1.3-6base-32k-step1647 self_attn.o_proj 671088640 0.001058 signed +xplain1.3-6base-32k-step1647 self_attn.v_proj 167772160 0.001018 signed + +### `xplain1.3-6base-64k-step1647` + + model module_type count std stat_mode +xplain1.3-6base-64k-step1647 final_norm 4096 0.006869 signed +xplain1.3-6base-64k-step1647 input_layernorm 163840 0.003692 signed +xplain1.3-6base-64k-step1647 embed_tokens 201326592 0.002014 signed +xplain1.3-6base-64k-step1647 post_attention_layernorm 163840 0.001730 signed +xplain1.3-6base-64k-step1647 mlp.gate_proj 2097152000 0.001716 signed +xplain1.3-6base-64k-step1647 mlp.up_proj 2097152000 0.001599 signed +xplain1.3-6base-64k-step1647 mlp.down_proj 2097152000 0.001552 signed +xplain1.3-6base-64k-step1647 self_attn.q_proj 671088640 0.001510 signed +xplain1.3-6base-64k-step1647 self_attn.k_proj 167772160 0.001416 signed +xplain1.3-6base-64k-step1647 self_attn.o_proj 671088640 0.001227 signed +xplain1.3-6base-64k-step1647 self_attn.v_proj 167772160 0.001183 signed + +### `xplain1.3-6base-8k-step2199` + + model module_type count std stat_mode +xplain1.3-6base-8k-step2199 final_norm 4096 0.002388 signed +xplain1.3-6base-8k-step2199 input_layernorm 163840 0.001203 signed +xplain1.3-6base-8k-step2199 embed_tokens 201326592 0.001027 signed +xplain1.3-6base-8k-step2199 mlp.gate_proj 2097152000 0.000944 signed +xplain1.3-6base-8k-step2199 mlp.up_proj 2097152000 0.000870 signed +xplain1.3-6base-8k-step2199 mlp.down_proj 2097152000 0.000842 signed +xplain1.3-6base-8k-step2199 self_attn.q_proj 671088640 0.000816 signed +xplain1.3-6base-8k-step2199 post_attention_layernorm 163840 0.000791 signed +xplain1.3-6base-8k-step2199 self_attn.k_proj 167772160 0.000770 signed +xplain1.3-6base-8k-step2199 self_attn.o_proj 671088640 0.000597 signed +xplain1.3-6base-8k-step2199 self_attn.v_proj 167772160 0.000571 signed + +### `xplain1.4-6base-16k-step2319` + + model module_type count std stat_mode +xplain1.4-6base-16k-step2319 final_norm 4096 0.005594 signed +xplain1.4-6base-16k-step2319 input_layernorm 163840 0.002340 signed +xplain1.4-6base-16k-step2319 embed_tokens 201326592 0.001679 signed +xplain1.4-6base-16k-step2319 mlp.gate_proj 2097152000 0.001364 signed +xplain1.4-6base-16k-step2319 post_attention_layernorm 163840 0.001322 signed +xplain1.4-6base-16k-step2319 mlp.up_proj 2097152000 0.001263 signed +xplain1.4-6base-16k-step2319 mlp.down_proj 2097152000 0.001219 signed +xplain1.4-6base-16k-step2319 self_attn.q_proj 671088640 0.001180 signed +xplain1.4-6base-16k-step2319 self_attn.k_proj 167772160 0.001101 signed +xplain1.4-6base-16k-step2319 self_attn.o_proj 671088640 0.000976 signed +xplain1.4-6base-16k-step2319 self_attn.v_proj 167772160 0.000919 signed + +### `xplain1.4-6base-32k-step1743` + + model module_type count std stat_mode +xplain1.4-6base-32k-step1743 final_norm 4096 0.006660 signed +xplain1.4-6base-32k-step1743 input_layernorm 163840 0.003153 signed +xplain1.4-6base-32k-step1743 embed_tokens 201326592 0.002007 signed +xplain1.4-6base-32k-step1743 mlp.gate_proj 2097152000 0.001612 signed +xplain1.4-6base-32k-step1743 post_attention_layernorm 163840 0.001604 signed +xplain1.4-6base-32k-step1743 mlp.up_proj 2097152000 0.001500 signed +xplain1.4-6base-32k-step1743 mlp.down_proj 2097152000 0.001444 signed +xplain1.4-6base-32k-step1743 self_attn.q_proj 671088640 0.001407 signed +xplain1.4-6base-32k-step1743 self_attn.k_proj 167772160 0.001313 signed +xplain1.4-6base-32k-step1743 self_attn.o_proj 671088640 0.001195 signed +xplain1.4-6base-32k-step1743 self_attn.v_proj 167772160 0.001136 signed + +### `xplain1.4-6base-64k-step1743` + + model module_type count std stat_mode +xplain1.4-6base-64k-step1743 final_norm 4096 0.007889 signed +xplain1.4-6base-64k-step1743 input_layernorm 163840 0.003853 signed +xplain1.4-6base-64k-step1743 embed_tokens 201326592 0.002247 signed +xplain1.4-6base-64k-step1743 post_attention_layernorm 163840 0.001784 signed +xplain1.4-6base-64k-step1743 mlp.gate_proj 2097152000 0.001755 signed +xplain1.4-6base-64k-step1743 mlp.up_proj 2097152000 0.001630 signed +xplain1.4-6base-64k-step1743 mlp.down_proj 2097152000 0.001564 signed +xplain1.4-6base-64k-step1743 self_attn.q_proj 671088640 0.001528 signed +xplain1.4-6base-64k-step1743 self_attn.k_proj 167772160 0.001424 signed +xplain1.4-6base-64k-step1743 self_attn.o_proj 671088640 0.001301 signed +xplain1.4-6base-64k-step1743 self_attn.v_proj 167772160 0.001245 signed + +### `xplain1.4-6base-8k-step2319` + + model module_type count std stat_mode +xplain1.4-6base-8k-step2319 final_norm 4096 0.003269 signed +xplain1.4-6base-8k-step2319 input_layernorm 163840 0.001287 signed +xplain1.4-6base-8k-step2319 embed_tokens 201326592 0.001185 signed +xplain1.4-6base-8k-step2319 mlp.gate_proj 2097152000 0.001025 signed +xplain1.4-6base-8k-step2319 mlp.up_proj 2097152000 0.000950 signed +xplain1.4-6base-8k-step2319 mlp.down_proj 2097152000 0.000910 signed +xplain1.4-6base-8k-step2319 post_attention_layernorm 163840 0.000906 signed +xplain1.4-6base-8k-step2319 self_attn.q_proj 671088640 0.000873 signed +xplain1.4-6base-8k-step2319 self_attn.k_proj 167772160 0.000821 signed +xplain1.4-6base-8k-step2319 self_attn.o_proj 671088640 0.000747 signed +xplain1.4-6base-8k-step2319 self_attn.v_proj 167772160 0.000709 signed + +### `xplain1.4-7base-32k-step1743` + + model module_type count std stat_mode +xplain1.4-7base-32k-step1743 final_norm 4096 0.006105 signed +xplain1.4-7base-32k-step1743 input_layernorm 163840 0.002731 signed +xplain1.4-7base-32k-step1743 embed_tokens 201326592 0.001716 signed +xplain1.4-7base-32k-step1743 post_attention_layernorm 163840 0.001456 signed +xplain1.4-7base-32k-step1743 mlp.gate_proj 2097152000 0.001452 signed +xplain1.4-7base-32k-step1743 mlp.up_proj 2097152000 0.001344 signed +xplain1.4-7base-32k-step1743 mlp.down_proj 2097152000 0.001307 signed +xplain1.4-7base-32k-step1743 self_attn.q_proj 671088640 0.001239 signed +xplain1.4-7base-32k-step1743 self_attn.k_proj 167772160 0.001151 signed +xplain1.4-7base-32k-step1743 self_attn.o_proj 671088640 0.000990 signed +xplain1.4-7base-32k-step1743 self_attn.v_proj 167772160 0.000926 signed +