nohup: ignoring input
当前指标：ZD
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:09<00:27,  9.21s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:18<00:18,  9.20s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:27<00:09,  9.24s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:30<00:00,  6.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:30<00:00,  7.54s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, a group of 3,000 people lived in a small village. The people were all very happy and had a peaceful life. One day, a visitor came to the village. He was a very handsome man, and the people of the village were very impressed by him. They wanted to know more about him, so they asked him to tell them about his life.
The visitor told the people of the village that he was from a very powerful and wealthy family. He said that he had everything
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 0 ---0.1303202509880066 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 1 ---0.14315810799598694 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 2 ---0.14602778851985931 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 3 ---0.14945906400680542 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 4 ---0.14816699922084808 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 5 ---0.14798566699028015 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 6 ---0.1493656039237976 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 7 ---0.14915773272514343 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 8 ---0.14589068293571472 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 9 ---0.14533302187919617 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 10 ---0.14568227529525757 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 11 ---0.14793498814105988 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 12 ---0.14989669620990753 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 13 ---0.1470077931880951 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 14 ---0.14632324874401093 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 15 ---0.14624223113059998 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 16 ---0.14805446565151215 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 17 ---0.14778870344161987 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 18 ---0.14963041245937347 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 19 ---0.14974866807460785 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 20 ---0.14976008236408234 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 21 ---0.15078411996364594 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 22 ---0.15186314284801483 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 23 ---0.15277639031410217 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 24 ---0.15071263909339905 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 25 ---0.15127883851528168 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 26 ---0.1505168378353119 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 27 ---0.14953136444091797 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 28 ---0.15086320042610168 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 29 ---0.15024136006832123 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 30 ---0.14679034054279327 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
ZD value of layer 31 ---0.14557501673698425 
[(0.15277639031410217, 23), (0.15186314284801483, 22), (0.15127883851528168, 25), (0.15086320042610168, 28), (0.15078411996364594, 21), (0.15071263909339905, 24), (0.1505168378353119, 26), (0.15024136006832123, 29), (0.14989669620990753, 12), (0.14976008236408234, 20), (0.14974866807460785, 19), (0.14963041245937347, 18), (0.14953136444091797, 27), (0.14945906400680542, 3), (0.1493656039237976, 6), (0.14915773272514343, 7), (0.14816699922084808, 4), (0.14805446565151215, 16), (0.14798566699028015, 5), (0.14793498814105988, 11), (0.14778870344161987, 17), (0.1470077931880951, 13), (0.14679034054279327, 30), (0.14632324874401093, 14), (0.14624223113059998, 15), (0.14602778851985931, 2), (0.14589068293571472, 8), (0.14568227529525757, 10), (0.14557501673698425, 31), (0.14533302187919617, 9), (0.14315810799598694, 1), (0.1303202509880066, 0)]
metric_name ZD: [23, 22, 25, 28, 21, 24, 26, 29, 12, 20, 19, 18, 27, 3, 6, 7, 4, 16, 5, 11, 17, 13, 30, 14, 15, 2, 8, 10, 31, 9, 1, 0]
当前指标：alpha
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:09<00:27,  9.33s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:18<00:18,  9.02s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:27<00:09,  9.03s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:29<00:00,  6.29s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:29<00:00,  7.32s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time there was a guy. He was a nice guy. He was a smart guy. He was a guy who was not very good at taking care of himself. He had a wife who loved him. He had a family who loved him. But he was not very good at taking care of himself. He was not very good at being a good husband. He was not very good at being a good father. He was not very good at being a good son. He was not very good at being
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 0 ---2.91736102104187 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 1 ---3.6096770763397217 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 2 ---4.369636058807373 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 3 ---3.645509958267212 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 4 ---4.020874977111816 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 5 ---3.767652988433838 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 6 ---3.7508127689361572 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 7 ---3.420032501220703 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 8 ---3.0236618518829346 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 9 ---2.772552013397217 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 10 ---2.9705560207366943 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 11 ---3.048171281814575 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 12 ---3.189865827560425 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 13 ---3.2777371406555176 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 14 ---3.2005908489227295 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 15 ---3.678138017654419 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 16 ---4.1130595207214355 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 17 ---3.8516407012939453 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 18 ---3.978748083114624 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 19 ---4.280331611633301 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 20 ---4.195464134216309 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 21 ---4.305949687957764 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 22 ---4.368300914764404 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 23 ---4.385020732879639 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 24 ---4.3416218757629395 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 25 ---4.0612688064575195 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 26 ---4.032166957855225 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 27 ---4.042068004608154 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 28 ---5.11770486831665 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 29 ---4.464050769805908 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 30 ---3.456982374191284 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha value of layer 31 ---3.2595152854919434 
[(5.11770486831665, 28), (4.464050769805908, 29), (4.385020732879639, 23), (4.369636058807373, 2), (4.368300914764404, 22), (4.3416218757629395, 24), (4.305949687957764, 21), (4.280331611633301, 19), (4.195464134216309, 20), (4.1130595207214355, 16), (4.0612688064575195, 25), (4.042068004608154, 27), (4.032166957855225, 26), (4.020874977111816, 4), (3.978748083114624, 18), (3.8516407012939453, 17), (3.767652988433838, 5), (3.7508127689361572, 6), (3.678138017654419, 15), (3.645509958267212, 3), (3.6096770763397217, 1), (3.456982374191284, 30), (3.420032501220703, 7), (3.2777371406555176, 13), (3.2595152854919434, 31), (3.2005908489227295, 14), (3.189865827560425, 12), (3.048171281814575, 11), (3.0236618518829346, 8), (2.9705560207366943, 10), (2.91736102104187, 0), (2.772552013397217, 9)]
metric_name alpha: [28, 29, 23, 2, 22, 24, 21, 19, 20, 16, 25, 27, 26, 4, 18, 17, 5, 6, 15, 3, 1, 30, 7, 13, 31, 14, 12, 11, 8, 10, 0, 9]
当前指标：alpha_hat
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:09<00:27,  9.12s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:17<00:17,  8.94s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:26<00:08,  8.85s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.20s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.21s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, in the world of marketing, there were two mighty forces that ruled the digital landscape: Google and Facebook. These two giants held the key to success for businesses looking to make their mark online.
But then, something extraordinary happened. A third contender emerged, armed with an innovative approach and a passion for transforming the way businesses engage with their audience. It was the dawn of Instagram marketing, and it was set to change the game forever.
The Power of Visual Storytelling
One of the most captivating
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 0 ---10.852300643920898 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 1 ---10.145772933959961 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 2 ---12.637839317321777 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 3 ---10.44900131225586 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 4 ---11.685830116271973 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 5 ---10.987555503845215 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 6 ---11.184784889221191 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 7 ---10.484479904174805 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 8 ---9.408217430114746 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 9 ---8.966535568237305 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 10 ---9.301414489746094 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 11 ---9.262565612792969 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 12 ---9.989272117614746 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 13 ---9.591448783874512 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 14 ---9.372644424438477 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 15 ---10.767301559448242 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 16 ---11.275540351867676 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 17 ---11.846827507019043 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 18 ---12.156525611877441 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 19 ---12.502681732177734 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 20 ---13.113412857055664 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 21 ---13.104711532592773 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 22 ---13.112133026123047 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 23 ---13.146225929260254 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 24 ---12.81999683380127 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 25 ---12.653292655944824 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 26 ---12.806440353393555 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 27 ---14.207252502441406 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 28 ---16.43989372253418 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 29 ---14.465442657470703 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 30 ---12.788350105285645 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
alpha_hat value of layer 31 ---13.2962064743042 
[(16.43989372253418, 28), (14.465442657470703, 29), (14.207252502441406, 27), (13.2962064743042, 31), (13.146225929260254, 23), (13.113412857055664, 20), (13.112133026123047, 22), (13.104711532592773, 21), (12.81999683380127, 24), (12.806440353393555, 26), (12.788350105285645, 30), (12.653292655944824, 25), (12.637839317321777, 2), (12.502681732177734, 19), (12.156525611877441, 18), (11.846827507019043, 17), (11.685830116271973, 4), (11.275540351867676, 16), (11.184784889221191, 6), (10.987555503845215, 5), (10.852300643920898, 0), (10.767301559448242, 15), (10.484479904174805, 7), (10.44900131225586, 3), (10.145772933959961, 1), (9.989272117614746, 12), (9.591448783874512, 13), (9.408217430114746, 8), (9.372644424438477, 14), (9.301414489746094, 10), (9.262565612792969, 11), (8.966535568237305, 9)]
metric_name alpha_hat: [28, 29, 27, 31, 23, 20, 22, 21, 24, 26, 30, 25, 2, 19, 18, 17, 4, 16, 6, 5, 0, 15, 7, 3, 1, 12, 13, 8, 14, 10, 11, 9]
当前指标：stable_rank
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:09<00:27,  9.05s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:18<00:18,  9.13s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:27<00:09,  9.05s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:29<00:00,  6.31s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:29<00:00,  7.32s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, there was a little girl who was very, very naughty. She was always making a mess and being a bother to her parents. One day, her parents decided that they had had enough and they sent her to live with her Aunt for a while. The girl was very sad to leave her home, but she knew that she had to go.
When she arrived at her Aunt’s house, she was surprised to find that it was much different than her own home. The girl had never seen such
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(76.8856, device='cuda:6')
spectral_norm tensor(32.1192, device='cuda:6')
frobenius_norm tensor(55.4457, device='cuda:6')
spectral_norm tensor(18.8411, device='cuda:6')
frobenius_norm tensor(14.8614, device='cuda:6')
spectral_norm tensor(1.9605, device='cuda:6')
frobenius_norm tensor(34.3062, device='cuda:6')
spectral_norm tensor(4.9343, device='cuda:6')
frobenius_norm tensor(98.5077, device='cuda:6')
spectral_norm tensor(10.0864, device='cuda:6')
frobenius_norm tensor(90.2663, device='cuda:6')
spectral_norm tensor(5.6548, device='cuda:6')
frobenius_norm tensor(90.4429, device='cuda:6')
spectral_norm tensor(6.7945, device='cuda:6')
stable_rank value of layer 0 ---92.50985717773438 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(79.7186, device='cuda:6')
spectral_norm tensor(12.8741, device='cuda:6')
frobenius_norm tensor(56.4302, device='cuda:6')
spectral_norm tensor(9.5045, device='cuda:6')
frobenius_norm tensor(18.3755, device='cuda:6')
spectral_norm tensor(1.3126, device='cuda:6')
frobenius_norm tensor(39.3655, device='cuda:6')
spectral_norm tensor(5.1106, device='cuda:6')
frobenius_norm tensor(99.7901, device='cuda:6')
spectral_norm tensor(7.4131, device='cuda:6')
frobenius_norm tensor(91.8738, device='cuda:6')
spectral_norm tensor(3.5568, device='cuda:6')
frobenius_norm tensor(91.7333, device='cuda:6')
spectral_norm tensor(5.1422, device='cuda:6')
stable_rank value of layer 1 ---213.65187072753906 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(76.9517, device='cuda:6')
spectral_norm tensor(9.4978, device='cuda:6')
frobenius_norm tensor(57.6473, device='cuda:6')
spectral_norm tensor(7.2243, device='cuda:6')
frobenius_norm tensor(15.2013, device='cuda:6')
spectral_norm tensor(1.0522, device='cuda:6')
frobenius_norm tensor(36.0830, device='cuda:6')
spectral_norm tensor(4.5946, device='cuda:6')
frobenius_norm tensor(102.2242, device='cuda:6')
spectral_norm tensor(8.5718, device='cuda:6')
frobenius_norm tensor(91.4413, device='cuda:6')
spectral_norm tensor(3.3492, device='cuda:6')
frobenius_norm tensor(91.8334, device='cuda:6')
spectral_norm tensor(5.1150, device='cuda:6')
stable_rank value of layer 2 ---229.96060180664062 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(77.2064, device='cuda:6')
spectral_norm tensor(10.1484, device='cuda:6')
frobenius_norm tensor(57.4959, device='cuda:6')
spectral_norm tensor(7.5375, device='cuda:6')
frobenius_norm tensor(17.8962, device='cuda:6')
spectral_norm tensor(1.0208, device='cuda:6')
frobenius_norm tensor(41.1775, device='cuda:6')
spectral_norm tensor(4.3115, device='cuda:6')
frobenius_norm tensor(107.6994, device='cuda:6')
spectral_norm tensor(9.9873, device='cuda:6')
frobenius_norm tensor(90.0023, device='cuda:6')
spectral_norm tensor(3.0939, device='cuda:6')
frobenius_norm tensor(89.8130, device='cuda:6')
spectral_norm tensor(4.4840, device='cuda:6')
stable_rank value of layer 3 ---268.34088134765625 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(76.3690, device='cuda:6')
spectral_norm tensor(9.9006, device='cuda:6')
frobenius_norm tensor(56.6955, device='cuda:6')
spectral_norm tensor(7.6416, device='cuda:6')
frobenius_norm tensor(19.1979, device='cuda:6')
spectral_norm tensor(1.1709, device='cuda:6')
frobenius_norm tensor(41.9950, device='cuda:6')
spectral_norm tensor(3.6805, device='cuda:6')
frobenius_norm tensor(112.9379, device='cuda:6')
spectral_norm tensor(11.9836, device='cuda:6')
frobenius_norm tensor(88.0967, device='cuda:6')
spectral_norm tensor(3.2062, device='cuda:6')
frobenius_norm tensor(87.9027, device='cuda:6')
spectral_norm tensor(4.8592, device='cuda:6')
stable_rank value of layer 4 ---240.6619415283203 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(76.0779, device='cuda:6')
spectral_norm tensor(9.3366, device='cuda:6')
frobenius_norm tensor(56.8678, device='cuda:6')
spectral_norm tensor(7.5451, device='cuda:6')
frobenius_norm tensor(16.0436, device='cuda:6')
spectral_norm tensor(1.0232, device='cuda:6')
frobenius_norm tensor(38.9231, device='cuda:6')
spectral_norm tensor(2.9655, device='cuda:6')
frobenius_norm tensor(112.7281, device='cuda:6')
spectral_norm tensor(12.0050, device='cuda:6')
frobenius_norm tensor(88.6070, device='cuda:6')
spectral_norm tensor(3.5774, device='cuda:6')
frobenius_norm tensor(88.3036, device='cuda:6')
spectral_norm tensor(4.7503, device='cuda:6')
stable_rank value of layer 5 ---226.9339141845703 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(77.5564, device='cuda:6')
spectral_norm tensor(9.4346, device='cuda:6')
frobenius_norm tensor(56.8842, device='cuda:6')
spectral_norm tensor(8.2064, device='cuda:6')
frobenius_norm tensor(17.3765, device='cuda:6')
spectral_norm tensor(1.1139, device='cuda:6')
frobenius_norm tensor(40.5982, device='cuda:6')
spectral_norm tensor(3.0531, device='cuda:6')
frobenius_norm tensor(113.1005, device='cuda:6')
spectral_norm tensor(12.2282, device='cuda:6')
frobenius_norm tensor(88.5713, device='cuda:6')
spectral_norm tensor(3.6645, device='cuda:6')
frobenius_norm tensor(88.2304, device='cuda:6')
spectral_norm tensor(5.4514, device='cuda:6')
stable_rank value of layer 6 ---209.63941955566406 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(74.5972, device='cuda:6')
spectral_norm tensor(9.6876, device='cuda:6')
frobenius_norm tensor(57.3617, device='cuda:6')
spectral_norm tensor(8.4875, device='cuda:6')
frobenius_norm tensor(17.1551, device='cuda:6')
spectral_norm tensor(1.1882, device='cuda:6')
frobenius_norm tensor(41.2336, device='cuda:6')
spectral_norm tensor(2.8647, device='cuda:6')
frobenius_norm tensor(111.0432, device='cuda:6')
spectral_norm tensor(11.8029, device='cuda:6')
frobenius_norm tensor(89.6853, device='cuda:6')
spectral_norm tensor(3.8356, device='cuda:6')
frobenius_norm tensor(89.3886, device='cuda:6')
spectral_norm tensor(6.4686, device='cuda:6')
stable_rank value of layer 7 ---192.40182495117188 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(74.6059, device='cuda:6')
spectral_norm tensor(8.0167, device='cuda:6')
frobenius_norm tensor(56.4533, device='cuda:6')
spectral_norm tensor(7.4413, device='cuda:6')
frobenius_norm tensor(18.0929, device='cuda:6')
spectral_norm tensor(1.1291, device='cuda:6')
frobenius_norm tensor(42.1143, device='cuda:6')
spectral_norm tensor(3.3892, device='cuda:6')
frobenius_norm tensor(111.3565, device='cuda:6')
spectral_norm tensor(11.8704, device='cuda:6')
frobenius_norm tensor(89.3811, device='cuda:6')
spectral_norm tensor(3.7573, device='cuda:6')
frobenius_norm tensor(89.2145, device='cuda:6')
spectral_norm tensor(6.2894, device='cuda:6')
stable_rank value of layer 8 ---201.4922332763672 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(74.5075, device='cuda:6')
spectral_norm tensor(8.3014, device='cuda:6')
frobenius_norm tensor(56.1731, device='cuda:6')
spectral_norm tensor(7.0673, device='cuda:6')
frobenius_norm tensor(20.9612, device='cuda:6')
spectral_norm tensor(1.3668, device='cuda:6')
frobenius_norm tensor(45.0585, device='cuda:6')
spectral_norm tensor(3.3602, device='cuda:6')
frobenius_norm tensor(112.6965, device='cuda:6')
spectral_norm tensor(12.7243, device='cuda:6')
frobenius_norm tensor(90.2687, device='cuda:6')
spectral_norm tensor(3.7826, device='cuda:6')
frobenius_norm tensor(89.5318, device='cuda:6')
spectral_norm tensor(6.5008, device='cuda:6')
stable_rank value of layer 9 ---199.4788360595703 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(75.8125, device='cuda:6')
spectral_norm tensor(8.0557, device='cuda:6')
frobenius_norm tensor(56.8728, device='cuda:6')
spectral_norm tensor(6.8490, device='cuda:6')
frobenius_norm tensor(18.1161, device='cuda:6')
spectral_norm tensor(1.2821, device='cuda:6')
frobenius_norm tensor(41.8618, device='cuda:6')
spectral_norm tensor(3.2646, device='cuda:6')
frobenius_norm tensor(110.1133, device='cuda:6')
spectral_norm tensor(12.3665, device='cuda:6')
frobenius_norm tensor(91.5807, device='cuda:6')
spectral_norm tensor(3.7899, device='cuda:6')
frobenius_norm tensor(90.7305, device='cuda:6')
spectral_norm tensor(6.3435, device='cuda:6')
stable_rank value of layer 10 ---198.48497009277344 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(72.1934, device='cuda:6')
spectral_norm tensor(8.1887, device='cuda:6')
frobenius_norm tensor(56.2865, device='cuda:6')
spectral_norm tensor(6.8306, device='cuda:6')
frobenius_norm tensor(17.7975, device='cuda:6')
spectral_norm tensor(1.2927, device='cuda:6')
frobenius_norm tensor(42.6298, device='cuda:6')
spectral_norm tensor(3.1474, device='cuda:6')
frobenius_norm tensor(108.6994, device='cuda:6')
spectral_norm tensor(12.0712, device='cuda:6')
frobenius_norm tensor(92.1148, device='cuda:6')
spectral_norm tensor(3.8644, device='cuda:6')
frobenius_norm tensor(91.3552, device='cuda:6')
spectral_norm tensor(7.2518, device='cuda:6')
stable_rank value of layer 11 ---189.51531982421875 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(72.9665, device='cuda:6')
spectral_norm tensor(10.0784, device='cuda:6')
frobenius_norm tensor(54.8966, device='cuda:6')
spectral_norm tensor(7.9653, device='cuda:6')
frobenius_norm tensor(20.8604, device='cuda:6')
spectral_norm tensor(1.2892, device='cuda:6')
frobenius_norm tensor(44.9508, device='cuda:6')
spectral_norm tensor(3.1262, device='cuda:6')
frobenius_norm tensor(107.3952, device='cuda:6')
spectral_norm tensor(12.6191, device='cuda:6')
frobenius_norm tensor(93.8434, device='cuda:6')
spectral_norm tensor(4.2624, device='cuda:6')
frobenius_norm tensor(92.6804, device='cuda:6')
spectral_norm tensor(6.7675, device='cuda:6')
stable_rank value of layer 12 ---187.5979766845703 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(73.6418, device='cuda:6')
spectral_norm tensor(8.6689, device='cuda:6')
frobenius_norm tensor(56.4655, device='cuda:6')
spectral_norm tensor(7.3055, device='cuda:6')
frobenius_norm tensor(19.4327, device='cuda:6')
spectral_norm tensor(1.1633, device='cuda:6')
frobenius_norm tensor(44.0239, device='cuda:6')
spectral_norm tensor(3.2448, device='cuda:6')
frobenius_norm tensor(107.4902, device='cuda:6')
spectral_norm tensor(13.0488, device='cuda:6')
frobenius_norm tensor(94.2244, device='cuda:6')
spectral_norm tensor(4.0377, device='cuda:6')
frobenius_norm tensor(92.7343, device='cuda:6')
spectral_norm tensor(5.5242, device='cuda:6')
stable_rank value of layer 13 ---212.75192260742188 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(72.2095, device='cuda:6')
spectral_norm tensor(8.4329, device='cuda:6')
frobenius_norm tensor(55.9640, device='cuda:6')
spectral_norm tensor(7.2572, device='cuda:6')
frobenius_norm tensor(19.2428, device='cuda:6')
spectral_norm tensor(1.1132, device='cuda:6')
frobenius_norm tensor(43.8044, device='cuda:6')
spectral_norm tensor(2.6452, device='cuda:6')
frobenius_norm tensor(109.8463, device='cuda:6')
spectral_norm tensor(13.1032, device='cuda:6')
frobenius_norm tensor(93.9992, device='cuda:6')
spectral_norm tensor(3.9854, device='cuda:6')
frobenius_norm tensor(92.6320, device='cuda:6')
spectral_norm tensor(5.2650, device='cuda:6')
stable_rank value of layer 14 ---234.5662841796875 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(78.9840, device='cuda:6')
spectral_norm tensor(9.9497, device='cuda:6')
frobenius_norm tensor(54.9467, device='cuda:6')
spectral_norm tensor(7.7205, device='cuda:6')
frobenius_norm tensor(20.9664, device='cuda:6')
spectral_norm tensor(1.2816, device='cuda:6')
frobenius_norm tensor(45.5146, device='cuda:6')
spectral_norm tensor(3.4394, device='cuda:6')
frobenius_norm tensor(112.3823, device='cuda:6')
spectral_norm tensor(14.2334, device='cuda:6')
frobenius_norm tensor(93.4551, device='cuda:6')
spectral_norm tensor(3.9940, device='cuda:6')
frobenius_norm tensor(92.3903, device='cuda:6')
spectral_norm tensor(5.1075, device='cuda:6')
stable_rank value of layer 15 ---213.3599090576172 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(76.3742, device='cuda:6')
spectral_norm tensor(9.9213, device='cuda:6')
frobenius_norm tensor(55.5387, device='cuda:6')
spectral_norm tensor(8.2110, device='cuda:6')
frobenius_norm tensor(19.7775, device='cuda:6')
spectral_norm tensor(1.1561, device='cuda:6')
frobenius_norm tensor(44.4344, device='cuda:6')
spectral_norm tensor(3.4351, device='cuda:6')
frobenius_norm tensor(114.8397, device='cuda:6')
spectral_norm tensor(14.0994, device='cuda:6')
frobenius_norm tensor(92.3233, device='cuda:6')
spectral_norm tensor(4.2256, device='cuda:6')
frobenius_norm tensor(91.3091, device='cuda:6')
spectral_norm tensor(4.5477, device='cuda:6')
stable_rank value of layer 16 ---215.97666931152344 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(76.7515, device='cuda:6')
spectral_norm tensor(10.3508, device='cuda:6')
frobenius_norm tensor(55.5269, device='cuda:6')
spectral_norm tensor(7.5316, device='cuda:6')
frobenius_norm tensor(21.6980, device='cuda:6')
spectral_norm tensor(1.2998, device='cuda:6')
frobenius_norm tensor(46.0819, device='cuda:6')
spectral_norm tensor(3.8492, device='cuda:6')
frobenius_norm tensor(115.8902, device='cuda:6')
spectral_norm tensor(13.8304, device='cuda:6')
frobenius_norm tensor(92.0595, device='cuda:6')
spectral_norm tensor(4.2038, device='cuda:6')
frobenius_norm tensor(91.2759, device='cuda:6')
spectral_norm tensor(4.6559, device='cuda:6')
stable_rank value of layer 17 ---209.34996032714844 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(75.6062, device='cuda:6')
spectral_norm tensor(9.9070, device='cuda:6')
frobenius_norm tensor(56.8777, device='cuda:6')
spectral_norm tensor(7.8718, device='cuda:6')
frobenius_norm tensor(19.7982, device='cuda:6')
spectral_norm tensor(1.1998, device='cuda:6')
frobenius_norm tensor(44.9770, device='cuda:6')
spectral_norm tensor(3.6232, device='cuda:6')
frobenius_norm tensor(116.1832, device='cuda:6')
spectral_norm tensor(13.2450, device='cuda:6')
frobenius_norm tensor(91.6447, device='cuda:6')
spectral_norm tensor(4.1425, device='cuda:6')
frobenius_norm tensor(91.0653, device='cuda:6')
spectral_norm tensor(4.6787, device='cuda:6')
stable_rank value of layer 18 ---211.72381591796875 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(75.5331, device='cuda:6')
spectral_norm tensor(9.8490, device='cuda:6')
frobenius_norm tensor(54.4802, device='cuda:6')
spectral_norm tensor(7.2052, device='cuda:6')
frobenius_norm tensor(20.8951, device='cuda:6')
spectral_norm tensor(1.2835, device='cuda:6')
frobenius_norm tensor(45.5737, device='cuda:6')
spectral_norm tensor(3.6876, device='cuda:6')
frobenius_norm tensor(116.8122, device='cuda:6')
spectral_norm tensor(12.4213, device='cuda:6')
frobenius_norm tensor(91.3425, device='cuda:6')
spectral_norm tensor(3.9625, device='cuda:6')
frobenius_norm tensor(90.9342, device='cuda:6')
spectral_norm tensor(4.0861, device='cuda:6')
stable_rank value of layer 19 ---235.54725646972656 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(74.6753, device='cuda:6')
spectral_norm tensor(9.4898, device='cuda:6')
frobenius_norm tensor(53.6751, device='cuda:6')
spectral_norm tensor(7.2770, device='cuda:6')
frobenius_norm tensor(22.0046, device='cuda:6')
spectral_norm tensor(1.3160, device='cuda:6')
frobenius_norm tensor(45.1627, device='cuda:6')
spectral_norm tensor(4.0906, device='cuda:6')
frobenius_norm tensor(116.7308, device='cuda:6')
spectral_norm tensor(12.2498, device='cuda:6')
frobenius_norm tensor(91.6521, device='cuda:6')
spectral_norm tensor(4.0443, device='cuda:6')
frobenius_norm tensor(91.2816, device='cuda:6')
spectral_norm tensor(4.0475, device='cuda:6')
stable_rank value of layer 20 ---232.97064208984375 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(73.7490, device='cuda:6')
spectral_norm tensor(9.8504, device='cuda:6')
frobenius_norm tensor(54.0573, device='cuda:6')
spectral_norm tensor(7.0908, device='cuda:6')
frobenius_norm tensor(22.6461, device='cuda:6')
spectral_norm tensor(1.3368, device='cuda:6')
frobenius_norm tensor(46.2102, device='cuda:6')
spectral_norm tensor(3.1179, device='cuda:6')
frobenius_norm tensor(117.5385, device='cuda:6')
spectral_norm tensor(11.7937, device='cuda:6')
frobenius_norm tensor(91.9506, device='cuda:6')
spectral_norm tensor(4.1673, device='cuda:6')
frobenius_norm tensor(91.5696, device='cuda:6')
spectral_norm tensor(3.9444, device='cuda:6')
stable_rank value of layer 21 ---249.41912841796875 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(71.8484, device='cuda:6')
spectral_norm tensor(9.1078, device='cuda:6')
frobenius_norm tensor(52.7300, device='cuda:6')
spectral_norm tensor(6.7125, device='cuda:6')
frobenius_norm tensor(23.8695, device='cuda:6')
spectral_norm tensor(1.3795, device='cuda:6')
frobenius_norm tensor(47.0670, device='cuda:6')
spectral_norm tensor(3.6429, device='cuda:6')
frobenius_norm tensor(117.4744, device='cuda:6')
spectral_norm tensor(11.7691, device='cuda:6')
frobenius_norm tensor(92.2259, device='cuda:6')
spectral_norm tensor(4.0605, device='cuda:6')
frobenius_norm tensor(91.8678, device='cuda:6')
spectral_norm tensor(3.5128, device='cuda:6')
stable_rank value of layer 22 ---269.9619445800781 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(71.9286, device='cuda:6')
spectral_norm tensor(9.4720, device='cuda:6')
frobenius_norm tensor(52.4818, device='cuda:6')
spectral_norm tensor(6.7927, device='cuda:6')
frobenius_norm tensor(25.1328, device='cuda:6')
spectral_norm tensor(1.6420, device='cuda:6')
frobenius_norm tensor(47.9436, device='cuda:6')
spectral_norm tensor(3.2829, device='cuda:6')
frobenius_norm tensor(117.6067, device='cuda:6')
spectral_norm tensor(11.1983, device='cuda:6')
frobenius_norm tensor(92.5378, device='cuda:6')
spectral_norm tensor(3.8395, device='cuda:6')
frobenius_norm tensor(92.2068, device='cuda:6')
spectral_norm tensor(3.6470, device='cuda:6')
stable_rank value of layer 23 ---270.76318359375 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(70.7268, device='cuda:6')
spectral_norm tensor(8.9255, device='cuda:6')
frobenius_norm tensor(49.9702, device='cuda:6')
spectral_norm tensor(6.5083, device='cuda:6')
frobenius_norm tensor(27.4158, device='cuda:6')
spectral_norm tensor(1.5623, device='cuda:6')
frobenius_norm tensor(50.0108, device='cuda:6')
spectral_norm tensor(2.8417, device='cuda:6')
frobenius_norm tensor(118.0258, device='cuda:6')
spectral_norm tensor(10.3830, device='cuda:6')
frobenius_norm tensor(92.8487, device='cuda:6')
spectral_norm tensor(3.5258, device='cuda:6')
frobenius_norm tensor(92.5677, device='cuda:6')
spectral_norm tensor(4.2689, device='cuda:6')
stable_rank value of layer 24 ---290.3305358886719 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(69.7075, device='cuda:6')
spectral_norm tensor(9.2668, device='cuda:6')
frobenius_norm tensor(49.8228, device='cuda:6')
spectral_norm tensor(6.4607, device='cuda:6')
frobenius_norm tensor(27.5467, device='cuda:6')
spectral_norm tensor(1.7434, device='cuda:6')
frobenius_norm tensor(50.2014, device='cuda:6')
spectral_norm tensor(3.0538, device='cuda:6')
frobenius_norm tensor(118.7821, device='cuda:6')
spectral_norm tensor(10.1132, device='cuda:6')
frobenius_norm tensor(93.4874, device='cuda:6')
spectral_norm tensor(3.6431, device='cuda:6')
frobenius_norm tensor(93.2013, device='cuda:6')
spectral_norm tensor(4.6891, device='cuda:6')
stable_rank value of layer 25 ---261.071044921875 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(70.0908, device='cuda:6')
spectral_norm tensor(9.0741, device='cuda:6')
frobenius_norm tensor(51.2286, device='cuda:6')
spectral_norm tensor(7.0144, device='cuda:6')
frobenius_norm tensor(28.7508, device='cuda:6')
spectral_norm tensor(1.6545, device='cuda:6')
frobenius_norm tensor(51.0959, device='cuda:6')
spectral_norm tensor(3.6542, device='cuda:6')
frobenius_norm tensor(119.5556, device='cuda:6')
spectral_norm tensor(11.0651, device='cuda:6')
frobenius_norm tensor(94.1382, device='cuda:6')
spectral_norm tensor(4.1865, device='cuda:6')
frobenius_norm tensor(93.8620, device='cuda:6')
spectral_norm tensor(4.2123, device='cuda:6')
stable_rank value of layer 26 ---247.0523681640625 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(68.9026, device='cuda:6')
spectral_norm tensor(9.8716, device='cuda:6')
frobenius_norm tensor(50.5827, device='cuda:6')
spectral_norm tensor(7.1128, device='cuda:6')
frobenius_norm tensor(30.7386, device='cuda:6')
spectral_norm tensor(1.9273, device='cuda:6')
frobenius_norm tensor(52.3594, device='cuda:6')
spectral_norm tensor(4.7373, device='cuda:6')
frobenius_norm tensor(120.4429, device='cuda:6')
spectral_norm tensor(12.6819, device='cuda:6')
frobenius_norm tensor(95.1424, device='cuda:6')
spectral_norm tensor(4.8977, device='cuda:6')
frobenius_norm tensor(94.6858, device='cuda:6')
spectral_norm tensor(4.1269, device='cuda:6')
stable_rank value of layer 27 ---209.96926879882812 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(68.7252, device='cuda:6')
spectral_norm tensor(10.2899, device='cuda:6')
frobenius_norm tensor(48.1675, device='cuda:6')
spectral_norm tensor(7.2302, device='cuda:6')
frobenius_norm tensor(31.7899, device='cuda:6')
spectral_norm tensor(1.9608, device='cuda:6')
frobenius_norm tensor(53.7083, device='cuda:6')
spectral_norm tensor(5.2134, device='cuda:6')
frobenius_norm tensor(119.7390, device='cuda:6')
spectral_norm tensor(12.8574, device='cuda:6')
frobenius_norm tensor(96.8375, device='cuda:6')
spectral_norm tensor(5.8801, device='cuda:6')
frobenius_norm tensor(96.0067, device='cuda:6')
spectral_norm tensor(4.0887, device='cuda:6')
stable_rank value of layer 28 ---195.3269500732422 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(68.5741, device='cuda:6')
spectral_norm tensor(9.4460, device='cuda:6')
frobenius_norm tensor(50.4763, device='cuda:6')
spectral_norm tensor(7.7513, device='cuda:6')
frobenius_norm tensor(32.9435, device='cuda:6')
spectral_norm tensor(2.0426, device='cuda:6')
frobenius_norm tensor(55.2837, device='cuda:6')
spectral_norm tensor(4.3637, device='cuda:6')
frobenius_norm tensor(119.4804, device='cuda:6')
spectral_norm tensor(13.5208, device='cuda:6')
frobenius_norm tensor(99.0342, device='cuda:6')
spectral_norm tensor(7.1489, device='cuda:6')
frobenius_norm tensor(97.3141, device='cuda:6')
spectral_norm tensor(3.2229, device='cuda:6')
stable_rank value of layer 29 ---242.49368286132812 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(65.2094, device='cuda:6')
spectral_norm tensor(11.1630, device='cuda:6')
frobenius_norm tensor(44.0073, device='cuda:6')
spectral_norm tensor(7.2343, device='cuda:6')
frobenius_norm tensor(39.1436, device='cuda:6')
spectral_norm tensor(2.5942, device='cuda:6')
frobenius_norm tensor(58.8802, device='cuda:6')
spectral_norm tensor(4.7271, device='cuda:6')
frobenius_norm tensor(123.5870, device='cuda:6')
spectral_norm tensor(15.2486, device='cuda:6')
frobenius_norm tensor(101.0766, device='cuda:6')
spectral_norm tensor(10.7552, device='cuda:6')
frobenius_norm tensor(97.5836, device='cuda:6')
spectral_norm tensor(3.7103, device='cuda:6')
stable_rank value of layer 30 ---185.66880798339844 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
frobenius_norm tensor(71.5148, device='cuda:6')
spectral_norm tensor(12.3063, device='cuda:6')
frobenius_norm tensor(48.6604, device='cuda:6')
spectral_norm tensor(8.2152, device='cuda:6')
frobenius_norm tensor(34.5621, device='cuda:6')
spectral_norm tensor(2.7818, device='cuda:6')
frobenius_norm tensor(57.8552, device='cuda:6')
spectral_norm tensor(3.7047, device='cuda:6')
frobenius_norm tensor(140.3628, device='cuda:6')
spectral_norm tensor(23.9930, device='cuda:6')
frobenius_norm tensor(115.5870, device='cuda:6')
spectral_norm tensor(19.0393, device='cuda:6')
frobenius_norm tensor(99.7985, device='cuda:6')
spectral_norm tensor(4.2950, device='cuda:6')
stable_rank value of layer 31 ---154.01223754882812 
[(290.3305358886719, 24), (270.76318359375, 23), (269.9619445800781, 22), (268.34088134765625, 3), (261.071044921875, 25), (249.41912841796875, 21), (247.0523681640625, 26), (242.49368286132812, 29), (240.6619415283203, 4), (235.54725646972656, 19), (234.5662841796875, 14), (232.97064208984375, 20), (229.96060180664062, 2), (226.9339141845703, 5), (215.97666931152344, 16), (213.65187072753906, 1), (213.3599090576172, 15), (212.75192260742188, 13), (211.72381591796875, 18), (209.96926879882812, 27), (209.63941955566406, 6), (209.34996032714844, 17), (201.4922332763672, 8), (199.4788360595703, 9), (198.48497009277344, 10), (195.3269500732422, 28), (192.40182495117188, 7), (189.51531982421875, 11), (187.5979766845703, 12), (185.66880798339844, 30), (154.01223754882812, 31), (92.50985717773438, 0)]
metric_name stable_rank: [24, 23, 22, 3, 25, 21, 26, 29, 4, 19, 14, 20, 2, 5, 16, 1, 15, 13, 18, 27, 6, 17, 8, 9, 10, 28, 7, 11, 12, 30, 31, 0]
当前指标：effective_rank
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:09<00:27,  9.27s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:18<00:18,  9.32s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:27<00:09,  9.18s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:29<00:00,  6.41s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:29<00:00,  7.45s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, I loved the sound of a dog’s nails clicking on the pavement.  I loved hearing a dog’s tail thumping against the floor when he got excited.  Now, I hate those sounds.  I have two dogs that are so much in love with each other that when they’re playing, it sounds like the floor is falling apart.  When they’re running down the hallway, they’re making the same noise.  When they run outside to go potty, the noise is deaf
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 0 ---2424.426513671875 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 1 ---2606.497802734375 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 2 ---2714.739501953125 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 3 ---2752.18798828125 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 4 ---2734.289306640625 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 5 ---2703.84765625 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 6 ---2724.106201171875 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 7 ---2702.1064453125 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 8 ---2669.786376953125 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 9 ---2662.1806640625 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 10 ---2663.650146484375 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 11 ---2661.160888671875 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 12 ---2689.672607421875 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 13 ---2650.41259765625 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 14 ---2673.497314453125 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 15 ---2690.130126953125 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 16 ---2709.111328125 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 17 ---2722.3828125 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 18 ---2743.938720703125 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 19 ---2763.34814453125 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 20 ---2768.924072265625 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 21 ---2781.080810546875 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 22 ---2777.256591796875 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 23 ---2794.071533203125 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 24 ---2777.125732421875 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 25 ---2782.5615234375 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 26 ---2779.262939453125 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 27 ---2767.594970703125 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 28 ---2780.562744140625 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 29 ---2792.9208984375 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 30 ---2712.323486328125 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
effective_rank value of layer 31 ---2732.52294921875 
[(2794.071533203125, 23), (2792.9208984375, 29), (2782.5615234375, 25), (2781.080810546875, 21), (2780.562744140625, 28), (2779.262939453125, 26), (2777.256591796875, 22), (2777.125732421875, 24), (2768.924072265625, 20), (2767.594970703125, 27), (2763.34814453125, 19), (2752.18798828125, 3), (2743.938720703125, 18), (2734.289306640625, 4), (2732.52294921875, 31), (2724.106201171875, 6), (2722.3828125, 17), (2714.739501953125, 2), (2712.323486328125, 30), (2709.111328125, 16), (2703.84765625, 5), (2702.1064453125, 7), (2690.130126953125, 15), (2689.672607421875, 12), (2673.497314453125, 14), (2669.786376953125, 8), (2663.650146484375, 10), (2662.1806640625, 9), (2661.160888671875, 11), (2650.41259765625, 13), (2606.497802734375, 1), (2424.426513671875, 0)]
metric_name effective_rank: [23, 29, 25, 21, 28, 26, 22, 24, 20, 27, 19, 3, 18, 4, 31, 6, 17, 2, 30, 16, 5, 7, 15, 12, 14, 8, 10, 9, 11, 13, 1, 0]
当前指标：head_diversity
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:08<00:26,  8.98s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:17<00:17,  8.96s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:26<00:08,  8.75s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.09s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.11s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, I was very sick. I was in the hospital for weeks and I was very, very tired. I was tired because my body was in constant pain, and I was tired because I was having to spend all my energy just to breathe. I was tired because I was being fed through a tube, and I was tired because I was so weak. I was tired because I was always alone and I was tired because I was scared. I was tired because I was losing hope. I was tired
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 0 ---0.9860825538635254 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 1 ---0.9932371377944946 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 2 ---0.993516206741333 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 3 ---0.9938094615936279 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 4 ---0.9929364919662476 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 5 ---0.9915013313293457 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 6 ---0.9929077625274658 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 7 ---0.9918708801269531 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 8 ---0.990802526473999 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 9 ---0.9907873868942261 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 10 ---0.9903575778007507 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 11 ---0.9904842376708984 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 12 ---0.9907500743865967 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 13 ---0.988161563873291 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 14 ---0.9888362288475037 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 15 ---0.9912642240524292 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 16 ---0.9916648864746094 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 17 ---0.9925468564033508 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 18 ---0.992446780204773 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 19 ---0.9933868646621704 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 20 ---0.993040919303894 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 21 ---0.9932280778884888 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 22 ---0.9927928447723389 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 23 ---0.9939982891082764 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 24 ---0.9935836791992188 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 25 ---0.9927812218666077 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 26 ---0.9929299354553223 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 27 ---0.9928981065750122 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 28 ---0.9935479164123535 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 29 ---0.9926327466964722 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 30 ---0.9917168617248535 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
head_diversity value of layer 31 ---0.9932112693786621 
[(0.9939982891082764, 23), (0.9938094615936279, 3), (0.9935836791992188, 24), (0.9935479164123535, 28), (0.993516206741333, 2), (0.9933868646621704, 19), (0.9932371377944946, 1), (0.9932280778884888, 21), (0.9932112693786621, 31), (0.993040919303894, 20), (0.9929364919662476, 4), (0.9929299354553223, 26), (0.9929077625274658, 6), (0.9928981065750122, 27), (0.9927928447723389, 22), (0.9927812218666077, 25), (0.9926327466964722, 29), (0.9925468564033508, 17), (0.992446780204773, 18), (0.9918708801269531, 7), (0.9917168617248535, 30), (0.9916648864746094, 16), (0.9915013313293457, 5), (0.9912642240524292, 15), (0.990802526473999, 8), (0.9907873868942261, 9), (0.9907500743865967, 12), (0.9904842376708984, 11), (0.9903575778007507, 10), (0.9888362288475037, 14), (0.988161563873291, 13), (0.9860825538635254, 0)]
metric_name head_diversity: [23, 3, 24, 28, 2, 19, 1, 21, 31, 20, 4, 26, 6, 27, 22, 25, 29, 17, 18, 7, 30, 16, 5, 15, 8, 9, 12, 11, 10, 14, 13, 0]
当前指标：coherence
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:08<00:26,  8.69s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:17<00:17,  8.78s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:26<00:08,  8.66s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.03s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.01s/it]
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
Once upon a time, the words “sustainable agriculture” conjured up images of hippies, granola, and Birkenstocks. Today, the concept is gaining acceptance across the political spectrum as an effective way to address some of the most pressing problems facing America. A growing number of businesses, cities, and communities are embracing sustainable agriculture as a way to reduce our carbon footprint, improve our health, and support local economies. Sustainable agriculture is a holistic approach to food production that emphasizes ecological balance and social equity. It
LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(128256, 4096)
    (layers): ModuleList(
      (0-31): 32 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear(in_features=4096, out_features=4096, bias=False)
          (k_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (v_proj): Linear(in_features=4096, out_features=1024, bias=False)
          (o_proj): Linear(in_features=4096, out_features=4096, bias=False)
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (up_proj): Linear(in_features=4096, out_features=14336, bias=False)
          (down_proj): Linear(in_features=14336, out_features=4096, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
        (post_attention_layernorm): LlamaRMSNorm((4096,), eps=1e-05)
      )
    )
    (norm): LlamaRMSNorm((4096,), eps=1e-05)
    (rotary_emb): LlamaRotaryEmbedding()
  )
  (lm_head): Linear(in_features=4096, out_features=128256, bias=False)
)
config: 
 LlamaConfig {
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 128000,
  "dtype": "float16",
  "eos_token_id": 128001,
  "head_dim": 128,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 14336,
  "max_position_embeddings": 131072,
  "mlp_bias": false,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 8,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": {
    "factor": 8.0,
    "high_freq_factor": 4.0,
    "low_freq_factor": 1.0,
    "original_max_position_embeddings": 8192,
    "rope_type": "llama3"
  },
  "rope_theta": 500000.0,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "vocab_size": 128256
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 0 ---0.04617950692772865 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 1 ---0.025356026366353035 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 2 ---0.017596283927559853 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 3 ---0.017325662076473236 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 4 ---0.0178972240537405 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 5 ---0.01778978481888771 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 6 ---0.017451299354434013 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 7 ---0.01878860592842102 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 8 ---0.01820538565516472 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 9 ---0.01830293983221054 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 10 ---0.01808197796344757 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 11 ---0.018941296264529228 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 12 ---0.01895361766219139 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 13 ---0.019280431792140007 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 14 ---0.017885111272335052 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 15 ---0.016964342445135117 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 16 ---0.01687796786427498 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 17 ---0.016494933515787125 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 18 ---0.017044615000486374 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 19 ---0.016093522310256958 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 20 ---0.015906821936368942 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 21 ---0.015557424165308475 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 22 ---0.015601491555571556 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 23 ---0.015353316441178322 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 24 ---0.015847351402044296 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 25 ---0.01560671441257 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 26 ---0.016048356890678406 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 27 ---0.0163591206073761 
Processing layer 28--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 28 ---0.01604018732905388 
Processing layer 29--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 29 ---0.015549903735518456 
Processing layer 30--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 30 ---0.018680352717638016 
Processing layer 31--subset--{'self_attn.q_proj': Linear(in_features=4096, out_features=4096, bias=False), 'self_attn.k_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.v_proj': Linear(in_features=4096, out_features=1024, bias=False), 'self_attn.o_proj': Linear(in_features=4096, out_features=4096, bias=False), 'mlp.gate_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.up_proj': Linear(in_features=4096, out_features=14336, bias=False), 'mlp.down_proj': Linear(in_features=14336, out_features=4096, bias=False)}
coherence value of layer 31 ---0.019824203103780746 
[(0.04617950692772865, 0), (0.025356026366353035, 1), (0.019824203103780746, 31), (0.019280431792140007, 13), (0.01895361766219139, 12), (0.018941296264529228, 11), (0.01878860592842102, 7), (0.018680352717638016, 30), (0.01830293983221054, 9), (0.01820538565516472, 8), (0.01808197796344757, 10), (0.0178972240537405, 4), (0.017885111272335052, 14), (0.01778978481888771, 5), (0.017596283927559853, 2), (0.017451299354434013, 6), (0.017325662076473236, 3), (0.017044615000486374, 18), (0.016964342445135117, 15), (0.01687796786427498, 16), (0.016494933515787125, 17), (0.0163591206073761, 27), (0.016093522310256958, 19), (0.016048356890678406, 26), (0.01604018732905388, 28), (0.015906821936368942, 20), (0.015847351402044296, 24), (0.01560671441257, 25), (0.015601491555571556, 22), (0.015557424165308475, 21), (0.015549903735518456, 29), (0.015353316441178322, 23)]
metric_name coherence: [0, 1, 31, 13, 12, 11, 7, 30, 9, 8, 10, 4, 14, 5, 2, 6, 3, 18, 15, 16, 17, 27, 19, 26, 28, 20, 24, 25, 22, 21, 29, 23]
当前指标：ZD
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.14s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.06s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:20<00:06,  6.87s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.72s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.82s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, I took an advanced placement class in high school, a class that promised to teach us the “basics” of calculus. It was an interesting class in its own right, but I was struck with the fact that we were learning calculus as a high schooler. The teacher was a nice guy, but I remember thinking, “If I can learn this in high school, why can’t high school students learn more math in high school?”\nI’m not saying that we should be teaching
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 0 ---0.14123240113258362 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 1 ---0.1311480700969696 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 2 ---0.14361177384853363 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 3 ---0.14743219316005707 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 4 ---0.14655494689941406 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 5 ---0.14689013361930847 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 6 ---0.14769357442855835 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 7 ---0.14733701944351196 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 8 ---0.14630182087421417 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 9 ---0.14524872601032257 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 10 ---0.1476735919713974 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 11 ---0.14676602184772491 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 12 ---0.14493051171302795 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 13 ---0.14517414569854736 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 14 ---0.14055506885051727 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 15 ---0.14331096410751343 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 16 ---0.14477485418319702 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 17 ---0.1436464488506317 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 18 ---0.1438734531402588 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 19 ---0.14420966804027557 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 20 ---0.1456553190946579 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 21 ---0.14289206266403198 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 22 ---0.1421867161989212 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 23 ---0.14454184472560883 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 24 ---0.14779578149318695 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 25 ---0.14544296264648438 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 26 ---0.14328131079673767 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
ZD value of layer 27 ---0.14147840440273285 
[(0.14779578149318695, 24), (0.14769357442855835, 6), (0.1476735919713974, 10), (0.14743219316005707, 3), (0.14733701944351196, 7), (0.14689013361930847, 5), (0.14676602184772491, 11), (0.14655494689941406, 4), (0.14630182087421417, 8), (0.1456553190946579, 20), (0.14544296264648438, 25), (0.14524872601032257, 9), (0.14517414569854736, 13), (0.14493051171302795, 12), (0.14477485418319702, 16), (0.14454184472560883, 23), (0.14420966804027557, 19), (0.1438734531402588, 18), (0.1436464488506317, 17), (0.14361177384853363, 2), (0.14331096410751343, 15), (0.14328131079673767, 26), (0.14289206266403198, 21), (0.1421867161989212, 22), (0.14147840440273285, 27), (0.14123240113258362, 0), (0.14055506885051727, 14), (0.1311480700969696, 1)]
metric_name ZD: [24, 6, 10, 3, 7, 5, 11, 4, 8, 20, 25, 9, 13, 12, 16, 23, 19, 18, 17, 2, 15, 26, 21, 22, 27, 0, 14, 1]
当前指标：alpha
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.07s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:13<00:13,  6.98s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:20<00:06,  6.87s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.74s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.81s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, a king had a very wise minister who was loved by everyone. One day, the king called for the minister and said, "I want to test your wisdom. I have three bags of gold coins, but two of them contain fake coins while the other one contains real coins. You need to identify the bag with real coins within just one weighing on my scales." The minister thought for a moment and then replied, "Yes, Your Majesty. I can do that." How did the minister manage
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 0 ---4.515918731689453 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 1 ---3.8895316123962402 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 2 ---3.5805065631866455 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 3 ---3.783604383468628 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 4 ---4.1970930099487305 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 5 ---4.019566535949707 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 6 ---6.430380821228027 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 7 ---4.559838771820068 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 8 ---5.161396503448486 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 9 ---4.962588310241699 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 10 ---4.414379596710205 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 11 ---4.310111999511719 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 12 ---4.061600685119629 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 13 ---6.176848411560059 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 14 ---3.9200167655944824 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 15 ---3.6569418907165527 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 16 ---4.209693431854248 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 17 ---3.776597261428833 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 18 ---4.984698295593262 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 19 ---3.751697301864624 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 20 ---3.6451375484466553 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 21 ---5.387862682342529 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 22 ---4.767049312591553 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 23 ---6.222294330596924 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 24 ---4.807549953460693 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 25 ---5.114707946777344 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 26 ---4.0341081619262695 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha value of layer 27 ---3.413137912750244 
[(6.430380821228027, 6), (6.222294330596924, 23), (6.176848411560059, 13), (5.387862682342529, 21), (5.161396503448486, 8), (5.114707946777344, 25), (4.984698295593262, 18), (4.962588310241699, 9), (4.807549953460693, 24), (4.767049312591553, 22), (4.559838771820068, 7), (4.515918731689453, 0), (4.414379596710205, 10), (4.310111999511719, 11), (4.209693431854248, 16), (4.1970930099487305, 4), (4.061600685119629, 12), (4.0341081619262695, 26), (4.019566535949707, 5), (3.9200167655944824, 14), (3.8895316123962402, 1), (3.783604383468628, 3), (3.776597261428833, 17), (3.751697301864624, 19), (3.6569418907165527, 15), (3.6451375484466553, 20), (3.5805065631866455, 2), (3.413137912750244, 27)]
metric_name alpha: [6, 23, 13, 21, 8, 25, 18, 9, 24, 22, 7, 0, 10, 11, 16, 4, 12, 26, 5, 14, 1, 3, 17, 19, 15, 20, 2, 27]
当前指标：alpha_hat
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.06s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.13s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.05s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.79s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.90s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, there was a young man named Alex who had a great passion for traveling and exploring new places. However, his parents were strict and never allowed him to leave their house. One day, Alex's curiosity got the better of him and he decided to sneak out of the house and explore the neighborhood. After wandering around for a few hours, he stumbled upon an old, abandoned building that caught his attention. He cautiously entered the building and found himself in a dimly lit room with a locked door.
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 0 ---17.878414154052734 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 1 ---10.724651336669922 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 2 ---11.09423542022705 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 3 ---13.750286102294922 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 4 ---12.229127883911133 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 5 ---13.23713493347168 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 6 ---15.239927291870117 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 7 ---13.34253215789795 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 8 ---12.03944206237793 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 9 ---14.739602088928223 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 10 ---12.660069465637207 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 11 ---11.740548133850098 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 12 ---12.344042778015137 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 13 ---12.779866218566895 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 14 ---10.932424545288086 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 15 ---10.816814422607422 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 16 ---12.136594772338867 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 17 ---10.876593589782715 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 18 ---12.142037391662598 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 19 ---11.50020694732666 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 20 ---11.685124397277832 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 21 ---14.077293395996094 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 22 ---14.83903980255127 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 23 ---17.10747718811035 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 24 ---14.79826831817627 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 25 ---16.314620971679688 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 26 ---14.33527946472168 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
alpha_hat value of layer 27 ---15.28555679321289 
[(17.878414154052734, 0), (17.10747718811035, 23), (16.314620971679688, 25), (15.28555679321289, 27), (15.239927291870117, 6), (14.83903980255127, 22), (14.79826831817627, 24), (14.739602088928223, 9), (14.33527946472168, 26), (14.077293395996094, 21), (13.750286102294922, 3), (13.34253215789795, 7), (13.23713493347168, 5), (12.779866218566895, 13), (12.660069465637207, 10), (12.344042778015137, 12), (12.229127883911133, 4), (12.142037391662598, 18), (12.136594772338867, 16), (12.03944206237793, 8), (11.740548133850098, 11), (11.685124397277832, 20), (11.50020694732666, 19), (11.09423542022705, 2), (10.932424545288086, 14), (10.876593589782715, 17), (10.816814422607422, 15), (10.724651336669922, 1)]
metric_name alpha_hat: [0, 23, 25, 27, 6, 22, 24, 9, 26, 21, 3, 7, 5, 13, 10, 12, 4, 18, 16, 8, 11, 20, 19, 2, 14, 17, 15, 1]
当前指标：stable_rank
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:22,  7.37s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.21s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.26s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.04s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.12s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time a poor man had a large field, but he was only able to work on it for three hours a day. He was very sad because of his poor life. One day a rich man said to him, "I want you to work for me for one year. I will pay you with rice. One heap   of rice will be given to you for the first day, and two heaps for the second day, and so on." "How much will I get on the last day?" the
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(74.6100, device='cuda:6')
spectral_norm tensor(22.8962, device='cuda:6')
frobenius_norm tensor(37.2340, device='cuda:6')
spectral_norm tensor(4.1839, device='cuda:6')
frobenius_norm tensor(12.2050, device='cuda:6')
spectral_norm tensor(1.2453, device='cuda:6')
frobenius_norm tensor(44.6291, device='cuda:6')
spectral_norm tensor(6.0001, device='cuda:6')
frobenius_norm tensor(126.6843, device='cuda:6')
spectral_norm tensor(37.7735, device='cuda:6')
frobenius_norm tensor(108.8807, device='cuda:6')
spectral_norm tensor(7.8392, device='cuda:6')
frobenius_norm tensor(115.3682, device='cuda:6')
spectral_norm tensor(6.5322, device='cuda:6')
stable_rank value of layer 0 ---108.18235778808594 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(56.7970, device='cuda:6')
spectral_norm tensor(10.4783, device='cuda:6')
frobenius_norm tensor(29.9237, device='cuda:6')
spectral_norm tensor(4.7115, device='cuda:6')
frobenius_norm tensor(18.3318, device='cuda:6')
spectral_norm tensor(1.4635, device='cuda:6')
frobenius_norm tensor(51.8109, device='cuda:6')
spectral_norm tensor(3.7208, device='cuda:6')
frobenius_norm tensor(123.2531, device='cuda:6')
spectral_norm tensor(20.7839, device='cuda:6')
frobenius_norm tensor(101.4097, device='cuda:6')
spectral_norm tensor(8.8381, device='cuda:6')
frobenius_norm tensor(101.5777, device='cuda:6')
spectral_norm tensor(13.5321, device='cuda:6')
stable_rank value of layer 1 ---91.95645141601562 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(65.4367, device='cuda:6')
spectral_norm tensor(5.5540, device='cuda:6')
frobenius_norm tensor(31.8038, device='cuda:6')
spectral_norm tensor(3.0331, device='cuda:6')
frobenius_norm tensor(14.7804, device='cuda:6')
spectral_norm tensor(1.2128, device='cuda:6')
frobenius_norm tensor(50.6930, device='cuda:6')
spectral_norm tensor(3.9768, device='cuda:6')
frobenius_norm tensor(138.8778, device='cuda:6')
spectral_norm tensor(23.4270, device='cuda:6')
frobenius_norm tensor(112.7775, device='cuda:6')
spectral_norm tensor(6.5452, device='cuda:6')
frobenius_norm tensor(115.6860, device='cuda:6')
spectral_norm tensor(7.9797, device='cuda:6')
stable_rank value of layer 2 ---157.4268035888672 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(67.9445, device='cuda:6')
spectral_norm tensor(7.3888, device='cuda:6')
frobenius_norm tensor(32.4597, device='cuda:6')
spectral_norm tensor(3.6898, device='cuda:6')
frobenius_norm tensor(17.2702, device='cuda:6')
spectral_norm tensor(1.2119, device='cuda:6')
frobenius_norm tensor(53.1248, device='cuda:6')
spectral_norm tensor(3.7626, device='cuda:6')
frobenius_norm tensor(153.6270, device='cuda:6')
spectral_norm tensor(22.2753, device='cuda:6')
frobenius_norm tensor(132.3810, device='cuda:6')
spectral_norm tensor(7.2662, device='cuda:6')
frobenius_norm tensor(131.2425, device='cuda:6')
spectral_norm tensor(17.0355, device='cuda:6')
stable_rank value of layer 3 ---143.3182830810547 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(64.7004, device='cuda:6')
spectral_norm tensor(6.6270, device='cuda:6')
frobenius_norm tensor(29.1071, device='cuda:6')
spectral_norm tensor(3.3637, device='cuda:6')
frobenius_norm tensor(20.8693, device='cuda:6')
spectral_norm tensor(1.5107, device='cuda:6')
frobenius_norm tensor(53.3331, device='cuda:6')
spectral_norm tensor(3.7430, device='cuda:6')
frobenius_norm tensor(156.7403, device='cuda:6')
spectral_norm tensor(22.8238, device='cuda:6')
frobenius_norm tensor(129.6389, device='cuda:6')
spectral_norm tensor(5.4833, device='cuda:6')
frobenius_norm tensor(128.7504, device='cuda:6')
spectral_norm tensor(8.4121, device='cuda:6')
stable_rank value of layer 4 ---200.63299560546875 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(63.2364, device='cuda:6')
spectral_norm tensor(5.9598, device='cuda:6')
frobenius_norm tensor(26.6552, device='cuda:6')
spectral_norm tensor(2.6609, device='cuda:6')
frobenius_norm tensor(19.8491, device='cuda:6')
spectral_norm tensor(1.5480, device='cuda:6')
frobenius_norm tensor(53.4192, device='cuda:6')
spectral_norm tensor(4.0186, device='cuda:6')
frobenius_norm tensor(149.4161, device='cuda:6')
spectral_norm tensor(21.1506, device='cuda:6')
frobenius_norm tensor(133.9768, device='cuda:6')
spectral_norm tensor(6.2399, device='cuda:6')
frobenius_norm tensor(132.0446, device='cuda:6')
spectral_norm tensor(8.4305, device='cuda:6')
stable_rank value of layer 5 ---187.18458557128906 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(64.2853, device='cuda:6')
spectral_norm tensor(5.5097, device='cuda:6')
frobenius_norm tensor(28.5616, device='cuda:6')
spectral_norm tensor(2.9881, device='cuda:6')
frobenius_norm tensor(20.6937, device='cuda:6')
spectral_norm tensor(1.4517, device='cuda:6')
frobenius_norm tensor(54.9730, device='cuda:6')
spectral_norm tensor(5.0745, device='cuda:6')
frobenius_norm tensor(151.2180, device='cuda:6')
spectral_norm tensor(14.3238, device='cuda:6')
frobenius_norm tensor(129.6978, device='cuda:6')
spectral_norm tensor(4.1238, device='cuda:6')
frobenius_norm tensor(127.6298, device='cuda:6')
spectral_norm tensor(7.3585, device='cuda:6')
stable_rank value of layer 6 ---278.4970703125 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(60.9057, device='cuda:6')
spectral_norm tensor(4.5698, device='cuda:6')
frobenius_norm tensor(23.8630, device='cuda:6')
spectral_norm tensor(2.5382, device='cuda:6')
frobenius_norm tensor(24.6983, device='cuda:6')
spectral_norm tensor(1.7165, device='cuda:6')
frobenius_norm tensor(60.3206, device='cuda:6')
spectral_norm tensor(3.5918, device='cuda:6')
frobenius_norm tensor(141.6723, device='cuda:6')
spectral_norm tensor(13.0737, device='cuda:6')
frobenius_norm tensor(133.2031, device='cuda:6')
spectral_norm tensor(5.1650, device='cuda:6')
frobenius_norm tensor(132.6786, device='cuda:6')
spectral_norm tensor(7.6574, device='cuda:6')
stable_rank value of layer 7 ---262.54864501953125 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(63.3943, device='cuda:6')
spectral_norm tensor(3.9526, device='cuda:6')
frobenius_norm tensor(27.2992, device='cuda:6')
spectral_norm tensor(2.5236, device='cuda:6')
frobenius_norm tensor(20.6254, device='cuda:6')
spectral_norm tensor(1.4007, device='cuda:6')
frobenius_norm tensor(55.2465, device='cuda:6')
spectral_norm tensor(3.4821, device='cuda:6')
frobenius_norm tensor(139.1350, device='cuda:6')
spectral_norm tensor(12.2201, device='cuda:6')
frobenius_norm tensor(135.6841, device='cuda:6')
spectral_norm tensor(4.7790, device='cuda:6')
frobenius_norm tensor(134.1169, device='cuda:6')
spectral_norm tensor(8.0838, device='cuda:6')
stable_rank value of layer 8 ---293.3992919921875 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(58.6742, device='cuda:6')
spectral_norm tensor(4.6933, device='cuda:6')
frobenius_norm tensor(23.2825, device='cuda:6')
spectral_norm tensor(2.6721, device='cuda:6')
frobenius_norm tensor(24.7824, device='cuda:6')
spectral_norm tensor(1.5406, device='cuda:6')
frobenius_norm tensor(60.4736, device='cuda:6')
spectral_norm tensor(6.3921, device='cuda:6')
frobenius_norm tensor(152.4723, device='cuda:6')
spectral_norm tensor(28.9447, device='cuda:6')
frobenius_norm tensor(124.9508, device='cuda:6')
spectral_norm tensor(5.0401, device='cuda:6')
frobenius_norm tensor(123.2868, device='cuda:6')
spectral_norm tensor(7.9764, device='cuda:6')
stable_rank value of layer 9 ---208.82086181640625 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(63.5265, device='cuda:6')
spectral_norm tensor(4.3337, device='cuda:6')
frobenius_norm tensor(27.1029, device='cuda:6')
spectral_norm tensor(2.6480, device='cuda:6')
frobenius_norm tensor(22.3527, device='cuda:6')
spectral_norm tensor(1.3822, device='cuda:6')
frobenius_norm tensor(57.0854, device='cuda:6')
spectral_norm tensor(4.1727, device='cuda:6')
frobenius_norm tensor(141.8071, device='cuda:6')
spectral_norm tensor(14.3111, device='cuda:6')
frobenius_norm tensor(133.1904, device='cuda:6')
spectral_norm tensor(5.2121, device='cuda:6')
frobenius_norm tensor(132.4129, device='cuda:6')
spectral_norm tensor(9.0536, device='cuda:6')
stable_rank value of layer 10 ---247.63009643554688 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(64.2904, device='cuda:6')
spectral_norm tensor(4.0832, device='cuda:6')
frobenius_norm tensor(28.0532, device='cuda:6')
spectral_norm tensor(2.6312, device='cuda:6')
frobenius_norm tensor(19.8152, device='cuda:6')
spectral_norm tensor(1.4528, device='cuda:6')
frobenius_norm tensor(55.3427, device='cuda:6')
spectral_norm tensor(4.3578, device='cuda:6')
frobenius_norm tensor(139.0032, device='cuda:6')
spectral_norm tensor(12.7478, device='cuda:6')
frobenius_norm tensor(135.5063, device='cuda:6')
spectral_norm tensor(5.4040, device='cuda:6')
frobenius_norm tensor(134.1321, device='cuda:6')
spectral_norm tensor(9.8842, device='cuda:6')
stable_rank value of layer 11 ---234.3900146484375 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(62.3252, device='cuda:6')
spectral_norm tensor(3.9938, device='cuda:6')
frobenius_norm tensor(26.8499, device='cuda:6')
spectral_norm tensor(2.5529, device='cuda:6')
frobenius_norm tensor(20.6057, device='cuda:6')
spectral_norm tensor(1.3954, device='cuda:6')
frobenius_norm tensor(55.6693, device='cuda:6')
spectral_norm tensor(4.6503, device='cuda:6')
frobenius_norm tensor(136.5591, device='cuda:6')
spectral_norm tensor(12.8626, device='cuda:6')
frobenius_norm tensor(136.9887, device='cuda:6')
spectral_norm tensor(5.2525, device='cuda:6')
frobenius_norm tensor(135.8750, device='cuda:6')
spectral_norm tensor(10.9823, device='cuda:6')
stable_rank value of layer 12 ---237.3602294921875 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(61.0881, device='cuda:6')
spectral_norm tensor(4.2847, device='cuda:6')
frobenius_norm tensor(25.6297, device='cuda:6')
spectral_norm tensor(2.9570, device='cuda:6')
frobenius_norm tensor(22.4836, device='cuda:6')
spectral_norm tensor(1.3594, device='cuda:6')
frobenius_norm tensor(57.5732, device='cuda:6')
spectral_norm tensor(5.3631, device='cuda:6')
frobenius_norm tensor(139.4029, device='cuda:6')
spectral_norm tensor(12.9032, device='cuda:6')
frobenius_norm tensor(135.3871, device='cuda:6')
spectral_norm tensor(5.2956, device='cuda:6')
frobenius_norm tensor(133.7299, device='cuda:6')
spectral_norm tensor(10.5605, device='cuda:6')
stable_rank value of layer 13 ---228.27247619628906 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(59.9385, device='cuda:6')
spectral_norm tensor(4.1989, device='cuda:6')
frobenius_norm tensor(25.3315, device='cuda:6')
spectral_norm tensor(2.3592, device='cuda:6')
frobenius_norm tensor(19.7791, device='cuda:6')
spectral_norm tensor(1.4382, device='cuda:6')
frobenius_norm tensor(54.6680, device='cuda:6')
spectral_norm tensor(4.8688, device='cuda:6')
frobenius_norm tensor(136.9790, device='cuda:6')
spectral_norm tensor(12.1678, device='cuda:6')
frobenius_norm tensor(136.4235, device='cuda:6')
spectral_norm tensor(5.1185, device='cuda:6')
frobenius_norm tensor(134.9485, device='cuda:6')
spectral_norm tensor(9.6906, device='cuda:6')
stable_rank value of layer 14 ---237.89956665039062 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(62.2632, device='cuda:6')
spectral_norm tensor(4.4386, device='cuda:6')
frobenius_norm tensor(26.5240, device='cuda:6')
spectral_norm tensor(2.5572, device='cuda:6')
frobenius_norm tensor(20.8667, device='cuda:6')
spectral_norm tensor(1.5160, device='cuda:6')
frobenius_norm tensor(55.3834, device='cuda:6')
spectral_norm tensor(4.4431, device='cuda:6')
frobenius_norm tensor(135.3356, device='cuda:6')
spectral_norm tensor(11.3039, device='cuda:6')
frobenius_norm tensor(137.9369, device='cuda:6')
spectral_norm tensor(5.1105, device='cuda:6')
frobenius_norm tensor(135.9716, device='cuda:6')
spectral_norm tensor(10.2615, device='cuda:6')
stable_rank value of layer 15 ---242.37692260742188 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(59.5504, device='cuda:6')
spectral_norm tensor(4.1705, device='cuda:6')
frobenius_norm tensor(23.7914, device='cuda:6')
spectral_norm tensor(2.3827, device='cuda:6')
frobenius_norm tensor(22.4299, device='cuda:6')
spectral_norm tensor(1.8284, device='cuda:6')
frobenius_norm tensor(57.2100, device='cuda:6')
spectral_norm tensor(5.3112, device='cuda:6')
frobenius_norm tensor(135.1444, device='cuda:6')
spectral_norm tensor(11.1679, device='cuda:6')
frobenius_norm tensor(137.6841, device='cuda:6')
spectral_norm tensor(5.1499, device='cuda:6')
frobenius_norm tensor(135.1051, device='cuda:6')
spectral_norm tensor(10.5823, device='cuda:6')
stable_rank value of layer 16 ---227.75900268554688 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(61.2952, device='cuda:6')
spectral_norm tensor(3.9247, device='cuda:6')
frobenius_norm tensor(24.2607, device='cuda:6')
spectral_norm tensor(2.3428, device='cuda:6')
frobenius_norm tensor(22.0993, device='cuda:6')
spectral_norm tensor(1.6527, device='cuda:6')
frobenius_norm tensor(56.9147, device='cuda:6')
spectral_norm tensor(4.7962, device='cuda:6')
frobenius_norm tensor(133.7680, device='cuda:6')
spectral_norm tensor(11.1640, device='cuda:6')
frobenius_norm tensor(138.4800, device='cuda:6')
spectral_norm tensor(5.2358, device='cuda:6')
frobenius_norm tensor(135.3299, device='cuda:6')
spectral_norm tensor(8.6134, device='cuda:6')
stable_rank value of layer 17 ---251.5305633544922 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(57.8086, device='cuda:6')
spectral_norm tensor(4.1784, device='cuda:6')
frobenius_norm tensor(23.1346, device='cuda:6')
spectral_norm tensor(2.7649, device='cuda:6')
frobenius_norm tensor(24.9173, device='cuda:6')
spectral_norm tensor(1.5424, device='cuda:6')
frobenius_norm tensor(60.8592, device='cuda:6')
spectral_norm tensor(5.2139, device='cuda:6')
frobenius_norm tensor(133.3184, device='cuda:6')
spectral_norm tensor(10.9756, device='cuda:6')
frobenius_norm tensor(140.8582, device='cuda:6')
spectral_norm tensor(5.6010, device='cuda:6')
frobenius_norm tensor(137.9481, device='cuda:6')
spectral_norm tensor(7.5009, device='cuda:6')
stable_rank value of layer 18 ---253.84030151367188 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(57.4449, device='cuda:6')
spectral_norm tensor(4.5447, device='cuda:6')
frobenius_norm tensor(21.3000, device='cuda:6')
spectral_norm tensor(2.4431, device='cuda:6')
frobenius_norm tensor(25.0779, device='cuda:6')
spectral_norm tensor(1.6237, device='cuda:6')
frobenius_norm tensor(59.6389, device='cuda:6')
spectral_norm tensor(4.9505, device='cuda:6')
frobenius_norm tensor(135.1317, device='cuda:6')
spectral_norm tensor(12.0810, device='cuda:6')
frobenius_norm tensor(140.5321, device='cuda:6')
spectral_norm tensor(5.5793, device='cuda:6')
frobenius_norm tensor(137.1833, device='cuda:6')
spectral_norm tensor(6.9554, device='cuda:6')
stable_rank value of layer 19 ---252.57142639160156 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(58.7482, device='cuda:6')
spectral_norm tensor(4.0858, device='cuda:6')
frobenius_norm tensor(22.3411, device='cuda:6')
spectral_norm tensor(2.3521, device='cuda:6')
frobenius_norm tensor(25.9143, device='cuda:6')
spectral_norm tensor(1.7104, device='cuda:6')
frobenius_norm tensor(60.9286, device='cuda:6')
spectral_norm tensor(4.8709, device='cuda:6')
frobenius_norm tensor(134.6000, device='cuda:6')
spectral_norm tensor(11.1297, device='cuda:6')
frobenius_norm tensor(141.2537, device='cuda:6')
spectral_norm tensor(5.9169, device='cuda:6')
frobenius_norm tensor(138.0343, device='cuda:6')
spectral_norm tensor(6.5506, device='cuda:6')
stable_rank value of layer 20 ---263.31256103515625 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(57.1627, device='cuda:6')
spectral_norm tensor(3.6813, device='cuda:6')
frobenius_norm tensor(19.9702, device='cuda:6')
spectral_norm tensor(2.2489, device='cuda:6')
frobenius_norm tensor(27.7738, device='cuda:6')
spectral_norm tensor(1.6762, device='cuda:6')
frobenius_norm tensor(63.2683, device='cuda:6')
spectral_norm tensor(5.2737, device='cuda:6')
frobenius_norm tensor(136.7462, device='cuda:6')
spectral_norm tensor(11.6961, device='cuda:6')
frobenius_norm tensor(141.2342, device='cuda:6')
spectral_norm tensor(5.6582, device='cuda:6')
frobenius_norm tensor(137.9897, device='cuda:6')
spectral_norm tensor(5.3100, device='cuda:6')
stable_rank value of layer 21 ---310.4990234375 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(58.1757, device='cuda:6')
spectral_norm tensor(4.0771, device='cuda:6')
frobenius_norm tensor(19.4699, device='cuda:6')
spectral_norm tensor(1.9257, device='cuda:6')
frobenius_norm tensor(27.1717, device='cuda:6')
spectral_norm tensor(2.0305, device='cuda:6')
frobenius_norm tensor(63.5510, device='cuda:6')
spectral_norm tensor(4.3551, device='cuda:6')
frobenius_norm tensor(136.9238, device='cuda:6')
spectral_norm tensor(12.4543, device='cuda:6')
frobenius_norm tensor(141.9774, device='cuda:6')
spectral_norm tensor(6.5433, device='cuda:6')
frobenius_norm tensor(139.0448, device='cuda:6')
spectral_norm tensor(5.1260, device='cuda:6')
stable_rank value of layer 22 ---289.32818603515625 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(59.5436, device='cuda:6')
spectral_norm tensor(4.0933, device='cuda:6')
frobenius_norm tensor(20.5584, device='cuda:6')
spectral_norm tensor(2.1754, device='cuda:6')
frobenius_norm tensor(27.5827, device='cuda:6')
spectral_norm tensor(1.8698, device='cuda:6')
frobenius_norm tensor(64.9988, device='cuda:6')
spectral_norm tensor(5.3918, device='cuda:6')
frobenius_norm tensor(138.1573, device='cuda:6')
spectral_norm tensor(14.3250, device='cuda:6')
frobenius_norm tensor(141.6219, device='cuda:6')
spectral_norm tensor(6.6344, device='cuda:6')
frobenius_norm tensor(138.9087, device='cuda:6')
spectral_norm tensor(5.2948, device='cuda:6')
stable_rank value of layer 23 ---271.5490417480469 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(56.8169, device='cuda:6')
spectral_norm tensor(3.8277, device='cuda:6')
frobenius_norm tensor(19.4813, device='cuda:6')
spectral_norm tensor(1.8291, device='cuda:6')
frobenius_norm tensor(31.0836, device='cuda:6')
spectral_norm tensor(2.3168, device='cuda:6')
frobenius_norm tensor(65.3215, device='cuda:6')
spectral_norm tensor(6.2412, device='cuda:6')
frobenius_norm tensor(135.7100, device='cuda:6')
spectral_norm tensor(12.7437, device='cuda:6')
frobenius_norm tensor(143.0019, device='cuda:6')
spectral_norm tensor(6.6948, device='cuda:6')
frobenius_norm tensor(141.1184, device='cuda:6')
spectral_norm tensor(5.4309, device='cuda:6')
stable_rank value of layer 24 ---266.88092041015625 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(55.3445, device='cuda:6')
spectral_norm tensor(3.8164, device='cuda:6')
frobenius_norm tensor(17.7185, device='cuda:6')
spectral_norm tensor(1.8025, device='cuda:6')
frobenius_norm tensor(33.9660, device='cuda:6')
spectral_norm tensor(3.0984, device='cuda:6')
frobenius_norm tensor(68.8981, device='cuda:6')
spectral_norm tensor(5.3780, device='cuda:6')
frobenius_norm tensor(134.2979, device='cuda:6')
spectral_norm tensor(12.3842, device='cuda:6')
frobenius_norm tensor(144.5099, device='cuda:6')
spectral_norm tensor(7.7998, device='cuda:6')
frobenius_norm tensor(143.8913, device='cuda:6')
spectral_norm tensor(6.7086, device='cuda:6')
stable_rank value of layer 25 ---216.0190887451172 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(51.7868, device='cuda:6')
spectral_norm tensor(3.9548, device='cuda:6')
frobenius_norm tensor(16.9575, device='cuda:6')
spectral_norm tensor(1.9295, device='cuda:6')
frobenius_norm tensor(40.6153, device='cuda:6')
spectral_norm tensor(3.1825, device='cuda:6')
frobenius_norm tensor(71.6628, device='cuda:6')
spectral_norm tensor(6.2830, device='cuda:6')
frobenius_norm tensor(134.3241, device='cuda:6')
spectral_norm tensor(10.3940, device='cuda:6')
frobenius_norm tensor(145.9831, device='cuda:6')
spectral_norm tensor(10.3400, device='cuda:6')
frobenius_norm tensor(143.8145, device='cuda:6')
spectral_norm tensor(5.9343, device='cuda:6')
stable_rank value of layer 26 ---213.61611938476562 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
frobenius_norm tensor(56.6802, device='cuda:6')
spectral_norm tensor(7.8725, device='cuda:6')
frobenius_norm tensor(18.0258, device='cuda:6')
spectral_norm tensor(2.1181, device='cuda:6')
frobenius_norm tensor(36.7123, device='cuda:6')
spectral_norm tensor(4.5755, device='cuda:6')
frobenius_norm tensor(66.9187, device='cuda:6')
spectral_norm tensor(10.3449, device='cuda:6')
frobenius_norm tensor(139.7623, device='cuda:6')
spectral_norm tensor(14.5234, device='cuda:6')
frobenius_norm tensor(145.0968, device='cuda:6')
spectral_norm tensor(16.8470, device='cuda:6')
frobenius_norm tensor(133.4643, device='cuda:6')
spectral_norm tensor(7.9846, device='cuda:6')
stable_rank value of layer 27 ---96.66703033447266 
[(310.4990234375, 21), (293.3992919921875, 8), (289.32818603515625, 22), (278.4970703125, 6), (271.5490417480469, 23), (266.88092041015625, 24), (263.31256103515625, 20), (262.54864501953125, 7), (253.84030151367188, 18), (252.57142639160156, 19), (251.5305633544922, 17), (247.63009643554688, 10), (242.37692260742188, 15), (237.89956665039062, 14), (237.3602294921875, 12), (234.3900146484375, 11), (228.27247619628906, 13), (227.75900268554688, 16), (216.0190887451172, 25), (213.61611938476562, 26), (208.82086181640625, 9), (200.63299560546875, 4), (187.18458557128906, 5), (157.4268035888672, 2), (143.3182830810547, 3), (108.18235778808594, 0), (96.66703033447266, 27), (91.95645141601562, 1)]
metric_name stable_rank: [21, 8, 22, 6, 23, 24, 20, 7, 18, 19, 17, 10, 15, 14, 12, 11, 13, 16, 25, 26, 9, 4, 5, 2, 3, 0, 27, 1]
当前指标：effective_rank
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.09s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:13<00:13,  6.97s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.01s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.70s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:27<00:00,  6.82s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, there was a small town called Harmonyville. The town had a population of 2,000 people, and 1,500 of them had access to the internet. A new survey found that 1,350 people in Harmonyville used the internet for social media, while 1,000 people used it for online shopping. How many people in Harmonyville used the internet for both social media and online shopping?

To determine how many people in Harmonyville
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 0 ---2094.5888671875 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 1 ---2108.23583984375 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 2 ---2268.78173828125 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 3 ---2322.7314453125 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 4 ---2320.885009765625 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 5 ---2335.18017578125 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 6 ---2337.6826171875 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 7 ---2348.204833984375 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 8 ---2334.47705078125 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 9 ---2323.93115234375 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 10 ---2351.43994140625 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 11 ---2331.3955078125 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 12 ---2333.96142578125 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 13 ---2327.25927734375 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 14 ---2299.207763671875 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 15 ---2312.1357421875 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 16 ---2319.4013671875 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 17 ---2322.21826171875 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 18 ---2311.623779296875 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 19 ---2325.7373046875 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 20 ---2334.143798828125 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 21 ---2338.0810546875 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 22 ---2328.93994140625 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 23 ---2365.697021484375 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 24 ---2355.1064453125 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 25 ---2361.042236328125 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 26 ---2345.724365234375 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
effective_rank value of layer 27 ---2320.07958984375 
[(2365.697021484375, 23), (2361.042236328125, 25), (2355.1064453125, 24), (2351.43994140625, 10), (2348.204833984375, 7), (2345.724365234375, 26), (2338.0810546875, 21), (2337.6826171875, 6), (2335.18017578125, 5), (2334.47705078125, 8), (2334.143798828125, 20), (2333.96142578125, 12), (2331.3955078125, 11), (2328.93994140625, 22), (2327.25927734375, 13), (2325.7373046875, 19), (2323.93115234375, 9), (2322.7314453125, 3), (2322.21826171875, 17), (2320.885009765625, 4), (2320.07958984375, 27), (2319.4013671875, 16), (2312.1357421875, 15), (2311.623779296875, 18), (2299.207763671875, 14), (2268.78173828125, 2), (2108.23583984375, 1), (2094.5888671875, 0)]
metric_name effective_rank: [23, 25, 24, 10, 7, 26, 21, 6, 5, 8, 20, 12, 11, 22, 13, 19, 9, 3, 17, 4, 27, 16, 15, 18, 14, 2, 1, 0]
当前指标：head_diversity
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.32s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.20s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.17s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.93s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.03s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time, in a small town, there lived a young girl named Lily. She was known for her love of music and her beautiful voice. One day, she decided to audition for the local symphony orchestra. She was nervous but excited, and after a few weeks, she received a call. She had been accepted!

Lily was thrilled and worked hard to prepare for her first performance. On the day of the concert, she was surrounded by all the other musicians, each playing their instruments with precision and
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 0 ---0.9926437139511108 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 1 ---0.990115761756897 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 2 ---0.9928579330444336 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 3 ---0.9908387064933777 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 4 ---0.9908434748649597 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 5 ---0.9908813834190369 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 6 ---0.9935818314552307 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 7 ---0.9881868362426758 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 8 ---0.9893801212310791 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 9 ---0.9851914644241333 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 10 ---0.9917221069335938 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 11 ---0.988156795501709 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 12 ---0.9904806017875671 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 13 ---0.9887334108352661 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 14 ---0.984466552734375 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 15 ---0.9898031949996948 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 16 ---0.9885467290878296 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 17 ---0.9883378744125366 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 18 ---0.987558901309967 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 19 ---0.9857799410820007 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 20 ---0.9886536598205566 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 21 ---0.988256573677063 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 22 ---0.9854484796524048 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 23 ---0.9908662438392639 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 24 ---0.9898425936698914 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 25 ---0.9891295433044434 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 26 ---0.9852111339569092 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
head_diversity value of layer 27 ---0.9841814637184143 
[(0.9935818314552307, 6), (0.9928579330444336, 2), (0.9926437139511108, 0), (0.9917221069335938, 10), (0.9908813834190369, 5), (0.9908662438392639, 23), (0.9908434748649597, 4), (0.9908387064933777, 3), (0.9904806017875671, 12), (0.990115761756897, 1), (0.9898425936698914, 24), (0.9898031949996948, 15), (0.9893801212310791, 8), (0.9891295433044434, 25), (0.9887334108352661, 13), (0.9886536598205566, 20), (0.9885467290878296, 16), (0.9883378744125366, 17), (0.988256573677063, 21), (0.9881868362426758, 7), (0.988156795501709, 11), (0.987558901309967, 18), (0.9857799410820007, 19), (0.9854484796524048, 22), (0.9852111339569092, 26), (0.9851914644241333, 9), (0.984466552734375, 14), (0.9841814637184143, 27)]
metric_name head_diversity: [6, 2, 0, 10, 5, 23, 4, 3, 12, 1, 24, 15, 8, 25, 13, 20, 16, 17, 21, 7, 11, 18, 19, 22, 26, 9, 14, 27]
当前指标：coherence
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.31s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:14<00:14,  7.20s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:21<00:07,  7.12s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  6.93s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:28<00:00,  7.03s/it]
Setting `pad_token_id` to `eos_token_id`:151643 for open-end generation.
Once upon a time，there lived a king.He had three daughters.One day，the king asked his daughters，"Which of you loves me most？" The two elder daughters answered at once. "I love you most，Father，" they said.But the youngest daughter was quiet for a while.Then she said， "I love you more than they do，Father." The king was not happy with her answer.So he gave each of them a coin and said， "Go to the street and spend this coin in any
Qwen2ForCausalLM(
  (model): Qwen2Model(
    (embed_tokens): Embedding(152064, 3584)
    (layers): ModuleList(
      (0-27): 28 x Qwen2DecoderLayer(
        (self_attn): Qwen2Attention(
          (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
          (k_proj): Linear(in_features=3584, out_features=512, bias=True)
          (v_proj): Linear(in_features=3584, out_features=512, bias=True)
          (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
        )
        (mlp): Qwen2MLP(
          (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
          (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
        (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
      )
    )
    (norm): Qwen2RMSNorm((3584,), eps=1e-06)
    (rotary_emb): Qwen2RotaryEmbedding()
  )
  (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
)
config: 
 Qwen2Config {
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "dtype": "float16",
  "eos_token_id": 151643,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "layer_types": [
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention",
    "full_attention"
  ],
  "max_position_embeddings": 131072,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
  "rope_scaling": null,
  "rope_theta": 1000000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "transformers_version": "4.57.3",
  "use_cache": true,
  "use_mrope": false,
  "use_sliding_window": false,
  "vocab_size": 152064
}

Processing layer 0--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 0 ---0.02408885583281517 
Processing layer 1--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 1 ---0.056700460612773895 
Processing layer 2--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 2 ---0.02794474922120571 
Processing layer 3--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 3 ---0.018478330224752426 
Processing layer 4--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 4 ---0.0186083372682333 
Processing layer 5--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 5 ---0.017582347616553307 
Processing layer 6--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 6 ---0.017218835651874542 
Processing layer 7--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 7 ---0.01599242351949215 
Processing layer 8--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 8 ---0.01598191447556019 
Processing layer 9--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 9 ---0.01899208500981331 
Processing layer 10--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 10 ---0.015675809234380722 
Processing layer 11--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 11 ---0.01689709722995758 
Processing layer 12--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 12 ---0.016846707090735435 
Processing layer 13--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 13 ---0.01713874563574791 
Processing layer 14--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 14 ---0.016889071092009544 
Processing layer 15--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 15 ---0.017316650599241257 
Processing layer 16--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 16 ---0.0172375850379467 
Processing layer 17--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 17 ---0.016330666840076447 
Processing layer 18--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 18 ---0.017193637788295746 
Processing layer 19--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 19 ---0.01639077439904213 
Processing layer 20--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 20 ---0.016309073194861412 
Processing layer 21--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 21 ---0.0156699325889349 
Processing layer 22--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 22 ---0.016703739762306213 
Processing layer 23--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 23 ---0.016167931258678436 
Processing layer 24--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 24 ---0.01581120304763317 
Processing layer 25--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 25 ---0.016396205872297287 
Processing layer 26--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 26 ---0.017451899126172066 
Processing layer 27--subset--{'self_attn.q_proj': Linear(in_features=3584, out_features=3584, bias=True), 'self_attn.k_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.v_proj': Linear(in_features=3584, out_features=512, bias=True), 'self_attn.o_proj': Linear(in_features=3584, out_features=3584, bias=False), 'mlp.gate_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.up_proj': Linear(in_features=3584, out_features=18944, bias=False), 'mlp.down_proj': Linear(in_features=18944, out_features=3584, bias=False)}
coherence value of layer 27 ---0.02082100510597229 
[(0.056700460612773895, 1), (0.02794474922120571, 2), (0.02408885583281517, 0), (0.02082100510597229, 27), (0.01899208500981331, 9), (0.0186083372682333, 4), (0.018478330224752426, 3), (0.017582347616553307, 5), (0.017451899126172066, 26), (0.017316650599241257, 15), (0.0172375850379467, 16), (0.017218835651874542, 6), (0.017193637788295746, 18), (0.01713874563574791, 13), (0.01689709722995758, 11), (0.016889071092009544, 14), (0.016846707090735435, 12), (0.016703739762306213, 22), (0.016396205872297287, 25), (0.01639077439904213, 19), (0.016330666840076447, 17), (0.016309073194861412, 20), (0.016167931258678436, 23), (0.01599242351949215, 7), (0.01598191447556019, 8), (0.01581120304763317, 24), (0.015675809234380722, 10), (0.0156699325889349, 21)]
metric_name coherence: [1, 2, 0, 27, 9, 4, 3, 5, 26, 15, 16, 6, 18, 13, 11, 14, 12, 22, 25, 19, 17, 20, 23, 7, 8, 24, 10, 21]