internlm

/

internlm-chat-7b

like 101

Follow

Intern Large Models 1.01k

v1.0.3 : AttributeError: 'InternLMConfig' object has no attribute 'rotary'

hello, why is "rotary" removed from InternLMConfig, but InternLMAttention is still called?

    def _init_rope(self):
        if self.config.rotary["type"] == "origin":
            self.rotary_emb = InternLMRotaryEmbedding(
                self.head_dim,
                max_position_embeddings=self.max_position_embeddings,
                base=self.config.rotary["base"],
            )
        elif self.config.rotary["type"] == "dynamic":
            self.rotary_emb = InternLMDynamicNTKScalingRotaryEmbedding(
                self.head_dim,
                max_position_embeddings=self.max_position_embeddings,
                base=self.config.rotary["base"],
                scaling_factor=self.config.rotary.get("scaling_factor", 1.0),
            )
        else:
            raise ValueError("Currently we only support rotary embedding's type being one of ('origin', 'dynamic').")
        return self.rotary_emb

You can add this to the config.json.

  "rotary": {
    "base": 10000,
    "type": "dynamic"
  }

It seems the maintainer delete it incorrectly.