⚠️ DO NOT MERGE: feat(transformers): Add native Transformers config
What does this PR do?
→ Adds rope_parameters, so the checkpoint loads directly with AutoModel / AutoConfig in Transformers 🤗 without trust_remote_code. auto_map is left untouched, so nothing changes for anyone on an older version.
→ Transformers support is being added in https://github.com/huggingface/transformers/pull/48416
→ ⚠️ This PR is a draft; please do not merge it until the Transformers PR above has landed 🤗
Produced by?
→ rope_parameters is the standardised form of the existing rope_scaling / rope_theta pair. GTE applies NTK scaling statically at construction by rescaling the base and dividing the inverse frequencies by factor ** (2 / head_dim), which is exactly a linear scaling with base rope_theta * factor, so the inverse frequencies are unchanged.
→ Outputs are reproduced against the original implementation on this checkpoint (max relative difference 1.1e-06), and the slow integration tests in the Transformers PR run against this revision.