In order to use the model with
- For Rotary Position Embeddings, in model_args, set use_rope = True, use_ape = False, use_alibi = False
- For Attention with Linear Biases, in model_args, set use_rope = False, use_ape = False, use_alibi = True
- For Adaptive Position Encoding, in model_args, set use_rope = True, use_ape = True, use_alibi = False Upon downloading the weights, please convert it into a .pt file to be used for inference