How to use maderix/llama-65b-4bit with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("maderix/llama-65b-4bit", dtype="auto")
Can you add some details of which scripts you used for the conversion? (maybe GTPQ?)
Updated in README
· Sign up or log in to comment