How to use michaelfeil/ct2fast-falcon-40b-instruct with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("michaelfeil/ct2fast-falcon-40b-instruct", dtype="auto")
Hi, thanks for providing this quantized version of Falcon-40b-instruct.Did you evaluate it on some benchmarks? Does it perform similarly to the non-quantized model?Thanks!
· Sign up or log in to comment
