How to use microsoft/bloom-deepspeed-inference-int8 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="microsoft/bloom-deepspeed-inference-int8")
# Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("microsoft/bloom-deepspeed-inference-int8") model = AutoModel.from_pretrained("microsoft/bloom-deepspeed-inference-int8")
Hi,
Do you have any pointer code to the approach of how you transforming the original model and saving the pt files?
· Sign up or log in to comment
