Giving multiple inputs to model.generate()
#16
by Ashitasaxena - opened
I am new to huggingface. I am using Pytorch for development. I have a query.
The model card for inference looks like this -
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xl", device_map="auto")
input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")
outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))
If I have a large list consisting of input_texts, how can I give them to the model.generate() function? Is there a way to perform this inference in batches?
Can someone provide code/references for this?