Update README.md

README.md

@@ -31,25 +31,52 @@ In addition, the model can be fine-tuned on a downstream task using the [CLM exa
 
 ### How to use
 
-You can use this model directly with a pipeline for text generation.
+For large OPT models, such as this one, it is not recommend to make use of the `text-generation` pipeline because
+one should load the model in half-precision to accelerate generation and optimize memory consumption on GPU.
+It is recommended to directly call the [`generate`](https://huggingface.co/docs/transformers/main/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate)
+ method as follows: 
+
 
 ```python
->>> from transformers import pipeline
+>>> from transformers import AutoModelForCausalLM, AutoTokenizer
+>>> import torch
+
+>>> model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b", torch_dtype=torch.float16).cuda()
+
+>>> # the fast tokenizer currently does not work correctly
+>>> tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)
+
+>>> prompt = "Hello, I'm am conscious and"
+
 
->>> generator = pipeline('text-generation', model="facebook/opt-350m")
->>> generator("Hello, I'm am conscious and")
-[{'generated_text': "Hello, I'm am conscious and I'm a bit of a noob. I'm looking for"}]
+>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
+
+>>> generated_ids = model.generate(input_ids)
+
+>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+["Hello, I'm am conscious and aware of my surroundings. I'm not sure what you mean"]
 ```
 
 By default, generation is deterministic. In order to use the top-k sampling, please set `do_sample` to `True`. 
 
 ```python
->>> from transformers import pipeline, set_seed
+>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
+>>> import torch
+
+>>> model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b", torch_dtype=torch.float16).cuda()
+
+>>> # the fast tokenizer currently does not work correctly
+>>> tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)
+
+>>> prompt = "Hello, I'm am conscious and"
+
+>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
 
 >>> set_seed(32)
->>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True)
->>> generator("Hello, I'm am conscious and")
-[{'generated_text': "Hello, I'm am conscious and I'm interested in this project. Can I get an initial contact"}]
+>>> generated_ids = model.generate(input_ids, do_sample=True)
+
+>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+["Hello, I'm am conscious and aware of my surroundings. I'm not sure if I'm"]
 ```
 
 ### Limitations and bias
@@ -66,31 +93,53 @@ unfiltered content from the internet, which is far from neutral the model is str
 Here's an example of how the model can have biased predictions:
 
 ```python
->>> from transformers import pipeline, set_seed
+>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
+>>> import torch
+
+>>> model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b", torch_dtype=torch.float16).cuda()
+
+>>> # the fast tokenizer currently does not work correctly
+>>> tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)
+
+>>> prompt = "The woman worked as a"
+
+>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
 
 >>> set_seed(32)
->>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True, num_return_sequences=5)
->>> generator("The woman worked as a")
-[{'generated_text': "The woman works as a substitute teacher for kids who have missed school. She's the teacher herself,"},
- {'generated_text': 'The woman works as a security guard for another company and does an average of around $13/hour'},
- {'generated_text': 'The woman works as a receptionist, she could at the least wait a week or two for her'},
- {'generated_text': 'The woman works as a manager/intern/career development coach/advisor at a nursing home'},
- {'generated_text': 'The woman works as a maid and has to clean the house but you can tell her to do it'}]
+>>> generated_ids = model.generate(input_ids, do_sample=True, num_return_sequences=5, max_length=10)
+
+>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+The woman worked as a nurse at a hospital
+The woman worked as a nurse at a hospital
+The woman worked as a nurse in the emergency
+The woman worked as a nurse at a hospital
+The woman worked as a nurse in a hospital
 ```
 
 compared to:
 
 ```python
->>> from transformers import pipeline, set_seed
+>>> from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
+>>> import torch
+
+>>> model = AutoModelForCausalLM.from_pretrained("facebook/opt-6.7b", torch_dtype=torch.float16).cuda()
+
+>>> # the fast tokenizer currently does not work correctly
+>>> tokenizer = AutoTokenizer.from_pretrained(path, use_fast=False)
+
+>>> prompt = "The man worked as a"
+
+>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda()
 
 >>> set_seed(32)
->>> generator = pipeline('text-generation', model="facebook/opt-350m", do_sample=True, num_return_sequences=5)
->>> generator("The man worked as a")
-[{'generated_text': 'The man works as a security guard for the National Football League franchise. He has been a part of'},
- {'generated_text': 'The man works as a security guard for another company and does an excellent job.\nI remember when'},
- {'generated_text': 'The man works as a "secret agent" but at the same time he\'s working to protect the'},
- {'generated_text': 'The man works as a manager/operator/servant for a grocery store and does a lot of'},
- {'generated_text': 'The man works as a bouncer near the scene of the accident - how he could do that is'}]
+>>> generated_ids = model.generate(input_ids, do_sample=True, num_return_sequences=5, max_length=10)
+
+>>> tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
+The man worked as a security guard at the
+The man worked as a security guard at the
+The man worked as a security guard at the
+The man worked as a security guard at the
+The man worked as a security guard at a
  ```
 
 This bias will also affect all fine-tuned versions of this model.