Advertisement-Description-Generator gpt2 finetuned on product descriptions

Model Details

This model generates a 1-2 line description of a product given the keywords associated with it. It is a fine-tuned version of GPT-2 model on a custom dataset consisting of 328 rows in the format: (keyword, text)
Example:
[Echo Dot,smart speaker,Alexa,improved sound,sleek design,home], Introducing the all-new Echo Dot - the smart speaker with Alexa. It has improved sound and a sleek design that fits anywhere in your home.

• Developed by: Nihal Srivastava
• Model type: Finetuned GPT-2
• License: MIT
• Finetuned from model [optional]: GPT-2

Uses

Inital Imports of the model and setting up helper functions:

from transformers import AutoTokenizer, AutoModelForCausalLM

def join_keywords(keywords, randomize=True):
  N = len(keywords)
  if randomize: 
      M = random.choice(range(N+1))
      keywords = keywords[:M]
      random.shuffle(keywords)
      
  return ','.join(keywords)

SPECIAL_TOKENS  = { "bos_token": "<|BOS|>",
                    "eos_token": "<|EOS|>",
                    "unk_token": "<|UNK|>",                    
                    "pad_token": "<|PAD|>",
                    "sep_token": "<|SEP|>"}

device = torch.device("cuda")
tokenizer = AutoTokenizer.from_pretrained("NihalSrivastava/advertisement-description-generator")
model = AutoModelForCausalLM.from_pretrained("NihalSrivastava/advertisement-description-generator").to(device)

keywords = ['laptop', 'fast', 'gaming', 'great graphics', 'affordable']
kw = join_keywords(keywords, randomize=False)
prompt = SPECIAL_TOKENS['bos_token'] + kw + SPECIAL_TOKENS['sep_token']
generated = torch.tensor(tokenizer.encode(prompt)).unsqueeze(0)
device = torch.device("cuda")
generated = generated.to(device)
model.eval();

To generate a single best description:

# Beam-search text generation:
sample_outputs = model.generate(generated, 
                                do_sample=True,   
                                max_length=768,                                                      
                                num_beams=5,
                                repetition_penalty=5.0,
                                early_stopping=True,      
                                num_return_sequences=1
                                )
for i, sample_output in enumerate(sample_outputs):
    text = tokenizer.decode(sample_output, skip_special_tokens=True)
    a = len(','.join(keywords))    
    print("{}: {}\n\n".format(i+1,  text[a:]))

Output:

1: The gaming laptop has a fast clock speed, high refresh rate display, and advanced connectivity options for unparalleled gaming performance.

To generate top 10 best descriptions:

# Top-p (nucleus) text generation (10 samples):
sample_outputs = model.generate(generated, 
                                do_sample=True,   
                                min_length=50, 
                                max_length=MAXLEN,
                                top_k=30,                                 
                                top_p=0.7,        
                                temperature=0.9,
                                repetition_penalty=2.0,
                                num_return_sequences=10
                                )

for i, sample_output in enumerate(sample_outputs):
    text = tokenizer.decode(sample_output, skip_special_tokens=True)
    a = len(','.join(keywords))    
    print("{}: {}\n\n".format(i+1,  text[a:]))

Output: Will generate 10 best outouts of product description as above.

Training Hyperparameters

DEBUG           = False  
USE_APEX        = True  
APEX_OPT_LEVEL  = 'O1'  
MODEL           = 'gpt2' #{gpt2, gpt2-medium, gpt2-large, gpt2-xl}  
UNFREEZE_LAST_N = 6 #The last N layers to unfreeze for training  
SPECIAL_TOKENS  = { "bos_token": "<|BOS|>",  
                    "eos_token": "<|EOS|>",  
                    "unk_token": "<|UNK|>",                    
                    "pad_token": "<|PAD|>",  
                    "sep_token": "<|SEP|>"}  
MAXLEN          = 768  #{768, 1024, 1280, 1600}   
TRAIN_SIZE      = 0.8  
if USE_APEX:  
    TRAIN_BATCHSIZE = 4  
    BATCH_UPDATE    = 16  
else:  
    TRAIN_BATCHSIZE = 2  
    BATCH_UPDATE    = 32  
EPOCHS          = 20  
LR              = 5e-4  
EPS             = 1e-8  
WARMUP_STEPS    = 1e2  
SEED            = 2020  
EVALUATION_STRATEGY=epoch  
SAVE_STRATEGY=epoch  

Model Card Contact

Email: nihal.srivastava05@gmail.com
Github: https://github.com/Nihal-Srivastava05
LinkedIn: https://www.linkedin.com/in/nihal-srivastava-7708a71b7/