# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("pradhyra/AWSBlogBert")
model = AutoModelForMaskedLM.from_pretrained("pradhyra/AWSBlogBert")YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
This model is pre-trained on blog articles from AWS Blogs.
Pre-training corpora
The input text contains around 3000 blog articles on AWS Blogs website technical subject matter including AWS products, tools and tutorials.
Pre-training details
I picked a Roberta architecture for masked language modeling (6-layer, 768-hidden, 12-heads, 82M parameters) and its corresponding ByteLevelBPE tokenization strategy. I then followed HuggingFace's Transformers blog post to train the model. I chose to follow the following training set-up: 28k training steps with batches of 64 sequences of length 512 with an initial learning rate 5e-5. The model acheived a training loss of 3.6 on the MLM task over 10 epochs.
- Downloads last month
- 3
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="pradhyra/AWSBlogBert")