Should You Mask 15% in Masked Language Modeling?
Paper • 2202.08005 • Published • 1
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("andreasmadsen/efficient_mlm_m0.80")
model = AutoModelForMaskedLM.from_pretrained("andreasmadsen/efficient_mlm_m0.80")Quick Links
This is a model checkpoint for "Should You Mask 15% in Masked Language Modeling" (code).
The original checkpoint is avaliable at princeton-nlp/efficient_mlm_m0.80. Unfortunately this checkpoint depends on code that isn't part of the official transformers
library. Additionally, the checkpoints contains unused weights due to a bug.
This checkpoint fixes the unused weights issue and uses the RobertaPreLayerNorm model from the transformers
library.
- Downloads last month
- 9
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="andreasmadsen/efficient_mlm_m0.80")