ConvBERT: Improving BERT with Span-based Dynamic Convolution
Paper • 2008.02496 • Published
# Load model directly
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("mrm8488/convbert-base-spanish")
model = AutoModel.from_pretrained("mrm8488/convbert-base-spanish")Quick Links
ConvBERT base pre-trained on large_spanish_corpus
The ConvBERT architecture is presented in the "ConvBERT: Improving BERT with Span-based Dynamic Convolution" paper.
Metrics on evaluation set
disc_accuracy = 0.9488542
disc_auc = 0.8833056
disc_loss = 0.15933733
disc_precision = 0.79224133
disc_recall = 0.27443287
global_step = 1000000
loss = 9.658503
masked_lm_accuracy = 0.6177698
masked_lm_loss = 1.7050561
sampled_masked_lm_accuracy = 0.5379228
Usage
from transformers import AutoModel, AutoTokenizer
model_name = "mrm8488/convbert-base-spanish"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
Created by Manuel Romero/@mrm8488 with the support of Narrativa
Made with ♥ in Spain
- Downloads last month
- 12
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="mrm8488/convbert-base-spanish")