| | --- |
| | language: |
| | - "eng" |
| | thumbnail: "URL to a thumbnail used in social sharing" |
| | tags: |
| | - "PyTorch" |
| | - "tensorflow" |
| | license: "apache-2.0" |
| |
|
| | --- |
| | |
| |
|
| | # vBERT-2021-LARGE |
| |
|
| | ### Model Info: |
| | <ul> |
| | <li> Authors: R&D AI Lab, VMware Inc. |
| | <li> Model date: April, 2022 |
| | <li> Model version: 2021-base |
| | <li> Model type: Pretrained language model |
| | <li> License: Apache 2.0 |
| | </ul> |
| |
|
| | #### Motivation |
| | Traditional BERT models struggle with VMware-specific words (Tanzu, vSphere, etc.), technical terms, and compound words. (<a href =https://medium.com/@rickbattle/weaknesses-of-wordpiece-tokenization-eb20e37fec99>Weaknesses of WordPiece Tokenization</a>) |
| |
|
| | We pretrained thevBERT model to address the aforementioned issues using our We have pretrained our vBERT model to address the aforementioned issues using our <a href=https://medium.com/vmware-data-ml-blog/pretraining-a-custom-bert-model-6e37df97dfc4>BERT Pretraining Library </a>. |
| | <br>We have replaced the first 1k unused tokens of BERT's vocabulary with VMware-specific terms to create a modified vocabulary. We then pretrained the 'bert-large-uncased' model for additional 66K steps (60k with MSL_128 and 6k with MSL_512) on VMware domain data. |
| |
|
| | #### Intended Use |
| | The model functions as a VMware-specific Language Model. |
| |
|
| |
|
| | #### How to Use |
| | Here is how to use this model to get the features of a given text in PyTorch: |
| |
|
| | ``` |
| | from transformers import BertTokenizer, BertModel |
| | tokenizer = BertTokenizer.from_pretrained('VMware/vbert-2021-large') |
| | model = BertModel.from_pretrained("VMware/vbert-2021-large") |
| | text = "Replace me by any text you'd like." |
| | encoded_input = tokenizer(text, return_tensors='pt') |
| | output = model(**encoded_input) |
| | ``` |
| |
|
| | and in TensorFlow: |
| |
|
| | ``` |
| | from transformers import BertTokenizer, TFBertModel |
| | tokenizer = BertTokenizer.from_pretrained('VMware/vbert-2021-large') |
| | model = TFBertModel.from_pretrained('VMware/vbert-2021-large') |
| | text = "Replace me by any text you'd like." |
| | encoded_input = tokenizer(text, return_tensors='tf') |
| | output = model(encoded_input) |
| | |
| | ``` |
| |
|
| | ### Training |
| |
|
| | #### - Datasets |
| | Publically available VMware text data such as VMware Docs, Blogs etc. were used for creating the pretraining corpus. Sourced in May, 2021. (~320,000 Documents) |
| | #### - Preprocessing |
| | <ul> |
| | <li>Decoding HTML |
| | <li>Decoding Unicode |
| | <li>Stripping repeated characters |
| | <li>Splitting compound word |
| | <li>Spelling correction |
| | </ul> |
| |
|
| | #### - Model performance measures |
| | We benchmarked vBERT on various VMware-specific NLP downstream tasks (IR, classification, etc). |
| | The model scored higher than the 'bert-base-uncased' model on all benchmarks. |
| |
|
| | ### Limitations and bias |
| | Since the model is further pretrained on the BERT model, it may have the same biases embedded within the original BERT model. |
| |
|
| | The data needs to be preprocessed using our internal vNLP Preprocessor (not available to the public) to maximize its performance. |
| |
|
