File size: 4,617 Bytes
7934b29 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 |
.. _nlp_model:
Model NLP
=========
The config file for NLP models contain three main sections:
- ``trainer``: contains the configs for PTL training. For more information, refer to :doc:`../core/core` and `PTL Trainer class API <https://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-class-api>`.
- ``exp_manager``: the configs of the experiment manager. For more information, refer to :doc:`../core/core`.
- ``model``: contains the configs of the datasets, model architecture, tokenizer, optimizer, scheduler, etc.
The following sub-sections of the model section are shared among most of the NLP models.
- ``tokenizer``: specifies the tokenizer
- ``language_model``: specifies the underlying model to be used as the encoder
- ``optim``: the configs of the optimizer and scheduler :doc:`../core/core`
The ``tokenizer`` and ``language_model`` sections have the following parameters:
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.tokenizer_name** | string | Tokenizer name will be filled automatically based on ``model.language_model.pretrained_model_name``. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.vocab_file** | string | Path to tokenizer vocabulary. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.tokenizer.tokenizer_model** | string | Path to tokenizer model (only for sentencepiece tokenizer). |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.pretrained_model_name** | string | Pre-trained language model name, for example: ``bert-base-cased`` or ``bert-base-uncased``. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.lm_checkpoint** | string | Path to the pre-trained language model checkpoint. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.config_file** | string | Path to the pre-trained language model config file. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.language_model.config** | dictionary | Config of the pre-trained language model. |
+------------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
The parameter **model.language_model.pretrained_model_name** can be one of the following:
- ``megatron-bert-345m-uncased``, ``megatron-bert-345m-cased``, ``biomegatron-bert-345m-uncased``, ``biomegatron-bert-345m-cased``, ``bert-base-uncased``, ``bert-large-uncased``, ``bert-base-cased``, ``bert-large-cased``
- ``distilbert-base-uncased``, ``distilbert-base-cased``
- ``roberta-base``, ``roberta-large``, ``distilroberta-base``
- ``albert-base-v1``, ``albert-large-v1``, ``albert-xlarge-v1``, ``albert-xxlarge-v1``, ``albert-base-v2``, ``albert-large-v2``, ``albert-xlarge-v2``, ``albert-xxlarge-v2``
|