Fill-Mask
Transformers
PyTorch
Norwegian
Norwegian Bokmål
Norwegian Nynorsk
BERT
NorBERT
Norwegian
encoder
custom_code
Instructions to use ltg/norbert3-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ltg/norbert3-base with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="ltg/norbert3-base", trust_remote_code=True)# Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("ltg/norbert3-base", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
Fix compatibility with transformers v5
Browse files- modeling_norbert.py +3 -3
modeling_norbert.py
CHANGED
|
@@ -225,11 +225,11 @@ class NorbertPreTrainedModel(PreTrainedModel):
|
|
| 225 |
std = math.sqrt(2.0 / (5.0 * self.hidden_size))
|
| 226 |
|
| 227 |
if isinstance(module, nn.Linear) or isinstance(module, nn.Embedding):
|
| 228 |
-
nn.init.trunc_normal_(module.weight
|
| 229 |
elif isinstance(module, nn.LayerNorm) and module.weight is not None:
|
| 230 |
-
module.weight.
|
| 231 |
if hasattr(module, "bias") and module.bias is not None:
|
| 232 |
-
module.bias.
|
| 233 |
|
| 234 |
|
| 235 |
class NorbertModel(NorbertPreTrainedModel):
|
|
|
|
| 225 |
std = math.sqrt(2.0 / (5.0 * self.hidden_size))
|
| 226 |
|
| 227 |
if isinstance(module, nn.Linear) or isinstance(module, nn.Embedding):
|
| 228 |
+
nn.init.trunc_normal_(module.weight, mean=0.0, std=std, a=-2*std, b=2*std)
|
| 229 |
elif isinstance(module, nn.LayerNorm) and module.weight is not None:
|
| 230 |
+
module.weight.fill_(1.0)
|
| 231 |
if hasattr(module, "bias") and module.bias is not None:
|
| 232 |
+
module.bias.zero_()
|
| 233 |
|
| 234 |
|
| 235 |
class NorbertModel(NorbertPreTrainedModel):
|