Instructions to use chandar-lab/NeoBERT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use chandar-lab/NeoBERT with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="chandar-lab/NeoBERT", trust_remote_code=True)# Load model directly from transformers import AutoModelForMaskedLM model = AutoModelForMaskedLM.from_pretrained("chandar-lab/NeoBERT", trust_remote_code=True, dtype="auto") - Notebooks
- Google Colab
- Kaggle
Tom Aarsen commited on
Commit ·
30e2384
1
Parent(s): 12e66dc
Cast attention_mask to bool in SDPA
Browse filesI'm pretty sure this is correct. It was an int tensor before, which SDPA doesn't like
model.py
CHANGED
|
@@ -190,7 +190,7 @@ class EncoderBlock(nn.Module):
|
|
| 190 |
query=xq.transpose(1, 2),
|
| 191 |
key=xk.transpose(1, 2),
|
| 192 |
value=xv.transpose(1, 2),
|
| 193 |
-
attn_mask=attention_mask,
|
| 194 |
dropout_p=0,
|
| 195 |
).transpose(1, 2)
|
| 196 |
|
|
|
|
| 190 |
query=xq.transpose(1, 2),
|
| 191 |
key=xk.transpose(1, 2),
|
| 192 |
value=xv.transpose(1, 2),
|
| 193 |
+
attn_mask=attention_mask.bool(),
|
| 194 |
dropout_p=0,
|
| 195 |
).transpose(1, 2)
|
| 196 |
|