# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForMaskedLM
tokenizer = AutoTokenizer.from_pretrained("nasa-impact/bert-e-base-mlm")
model = AutoModelForMaskedLM.from_pretrained("nasa-impact/bert-e-base-mlm", device_map="auto")Quick Links
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
This model is further trained on top of scibert-base using masked language modeling loss (MLM). The corpus is roughly abstracts from 270,000 earth science-based publications.
The tokenizer used is AutoTokenizer, which is trained on the same corpus.
Stay tuned for further downstream task tests and updates to the model.
in the works
- MLM + NSP task loss
- Add more data sources for training
- Test using downstream tasks
- Downloads last month
- 57
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="nasa-impact/bert-e-base-mlm")