| --- |
| language: en |
| license: apache-2.0 |
| --- |
| |
| # AstroBERT Small |
|
|
| This is a `22.7M` parameter [BERT](https://arxiv.org/abs/1810.04805) encoder-only model trained on [ArXiv abstracts](https://huggingface.co/NeuML/txtai-arxiv) categorized as `astro-ph` and [Wikipedia articles](https://huggingface.co/datasets/NeuML/wikipedia-20260401) labeled as `astronomy` related. |
|
|
| This is a domain-specialized small model that often performs as good as models 10-100x larger. It demonstrates that narrowing down a model to a small domain requires less overall parameters than models generalized for all problems. |
|
|
| ## Usage |
|
|
| `astrobert-small` can be loaded using [Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index) as follows. |
|
|
| ```python |
| from transformers import AutoModel |
| |
| model = AutoModel.from_pretrained("neuml/astrobert-small") |
| ``` |
|
|
| The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on. |
|
|
| ## More Information |
|
|
| Read more about the model in [this article](https://hf.co/blog/NeuML/astrobert-small). |
|
|
