---
language: en
license: apache-2.0
---

# AstroBERT Small

This is a `22.7M` parameter [BERT](https://arxiv.org/abs/1810.04805) encoder-only model trained on [ArXiv abstracts](https://huggingface.co/NeuML/txtai-arxiv) categorized as `astro-ph` and [Wikipedia articles](https://huggingface.co/datasets/NeuML/wikipedia-20260401) labeled as `astronomy` related.

This is a domain-specialized small model that often performs as good as models 10-100x larger. It demonstrates that narrowing down a model to a small domain requires less overall parameters than models generalized for all problems.

## Usage

`astrobert-small` can be loaded using [Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index) as follows.

```python
from transformers import AutoModel

model = AutoModel.from_pretrained("neuml/astrobert-small")
```

The model is intended to be further fine-tuned for a specific task such as Text Classification, Entity Extraction, Sentence Embeddings and so on.

## More Information

Read more about the model in [this article](https://hf.co/blog/NeuML/astrobert-small).