|
|
--- |
|
|
tags: ['tokenizer', 'bert', 'wordpiece'] |
|
|
language: en |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# bert-astronomy-tokenizer |
|
|
|
|
|
## Description |
|
|
WordPiece tokenizer (30k vocab) shared across all astronomy models |
|
|
|
|
|
## Tokenizer Details |
|
|
- **Type**: WordPiece (BERT-style) |
|
|
- **Vocabulary Size**: 30,000 tokens |
|
|
- **Special Tokens**: `[PAD]`, `[UNK]`, `[CLS]`, `[SEP]`, `[MASK]` |
|
|
- **Trained On**: 95,000 Wikipedia documents (full corpus train split) |
|
|
- **Normalization**: Lowercase, NFD, strip accents |
|
|
- **Max Length**: 256 tokens |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import PreTrainedTokenizerFast |
|
|
|
|
|
tokenizer = PreTrainedTokenizerFast.from_pretrained("vraj1/bert-astronomy-tokenizer") |
|
|
|
|
|
# Tokenize text |
|
|
text = "The Hubble telescope orbits Earth." |
|
|
tokens = tokenizer.tokenize(text) |
|
|
print(tokens) |
|
|
# Output: ['the', 'hub', '##ble', 'telescope', 'orbit', '##s', 'earth', '.'] |
|
|
``` |
|
|
|
|
|
## Research Context |
|
|
This tokenizer is part of a research project studying the effect of corpus composition on language model performance. |
|
|
|
|
|
**Project**: Effect of Corpus on Language Model Performance |
|
|
**Institution**: [Your University] |
|
|
**Course**: NLP - Master's Computer Science |
|
|
**Date**: November 2024 |
|
|
|
