| --- |
| license: cc-by-nc-nd-4.0 |
| language: |
| - en |
| pipeline_tag: token-classification |
| tags: |
| - gpt-neox |
| - tokenizer |
| - pythia |
| - luna |
| - text-generation |
| --- |
| |
| # LUNA Tokenizer |
|
|
| The **shared tokenizer** for the [LUNA model family](https://huggingface.co/ASTERIZER) — used by both [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M) and [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M). |
|
|
| It is the **EleutherAI / Pythia-160m (GPT-NeoX) tokenizer**, extracted into a standalone repository so both models — and any future LUNA variants — reference a single canonical source. |
|
|
| ## Details |
|
|
| | Property | Value | |
| |---|---| |
| | **Tokenizer class** | `GPTNeoXTokenizer` | |
| | **Vocabulary size** | 50,304 (50,254 + EOS padding to multiple of 128) | |
| | **BOS token** | `<|endoftext|>` (id 0) | |
| | **EOS token** | `<|endoftext|>` (id 0) | |
| | **UNK token** | `<|endoftext|>` | |
| | **`add_prefix_space`** | false | |
| | **Upstream** | EleutherAI/Pythia-160m (gpt-neox-20b lineage) | |
| | **License** | Apache-2.0 (upstream EleutherAI tokenizer) | |
|
|
| ## Files |
|
|
| | File | Description | |
| |---|---| |
| | `tokenizer.json` | Tokenizer vocabulary + merges (Hugging Face tokenizers format) | |
| | `tokenizer_config.json` | Tokenizer configuration (`GPTNeoXTokenizer`, special tokens) | |
| | `config.json` | Model-side config (`gpt_neox`, `vocab_size = 50304`) for reference | |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("ASTERIZER/LUNA-Tokenizer") |
| ids = tok.encode("Hello, LUNA!") |
| print(tok.decode(ids)) |
| ``` |
|
|
| ## Related |
|
|
| - [LUNA-100M](https://huggingface.co/ASTERIZER/LUNA-100M) |
| - [LUNA-300M](https://huggingface.co/ASTERIZER/LUNA-300M) |
| - [LUNA-100M Program](https://huggingface.co/collections/ASTERIZER/luna-100m-program-6a9115383c18e52460bf67c9) |
| - [LUNA-300M Program](https://huggingface.co/collections/ASTERIZER/luna-300m-program-6a91195ab75671026f83ac94) |
|
|
