Initialize NeoLM Small with trained tokenizer and model card

README.md

@@ -0,0 +1,82 @@
+---
+language:
+- en
+datasets:
+- MaagDeveloper/NeoCorpus
+tags:
+- neolm
+- tokenizer
+- byte-level-bpe
+- pretraining
+---
+
+# NeoLM-Small-Base
+
+NeoLM-Small-Base is a planned decoder-only language model
+in the NeoLM family, developed from scratch.
+
+**Current status: Tokenizer only. Model weights are not available yet.**
+
+## Tokenizer
+
+| Property | Value |
+|---|---|
+| Algorithm | ByteLevel BPE |
+| Vocabulary size | 49,152 |
+| Training data | NeoCorpus / foundation / train |
+| Training documents | 1,889,952 |
+| Pre-tokenization | GPT-style regex, 1–3 digit groups |
+| EOS token | `<|endoftext|>` (ID 0) |
+| PAD token | `<|pad|>` (ID 1) |
+| BOS token | None |
+
+The tokenizer was trained from scratch on the training split only.
+
+## Load the Tokenizer
+
+    from transformers import AutoTokenizer
+
+    tokenizer = AutoTokenizer.from_pretrained(
+        "MaagDeveloper/NeoLM-Small-Base"
+    )
+
+## Dataset
+
+Source: [NeoCorpus](https://huggingface.co/datasets/MaagDeveloper/NeoCorpus)
+
+Configuration: `foundation`
+
+The source dataset is derived from NVIDIA Nemotron-ClimbMix
+through Andrej Karpathy's globally shuffled redistribution.
+
+The dataset is published under CC BY-NC 4.0 licensing conditions.
+
+## Model Development
+
+The following details have not been finalized:
+
+- Transformer architecture and parameter count
+- Context length
+- Training configuration and compute budget
+- Evaluation metrics and benchmarks
+- Pretraining checkpoints
+
+Training stages will be recorded as versioned revisions
+after the corresponding checkpoints are available.
+
+## Limitations
+
+This repository currently contains a tokenizer, not a trained
+language model. It cannot generate text or perform inference.
+
+The tokenizer has passed local encoding, decoding, EOS,
+and save/reload checks.
+
+## Attribution
+
+Dataset sources:
+- [NVIDIA Nemotron-ClimbMix](https://huggingface.co/datasets/nvidia/Nemotron-ClimbMix)
+- [Karpathy ClimbMix Shuffle](https://huggingface.co/datasets/karpathy/climbmix-400b-shuffle)
+
+NeoLM is an independent project and is not affiliated with
+NVIDIA, Andrej Karpathy, or Hugging Face.

tokenizer_config.json

@@ -0,0 +1,8 @@
+{
+  "backend": "tokenizers",
+  "bos_token": null,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "TokenizersBackend"
+}