Create README.md

README.md

@@ -0,0 +1,99 @@
+---
+license: mit
+tags:
+- chess,
+- tiktoken,
+- tokenizer
+---
+# Chess BPE Tokenizer
+
+A BPE tokenizer trained on chess moves using [rustbpe](https://github.com/karpathy/rustbpe) with tiktoken inference.
+
+## Installation
+
+```bash
+pip install rustbpe tiktoken datasets huggingface_hub
+```
+
+## Quick Start
+
+### Load from HuggingFace & Inference
+
+```python
+from chess_tokenizer import load_tiktoken
+
+enc = load_tiktoken("ItsMaxNorm/chess-bpe-tokenizer")
+
+# Encode chess moves
+ids = enc.encode("w.♘g1♘f3.. b.♟c7♟c5.. w.♙d2♙d4..")
+print(ids)  # [token_ids...]
+
+# Decode back
+text = enc.decode(ids)
+print(text)  # "w.♘g1♘f3.. b.♟c7♟c5.. w.♙d2♙d4.."
+```
+
+
+### Or simply load using tiktoken
+```python
+config = json.load(open(hf_hub_download("ItsMaxNorm/bpess", "config.json")))
+vocab = json.load(open(hf_hub_download("ItsMaxNorm/bpess", "vocab.json")))
+return tiktoken.Encoding(
+    name="chess", pat_str=config["pattern"],
+    mergeable_ranks={k.encode('utf-8', errors='replace'): v for k, v in vocab.items()},
+    special_tokens={}
+)
+```
+### Train Your Own
+
+```python
+from chess_tokenizer import train, upload
+
+# Train on chess dataset
+tok = train(vocab_size=4096, split="train[0:10000]")
+
+# Upload to HuggingFace
+upload(tok, "YOUR_USERNAME/chess-bpe-tokenizer")
+```
+
+### Full Pipeline
+
+```bash
+python chess_tokenizer.py
+```
+
+## Move Format
+
+The tokenizer is trained on custom chess notation:
+
+| Move | Meaning |
+|------|---------|
+| `w.♘g1♘f3..` | White knight g1 to f3 |
+| `b.♟c7♟c5..` | Black pawn c7 to c5 |
+| `b.♟c5♟d4.x.` | Black pawn captures on d4 |
+| `w.♔e1♔g1♖h1♖f1..` | White kingside castle |
+| `b.♛d7♛d5..+` | Black queen to d5 with check |
+
+### Piece Symbols
+
+| White | Black | Piece |
+|-------|-------|-------|
+| ♔ | ♚ | King |
+| ♕ | ♛ | Queen |
+| ♖ | ♜ | Rook |
+| ♗ | ♝ | Bishop |
+| ♘ | ♞ | Knight |
+| ♙ | ♟ | Pawn |
+
+## API
+
+| Function | Description |
+|----------|-------------|
+| `train(vocab_size, split)` | Train BPE on angeluriot/chess_games |
+| `save(tok, path)` | Save vocab.json + config.json |
+| `upload(tok, repo_id)` | Push to HuggingFace Hub |
+| `load_tiktoken(repo_id)` | Load as tiktoken Encoding |
+
+## License
+
+MIT