Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,29 @@
+# TinyGuardrail Tokenizer
+
+Advanced BPE-based tokenizer for TinyGuardrail safety model.
+
+## Specifications
+
+- **Vocabulary Size**: 16,000
+- **Max Length**: 512
+- **Min Frequency**: 2
+- **Special Tokens**: <pad>, <unk>, <cls>, <sep>
+- **BPE Merges**: 141
+
+## Usage
+
+```python
+from src.data.tokenizer import load_tokenizer
+
+# Load from HuggingFace
+tokenizer = load_tokenizer(hf_repo="2796gauravc/tinyguardrail-tokenizer")
+
+# Or load from local path
+tokenizer = load_tokenizer("outputs/tokenizer.pkl")
+
+# Encode text
+tokens = tokenizer.encode("Your text here")
+
+# Decode tokens
+text = tokenizer.decode(tokens)
+```