Create README.md

README.md

@@ -0,0 +1,19 @@
+🔤 bvv241-max: Cross-Model Unicode Tokenizer with Shared Token Space (vocab_size=131072, n_embed=1024)
+🧠 Overview
+Constructed by matching common tokens across multiple SoTA tokenizer vocabularies:
+
+o200k_base, cl100k, Mistral Nemo, Qwen3, QwQ, DeepSeek-R1
+
+We found ≈19,000 common text tokens, reindexed as a universal Unicode-aligned vocabulary in a vocab_size=131072 space. 
+Combined with Unicode monograms.
+
+🧊 Embedding Matrix
+
+Precomputed frozen embeddings: 131072 x 1024
+Delivered in tensor normalized_embeddings_weights.pt
+
+💡 Use Cases
+Multilingual base modeling
+Joint-instruction alignment models
+Shared embedding space for MoE architectures
+