Release GCT v1 consensus tokenizer

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+consensus_report.json filter=lfs diff=lfs merge=lfs -text
+L/rules.json filter=lfs diff=lfs merge=lfs -text
+M/rules.json filter=lfs diff=lfs merge=lfs -text
+XL/rules.json filter=lfs diff=lfs merge=lfs -text

L/merges.txt

@@ -0,0 +1 @@
+# GCT deterministic longest-match representation; no corpus-trained BPE merges

L/rules.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03a6d2caef450e7c35e2a5ca217e1697d7375d2c6bca1b6ee0bc478b597d4b23
+size 27235982

M/merges.txt

@@ -0,0 +1 @@
+# GCT deterministic longest-match representation; no corpus-trained BPE merges

M/rules.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:099a776c87611758ac1717b7ded723981958a0bf014f42407d126b8849dfd549
+size 14715191

README.md

@@ -1,3 +1,14 @@
----
-license: apache-2.0
----
+# GCT-Tokenizer-v1
+
+GCT v1 is a corpus-free multilingual tokenizer built from vocabulary consensus across six open frontier tokenizers: DeepSeek, Kimi, Qwen, GLM, Gemma, and Mistral.
+
+It provides four deterministic variants:
+
+- GCT-S — 32,768 tokens
+- GCT-M — 65,536 tokens
+- GCT-L — 131,072 tokens
+- GCT-XL — 262,144 tokens
+
+GCT prioritizes tokens independently selected by multiple existing tokenizers. This produces a strong shared Latin/technical vocabulary and universal UTF-8 byte fallback, while sacrificing some compression efficiency and multilingual balance compared with individually corpus-trained tokenizers.
+
+The tokenizer uses no corpus-trained merges and performs exact lossless round-tripping in its reference implementation.

S/merges.txt

@@ -0,0 +1 @@
+# GCT deterministic longest-match representation; no corpus-trained BPE merges

XL/merges.txt

@@ -0,0 +1 @@
+# GCT deterministic longest-match representation; no corpus-trained BPE merges

XL/rules.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5aa8c5ffb41d21abbcc1cc9606fe3d91bfd8eb561feb379a7865074753c26f6d
+size 49861459

consensus_report.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad82d972f3a881c5daef6071e2278341ec475c5f585b55c00374ac59d7743bd3
+size 113391375