Upload folder using huggingface_hub

LICENSE

@@ -0,0 +1,189 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work.
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to the Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by the Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding any notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2025-2026 AENEA Global Ltd
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,205 @@
+---
+language:
+  - en
+  - de
+  - fr
+  - es
+  - pt
+  - it
+  - nl
+  - pl
+  - ro
+  - cs
+  - sv
+  - da
+  - "no"
+  - fi
+  - hu
+  - hr
+  - bg
+  - tr
+  - ca
+  - ru
+  - uk
+  - sr
+  - zh
+  - ja
+  - ko
+  - ar
+  - fa
+  - he
+  - hi
+  - bn
+  - th
+  - vi
+  - ka
+  - hy
+  - el
+  - yi
+  - ur
+  - ta
+  - te
+  - gu
+  - pa
+  - ml
+  - kn
+  - am
+  - si
+  - my
+  - km
+  - mr
+  - ne
+  - or
+  - bo
+  - dv
+  - eu
+  - gl
+  - gd
+  - et
+  - sk
+  - lt
+  - sl
+  - lv
+  - af
+  - sq
+  - sw
+  - is
+  - tl
+  - cy
+  - ga
+  - br
+  - la
+  - mk
+  - id
+  - code
+license: apache-2.0
+library_name: tokenizers
+tags:
+  - tokenizer
+  - bpe
+  - multilingual
+  - code
+  - quartz
+  - aenea
+  - coding
+  - python
+  - flores
+pipeline_tag: text-generation
+---
+
+# QT_V.2 Code 114K — Multilingual Coding Tokenizer
+
+**Lowest total tokens on our 66-test field benchmark of any tokenizer at any vocab size.** 114,688 vocabulary optimised for multilingual coding models. Trained with doubled code weight (37% of corpus) including 450K high-quality Python functions from CodeSearchNet. Beats Llama 3, Tekken, and Qwen 2.5 on total tokens while using 10–37% less vocabulary. Validated on FLORES-200 across 204 languages.
+
+Part of the **QT_V.2 tokenizer family** by [Quartz Data Infrastructure](https://quartz.host), the open data layer behind [AENEA](https://aenea.app).
+
+## FLORES-200 Results (204 Languages · 1,012 Parallel Sentences)
+
+| Metric | QT Code 114K | QT 96K | QT 64K | Llama 3 (128K) | Tekken (131K) | Qwen 2.5 (152K) |
+|---|---|---|---|---|---|---|
+| **Total tokens** | 13,007,924 | **12,961,617** | 13,592,357 | 16,764,198 | 14,421,539 | 15,425,680 |
+| **Equity ratio** | 43.3× | **31.6×** | 41.0× | 118.6× | 127.9× | 77.7× |
+| Mean fertility | 4.03 | **3.94** | 4.18 | 5.72 | 5.34 | 4.91 |
+
+QT Code 114K uses **22.4% fewer tokens than Llama 3** and **9.8% fewer than Tekken** across all 204 FLORES languages — with 10–37% less vocabulary.
+
+### Key FLORES Languages (tok/word)
+
+| Language | QT Code | Llama 3 | Tekken | Qwen 2.5 |
+|---|---|---|---|---|
+| Japanese | **32.1** | 38.9 | 41.3 | 35.8 |
+| Tibetan | **46.5** | 149.8 | 168.4 | 98.0 |
+| Sinhala | **3.58** | 11.37 | 16.60 | 9.17 |
+| Amharic | **3.40** | 11.95 | 11.98 | 6.45 |
+| Georgian | **3.46** | 15.47 | 3.93 | 8.33 |
+| Odia | **4.10** | 16.90 | 18.30 | 13.65 |
+
+## Field Benchmark (66 Tests)
+
+| Metric | Value |
+|---|---|
+| **Total tokens** | **3,314** (lowest of any tokenizer) |
+| vs Llama 3 (128K) | 41.2% fewer tokens |
+| vs Tekken (131K) | 23.8% fewer tokens |
+| vs Qwen 2.5 (152K) | 36.1% fewer tokens |
+
+### Code Performance
+
+| Language | QT Code | QT 96K | QT 64K | Llama 3 | Tekken | Qwen 2.5 |
+|---|---|---|---|---|---|---|
+| Python | **110** | 115 | 125 | 97 | 112 | 105 |
+| JavaScript | **67** | 71 | 71 | 65 | 69 | 64 |
+| Rust | **111** | 113 | 117 | 108 | 111 | 107 |
+
+Python compression improved from 125 (64K) to 115 (96K) to **110** (Code 114K) — closing the gap versus Llama 3's 97 from 28.9% to 13.4%.
+
+### Category Totals (lower is better)
+
+| Category | QT Code | Llama 3 | Tekken | Qwen 2.5 |
+|---|---|---|---|---|
+| Natural Languages (20) | **1,033** | 1,599 | 1,038 | 1,535 |
+| V1 Expansion (14) | **662** | 1,758 | 1,092 | 1,509 |
+| V2 New Scripts (3) | **188** | 692 | 740 | 523 |
+| Celtic / Brythonic (8) | **312** | 391 | 341 | 384 |
+| Code (3) | 288 | **270** | 292 | 276 |
+| **TOTAL (66 tests)** | **3,314** | 5,639 | 4,347 | 5,183 |
+
+## When to Use This Variant
+
+**QT_V.2 Code 114K** is designed for multilingual coding assistants and code generation models. It wins Natural Languages outright (1,033 — beating Tekken's 1,038) while offering competitive code compression. Ideal for models that must serve both code and diverse natural language users.
+
+Also available: [QT_V.2 64K](https://huggingface.co/QuartzOpen/QT_V.2_64K) (smallest embedding) · [QT_V.2 96K](https://huggingface.co/QuartzOpen/QT_V.2_96K) (best all-round)
+
+## Usage
+
+```python
+from tokenizers import Tokenizer
+tok = Tokenizer.from_file("tokenizer.json")
+encoded = tok.encode("def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)")
+print(encoded.tokens)
+```
+
+## Specifications
+
+| Spec | Value |
+|---|---|
+| Vocabulary | 114,688 |
+| Languages | 71 natural + 15 code (incl. CodeSearchNet) |
+| Script families | 26 |
+| Pretokenizer | Llama 3 regex |
+| Arithmetic | Single-digit splitting |
+| Max token length | 15 chars |
+| Avg token length | 6.24 chars |
+| Compression | 3.60 chars/token |
+
+## Training
+
+Byte-level BPE with Llama 3 regex pretokenizer. Code-heavy corpus:
+
+| Category | Share | Sources |
+|---|---|---|
+| Wikipedia | 37.3% | 71 languages (wiki_ultra_clean v7.3) |
+| Code | 37.4% | 14 languages + CodeSearchNet Python (450K functions) |
+| Stack Exchange | 25.3% | 49 sites (se_ultra_clean v1) |
+
+## Files
+
+`tokenizer.json` · `vocab.json` · `merges.txt` · `training_report.json`
+
+## Contact
+
+Open-source: quartzopensource@gmail.com
+Commercial licensing & enterprise: commercial@aeneaglobal.com
+
+## License
+
+Apache 2.0 — Copyright 2025-2026 AENEA Global Ltd
+
+```bibtex
+@misc{qt_v2_2026,
+  title={QT_V.2: A Multilingual BPE Tokenizer Family},
+  author={AENEA Global Ltd},
+  year={2026},
+  url={https://quartz.host},
+}
+```

special_tokens_map.json

@@ -0,0 +1,5 @@
+{
+  "bos_token": "<|bos|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<pad>"
+}

tokenizer_config.json

@@ -0,0 +1,15 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_type": "bpe",
+  "bos_token": "<|bos|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<pad>",
+  "unk_token": null,
+  "clean_up_tokenization_spaces": false,
+  "model_max_length": 131072,
+  "added_tokens_decoder": {
+    "0": {"content": "<pad>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "1": {"content": "<|bos|>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true},
+    "2": {"content": "<|endoftext|>", "lstrip": false, "normalized": false, "rstrip": false, "single_word": false, "special": true}
+  }
+}

training_report.json

@@ -0,0 +1,338 @@
+{
+  "vocab_size": 114688,
+  "avg_token_length_chars": 6.24,
+  "max_token_length_chars": 15,
+  "single_byte_tokens": 256,
+  "special_tokens": [
+    {
+      "token": "<pad>",
+      "id": 0
+    },
+    {
+      "token": "<|bos|>",
+      "id": 1
+    },
+    {
+      "token": "<|endoftext|>",
+      "id": 2
+    },
+    {
+      "token": "<",
+      "id": 30
+    },
+    {
+      "token": "[",
+      "id": 61
+    },
+    {
+      "token": "[User",
+      "id": 984
+    },
+    {
+      "token": "[System",
+      "id": 1019
+    },
+    {
+      "token": "[]",
+      "id": 2190
+    },
+    {
+      "token": "['",
+      "id": 2206
+    },
+    {
+      "token": "[\"",
+      "id": 3088
+    },
+    {
+      "token": "</",
+      "id": 3103
+    },
+    {
+      "token": "[i",
+      "id": 3156
+    },
+    {
+      "token": "<T",
+      "id": 4720
+    },
+    {
+      "token": "[:",
+      "id": 6758
+    },
+    {
+      "token": "[Oracle",
+      "id": 8458
+    },
+    {
+      "token": "<String",
+      "id": 8677
+    },
+    {
+      "token": "<int",
+      "id": 8793
+    },
+    {
+      "token": "[string",
+      "id": 9942
+    },
+    {
+      "token": "<string",
+      "id": 10055
+    },
+    {
+      "token": "[str",
+      "id": 10751
+    }
+  ],
+  "compression_ratio_chars_per_token": 3.597,
+  "fertility_tokens_per_word": 4.042,
+  "training_files": 1086,
+  "training_time_seconds": 308.3,
+  "config": {
+    "preset": "llama3",
+    "vocab_size": 114688,
+    "min_frequency": 2,
+    "bpe_dropout": null,
+    "single_digit": true,
+    "max_token_length": 16,
+    "special_tokens": [
+      "<pad>",
+      "<|bos|>",
+      "<|endoftext|>"
+    ],
+    "sample_ratio": 0.12
+  },
+  "per_language_evaluation": {
+    "ar": {
+      "chars": 135,
+      "bytes": 250,
+      "tokens": 38,
+      "words": 20,
+      "chars_per_token": 3.553,
+      "bytes_per_token": 6.579,
+      "tokens_per_word": 1.9,
+      "byte_fragmentation_ratio": 0.132
+    },
+    "arithmetic": {
+      "chars": 88,
+      "bytes": 88,
+      "tokens": 78,
+      "words": 18,
+      "chars_per_token": 1.128,
+      "bytes_per_token": 1.128,
+      "tokens_per_word": 4.333,
+      "byte_fragmentation_ratio": 0.987
+    },
+    "bn": {
+      "chars": 129,
+      "bytes": 347,
+      "tokens": 68,
+      "words": 21,
+      "chars_per_token": 1.897,
+      "bytes_per_token": 5.103,
+      "tokens_per_word": 3.238,
+      "byte_fragmentation_ratio": 0.0
+    },
+    "de": {
+      "chars": 177,
+      "bytes": 180,
+      "tokens": 29,
+      "words": 23,
+      "chars_per_token": 6.103,
+      "bytes_per_token": 6.207,
+      "tokens_per_word": 1.261,
+      "byte_fragmentation_ratio": 0.069
+    },
+    "el": {
+      "chars": 158,
+      "bytes": 289,
+      "tokens": 52,
+      "words": 26,
+      "chars_per_token": 3.038,
+      "bytes_per_token": 5.558,
+      "tokens_per_word": 2.0,
+      "byte_fragmentation_ratio": 0.154
+    },
+    "en": {
+      "chars": 224,
+      "bytes": 224,
+      "tokens": 40,
+      "words": 30,
+      "chars_per_token": 5.6,
+      "bytes_per_token": 5.6,
+      "tokens_per_word": 1.333,
+      "byte_fragmentation_ratio": 0.175
+    },
+    "es": {
+      "chars": 162,
+      "bytes": 168,
+      "tokens": 36,
+      "words": 26,
+      "chars_per_token": 4.5,
+      "bytes_per_token": 4.667,
+      "tokens_per_word": 1.385,
+      "byte_fragmentation_ratio": 0.25
+    },
+    "fa": {
+      "chars": 153,
+      "bytes": 281,
+      "tokens": 42,
+      "words": 28,
+      "chars_per_token": 3.643,
+      "bytes_per_token": 6.69,
+      "tokens_per_word": 1.5,
+      "byte_fragmentation_ratio": 0.024
+    },
+    "fr": {
+      "chars": 175,
+      "bytes": 183,
+      "tokens": 39,
+      "words": 26,
+      "chars_per_token": 4.487,
+      "bytes_per_token": 4.692,
+      "tokens_per_word": 1.5,
+      "byte_fragmentation_ratio": 0.333
+    },
+    "he": {
+      "chars": 136,
+      "bytes": 249,
+      "tokens": 43,
+      "words": 22,
+      "chars_per_token": 3.163,
+      "bytes_per_token": 5.791,
+      "tokens_per_word": 1.955,
+      "byte_fragmentation_ratio": 0.14
+    },
+    "hi": {
+      "chars": 137,
+      "bytes": 353,
+      "tokens": 67,
+      "words": 30,
+      "chars_per_token": 2.045,
+      "bytes_per_token": 5.269,
+      "tokens_per_word": 2.233,
+      "byte_fragmentation_ratio": 0.0
+    },
+    "hy": {
+      "chars": 140,
+      "bytes": 257,
+      "tokens": 48,
+      "words": 20,
+      "chars_per_token": 2.917,
+      "bytes_per_token": 5.354,
+      "tokens_per_word": 2.4,
+      "byte_fragmentation_ratio": 0.229
+    },
+    "ja": {
+      "chars": 70,
+      "bytes": 210,
+      "tokens": 41,
+      "words": 1,
+      "chars_per_token": 1.707,
+      "bytes_per_token": 5.122,
+      "tokens_per_word": 41.0,
+      "byte_fragmentation_ratio": 0.0
+    },
+    "javascript": {
+      "chars": 212,
+      "bytes": 212,
+      "tokens": 53,
+      "words": 31,
+      "chars_per_token": 4.0,
+      "bytes_per_token": 4.0,
+      "tokens_per_word": 1.71,
+      "byte_fragmentation_ratio": 0.245
+    },
+    "ka": {
+      "chars": 126,
+      "bytes": 346,
+      "tokens": 47,
+      "words": 14,
+      "chars_per_token": 2.681,
+      "bytes_per_token": 7.362,
+      "tokens_per_word": 3.357,
+      "byte_fragmentation_ratio": 0.085
+    },
+    "ko": {
+      "chars": 73,
+      "bytes": 185,
+      "tokens": 33,
+      "words": 16,
+      "chars_per_token": 2.212,
+      "bytes_per_token": 5.606,
+      "tokens_per_word": 2.062,
+      "byte_fragmentation_ratio": 0.061
+    },
+    "math": {
+      "chars": 152,
+      "bytes": 172,
+      "tokens": 83,
+      "words": 28,
+      "chars_per_token": 1.831,
+      "bytes_per_token": 2.072,
+      "tokens_per_word": 2.964,
+      "byte_fragmentation_ratio": 0.771
+    },
+    "python": {
+      "chars": 202,
+      "bytes": 202,
+      "tokens": 60,
+      "words": 32,
+      "chars_per_token": 3.367,
+      "bytes_per_token": 3.367,
+      "tokens_per_word": 1.875,
+      "byte_fragmentation_ratio": 0.5
+    },
+    "ru": {
+      "chars": 190,
+      "bytes": 357,
+      "tokens": 45,
+      "words": 23,
+      "chars_per_token": 4.222,
+      "bytes_per_token": 7.933,
+      "tokens_per_word": 1.957,
+      "byte_fragmentation_ratio": 0.022
+    },
+    "th": {
+      "chars": 123,
+      "bytes": 367,
+      "tokens": 51,
+      "words": 2,
+      "chars_per_token": 2.412,
+      "bytes_per_token": 7.196,
+      "tokens_per_word": 25.5,
+      "byte_fragmentation_ratio": 0.0
+    },
+    "uk": {
+      "chars": 157,
+      "bytes": 291,
+      "tokens": 43,
+      "words": 22,
+      "chars_per_token": 3.651,
+      "bytes_per_token": 6.767,
+      "tokens_per_word": 1.955,
+      "byte_fragmentation_ratio": 0.116
+    },
+    "vi": {
+      "chars": 143,
+      "bytes": 190,
+      "tokens": 41,
+      "words": 34,
+      "chars_per_token": 3.488,
+      "bytes_per_token": 4.634,
+      "tokens_per_word": 1.206,
+      "byte_fragmentation_ratio": 0.146
+    },
+    "zh": {
+      "chars": 62,
+      "bytes": 186,
+      "tokens": 34,
+      "words": 1,
+      "chars_per_token": 1.824,
+      "bytes_per_token": 5.471,
+      "tokens_per_word": 34.0,
+      "byte_fragmentation_ratio": 0.0
+    }
+  }
+}