IlhaEmbed-97M (vocab-pruned) initial release

README.md

@@ -0,0 +1,199 @@
+---
+license: apache-2.0
+language:
+  - zh
+library_name: sentence-transformers
+pipeline_tag: sentence-similarity
+tags:
+  - embeddings
+  - clinical
+  - healthcare
+  - traditional-chinese
+  - taiwan
+  - medical
+  - fhir
+  - on-premise
+  - onnx
+base_model: ibm-granite/granite-embedding-97m-multilingual-r2
+---
+
+# IlhaEmbed
+
+**English** | [**繁體中文**](#繁體中文)
+
+**The clinical embedding model that reads how Taiwan actually writes medicine.**
+
+`斷腦筋` is a stroke. `不辣咖` is a blood culture. `H/T` is hypertension, `V(H/T)` means the patient
+attends an outside clinic *for* hypertension. Generic and general-Chinese embedders read none of these —
+they were never trained on how Taiwanese clinicians, nurses, and care workers really write in charts,
+remarks, and community-care sheets. **IlhaEmbed was.**
+
+*Ilha* — from *Ilha Formosa*, "beautiful island." An embedder that understands this island's clinical
+mother tongue.
+
+## Why IlhaEmbed
+
+| | What it means for you |
+|---|---|
+| 🩺 **Reads Taiwan's clinical language** | Slang, abbreviations, Taigi, hand-written remark shorthand — the dirty free-text in every real Taiwanese medical record. Where a generic embedder returns noise, IlhaEmbed returns the right concept. |
+| 🔒 **Runs fully on-premise** | 37 MB, INT8, CPU-only ONNX. No GPU, no cloud, no API key — **no patient data ever leaves the hospital.** Built for the offline reality of clinical infrastructure. |
+| 🧾 **Transparent, auditable lineage** | Apache-2.0 weights, a documented base model (IBM Granite ModernBERT), and a fully traceable training pipeline. Provenance you can hand to a procurement or infosec review, not a black box. |
+| 🪶 **Open and commercial-ready** | Apache-2.0 — use it, ship it, embed it in a product, no copyleft strings. |
+
+## What it's for
+
+IlhaEmbed turns messy Taiwanese clinical text into structured meaning. Drop-in for:
+
+- **Clinical text normalization** — map free-text abbreviations/slang to canonical concepts (`H/T` → 高血壓).
+- **Dirty-data → structured routing** — the semantic layer of an importer that reads a spreadsheet cell and routes each fact to its correct home (medication, condition, care source…).
+- **Terminology / code matching** — retrieve the right ICD / LOINC / drug concept for a surface term.
+- **De-siloing search** — find the record from the clue a nurse actually has, not the exact field name.
+
+It is the normalization layer of a fully Taiwan-origin clinical pipeline:
+**`Breeze-ASR-26` (speech) → `IlhaEmbed` (meaning) → `FHIR` (structure).**
+
+## Benchmarks
+
+**Task: jargon top-1** — given a Taiwan-clinical surface term, retrieve its canonical clinical concept
+from a combined concept pool. Each register (slang, abbreviation, apposition) is held out separately;
+self-matches excluded; macro-averaged over registers. Same methodology and pool for every model.
+
+| Model | Deployable on-prem | slang | abbrev | apposition | **jargon top-1 (macro)** |
+|---|---|---|---|---|---|
+| **IlhaEmbed** | ✅ 37 MB CPU | **0.84** | **0.82** | **0.90** | **0.85** |
+| jina-embeddings-v2-base-zh | ❌ | 0.06 | 0.14 | 0.45 | 0.22 |
+| ckip-base | ✅ | 0.00 | 0.01 | 0.36 | 0.12 |
+| bge-small-zh | ✅ | 0.00 | 0.00 | 0.33 | 0.11 |
+
+**~4× the best general model, ~8× bge** — on exactly the vocabulary generic and general-Chinese embedders
+were never trained to read. We do **not** claim to top general leaderboards (MTEB): IlhaEmbed is a
+*specialist* that wins where generic models fail, while staying small enough to run on the ward.
+
+*(Evaluation methodology described above; the register-held-out protocol is reproducible. The raw
+evaluation pairs carry third-party copyright and are not redistributed. A dedicated Han-Taigi →
+standard-concept benchmark is in progress and reported separately once its concept mapping is clean.)*
+
+## Usage
+
+```python
+# ONNX (deployment path — no torch, CPU)
+from tokenizers import Tokenizer
+import onnxruntime as ort, numpy as np
+
+tok = Tokenizer.from_file("tokenizer.json")
+sess = ort.InferenceSession("model_int8.onnx")
+# ... encode (max_len 32), run, mean-pool, L2-normalize → 384-d vector
+```
+
+```python
+# sentence-transformers (research path)
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("weemed/IlhaEmbed")
+model.encode(["斷腦筋", "H/T", "定期心內門診-戒菸"])
+```
+
+- **Dimensions:** 384 · **Max sequence:** 32 tokens · **Vocab:** 25.5k (pruned to Traditional-Chinese + clinical) · **Size:** 37 MB (INT8 ONNX) / 152 MB (fp32 safetensors).
+
+## How it was built
+
+- **Base:** IBM Granite ModernBERT (Apache-2.0) — a transparent, permissively-licensed foundation.
+- **Distillation:** a Taiwan-clinical teacher signal supplies the domain knowledge; the student learns to place Taiwanese slang, abbreviations, and Taigi near their canonical concepts.
+- **Vocabulary pruning:** the base 180k multilingual BPE vocab was pruned to the 25.5k tokens Traditional-Chinese clinical text actually uses — a **68% size cut (117 MB → 37 MB) with zero accuracy loss** (jargon top-1 held; real-pipeline routing bit-identical).
+
+## Intended use & limitations
+
+IlhaEmbed is a **semantic representation tool**, not a diagnostic system. It surfaces likely meanings and matches for a human to confirm — it must never auto-decide clinical facts without review. It is tuned for Traditional-Chinese Taiwanese clinical text; general or Simplified-Chinese prose is out of scope.
+
+## License & citation
+
+Apache-2.0. Base model: IBM Granite (Apache-2.0). Training-pair sources documented in `SOURCES.md`; the released weights are a distributable derivative, the raw third-party pairs are not redistributed.
+
+*IlhaEmbed — reads the clinical mother tongue of this island.*
+
+---
+
+<a name="繁體中文"></a>
+
+# IlhaEmbed（繁體中文）
+
+[**English**](#ilhaembed) | **繁體中文**
+
+**讀得懂台灣臨床怎麼寫的語意模型。**
+
+`斷腦筋` 是中風。`不辣咖` 是血液培養。`H/T` 是高血壓，`V(H/T)` 是這位病人在外院看高血壓。通用模型與一般中文
+模型全都讀不懂——它們從沒學過台灣的醫師、護理師、照服員真正怎麼在病歷、備註、社區照護表上書寫。**IlhaEmbed 學過。**
+
+*Ilha* 取自 *Ilha Formosa*（美麗島）。一顆讀得懂這座島臨床母語的語意模型。
+
+## 為什麼選 IlhaEmbed
+
+| | 對你的意義 |
+|---|---|
+| 🩺 **讀得懂台灣臨床語言** | 行話、縮寫、台語、手寫備註簡寫——每一份真實台灣病歷裡的髒 free-text。通用模型讀成雜訊的地方，IlhaEmbed 讀出正確概念。 |
+| 🔒 **完全地端執行** | 37 MB、INT8、純 CPU ONNX。免 GPU、免雲端、免 API key——**病人資料永不離開醫院。** 為臨床基礎設施的離線現實而生。 |
+| 🧾 **透明可稽核的血統** | Apache-2.0 權重、有文件的底模（IBM Granite ModernBERT）、可完整追溯的訓練管線。是可以交給採購與資安審查的來源證明，不是黑盒子。 |
+| 🪶 **開源且可商用** | Apache-2.0——自己用、包進產品、隨貨出貨都行，沒有 copyleft 綁約。 |
+
+## 用來做什麼
+
+IlhaEmbed 把凌亂的台灣臨床文字轉成結構化的意義。可直接用於：
+
+- **臨床文字正規化**——把 free-text 縮寫／行話對應到標準概念（`H/T` → 高血壓）。
+- **髒資料 → 結構化路由**——匯入器的語意層：讀一格試算表、把每個事實路由到它正確的歸宿（用藥、疾病、就醫來源…）。
+- **術語／代碼比對**——為一個表面詞找到正確的 ICD／LOINC／藥品概念。
+- **打破資訊孤島的搜尋**——用護理師手上真正有的線索找到紀錄，而不是要求精確欄位名。
+
+它是一條**全台灣血統**臨床管線的正規化層：
+**`Breeze-ASR-26`（聽音）→ `IlhaEmbed`（理解）→ `FHIR`（結構）。**
+
+## 效能指標
+
+**任務：jargon top-1**——給一個台灣臨床表面詞，於合併概念池中檢索其標準臨床概念。各語域（行話 slang／縮寫 abbrev／同位語 apposition）分別留出、排除自我匹配、macro 平均；所有模型使用同一方法學與同一概念池。
+
+| 模型 | 可地端部署 | slang | abbrev | apposition | jargon top-1（macro） |
+|---|---|---|---|---|---|
+| **IlhaEmbed** | ✅ 37 MB CPU | **0.84** | **0.82** | **0.90** | **0.85** |
+| jina-embeddings-v2-base-zh | ❌ | 0.06 | 0.14 | 0.45 | 0.22 |
+| ckip-base | ✅ | 0.00 | 0.01 | 0.36 | 0.12 |
+| bge-small-zh | ✅ | 0.00 | 0.00 | 0.33 | 0.11 |
+
+**約為最佳通用模型的 4 倍、bge 的近 8 倍**——就贏在通用與一般中文模型從未學過的在地詞彙。我們**不宣稱**在通用排行榜（MTEB）霸榜：IlhaEmbed 是**專科模型**，只贏在通用模型會爆掉的地方，同時小到能在病房裡跑。
+
+*（評估方法學如上，register-held-out 協定可重現；原始評估配對含第三方著作權，不隨附散佈。專門的「漢字台語→標準概念」基準進行中。）*
+
+## 使用方式
+
+```python
+# ONNX（部署路徑——免 torch、CPU）
+from tokenizers import Tokenizer
+import onnxruntime as ort, numpy as np
+
+tok = Tokenizer.from_file("tokenizer.json")
+sess = ort.InferenceSession("model_int8.onnx")
+# ... 編碼（max_len 32）、推論、mean-pool、L2 normalize → 384 維向量
+```
+
+```python
+# sentence-transformers（研究路徑）
+from sentence_transformers import SentenceTransformer
+model = SentenceTransformer("weemed/IlhaEmbed")
+model.encode(["斷腦筋", "H/T", "定期心內門診-戒菸"])
+```
+
+- **維度：** 384 · **最大序列：** 32 tokens · **詞表：** 25.5k（剪枝至繁中＋臨床）· **大小：** 37 MB（INT8 ONNX）／152 MB（fp32 safetensors）。
+
+## 怎麼做出來的
+
+- **底模：** IBM Granite ModernBERT（Apache-2.0）——透明、寬鬆授權的基礎。
+- **蒸餾：** 由台灣臨床 teacher 訊號提供領域知識；學生學會把台灣行話、縮寫、台語擺到它們的標準概念旁邊。
+- **詞彙剪枝：** 把底模 180k 的多語 BPE 詞表剪到繁中臨床文字真正會用到的 25.5k token——**體積砍 68%（117 MB → 37 MB）、準確度零損**（jargon top-1 不掉；真實管線路由逐位一致）。
+
+## 適用範圍與限制
+
+IlhaEmbed 是**語意表示工具**，不是診斷系統。它浮現可能的意義與比對供人確認——絕不可在未經審核下自動判定臨床事實。它為繁體中文的台灣臨床文字調校；一般或簡體中文散文不在範圍內。
+
+## 授權與引用
+
+Apache-2.0。底模：IBM Granite（Apache-2.0）。訓練配對來源記於 `SOURCES.md`；釋出的權重是可散佈的衍生作，原始第三方配對不隨附散佈。
+
+*IlhaEmbed——讀得懂這座島的臨床母語。*

SOURCES.md

@@ -0,0 +1,52 @@
+# Training-data provenance — TW clinical terminology embedder
+
+Every corpus mined for the CODER-TW fine-tune, recorded for **ground-truth /
+reproducibility / licensing**. Rule: anything listed here is IN the training
+data and must NOT be reused as a held-out test set. Access date: **2026-07-18**.
+
+## Specialized (surface → canonical) — the scarce clinical signal
+
+| id | source | how obtained | endpoint / file | rows | license | notes |
+|----|--------|--------------|-----------------|-----:|---------|-------|
+| moe-twblg | 教育部臺灣台語常用詞辭典 | open-data dump | `github.com/g0v/moedict-data-twblg` → `dict-twblg.json` | 778 | MOE open data | Taigi 漢字+台羅+華語定義; medical-regex filtered from 14,489 |
+| itaigi | iTaigi 愛台語 (g0v) | reverse-eng API | `itaigi.tw/平臺項目列表/揣列表?關鍵字=` | 1,288 | CC (條目標「會使公開」) | crowd Taigi readings + votes |
+| slang-blog | 陳志金「巷子內醫療用語」/ udn 詹廖明義 / vocus | manual WebFetch | `snore123.blogspot.com/2019/05/medword.html`, `blog.udn.com/ptsafetyrm/3771916`, `vocus.cc/article/6541c172…` | 62 | **作者著作權** | 口語黑話(摸咪/掐水/歐卡)+書面(Endo/Foley/NG) |
+| abbr-pdf | 醫院「可使用縮寫表」+ 護理教材 | curl + pdftotext | nutc, mhchcm, sijhih, kmu(失敗), wagners(需OCR), hpa | 398 | **醫院/作者著作權** | PDF 抽取,有版面噪音 |
+| wiki-redirect | 中文維基百科 重定向 | MediaWiki API | `zh.wikipedia.org/w/api.php prop=redirects` | 284 | CC BY-SA | 別名→條目;醫學 redirect 覆蓋稀疏 |
+| wiki-appos | 中文維基百科 內文同位語 | MediaWiki API | `…prop=extracts&exintro` + Hearst patterns | 371 | CC BY-SA | 「又稱/俗稱/簡稱/縮寫為」→ 挖出 CVA/COPD/心梗 等縮寫 |
+| rsroc-weiei | 中華民國放射線醫學會 衛教 | curl crawl | `rsroc.org.tw/knowledge/news/content.asp?ID=1..119` | 34 | **學會著作權** | 影像縮寫 LDCT/CTA/RFA/TACE;高精度 apposition |
+
+## Bulk (formal synonym) — abundant, saturates cross-lingual
+
+| id | source | how obtained | file | rows | license | notes |
+|----|--------|--------------|------|-----:|---------|-------|
+| icd-loinc | 衛福部 ICD-10-CM/PCS 中文版 + LOINC-NHI | hygieia local | `core/data/icd10_cm_2023.csv.gz` 等 | 63,529 | gov public / LOINC | zh↔en cross-lingual pairs |
+| snomed-syn | SNOMED CT description synonyms | hygieia local | `core/data/terminology_sources/snomed/description.csv.gz` | 44,973 | **SNOMED CT license (UMLS/UTS)** | en synonym→FSN, high-signal subset |
+
+## Not used / dropped (recorded so they aren't re-attempted blindly)
+- iTaigi 2,500-seed expansion — process kept dying mid-run; 1,288 base run already folded in.
+- icd_term_bridge.csv.gz (204k) — token-level alignment noise ("abandonment→照顧或"), unusable.
+- Common Crawl — the right web-scale corpus for the apposition pattern, but a
+  petabyte S3/Athena/Spark project; deferred. Targeted TW-domain crawl is the
+  lighter substitute (rsroc above; extend to more hospital 衛教 domains next).
+
+## Open-source caveat (carried from core/data/README)
+Bulk gov/CC/open-licensed parts are redistributable; the **abbr-pdf / slang-blog
+/ rsroc** raw text is third-party copyright — release the *trained embedder
+weights* (derived work), not the raw pairs, unless per-source consent obtained.
+
+## Model produced from these corpora (v2, 2026-07-18)
+
+- **Base**: CODER (`GanjinZero/coder_all`, Apache-2.0), CLS pooler, BERT-base 768d.
+- **Train**: 119,242 pairs = specialized 1,652×8 (upsampled) + bulk 108,502;
+  InfoNCE / in-batch negatives, 4 epochs, RTX 4080.
+- **Held-out (247 specialized, 500 xling), never trained on:**
+  | | base | v2 fp32 | v2 int8 |
+  |---|---|---|---|
+  | specialized top1 | 0.279 | 0.453 | 0.433 |
+  | specialized top5 | 0.429 | 0.628 | 0.615 |
+  | xling top1 | 0.708 | 0.978 | — |
+- **Deployable**: `coder_tw_v2_int8.onnx` 178.7 MB (25% of fp32), CPU inference, fits 2 GB tier.
+- **Honest limits**: specialized top1 0.43 = usable for a suggest-with-review tier,
+  not autonomous. Chinese colloquial signal is web-scale-sparse (see Common Crawl
+  note); the gains came mostly from apposition-mined clinical abbreviations.

config.json

@@ -0,0 +1,45 @@
+{
+  "architectures": [
+    "ModernBertModel"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 25460,
+  "classifier_activation": "silu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "cls",
+  "cls_token_id": 25460,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 25464,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 150000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "silu",
+  "hidden_size": 384,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1536,
+  "layer_norm_eps": 1e-05,
+  "local_attention": 128,
+  "local_rope_theta": 160000.0,
+  "max_position_embeddings": 32768,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 25461,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 25464,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "transformers_version": "4.57.1",
+  "vocab_size": 25526
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:261af96cef3d72ea9e1877640083fa8d9fe6403333fec4a8cd54ef5e09f92299
+size 152499624

model_int8.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3449f8bade1c8836323c9e5b802a48c35be97ced9fafcb7c16a2fe2d471a2705
+size 38537426

special_tokens_map.json

@@ -0,0 +1,44 @@
+{
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|return|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "<|return|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,555 @@
+{
+  "added_tokens_decoder": {
+    "25460": {
+      "content": "<|startoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25461": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25462": {
+      "content": "<|reserved_200000|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25463": {
+      "content": "<|reserved_200001|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25464": {
+      "content": "<|return|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25465": {
+      "content": "<|constrain|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25466": {
+      "content": "<|reserved_200004|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25467": {
+      "content": "<|channel|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25468": {
+      "content": "<|start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25469": {
+      "content": "<|end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25470": {
+      "content": "<|message|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25471": {
+      "content": "<|reserved_200009|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25472": {
+      "content": "<|reserved_200010|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25473": {
+      "content": "<|reserved_200011|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25474": {
+      "content": "<|call|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25475": {
+      "content": "<|reserved_200013|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25476": {
+      "content": "<|reserved_200014|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25477": {
+      "content": "<|reserved_200015|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25478": {
+      "content": "<|reserved_200016|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25479": {
+      "content": "<|reserved_200017|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25480": {
+      "content": "<|endofprompt|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25481": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "25482": {
+      "content": "<|reserved_200020|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25483": {
+      "content": "<|reserved_200021|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25484": {
+      "content": "<|reserved_200022|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25485": {
+      "content": "<|reserved_200023|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25486": {
+      "content": "<|reserved_200024|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25487": {
+      "content": "<|reserved_200025|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25488": {
+      "content": "<|reserved_200026|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25489": {
+      "content": "<|reserved_200027|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25490": {
+      "content": "<|reserved_200028|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25491": {
+      "content": "<|reserved_200029|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25492": {
+      "content": "<|reserved_200030|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25493": {
+      "content": "<|reserved_200031|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25494": {
+      "content": "<|reserved_200032|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25495": {
+      "content": "<|reserved_200033|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25496": {
+      "content": "<|reserved_200034|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25497": {
+      "content": "<|reserved_200035|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25498": {
+      "content": "<|reserved_200036|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25499": {
+      "content": "<|reserved_200037|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25500": {
+      "content": "<|reserved_200038|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25501": {
+      "content": "<|reserved_200039|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25502": {
+      "content": "<|reserved_200040|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25503": {
+      "content": "<|reserved_200041|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25504": {
+      "content": "<|reserved_200042|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25505": {
+      "content": "<|reserved_200043|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25506": {
+      "content": "<|reserved_200044|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25507": {
+      "content": "<|reserved_200045|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25508": {
+      "content": "<|reserved_200046|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25509": {
+      "content": "<|reserved_200047|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25510": {
+      "content": "<|reserved_200048|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25511": {
+      "content": "<|reserved_200049|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25512": {
+      "content": "<|reserved_200050|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25513": {
+      "content": "<|reserved_200051|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25514": {
+      "content": "<|reserved_200052|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25515": {
+      "content": "<|reserved_200053|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25516": {
+      "content": "<|reserved_200054|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25517": {
+      "content": "<|reserved_200055|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25518": {
+      "content": "<|reserved_200056|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25519": {
+      "content": "<|reserved_200057|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25520": {
+      "content": "<|reserved_200058|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25521": {
+      "content": "<|reserved_200059|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25522": {
+      "content": "<|reserved_200060|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25523": {
+      "content": "<|reserved_200061|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25524": {
+      "content": "<|reserved_200062|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    },
+    "25525": {
+      "content": "<|reserved_200063|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": true,
+      "special": false
+    }
+  },
+  "backend": "tokenizers",
+  "bos_token": "<|startoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<|startoftext|>",
+  "eos_token": "<|return|>",
+  "extra_special_tokens": {},
+  "is_local": true,
+  "mask_token": "[MASK]",
+  "max_length": 32768,
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 32768,
+  "pad_to_multiple_of": null,
+  "pad_token": "<|endoftext|>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "<|return|>",
+  "stride": 0,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first"
+}