Upload 2 files

DATA_SOURCES.md

@@ -0,0 +1,49 @@
+# VOCAB_SOURCES.md
+
+This document lists the lexical sources used to construct the PipeOwl vocabulary.
+
+Only vocabulary tokens were extracted from these sources. No dictionary definitions or explanatory text are included in the model assets.
+
+## vocab_size:495090
+
+Line numbers refer to the JSON vocabulary file where the first line contains '[' and the final line contains ']'.
+
+Actual vocabulary tokens therefore begin at line 2.
+
+### fallback 
+#### line 2-371
+
+Symbolic tokens and byte-level fallback tokens.
+
+These tokens ensure full input coverage for unknown or out-of-vocabulary strings.
+
+### chinese 
+#### line 372-161561
+  
+Some vocabulary entries were extracted from the
+MOE Revised Mandarin Dictionary.
+
+Source: Ministry of Education, Taiwan
+Website: https://language.moe.gov.tw/001/Upload/Files/site_content/M0001/respub/index.html
+
+License: CC BY-ND 3.0 TW
+
+Only vocabulary tokens were used. Dictionary definitions and explanations are not included.
+  
+### english 
+#### line 161562-494920
+
+Source:
+https://www.kaggle.com/datasets/rtatman/english-word-frequency/data
+
+License: MIT
+
+### math 
+#### line 494921-494930
+
+Mathematical symbols and operators.
+
+### japanese 
+#### line 494931-495091
+
+Japanese vocabulary tokens compiled from publicly available lexical resources.