initial commit

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.json filter=lfs diff=lfs merge=lfs -text
+*.tfevents* filter=lfs diff=lfs merge=lfs -text
+*.pbtxt filter=lfs diff=lfs merge=lfs -text
+*.index filter=lfs diff=lfs merge=lfs -text
+*.meta filter=lfs diff=lfs merge=lfs -text
+*.data-* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,122 @@
----
-license: cc-by-4.0
----
+---
+language:
+- yue
+- zh
+language_details: "yue-Hant-HK; zh-Hant-HK"
+license: cc-by-4.0
+datasets:
+- SolarisCipher/hk_content_corpus
+metrics:
+- accuracy
+- exact_match
+tags:
+- ELECTRA
+- pretrained
+- masked-language-model
+- replaced-token-detection
+- feature-extraction
+library_name: transformers
+---
+
+# HKELECTRA - ELECTRA Pretrained Models for Hong Kong Content
+
+This repository contains **pretrained ELECTRA models** trained on Hong Kong Cantonese and Traditional Chinese content, focused on studying diglossia effects for NLP modeling.  
+
+The repo includes:  
+
+- `generator/` : HuggingFace Transformers format **generator** model for masked token prediction.
+- `discriminator/` : HuggingFace Transformers format **discriminator** model for replaced token detection.
+- `tf_checkpoint/` : Original **TensorFlow checkpoint** from pretraining (requires TensorFlow to load).
+- `runs/` : **TensorBoard log** of pretraining.
+
+**Note:** Because this repo contains multiple models with different purposes, there is **no `pipeline_tag`**. Users should select the appropriate model and pipeline for their use case. TensorFlow checkpoint requires TensorFlow >= 2.X to load manually.  
+
+This model is also available at Zenodo: https://doi.org/10.5281/zenodo.16889492
+
+## Model Details
+
+### Model Description
+
+**Architecture:** ELECTRA (small/base/large)  
+**Pretraining:** from scratch (no base model)  
+**Languages:** Hong Kong Cantonese, Traditional Chinese  
+**Intended Use:** Research, feature extraction, masked token prediction  
+**License:** cc-by-4.0
+
+## Usage Examples
+
+### Load Generator (Masked LM)
+
+```python
+from transformers import ElectraTokenizer, ElectraForMaskedLM, pipeline
+
+tokenizer = ElectraTokenizer.from_pretrained("SolarisCipher/HKELECTRA/generator/small")
+model = ElectraForMaskedLM.from_pretrained("SolarisCipher/HKELECTRA/generator/small")
+
+unmasker = pipeline("fill-mask", model=model, tokenizer=tokenizer)
+unmasker("從中環[MASK]到尖沙咀。")
+```
+
+### Load Discriminator (Feature Extraction / Replaced Token Detection)
+
+```python
+from transformers import ElectraTokenizer, ElectraForPreTraining
+
+tokenizer = ElectraTokenizer.from_pretrained("SolarisCipher/HKELECTRA/discriminator/small")
+model = ElectraForPreTraining.from_pretrained("SolarisCipher/HKELECTRA/discriminator/small")
+
+inputs = tokenizer("從中環坐車到[MASK]。", return_tensors="pt")
+outputs = model(**inputs)  # logits for replaced token detection
+```
+
+## Citation
+
+If you use this model in your work, please cite our dataset and the original research:
+
+Dataset (Upstream SQL Dump)  
+```bibtex
+@dataset{yung_2025_16875235,
+  author       = {Yung, Yiu Cheong},
+  title        = {HK Web Text Corpus (MySQL Dump, raw version)},
+  month        = aug,
+  year         = 2025,
+  publisher    = {Zenodo},
+  doi          = {10.5281/zenodo.16875235},
+  url          = {https://doi.org/10.5281/zenodo.16875235},
+}
+```
+
+Dataset (Cleaned Corpus)  
+```bibtex
+@dataset{yung_2025_16882351,
+  author       = {Yung, Yiu Cheong},
+  title        = {HK Content Corpus (Cantonese \& Traditional Chinese)},
+  month        = aug,
+  year         = 2025,
+  publisher    = {Zenodo},
+  doi          = {10.5281/zenodo.16882351},
+  url          = {https://doi.org/10.5281/zenodo.16882351},
+}
+```
+
+Research Paper  
+```bibtex
+@article{10.1145/3744341,
+author = {Yung, Yiu Cheong and Lin, Ying-Jia and Kao, Hung-Yu},
+title = {Exploring the Effectiveness of Pre-training Language Models with Incorporation of Diglossia for Hong Kong Content},
+year = {2025},
+issue_date = {July 2025},
+publisher = {Association for Computing Machinery},
+address = {New York, NY, USA},
+volume = {24},
+number = {7},
+issn = {2375-4699},
+url = {https://doi.org/10.1145/3744341},
+doi = {10.1145/3744341},
+journal = {ACM Trans. Asian Low-Resour. Lang. Inf. Process.},
+month = jul,
+articleno = {71},
+numpages = {16},
+keywords = {Hong Kong, diglossia, ELECTRA, language modeling}
+}
+```

discriminator/base/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:27e25b1740e0f239391ef69c0211697e9c84866e53da8f8484282a4c5eef765c
+size 467

discriminator/base/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f719031dc799e49b82f22325d945c88da4fcc54776962fb330d7873824990ef9
+size 409157229

discriminator/large-B96-L1e4-2M/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:074daf8b9ddafc0d853b3d940c827333d068f91c512338bf1f6262fcd02a99ba
+size 469

discriminator/large-B96-L1e4-2M/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3d35082dcb15c82840c62e5346a13d85ab7b7418b1533e6ed3c64d9c3daa0417
+size 1302254445

discriminator/small/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98aa105c2183d4189722d8ac81a84a3e4d1e9152818734caf4546c8acaf5693f
+size 466

discriminator/small/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ed3f25aa25a21a5456f11856f3f56a9929e51d4cc56de616436f187631570a52
+size 49470697

generator/base/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:141c7eb034490f4265437464c02914475a0583ad27592cdb124805b1b0e14fc9
+size 463

generator/base/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f8fd590384b609320d02c17606a79d65b0fd6d6b1d82f3b3723b2826fcf192ed
+size 106151079

generator/large-B96-L1e4-2M/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:53578d9921c142686415b42b934dbc6559b1fd1c50e34f1ace768cd9a916652e
+size 464

generator/large-B96-L1e4-2M/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4ac454800b2036161ff0a824a3e0204fb877bf7bfdf3a4ad1c1bf75018adb36
+size 166827367

generator/small/config.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d5710f85b07df9f04af744d1a2df051d3073d5cf2e8a80bb3b13cc29c6260ff6
+size 461

generator/small/pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9ab3011c178489ba5d18ac347fcef5a4cd38ee005a26a83a46ea9fcf569c51a
+size 13716839

runs/base/events.out.tfevents.1665678463.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e564732c4178b7e7f6e217e8f8d9668b82aef5cb5bdaa752c145fa467f492d2
+size 49916565

runs/base/events.out.tfevents.1665824088.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d0c921aa6674b0f94f5106c5a92a7835d39cca2a6b87359fe70f970b97011ab9
+size 49846765

runs/base/events.out.tfevents.1665911274.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f5e54937cc53900a23d4b78088e42c4eda5f4b26db8d472b8c58dbfc7d4f6fb9
+size 49902346

runs/base/events.out.tfevents.1666038145.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:84b6df06a389b08f3c20828f1041d133eb2cf499433d6112dce111a545b41400
+size 49810190

runs/base/events.out.tfevents.1666058059.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1930b98a03e50642bb961ccd309f38755bebf3bf9744dc085073452485f7415e
+size 32343601

runs/base/events.out.tfevents.1666062046.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d5f9d5019a319e4b3cc27c0338960565496a2f8733e01fc021bdf53a5dfe79f
+size 40824673

runs/base/events.out.tfevents.1666063769.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3f90cd477e96f2472bb67c35b67220480b1c975fa25f917feeee787fd6cdd7d
+size 7199003

runs/base/events.out.tfevents.1666064251.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:70d246dbecaa675b184798c543f93b79655afeecf5247e1bb5f4c5c188d296dc
+size 32763408

runs/large-B96-L1e4-2M/events.out.tfevents.1665679361.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2b96e7481370e16de2da96d5a22e5209e9dd01f373a635cc2bb06a1a9ffc19f9
+size 64106492

runs/large-B96-L1e4-2M/events.out.tfevents.1665694939.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b1bd83650afe805616cbe486ee31cf88fe1ef08b560348f360bc86504a71cc38
+size 80915759

runs/large-B96-L1e4-2M/events.out.tfevents.1665722154.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9ba072b621eb16ac2c10c52f3682806d9d3bba16e80742aba11e74659c0ae63c
+size 115581525

runs/large-B96-L1e4-2M/events.out.tfevents.1665862500.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:745de7733886d5b33223a38ba1af336ca25fb6f186094e55f1d0966420a4e338
+size 80941568

runs/large-B96-L1e4-2M/events.out.tfevents.1665911345.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a1c1f7c011354deba627424d6dd76f6b7858f8abdef171e36b6a2cdc0904ea78
+size 80946451

runs/large-B96-L1e4-2M/events.out.tfevents.1665965743.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e121a01639e86bc6f4eb2e62519161ebd93ca9acd4b47f64342ddca4a5c004a5
+size 80922987

runs/large-B96-L1e4-2M/events.out.tfevents.1665993991.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:280d7c61568c5759d10c7e6c76ff4193b9b842e12a26f4554e776132607b09d0
+size 64145720

runs/large-B96-L1e4-2M/events.out.tfevents.1666062134.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e42b79cbb0ab088c11db74994dd5d0633ffed7d281ede0ab94f74ef2c3137dfd
+size 64148675

runs/large-B96-L1e4-2M/events.out.tfevents.1666134343.hk-electra-tpuv3b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13ab7c99dcec3eeaba09d4e23153c5fa48609ef6f631ac109a925999252dcc9d
+size 14286111

runs/large-B96-L1e4-2M/events.out.tfevents.1666135134.hk-electra-tpuv3b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfdc9ecce9871107a80849c07e5b4c58b31a08a523b104e6ab4d46a3782468cf
+size 14286114

runs/large-B96-L1e4-2M/events.out.tfevents.1666135271.hk-electra-tpuv3b

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0e5d7edf2eeed99e34f7be097fe939c8124a4e77a574db122cebfdfec02fbbef
+size 65256032

runs/small/events.out.tfevents.1665488622.hk-electra-tpuv3

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:80980ce88819c7845a7ec2e3b75b8f6f56fa37b1b29f14b79bdf1e7b3f562a1c
+size 32978061

tf_checkpoint/base/checkpoint

@@ -0,0 +1,2 @@
+model_checkpoint_path: "model.ckpt-1000000"
+all_model_checkpoint_paths: "model.ckpt-1000000"

tf_checkpoint/base/graph.pbtxt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e6d180eb2e06c4114a75ae7134a67a44a2773ef4f519ee561ea27ada4d32144
+size 20279384

tf_checkpoint/base/model.ckpt-1000000.data-00000-of-00001

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0628f6bf5a4067e76649aa69c87d1ce0eed58b519236fab45753a2949fe339a9
+size 1345949300

tf_checkpoint/base/model.ckpt-1000000.index

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e925a95f98a85f4b579ba29206ec12517e5bc31b1df57353368489a35455dff7
+size 18063

tf_checkpoint/base/model.ckpt-1000000.meta

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d965ae35f21f71f5018e5de76082f72148521473f3e4e36a0f770aae7155e70e
+size 8971897

tf_checkpoint/large-B96-L1e4-2M/checkpoint

@@ -0,0 +1,2 @@
+model_checkpoint_path: "model.ckpt-2000000"
+all_model_checkpoint_paths: "model.ckpt-2000000"

tf_checkpoint/large-B96-L1e4-2M/graph.pbtxt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:40077e936a82ea74e8024c93870792e70c70530978fa039047b3948626bf8448
+size 39970200

tf_checkpoint/large-B96-L1e4-2M/model.ckpt-2000000.data-00000-of-00001

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:38c0f896ad9058c4e5cc51b6e93fe10df45fb1ddf9f8e849f2ec43db2a9ca302
+size 4140317300

tf_checkpoint/large-B96-L1e4-2M/model.ckpt-2000000.index

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a925e88b46df901a20b60bca263e50b16bedd34b8881ae9eb7cf503b237e6149
+size 34448

tf_checkpoint/large-B96-L1e4-2M/model.ckpt-2000000.meta

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a22be54f295fdd13263d265ec243478b267588a26cb62389865d62c2e7cf9e75
+size 17762941