|
|
--- |
|
|
license: apache-2.0 |
|
|
datasets: |
|
|
- OmniAICreator/Japanese-Novels-23M |
|
|
- NilanE/ParallelFiction-Ja_En-100k |
|
|
- globis-university/aozorabunko-clean |
|
|
- joujiboi/Galgame-VisualNovel-Reupload |
|
|
- CC100 |
|
|
- AnimeText |
|
|
language: |
|
|
- ja |
|
|
pipeline_tag: fill-mask |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# Custom Japanese BERT (4-layer) |
|
|
|
|
|
This model is a tiny Japanese BERT model with 4 layers, optimized for speed. |
|
|
|
|
|
## Model Background |
|
|
- **Architecture:** BERT (4 layers, 256 hidden size, 4 heads, 1024 FFN) |
|
|
- **Distillation:** Distilled from a fine-tuned version of `tohoku-nlp/bert-base-japanese-char-v2`. |
|
|
- **Initialization:** The student model was randomly initialized. |
|
|
- **Tokenizer:** Japanese Character-level tokenizer, shared with the teacher. |
