metadata
license: apache-2.0
datasets:
- OmniAICreator/Japanese-Novels-23M
- NilanE/ParallelFiction-Ja_En-100k
- globis-university/aozorabunko-clean
- joujiboi/Galgame-VisualNovel-Reupload
- CC100
- AnimeText
language:
- ja
pipeline_tag: fill-mask
library_name: transformers
Custom Japanese BERT (4-layer)
This model is a tiny Japanese BERT model with 4 layers, optimized for speed.
Model Background
- Architecture: BERT (4 layers, 256 hidden size, 4 heads, 1024 FFN)
- Distillation: Distilled from a fine-tuned version of
tohoku-nlp/bert-base-japanese-char-v2. - Initialization: The student model was randomly initialized.
- Tokenizer: Japanese Character-level tokenizer, shared with the teacher.