File size: 2,147 Bytes
4a0cc0a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: apache-2.0
language:
- zh
- en
tags:
- bert
- fill-mask
- chinese
- modernbert
- masked-language-modeling
pipeline_tag: fill-mask
library_name: pytorch
---
# BERTc-165M
BERTc-165M is a char-level Chinese Modern BERTc masked language model trained from scratch.
It uses a custom ModernBERT-style PyTorch architecture from the BERTc repository, with
ScaledSinusoidal positional embeddings, GeGLU MLPs, no linear biases, tied input/output
embeddings, and a SentencePiece-based char/BPE tokenizer.
## Model Details
- Parameters: 165M
- Architecture: 12L / 1024H / 2752I / 16 heads
- Vocabulary size: 12,536
- Max position length: 1,024
- Pretraining data: 17.65B-token BERTc mixed corpus
- License: Apache-2.0
## Reported Downstream Results
These are internal BERTc evaluations using fine-tuned heads:
- PD-1998 CWS/POS/NER multi-task: score 1.4689 (CWS 0.9836 / POS 0.9753 / NER 0.9632)
- SIGHAN-15 Chinese spelling correction: SIGHAN-15 sentence F1 0.8308
First Modern BERTc backbone to reach broad MT/CSC SOTA at the 165M scale.
## Files
- `model.safetensors`: `ModernBertForMLM` state dict.
- `config.json`: architecture configuration.
- `model.py`: model implementation used by the original training code.
- `piece.model`: tokenizer model; load with `piece_tokenizer` using `cn_dict="no"`.
- `mask_token_id.txt`: mask token id.
## Loading
```python
import json
import torch
from safetensors.torch import load_file
from model import ModernBertConfig, ModernBertForMLM
with open("config.json") as f:
cfg = ModernBertConfig(**json.load(f))
model = ModernBertForMLM(cfg)
state = load_file("model.safetensors")
model.load_state_dict(state, strict=True)
model.eval()
```
Tokenization in the original code uses the sibling `piece_tokenizer` package:
```python
import piece_tokenizer as pt
tok = pt.Tokenizer()
tok.load("piece.model", cn_dict="no")
ids = tok.encode_as_ids("中文测试")
```
## Intended Use
Use this model as a Chinese encoder/MLM backbone for fine-tuning tasks such as CWS,
POS, NER, and Chinese spelling correction. This release is not an instruction model
and is not intended for text generation.
|