---
license: apache-2.0
language:
- zh
- en
tags:
- bert
- fill-mask
- chinese
- modernbert
- masked-language-modeling
pipeline_tag: fill-mask
library_name: pytorch
---

# BERTc-165M

BERTc-165M is a char-level Chinese Modern BERTc masked language model trained from scratch.
It uses a custom ModernBERT-style PyTorch architecture from the BERTc repository, with
ScaledSinusoidal positional embeddings, GeGLU MLPs, no linear biases, tied input/output
embeddings, and a SentencePiece-based char/BPE tokenizer.

## Model Details

- Parameters: 165M
- Architecture: 12L / 1024H / 2752I / 16 heads
- Vocabulary size: 12,536
- Max position length: 1,024
- Pretraining data: 17.65B-token BERTc mixed corpus
- License: Apache-2.0

## Reported Downstream Results

These are internal BERTc evaluations using fine-tuned heads:

- PD-1998 CWS/POS/NER multi-task: score 1.4689 (CWS 0.9836 / POS 0.9753 / NER 0.9632)
- SIGHAN-15 Chinese spelling correction: SIGHAN-15 sentence F1 0.8308

First Modern BERTc backbone to reach broad MT/CSC SOTA at the 165M scale.

## Files

- `model.safetensors`: `ModernBertForMLM` state dict.
- `config.json`: architecture configuration.
- `model.py`: model implementation used by the original training code.
- `piece.model`: tokenizer model; load with `piece_tokenizer` using `cn_dict="no"`.
- `mask_token_id.txt`: mask token id.

## Loading

```python
import json
import torch
from safetensors.torch import load_file
from model import ModernBertConfig, ModernBertForMLM

with open("config.json") as f:
    cfg = ModernBertConfig(**json.load(f))

model = ModernBertForMLM(cfg)
state = load_file("model.safetensors")
model.load_state_dict(state, strict=True)
model.eval()
```

Tokenization in the original code uses the sibling `piece_tokenizer` package:

```python
import piece_tokenizer as pt

tok = pt.Tokenizer()
tok.load("piece.model", cn_dict="no")
ids = tok.encode_as_ids("中文测试")
```

## Intended Use

Use this model as a Chinese encoder/MLM backbone for fine-tuning tasks such as CWS,
POS, NER, and Chinese spelling correction. This release is not an instruction model
and is not intended for text generation.