BERTc-165M

BERTc-165M is a char-level Chinese Modern BERTc masked language model trained from scratch. It uses a custom ModernBERT-style PyTorch architecture from the BERTc repository, with ScaledSinusoidal positional embeddings, GeGLU MLPs, no linear biases, tied input/output embeddings, and a SentencePiece-based char/BPE tokenizer.

Model Details

• Parameters: 165M
• Architecture: 12L / 1024H / 2752I / 16 heads
• Vocabulary size: 12,536
• Max position length: 1,024
• Pretraining data: 17.65B-token BERTc mixed corpus
• License: Apache-2.0

Reported Downstream Results

These are internal BERTc evaluations using fine-tuned heads:

• PD-1998 CWS/POS/NER multi-task: score 1.4689 (CWS 0.9836 / POS 0.9753 / NER 0.9632)
• SIGHAN-15 Chinese spelling correction: SIGHAN-15 sentence F1 0.8308

First Modern BERTc backbone to reach broad MT/CSC SOTA at the 165M scale.

Files

• model.safetensors: ModernBertForMLM state dict.
• config.json: architecture configuration.
• model.py: model implementation used by the original training code.
• piece.model: tokenizer model; load with piece_tokenizer using cn_dict="no".
• mask_token_id.txt: mask token id.

Loading

import json
import torch
from safetensors.torch import load_file
from model import ModernBertConfig, ModernBertForMLM

with open("config.json") as f:
    cfg = ModernBertConfig(**json.load(f))

model = ModernBertForMLM(cfg)
state = load_file("model.safetensors")
model.load_state_dict(state, strict=True)
model.eval()

Tokenization in the original code uses the sibling piece_tokenizer package:

import piece_tokenizer as pt

tok = pt.Tokenizer()
tok.load("piece.model", cn_dict="no")
ids = tok.encode_as_ids("中文测试")

Intended Use

Use this model as a Chinese encoder/MLM backbone for fine-tuning tasks such as CWS, POS, NER, and Chinese spelling correction. This release is not an instruction model and is not intended for text generation.