README.md

metadata

license: apache-2.0
language:
  - zh
  - en
tags:
  - bert
  - fill-mask
  - chinese
  - modernbert
  - masked-language-modeling
pipeline_tag: fill-mask
library_name: pytorch

BERTc-315M

BERTc-315M is a char-level Chinese Modern BERTc masked language model trained from scratch. It uses a custom ModernBERT-style PyTorch architecture from the BERTc repository, with ScaledSinusoidal positional embeddings, GeGLU MLPs, no linear biases, tied input/output embeddings, and a SentencePiece-based char/BPE tokenizer.

Model Details

• Parameters: 315M
• Architecture: 24L / 1024H / 2752I / 16 heads
• Vocabulary size: 12,536
• Max position length: 1,024
• Pretraining data: 17.65B-token BERTc mixed corpus
• License: Apache-2.0

Reported Downstream Results

These are internal BERTc evaluations using fine-tuned heads:

• PD-1998 CWS/POS/NER multi-task: score 1.4712 (CWS 0.9840 / POS 0.9800 / NER 0.9660)
• SIGHAN-15 Chinese spelling correction: SIGHAN-15 sentence F1 0.8346

Current strongest Modern BERTc backbone, trained on the full 17.65B-token corpus.

Files

• model.safetensors: ModernBertForMLM state dict.
• config.json: architecture configuration.
• model.py: model implementation used by the original training code.
• piece.model: tokenizer model; load with piece_tokenizer using cn_dict="no".
• mask_token_id.txt: mask token id.

Loading

import json
import torch
from safetensors.torch import load_file
from model import ModernBertConfig, ModernBertForMLM

with open("config.json") as f:
    cfg = ModernBertConfig(**json.load(f))

model = ModernBertForMLM(cfg)
state = load_file("model.safetensors")
model.load_state_dict(state, strict=True)
model.eval()

Tokenization in the original code uses the sibling piece_tokenizer package:

import piece_tokenizer as pt

tok = pt.Tokenizer()
tok.load("piece.model", cn_dict="no")
ids = tok.encode_as_ids("中文测试")

Intended Use

Use this model as a Chinese encoder/MLM backbone for fine-tuning tasks such as CWS, POS, NER, and Chinese spelling correction. This release is not an instruction model and is not intended for text generation.