yuyijiong/LFM2.5-Encoder-350M-corpus-cleaner

LFM2.5-Encoder-350M Corpus Cleaner

English | 中文

---

Try this model in HF space

English

Overview

LFM2.5-Encoder-350M Corpus Cleaner is a token-classification model built on LFM2.5-Encoder. Given noisy pretraining text, it labels each token as KEEP or DELETE, so you can strip noise in one pass and recover clean, usable text—without rewriting the document.

Label	ID	Meaning
KEEP	0	Content worth keeping for pretraining
DELETE	1	Noise to remove (ads, boilerplate, garbled text, etc.)

Note: This is v0.0.1. It may not cover every edge case. If you hit issues, please leave a comment or open an issue.

Why this model?

Raw pretraining corpora are often full of noise: garbled bytes, site chrome, ads, navigation, references, and other low-value spans. Common approaches fall short:

1. Hand-written rules — brittle and hard to maintain; they miss the long tail of ad and boilerplate patterns.
2. FastText / BERT quality scorers — can only drop low-scoring documents, not clean them into usable text.
3. LLM rewrite / cleaning — highest quality, but slow. At 200 tokens/s with 500 concurrent workers, cleaning 1 TB of text can take on the order of ~120 days.

Our approach

We train a token-classification model on LFM2.5-Encoder to mark noisy spans directly, then reconstruct clean text by keeping only KEEP tokens.

Training data: ~3M aligned (raw → cleaned) examples. Clean targets were produced with Qwen3.5-9B + a cleaning prompt on noisy web-scale source text.

Advantages

1. Orders of magnitude faster than generative LLMs — no GPU required for practical use. Measured throughput (this model, token classification):

   • CPU: ~3k tokens/s
   • GPU (single H20, PyTorch): ~160k tokens/s
   • vLLM can push throughput even higher — cleaning terabyte-scale corpora in days is realistic.

2. All-in-one, out of the box — no rule stack to tune. Works as both cleaning and filtering: removes ads, page shells, bibliographies, formatting junk, and other training-irrelevant content while preserving useful prose. Low-quality samples (e.g. pure ads) may be deleted entirely; post-clean length is a useful quality signal.

3. 8k context — reduces painful document chunking during preprocessing and inference.

Quick start

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

device = torch.device("cpu")
model_id = "yuyijiong/LFM2.5-Encoder-350M-corpus-cleaner"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
)
model.to(device)
model.eval()

text = "Your noisy pretraining document here ..."
enc = tokenizer(
    text,
    return_offsets_mapping=True,
    truncation=True,
    max_length=8192,
    return_tensors="pt",
)
offsets = enc.pop("offset_mapping")[0].tolist()
inputs = {k: v.to(device) for k, v in enc.items()}

with torch.inference_mode():
    logits = model(**inputs).logits[0]
    labels = logits.argmax(dim=-1).tolist()

# Reconstruct: keep spans where label == 0 (KEEP); merge overlapping offsets
ranges = [(s, e) for (s, e), lab in zip(offsets, labels) if lab == 0 and e > s]
ranges.sort()
merged = []
for s, e in ranges:
    if merged and s <= merged[-1][1]:
        merged[-1] = (merged[-1][0], max(merged[-1][1], e))
    else:
        merged.append((s, e))
cleaned = "".join(text[s:e] for s, e in merged)
print(cleaned)

Requires transformers with LFM2 support and trust_remote_code=True (custom modeling code is bundled).

Model details

Base encoder	LFM2.5-Encoder-350M (bidirectional)
Task	Token classification (KEEP / DELETE)
Parameters	~350M
Recommended max_length	8192
Version	v0.0.1

Citation

@misc{LFM2.5-Encoder-350M-corpus-cleaner,
  author       = {Yijiong Yu},
  title        = {LFM2.5-Encoder-350M-corpus-cleaner},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Repository},
  howpublished = {\url{https://huggingface.co/yuyijiong/LFM2.5-Encoder-350M-corpus-cleane}},
}

---

中文

简介

LFM2.5-Encoder-350M Corpus Cleaner 是基于 LFM2.5-Encoder 的 Token 分类模型。对任意含噪声的预训练文本，模型逐 token 预测 保留（KEEP） 或 删除（DELETE），一键剔除噪声并还原可用正文，无需整段重写。

标签	ID	含义
KEEP	0	值得保留的训练内容
DELETE	1	应删除的噪声（广告、壳层、乱码等）

提示： 当前为 v0.0.1 版本，可能无法覆盖所有场景。使用中如有问题，欢迎评论或提 Issue。

动机

原始预训练数据往往包含大量噪声（乱码、网页目录、广告等）。常见清洗方式各有局限：

1. 大量手工规则 — 不够灵活，难以覆盖千奇百怪的广告与版式噪声。
2. FastText / BERT 质量打分 — 只能丢弃低分样本，无法把脏数据清洗成可用数据。
3. LLM 清洗或重写 — 质量最好，但速度慢。按约 200 tokens/s、500 并发估算，清洗 1 TB 数据约需 120 天。

我们的方案

训练基于 LFM2.5-Encoder 的 Token 分类模型，直接标出原文中的噪声片段，再拼接保留内容得到干净文本。

训练数据： 使用 Qwen3.5-9B + 清洗 prompt，在互联网原始语料上生成约 300 万 条「原始文本 → 清洗后文本」对齐样本。

优势

1. 速度远超生成式 LLM（可达数量级以上），无需 GPU 也可实用部署：

   • 纯 CPU： 约 3k tokens/s
   • 单卡 H20（纯 PyTorch）： 约 160k tokens/s
   • 配合 vLLM 还可更快 — 几天内洗完 TB 级数据 具备可行性。

2. All-in-one，开箱即用 — 无需维护规则库，同时承担清洗与筛选：可去除广告、网页壳、参考文献、格式性文本等对训练无用的内容，并保留正文；对极低质量样本（如纯广告）会整段删空，因此清洗后文本长度也可作为质量参考。

3. 支持 8k 上下文 — 减轻长文档切分带来的工程负担。

快速上手

from transformers import AutoModelForTokenClassification, AutoTokenizer
import torch

device = torch.device("cpu")
model_id = "yuyijiong/LFM2.5-Encoder-350M-corpus-cleaner"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForTokenClassification.from_pretrained(
    model_id, trust_remote_code=True, torch_dtype=torch.float32
)
model.to(device)
model.eval()

text = "这里放待清洗的预训练文本 ..."
enc = tokenizer(
    text,
    return_offsets_mapping=True,
    truncation=True,
    max_length=8192,
    return_tensors="pt",
)
offsets = enc.pop("offset_mapping")[0].tolist()
inputs = {k: v.to(device) for k, v in enc.items()}

with torch.inference_mode():
    logits = model(**inputs).logits[0]
    labels = logits.argmax(dim=-1).tolist()

# 按 offset 拼接 label==0（KEEP）的片段；合并重叠 offset（LFM 对部分汉字会切出重叠碎片）
ranges = [(s, e) for (s, e), lab in zip(offsets, labels) if lab == 0 and e > s]
ranges.sort()
merged = []
for s, e in ranges:
    if merged and s <= merged[-1][1]:
        merged[-1] = (merged[-1][0], max(merged[-1][1], e))
    else:
        merged.append((s, e))
cleaned = "".join(text[s:e] for s, e in merged)
print(cleaned)

需安装支持 LFM2 的 transformers，并设置 trust_remote_code=True（仓库内已包含自定义建模代码）。

模型信息

基座	LFM2.5-Encoder-350M（双向）
任务	Token 分类（KEEP / DELETE）
参数量	约 350M
推荐 max_length	8192
版本	v0.0.1

引用

@misc{LFM2.5-Encoder-350M-corpus-cleaner,
  author       = {Yijiong Yu},
  title        = {LFM2.5-Encoder-350M-corpus-cleaner},
  year         = {2026},
  publisher    = {Hugging Face},
  journal      = {Hugging Face Repository},
  howpublished = {\url{https://huggingface.co/yuyijiong/LFM2.5-Encoder-350M-corpus-cleane}},
}