Text Classification
Transformers
Safetensors
Japanese
japanese
data-filtering
pretraining-data
common-crawl
Instructions to use minhhien0811/ja-filter-classifier-modernbert-4class with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use minhhien0811/ja-filter-classifier-modernbert-4class with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="minhhien0811/ja-filter-classifier-modernbert-4class")# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("minhhien0811/ja-filter-classifier-modernbert-4class", dtype="auto") - Notebooks
- Google Colab
- Kaggle
| { | |
| "label_order": [ | |
| "reject", | |
| "low_value", | |
| "keep", | |
| "high_value" | |
| ], | |
| "mapping": { | |
| "X": "reject", | |
| "R": "reject", | |
| "KL": "low_value", | |
| "D": "low_value", | |
| "K": "keep", | |
| "KH": "high_value" | |
| }, | |
| "rows": 198874, | |
| "train_rows": 178986, | |
| "test_rows": 19888, | |
| "source_d_counts": { | |
| "KL": 17014, | |
| "D": 137652, | |
| "X": 30408, | |
| "R": 1374, | |
| "K": 12082, | |
| "KH": 344 | |
| }, | |
| "label_counts": { | |
| "low_value": 154666, | |
| "reject": 31782, | |
| "keep": 12082, | |
| "high_value": 344 | |
| }, | |
| "train_label_counts": { | |
| "reject": 28604, | |
| "low_value": 139199, | |
| "keep": 10874, | |
| "high_value": 309 | |
| }, | |
| "test_label_counts": { | |
| "low_value": 15467, | |
| "reject": 3178, | |
| "keep": 1208, | |
| "high_value": 35 | |
| }, | |
| "test_size": 0.1, | |
| "seed": 20260604, | |
| "head_chars": 10000, | |
| "tail_chars": 2000, | |
| "weight_floor": 0.2 | |
| } |