Update README.md

README.md

@@ -1,20 +1,19 @@
-
 ---
 language:
 - en
 license: apache-2.0
 datasets:
-- HuggingFaceFW/finepdfs_fw_edu_labeled
+- HuggingFaceFW/finepdfs_eng_Latn_labeled
 ---
 
-# FinePDFs-Edu classifier (English)
+# FinePDFs-OCR-Quality classifier (English)
 
 ## Model summary
-This is a classifier for judging the educational value of web pages. It was developed to filter and curate educational content from web datasets and was trained on 0 [annotations](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_fw_edu_labeled) generated by [Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) for web samples from [FinePDFs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs) dataset.
+This is a classifier for judging the educational value of web pages. It was developed to filter and curate educational content from web datasets and was trained on 1304547
+ [annotations](https://huggingface.co/datasets/HuggingFaceFW/finepdfs_eng_Latn_labeled) generated by [Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) for web samples from [FinePDFs](https://huggingface.co/datasets/HuggingFaceFW/finepdfs) dataset.
 
-We used this classifier to build [FinePDFs-Edu](https://huggingface.co/datasets/HuggingFaceFW/finepdfs-edu) dataset.
 ### How to use in transformers
-To load the FinePDFs-Edu classifier, use the following code:
+To load the FinePDFs-OCR-Quality classifier, use the following code:
 
 ```python
 from transformers import AutoTokenizer, AutoModelForSequenceClassification
@@ -83,42 +82,33 @@ print(max(scores))
 ```
 
 ## Training
-The classifier was trained on 11153120 pairs of web samples and their scores from 0 to 5, generated by Qwen3-235B-A22B-Instruct-2507. The samples were annotated based on their educational quality with 0 being not educational and 5 being highly educational. 
+The classifier was trained on 1304547 pairs of web samples and their scores from 0 to 5, generated by Qwen3-235B-A22B-Instruct-2507. The samples were annotated based on their educational quality with 0 being not educational and 5 being highly educational. 
 
 Below is the prompt used for Qwen3-235B-A22B-Instruct-2507 annotations:
 ```
-Below is an extract from a PDF file. Evaluate whether the extract has a high educational
-value and could be useful in an educational setting for teaching from primary school to
-grade school levels using the additive 5-point scoring system described below. Points are
-accumulated based on the satisfaction of each criterion:
-- Add 1 point if the extract provides some basic information relevant to educational topics, even if it includes some irrelevant or non-academic content like advertisements and
-promotional material.
-- Add another point if the extract addresses certain elements pertinent to education but
-does not align closely with educational standards. It might mix educational content with
-non-educational material, offering a superficial overview of potentially useful topics, or
-presenting information in a disorganized manner and incoherent writing style.
-- Award a third point if the extract is appropriate for educational use and introduces key
-concepts relevant to school curricula. It is coherent though it may not be comprehensive
-or could include some extraneous information. It may resemble an introductory section of
-a textbook or a basic tutorial that is suitable for learning but has notable limitations like
-treating concepts that are too complex for grade school students.
-- Grant a fourth point if the extract highly relevant and beneficial for educational purposes
-for a level not higher than grade school, exhibiting a clear and consistent writing style. It
-could be similar to a chapter from a textbook or a tutorial, offering substantial educational
-content, including exercises and solutions, with minimal irrelevant information, and the
-concepts aren’t too advanced for grade school students. The content is coherent, focused,
-and valuable for structured learning.
-- Bestow a fifth point if the extract is outstanding in its educational value, perfectly suited for
-teaching either at primary school or grade school. It follows detailed reasoning, the writing
-style is easy to follow and offers profound and thorough insights into the subject matter,
-devoid of any non-educational or complex content.
-The extract: {example}.
+Below is an extract from a PDF file. Evaluate the quality of the document extraction using the 4-point string system described below. Select the single score that best represents the extraction quality level:
+
+**Score 0: Garbage Text Present**
+- Award 0 points if there are any garbage artifacts present in the text, regardless of how much legitimate content surrounds them. This includes OCR corruption like random character sequences (e.g., "7*/3./ +*/ 6- 4603"), unreadable symbol combinations, corrupted encoding artifacts, or any form of garbled text that renders portions of the document incomprehensible. Even if 90% of the text is perfectly readable, the presence of any garbage characters results in a score of 0.
+
+**Score 1: Clear Formatting Issues**
+- Award 1 point if there are no garbage characters but clear formatting problems are present. This includes broken mathematical equations or formulas that are unreadable, excessive or irregular spacing that disrupts readability, malformed tables or lists, severely corrupted line breaks, or other structural formatting issues that significantly impact the document's usability while keeping the text itself readable.
+
+**Score 2: Minor Formatting Problems**
+- Award 2 points if there are no garbage characters but minor formatting issues exist. This includes scattered extra spaces within words or sentences (e.g., "A t t h e S how"), inconsistent spacing, minor alignment issues, occasional broken line formatting, or small structural problems that don't severely impact readability but indicate imperfect extraction quality.
+
+**Score 3: Clean Extraction**
+- Award 3 points if there are no OCR garbage artifacts, no significant formatting issues, and the text extraction preserves the document's structure and readability effectively. The content should be clean, properly formatted, and easily readable with minimal to no extraction artifacts.
+
+## Evaluation Process
+The extract: {example}
+
 After examining the extract:
-- Briefly justify your total score, up to 100 words.
-- Conclude with the score using the format: "Educational score: <total points>"\
+- Briefly justify your score, focusing specifically on the presence of garbage text, formatting issues, and overall extraction quality, up to 100 words.
+- Conclude with the score using the format: "Document extraction score: <total points>"\
 ```
 
-We added a classification head with a single regression output to answerdotai/ModernBERT-large, unroze the last 4 layers and trained the model for 5000 steps with a learning rate of 3e-4.
+We added a classification head with a single regression output to answerdotai/ModernBERT-large, unfroze the last 4 layers and trained the model for 5000 steps with a learning rate of 3e-4.
 
 **Training Details:**
 
@@ -131,7 +121,7 @@ We added a classification head with a single regression output to answerdotai/Mo
 
 **Classification report**
 
-We treat the regression model's predictions as discrete classes to calculate the metrics on a hold-out set of 0 Qwen3-235B-A22B-Instruct-2507-annotated samples.
+We treat the regression model's predictions as discrete classes to calculate the metrics on a hold-out set of 10000 Qwen3-235B-A22B-Instruct-2507-annotated samples.
 ```
 Validation Report:
 |   class |   precision |   recall |   f1-score |   support |
@@ -144,7 +134,7 @@ Validation Report:
 
 **Confusion matrix**
 
-We verify that the predicted educational scores are indeed close to their ground truth, and are mostry impacted by the noisy annotation.
+We verify that the predicted ocr quality scores are indeed close to their ground truth, and are mostry impacted by the noisy annotation.
 ```
 Confusion Matrix:
 |   class  |   0 |    1 |    2 |    3 |
@@ -157,11 +147,12 @@ Confusion Matrix:
 
 
 ## Limitations
-While the FinePDFs-Edu classifier performs well in distinguishing high-quality educational content for FinePDFs dataset, there are some limitations:
+While the FinePDFs-OCR-Quality classifier performs well in distinguishing high-quality PDF extraction for FinePDFs dataset, there are some limitations:
 
-- Scope: The model's performance might change for other datasets, in particular for out of distribution samples. It is also focused on educational content relevant to primary and grade school levels and may not perform as well on content intended for higher education or specialized domains.
-- Bias: The model's performance is dependent on the quality and representativeness of the training data and the LLM used for the annotation. Biases in both can affect the classifier's judgments. It might overfit to academic looking content for the higher scores and we recommend using int_score >= 1.35 (top 10% for english) as a threshold for data curation.
-- Context: The classifier evaluates individual web pages or extracts without considering broader context, which might impact its effectiveness in certain scenarios.
+- Scope: The model evaluates OCR quality using the recognized text only. Its behavior can vary across languages, scripts, and formatting (tables, math, mixed inline code). It is tuned on common, printed materials and may be less reliable on handwriting-heavy documents, highly technical notation, or unconventional orthography.
+- Bias: Performance depends on the representativeness of the text produced by the OCR pipeline and the data used to train/annotate the classifier. If training skewed toward clean, Latin-script outputs or specific OCR engines, the classifier may systematically favor those and under-score text from other scripts, noisy sources, or different OCR models.
+- Context: The classifier scores individual pages/snippets of post-OCR text without access to the original images, layout, or broader document context. It does not model downstream usage (e.g., NER, search, or translation) and cannot recover layout fidelity, tables, or figures lost during OCR.
 
+Thresholds / Recommendation: In our evaluations, applying classifier-score filtering provided no measurable downstream performance benefit. We therefore do not recommend using any score threshold for curation or routing.
 The training and inference code is available on GitHub 
-https://github.com/huggingface/finepdfs/tree/main/classification
+https://github.com/huggingface/finepdfs/tree/main/classification