MohammadKhosravi/roberta-large-cefr-classifier-WCE

RoBERTa Large - CEFR Classifier (WCE Method)

This repository contains a publication-grade, fine-tuned transformer model optimized for mapping English student prose directly to the Common European Framework of Reference for Languages (CEFR) proficiency scales (A1–C2).

The model is built on top of the deep contextual architecture of RoBERTa-Large and is fine-tuned on a massive, structural subset of the EFCAMDAT corpus comprising 104,125 rows. It implements a Weighted Cross-Entropy (WCE) nominal framework designed to explicitly eliminate the accuracy penalties caused by severe minority class data scarcity in advanced cohorts.

Model Description

Architectural Modification

The base model originates from FacebookAI/roberta-large (355M parameters). To repurpose the architecture for multi-class sequence classification, the pre-trained Masked Language Modeling (MLM) head (lm_head) was completely excised. It was replaced with a randomly initialized linear classification head mapped to 6 discrete output nodes corresponding to the target CEFR levels:

$$\text{Outputs}\in \{\text{A1},\text{A2},\text{B1},\text{B2},\text{C1},\text{C2}\}\to \{0,1,2,3,4,5\}\backslash text\{Outputs\}\; \backslash in\; \backslash \{\backslash text\{A1\},\; \backslash text\{A2\},\; \backslash text\{B1\},\; \backslash text\{B2\},\; \backslash text\{C1\},\; \backslash text\{C2\}\backslash \}\; \backslash rightarrow\; \backslash \{0,\; 1,\; 2,\; 3,\; 4,\; 5\backslash \}$$

Optimization Framework (Weighted Cross-Entropy)

Standard cross-entropy loss optimizations evaluate errors symmetrically, causing deep neural networks to heavily favor majority classes in skewed datasets to maximize global accuracy. In this nominal framework, categorical cross-entropy loss is scaled using inverse class frequency weights calculated across the entire 104k dataset rows:

$${\mathcal{L}}_{\text{WCE}}=-\frac{1}{B}\sum _{b=1}^B{w}_{y_b}\log \left(\frac{e^{z_{b,y_b}}}{\sum _{c=0}^{C-1}{e}^{z_{b,c}}}\right)\backslash mathcal\{L\}\_\{\backslash text\{WCE\}\}\; =\; -\backslash frac\{1\}\{B\}\; \backslash sum\_\{b=1\}^\{B\}\; w\_\{y\_b\}\; \backslash log\; \backslash left(\; \backslash frac\{e^\{z\_\{b,\; y\_b\}\}\}\{\backslash sum\_\{c=0\}^\{C-1\}\; e^\{z\_\{b,\; c\}\}\}\; \backslash right)$$

The inverse scaling penalty factor for each class $c$ is determined by the formulation:

$$w_c=\frac{N_{\text{total}}}{C\times N_c}w\_c\; =\; \backslash frac\{N\_\{\backslash text\{total\}\}\}\{C\; \backslash times\; N\_c\}$$

This directly forces the loss function to scale up penalties when the architecture misclassifies rare classes like C1 and C2, giving the model an unbiased optimization trajectory despite severe class imbalances.

Note: For an alternative optimization approach that evaluates CEFR categories sequentially using distance constraints, please refer to the companion architecture: RoBERTa Large - CEFR Classifier (Joint WCE+Ordinal Method).

Intended Uses & Limitations

Recommended Applications

• Automated Essay Scoring (AES): Instantaneous evaluation of continuous multi-paragraph student text blocks.
• Educational Analytics: Longitudinal progress tracking of language learners across digital learning platforms.
• Curriculum Alignment: Diagnostic screening of raw reading or writing materials to match specific target student profiles.

Technical Limitations

• Domain Specificity: The model was fine-tuned heavily on non-native English learner compositions. Performance may vary on native speech, fiction prose, or specialized academic/legal discourse.
• Context Length Constraints: Standard tokenization caps the input length at 512 subword tokens. Texts exceeding this threshold will undergo truncation, potentially missing macro-syntactic cues embedded at the tail end of longer essays.

Training and Evaluation Data

The model was developed using a dedicated, customized corpus extracted from the Cambridge English Profile Lab EFCAMDAT Database.

Sampling and Structural Adjustments

• Sampling Protocol: Slices of continuous student prose were extracted sequentially from the shuffled master database following indices explicitly partitioned for downstream pipeline testing to ensure zero data pollution.
• B2 Ceiling Correction: To balance data allocation requirements across external components, a dedicated partition of 10,570 B2 rows was held out, and the remaining 23,529 B2 records were migrated completely to this dataset to maximize deep contextual boundary exposure.
• Retained Dimensions: Each row tracks a unique sequential text identifier, the curriculum tier (1–16), the target CEFR alphanumeric level, and the sanitized continuous student essay text string.

Final Training Partition Inventory

The final dataset consists of 104,125 rows with a heavily stratified 90/10 training and internal validation split:

• Level A1: 25,000 samples | 24.01% allocation
• Level A2: 25,000 samples | 24.01% allocation
• Level B1: 25,000 samples | 24.01% allocation
• Level B2: 23,529 samples | 22.60% allocation
• Level C1: 4,614 samples | 4.43% allocation
• Level C2: 982 samples | 0.94% allocation

---

Training Procedure

Training Hyperparameters

• Learning Rate: 2e-05
• Per-Device Train Batch Size: 16
• Per-Device Eval Batch Size: 16
• Gradient Accumulation Steps: 2 (Effectively maximizing total training batch size to 32 per step)
• Total Seed Configuration: 42
• Optimizer: Fused AdamW (betas=(0.9,0.999), epsilon=1e-08)
• Total Training Epochs: 4
• Precision Mode: Native Automatic Mixed Precision (fp16=True)

Training Progress Logs

Training Loss	Epoch	Step	Validation Loss	Strict Accuracy	Adjacent Accuracy	Macro F1	MAE	QWK
0.3196	1.0	2929	0.2253	97.01%	98.94%	0.9573	0.0443	0.9735
0.1663	2.0	5858	0.1833	97.99%	99.41%	0.9663	0.0284	0.9834
0.0604	3.0	8787	0.1882	98.27%	99.46%	0.9716	0.0248	0.9855
0.0359	4.0	11716	0.1928	98.41%	99.49%	0.9732	0.0230	0.9866

Framework Versions

• Transformers 5.12.0
• Pytorch 2.11.0+cu128
• Datasets 4.0.0
• Tokenizers 0.22.2

---

Per-Class Evaluation Distribution (Internal Validation Split)

This table tracks the performance breakdown on the internal 10% stratified validation partition derived from the training inventory:

CEFR Level	Validation Count	Accuracy (%)	Error Rate (%)
A1	2,500	99.48%	0.52%
A2	2,500	98.72%	1.28%
B1	2,500	98.40%	1.60%
B2	2,353	97.79%	2.21%
C1	462	95.24%	4.76%
C2	98	92.86%	7.14%

---

🧪 Out-of-Distribution Robustness Benchmark (Balanced Unseen Dataset)

To establish true generalizability and rule out internal data leakage, the finalized model was subjected to an Independent Out-of-Distribution (OOD) Stress-Testing Routine.

Benchmark Methodology

The evaluation architecture was frozen in pure inference mode (model.eval(), torch.no_grad()) and fed an entirely separate, non-overlapping dataset slice from EFCAMDAT originally reserved for testing a distinct Plug-and-Play Language Model (PPLM) classifier module. To remove all baseline prevalence bias and simulate a strict diagnostic environment, the evaluation subset was forcefully downsampled into a perfectly balanced profile anchored to the absolute minority class floor ($N = 928$ rows per class across all 6 categories, creating a global verification volume of 5,568 samples).

Global Generalization Metrics

• Strict Accuracy: 97.72%
• Adjacent Accuracy (Distance $\leq$ 1): 98.80%
• Micro F1 Score: 0.9772
• Macro F1 Score: 0.9772
• Mean Absolute Error (MAE): 0.0408
• Quadratic Weighted Kappa (QWK): 0.9838

Per-Class Diagnostic Stress Test Breakdown

CEFR Level	Evaluated Samples	Accuracy (%)	Error Rate (%)
A1	928	99.68%	0.32%
A2	928	98.60%	1.40%
B1	928	98.38%	1.62%
B2	928	98.38%	1.62%
C1	928	95.91%	4.09%
C2	928	95.47%	4.53%

---

How to Use

You can easily instantiate this model for high-throughput batch inference using the code snippet below:

import torch
import numpy as np
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Define target repository path
MODEL_ID = "MohammadKhosravi/roberta-large-cefr-classifier-WCE"

# 1. Load native tokenizer and weighted cross entropy classifier
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID)

# Place model on optimized GPU accelerator if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
model.eval()

# 2. Prepare sample student essays for diagnostic assessment
sample_texts = [
    "I like to play football with my friends on Sunday. It is very fun.",
    "Although the economic implications of the policy change are highly controversial, empirical data suggests a clear trend toward market stabilization."
]

# 3. Compute inference pipeline
inputs = tokenizer(sample_texts, padding=True, truncation=True, max_length=512, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    predictions = torch.argmax(logits, dim=-1).cpu().numpy()

# 4. Map output tokens to clean strings
id2label = model.config.id2label
for text, pred_id in zip(sample_texts, predictions):
    print(f"\nText Sample: {text[:80]}...")
    print(f"Predicted CEFR Level: {id2label[pred_id]}")