balastml/COPAL

T5-CEFR — English Sentence CEFR Level Classifier

A fine-tuned t5-base that predicts the CEFR level (a1, a2, b1, b2, c1) of an English sentence. It is framed as a text-to-text task: given a sentence, the model generates the level token.

Input format is fixed. The model was trained with the prefix classify cefr: . You must use it:

classify cefr: <your sentence>

Output is one of: a1, a2, b1, b2, c1.

Quick start (Transformers)

from transformers import T5TokenizerFast, T5ForConditionalGeneration
import torch

tok = T5TokenizerFast.from_pretrained("your-username/t5-cefr-v3", model_max_length=96)
model = T5ForConditionalGeneration.from_pretrained("your-username/t5-cefr-v3").eval()

def cefr(sentence: str) -> str:
    enc = tok("classify cefr: " + sentence, return_tensors="pt", truncation=True, max_length=96)
    with torch.no_grad():
        out = model.generate(**enc, max_length=8, num_beams=1)
    return tok.decode(out[0], skip_special_tokens=True).strip().lower()

print(cefr("She eats an apple every morning."))                  # a1
print(cefr("Despite the heavy traffic, we arrived on time."))    # b2

Or use the bundled script:

pip install -r requirements.txt
python predict.py --text "If I had known, I would have left earlier."
python predict.py        # interactive

GGUF / llama.cpp

Quantized GGUF builds are included for CPU inference with llama.cpp:

File	Size	Notes
t5-cefr-v3-f16.gguf	427 MB	full precision
t5-cefr-v3-q4_k_m.gguf	140 MB	4-bit, near-lossless

T5 is an encoder-decoder model, so use llama-completion (not llama-cli), and note that Ollama does not support encoder-decoder models:

llama-completion -m t5-cefr-v3-q4_k_m.gguf \
  -p "classify cefr: The intricate argument was difficult to follow." \
  -n 4 --no-warmup --simple-io
# -> c1

To serve an HTTP API, use llama-server from llama.cpp (it supports T5 encoder-decoder).

Intended use & limitations

• Use: quick, lightweight CEFR difficulty estimation of single English sentences for language-learning tools, content grading, and exercise generation.
• Scope: sentence-level only (not documents), English only, levels a1–c1 (C2 was folded into C1).
• Fuzziness: CEFR labelling is inherently subjective; adjacent levels (e.g. A2 vs B1) overlap. Treat the prediction as an estimate. The ±1 level accuracy is far higher than exact accuracy (see below).
• Not a chat/instruction model — it only emits a level token.

Training

• Base: google-t5/t5-base (220M), full fine-tune (not LoRA).
• Data:
  • CEFR-SP — human expert sentence-level CEFR annotations (label = rounded mean of two annotators; C2 folded into C1).
  • Synthetic sentences generated with DeepSeek-V4 across all levels to balance the corpus (the base CEFR-SP set is severely A1-sparse). Generation used diversity controls (rotating themes, opening-word caps, banned clichéd openers).
• Combined training set: ~9.2k sentences; minority classes oversampled.
• Trained in fp32 (T5 is unstable in fp16).

Evaluation

On the held-out combined test set (1,637 sentences):

Metric	Score
Exact accuracy	64.9%
±1 level accuracy	98.7%

Per-level exact: a1 0.49 · a2 0.76 · b1 0.60 · b2 0.59 · c1 0.82.

On a clean hand-written reference set (10 textbook-style sentences, 2 per level):

Metric	Score
Exact accuracy	90% (9/10)
±1 level accuracy	100% (10/10)

The model almost never makes a far-off prediction (the ±1 accuracy is ~99%).

License

Released under CC BY-NC-SA 4.0, inherited from the CEFR-SP training data (cc-by-nc-sa-4.0). Non-commercial use; share alike. The base t5-base is Apache-2.0.