README.md

---
license: cc-by-nc-sa-4.0
language:
- en
base_model:
- google-t5/t5-base
library_name: transformers
pipeline_tag: text-classification
tags:
- text2text-generation
- cefr
- text-classification
- language-learning
- education
- t5
- gguf
metrics:
- accuracy
---

# T5-CEFR — English Sentence CEFR Level Classifier

A fine-tuned [`t5-base`](https://huggingface.co/google-t5/t5-base) that predicts the **CEFR level** (`a1`, `a2`, `b1`, `b2`, `c1`) of an English **sentence**. It is framed as a text-to-text task: given a sentence, the model generates the level token.

> **Input format is fixed.** The model was trained with the prefix `classify cefr: `. You **must** use it:
> ```
> classify cefr: <your sentence>
> ```
> Output is one of: `a1`, `a2`, `b1`, `b2`, `c1`.

## Quick start (Transformers)

```python
from transformers import T5TokenizerFast, T5ForConditionalGeneration
import torch

tok = T5TokenizerFast.from_pretrained("your-username/t5-cefr-v3", model_max_length=96)
model = T5ForConditionalGeneration.from_pretrained("your-username/t5-cefr-v3").eval()

def cefr(sentence: str) -> str:
    enc = tok("classify cefr: " + sentence, return_tensors="pt", truncation=True, max_length=96)
    with torch.no_grad():
        out = model.generate(**enc, max_length=8, num_beams=1)
    return tok.decode(out[0], skip_special_tokens=True).strip().lower()

print(cefr("She eats an apple every morning."))                  # a1
print(cefr("Despite the heavy traffic, we arrived on time."))    # b2
```

Or use the bundled script:

```bash
pip install -r requirements.txt
python predict.py --text "If I had known, I would have left earlier."
python predict.py        # interactive
```

## GGUF / llama.cpp

Quantized GGUF builds are included for CPU inference with [llama.cpp](https://github.com/ggerganov/llama.cpp):

| File | Size | Notes |
|------|------|-------|
| `t5-cefr-v3-f16.gguf` | 427 MB | full precision |
| `t5-cefr-v3-q4_k_m.gguf` | 140 MB | 4-bit, near-lossless |

T5 is an **encoder-decoder** model, so use `llama-completion` (not `llama-cli`), and note that **Ollama does not support encoder-decoder models**:

```bash
llama-completion -m t5-cefr-v3-q4_k_m.gguf \
  -p "classify cefr: The intricate argument was difficult to follow." \
  -n 4 --no-warmup --simple-io
# -> c1
```

To serve an HTTP API, use `llama-server` from llama.cpp (it supports T5 encoder-decoder).

## Intended use & limitations

- **Use:** quick, lightweight CEFR difficulty estimation of single English sentences for language-learning tools, content grading, and exercise generation.
- **Scope:** sentence-level only (not documents), English only, levels `a1`–`c1` (C2 was folded into C1).
- **Fuzziness:** CEFR labelling is inherently subjective; adjacent levels (e.g. A2 vs B1) overlap. Treat the prediction as an estimate. The `±1 level` accuracy is far higher than exact accuracy (see below).
- Not a chat/instruction model — it only emits a level token.

## Training

- **Base:** `google-t5/t5-base` (220M), **full fine-tune** (not LoRA).
- **Data:**
  - [CEFR-SP](https://github.com/yukiar/CEFR-SP) — human expert sentence-level CEFR annotations (label = rounded mean of two annotators; C2 folded into C1).
  - **Synthetic** sentences generated with DeepSeek-V4 across all levels to balance the corpus (the base CEFR-SP set is severely A1-sparse). Generation used diversity controls (rotating themes, opening-word caps, banned clichéd openers).
- Combined training set: ~9.2k sentences; minority classes oversampled.
- Trained in fp32 (T5 is unstable in fp16).

## Evaluation

**On the held-out combined test set (1,637 sentences):**

| Metric | Score |
|--------|-------|
| Exact accuracy | **64.9%** |
| ±1 level accuracy | **98.7%** |

Per-level exact: a1 0.49 · a2 0.76 · b1 0.60 · b2 0.59 · c1 0.82.

**On a clean hand-written reference set (10 textbook-style sentences, 2 per level):**

| Metric | Score |
|--------|-------|
| Exact accuracy | **90%** (9/10) |
| ±1 level accuracy | **100%** (10/10) |

The model almost never makes a far-off prediction (the ±1 accuracy is ~99%).

## License

Released under **CC BY-NC-SA 4.0**, inherited from the CEFR-SP training data ([cc-by-nc-sa-4.0](https://github.com/yukiar/CEFR-SP)). Non-commercial use; share alike. The base `t5-base` is Apache-2.0.