README.md

---
language: en
library_name: transformers.js
pipeline_tag: text-classification
base_model: microsoft/deberta-v3-small
tags:
  - cefr
  - text-classification
  - onnx
---

# Speako CEFR Classifier

Fine-tuned [microsoft/deberta-v3-small](https://huggingface.co/microsoft/deberta-v3-small) that classifies English text into CEFR proficiency levels (A1–C2). Built for [Speako](https://speako.tre.systems/), a browser-based speaking-practice app that runs this model client-side via [Transformers.js](https://huggingface.co/docs/transformers.js).

## Files

- `onnx/model_quantized.onnx` (~172 MB) — INT8 dynamic-quantized, what the app loads (`dtype: 'q8'`)
- `onnx/model.onnx` (~568 MB) — FP32 export

Use the `v2` tag: the `main` revision's early history had an empty root `config.json`, and clients that cached it never revalidate.

```js
import { pipeline } from '@huggingface/transformers';

const classify = await pipeline('text-classification', 'robg/speako-cefr-deberta', {
  device: 'wasm', // the q8 model mis-executes on the WebGPU backend
  dtype: 'q8',
  revision: 'v2',
});
const [top] = await classify('I think studying abroad teaches independence.', { top_k: 1 });
// { label: 'B2', score: ... }
```

**Run the quantized model on CPU/WASM.** On the onnxruntime-web WebGPU backend it produces degenerate predictions (C1 for nearly everything).

## Training data

Written English text from three datasets, chunked to 5–50 words and augmented with synthetic ASR noise and disfluencies:

- [edesaras/CEFR-Sentence-Level-Annotations](https://huggingface.co/datasets/edesaras/CEFR-Sentence-Level-Annotations) (MIT)
- [Alex123321/english_cefr_dataset](https://huggingface.co/datasets/Alex123321/english_cefr_dataset) (Apache-2.0)
- [amontgomerie/cefr-levelled-english-texts](https://huggingface.co/datasets/amontgomerie/cefr-levelled-english-texts) (see dataset card)

## Measured accuracy

- Speak & Improve 2025 `eval-asr` reference transcripts (1,500-sample subsample, coarse `C` labels mapped to C1): **40.5% exact**, **89.7% within one level**. That eval set is 51% B2; a constant-B2 predictor scores 51%/95%, so treat exact-level predictions as rough estimates.
- Full Speako pipeline (Whisper transcription → this model on WASM), 40 S&I dev files: **70% exact**, **95% within one level**.

## Limitations

Trained on written text but typically applied to transcripts of spontaneous speech — a domain gap synthetic augmentation only partly closes. Not suitable for high-stakes assessment.