Add model card

aa3153c verified 2 days ago

2.52 kB

language: en
library_name: transformers.js
pipeline_tag: text-classification
base_model: microsoft/deberta-v3-small
tags:
  - cefr
  - text-classification
  - onnx

Speako CEFR Classifier

Fine-tuned microsoft/deberta-v3-small that classifies English text into CEFR proficiency levels (A1–C2). Built for Speako, a browser-based speaking-practice app that runs this model client-side via Transformers.js.

Files

onnx/model_quantized.onnx (~172 MB) — INT8 dynamic-quantized, what the app loads (dtype: 'q8')
onnx/model.onnx (~568 MB) — FP32 export

Use the v2 tag: the main revision's early history had an empty root config.json, and clients that cached it never revalidate.

import { pipeline } from '@huggingface/transformers';

const classify = await pipeline('text-classification', 'robg/speako-cefr-deberta', {
  device: 'wasm', // the q8 model mis-executes on the WebGPU backend
  dtype: 'q8',
  revision: 'v2',
});
const [top] = await classify('I think studying abroad teaches independence.', { top_k: 1 });
// { label: 'B2', score: ... }

Run the quantized model on CPU/WASM. On the onnxruntime-web WebGPU backend it produces degenerate predictions (C1 for nearly everything).

Training data

Written English text from three datasets, chunked to 5–50 words and augmented with synthetic ASR noise and disfluencies:

edesaras/CEFR-Sentence-Level-Annotations (MIT)
Alex123321/english_cefr_dataset (Apache-2.0)
amontgomerie/cefr-levelled-english-texts (see dataset card)

Measured accuracy

Speak & Improve 2025 eval-asr reference transcripts (1,500-sample subsample, coarse C labels mapped to C1): 40.5% exact, 89.7% within one level. That eval set is 51% B2; a constant-B2 predictor scores 51%/95%, so treat exact-level predictions as rough estimates.
Full Speako pipeline (Whisper transcription → this model on WASM), 40 S&I dev files: 70% exact, 95% within one level.

Limitations

Trained on written text but typically applied to transcripts of spontaneous speech — a domain gap synthetic augmentation only partly closes. Not suitable for high-stakes assessment.