| --- |
| license: mit |
| base_model: intfloat/multilingual-e5-small |
| language: [ru, kk, en] |
| pipeline_tag: text-classification |
| --- |
| |
| # TuranAssist intent classifier |
|
|
| Классификатор 53 интентов чат-бота TuranAssist (Университет «Туран»), ONNX int8. |
| Код и данные: https://github.com/AlexTrav/TuranAssist |
|
|
| # Трансформер multilingual-e5-small против baseline |
|
|
| Значения – доля верных ответов, в скобках 95% доверительный интервал Уилсона. |
|
|
| | Модель | CV acc | Test top-1 | Test macro-F1 | Порог | Test с порогом | OOD отклонено | Внешний тест | Сценарии | p50, мс | p95, мс | |
| |---|---|---|---|---|---|---|---|---|---|---| |
| | `baseline: tfidf_char` | 0.598 | 0.805 [0.758; 0.845] | 0.811 | 0.356 | 0.698 [0.646; 0.746] | 0.808 [0.707; 0.880] | 0.636 [0.569; 0.699] | 0.831 [0.722; 0.903] | 0.8 | 1.6 | |
| | `e5_frozen_logreg` | 0.801 | 0.893 [0.854; 0.922] | 0.896 | 0.589 | 0.629 [0.575; 0.680] | 0.859 [0.765; 0.919] | 0.598 [0.530; 0.662] | 0.800 [0.687; 0.879] | – | – | |
| | `e5_finetuned` | 0.624 | 0.695 [0.642; 0.743] | 0.660 | 0.049 | 0.129 [0.096; 0.170] | 0.987 [0.931; 0.998] | 0.378 [0.315; 0.445] | 0.262 [0.170; 0.380] | – | – | |
| | `e5_frozen_logreg_onnx_int8` | 0.801 | 0.881 [0.840; 0.912] | 0.884 | 0.589 | 0.566 [0.511; 0.619] | 0.859 [0.765; 0.919] | 0.579 [0.511; 0.644] | 0.723 [0.604; 0.817] | 6.6 | 13.7 | |
|
|
| Выбран вариант `e5_frozen_logreg` (по точности кросс-валидации), в продакшн идёт ONNX int8. |
|
|
| ONNX: размер, память сессии, совпадение с PyTorch: |
|
|
| - fp32: 449 МБ, память сессии ~375 МБ, совпадение top-1 с PyTorch 1.000, p50 12.1 мс, p95 21.1 мс (CPU, 1 поток) |
| - int8: 113 МБ, память сессии ~-303 МБ, совпадение top-1 с PyTorch 0.950, p50 6.6 мс, p95 13.7 мс (CPU, 1 поток) |
|
|
| Точность по стилям теста (int8, с порогом): |
|
|
| - long: 0.509 [0.379; 0.639] |
| - mixed: 0.857 [0.487; 0.974] |
| - plain: 0.737 [0.643; 0.814] |
| - slang: 0.472 [0.344; 0.603] |
| - translit: 0.189 [0.106; 0.314] |
| - typo: 0.736 [0.604; 0.836] |
|
|
| По языкам (int8, с порогом): |
|
|
| - en: 0.792 [0.665; 0.880] |
| - kk: 0.698 [0.565; 0.805] |
| - ru: 0.476 [0.410; 0.543] |
|
|
| Самые слабые интенты (F1): |
|
|
| - `goodbye`: F1 0.63 |
| - `admission_steps`: F1 0.71 |
| - `master_admission`: F1 0.73 |
| - `practice`: F1 0.73 |
| - `ent_discounts`: F1 0.77 |
| - `academic_calendar`: F1 0.80 |
| - `academic_integrity`: F1 0.80 |
| - `admission_deadlines`: F1 0.80 |
|
|
| Частые ошибки (верный → предсказанный): |
|
|
| - `greeting` → `goodbye`: 1 |
| - `thanks` → `goodbye`: 1 |
| - `capabilities` → `international_exams`: 1 |
| - `admission_steps` → `transfer_restore`: 1 |
| - `ent_subjects` → `ent_discounts`: 1 |
| - `creative_exam` → `appeal`: 1 |
| - `college_graduates` → `goodbye`: 1 |
| - `programs_list` → `ent_discounts`: 1 |
| - `programs_list` → `transfer_restore`: 1 |
| - `study_forms` → `goodbye`: 1 |
|
|
