| --- |
| language: en |
| base_model: microsoft/deberta-v3-large |
| pipeline_tag: text-classification |
| tags: |
| - semeval-2026 |
| - word-sense-disambiguation |
| - plausibility |
| - cross-encoder |
| - deberta-v3 |
| --- |
| |
| # AmbiStory E13: DeBERTa-v3-large plausibility scorer (SemEval 2026 Task 5) |
|
|
| Rates how plausible (1–5) a given sense of a homonym is in a short story, the task of |
| [SemEval 2026 Task 5 (AmbiStory)](https://github.com/Janosch-Gehring/semeval26-05-scripts). |
| Code, training scripts, evaluation and the Streamlit app: **[github.com/Seronic2001/ambistory](https://github.com/Seronic2001/ambistory)**. |
|
|
| ## Model |
|
|
| - `microsoft/deberta-v3-large` cross-encoder. Segment A is the story with the homonym quoted; segment B is |
| `homonym: gloss. Example: sentence`. |
| - Mean pooling, then an ordinal (proportional-odds) head over the 5 ratings. The score is the expected rating. |
| - Trained on all of train+dev for 6 epochs with KL to the annotator distribution, a within-setup ranking loss and MSE, |
| story-grouped batches and layer-wise LR decay. The checkpoint was chosen on an inner homonym-grouped split |
| (`kaggle/e13_train.py --deploy`). |
| - Two seeds (`model_seed42`, `model_seed43`), stored as fp16 safetensors (828 MB each). The app averages them. |
|
|
| These are custom PyTorch modules, not a `transformers` pipeline. Load them with `load_deployed` from the repo: |
|
|
| ```python |
| from huggingface_hub import snapshot_download |
| import sys; sys.path[:0] = ["ambistory", "ambistory/kaggle"] # a clone of the GitHub repo |
| from e13_train import load_deployed, build_inputs |
| |
| path = snapshot_download("Seronic/ambistory-e13") |
| model, tok, meta = load_deployed(f"{path}/model_seed42") |
| ``` |
|
|
| ## Results (official 930-item test set) |
|
|
| | | Spearman ρ | Accuracy within SD | |
| |---|---|---| |
| | 2 seeds averaged | 0.681 | 78.0% | |
| | 2 seeds + within-setup soft-rank blend (as in the app) | **0.702** | **80.1%** | |
|
|
| Accuracy counts a prediction as correct when `|pred − mean| < max(annotator SD, 1)`, as in the official scorer. |
| These scores were computed after the official evaluation period. The test set was never used for training or model |
| selection. The cross-validated version of this model (5 folds × 3 seeds) reaches 0.719, and blending it with a |
| zero-shot Qwen2.5-32B judge reaches 0.811 (see the GitHub README). |
|
|
| ## Limitations |
|
|
| Trained only on AmbiStory (English, short constructed stories, 275 homonyms). On CPU it takes about 1.2 s per |
| story–meaning pair per model, and PyTorch dynamic int8 quantization of the feed-forward layers breaks it (correlation with the |
| full-precision scores drops to ~0.1), so the weights are served unquantized. |
|
|
