File size: 2,623 Bytes
6b398e5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
language: en
base_model: microsoft/deberta-v3-large
pipeline_tag: text-classification
tags:
- semeval-2026
- word-sense-disambiguation
- plausibility
- cross-encoder
- deberta-v3
---
# AmbiStory E13: DeBERTa-v3-large plausibility scorer (SemEval 2026 Task 5)
Rates how plausible (1–5) a given sense of a homonym is in a short story, the task of
[SemEval 2026 Task 5 (AmbiStory)](https://github.com/Janosch-Gehring/semeval26-05-scripts).
Code, training scripts, evaluation and the Streamlit app: **[github.com/Seronic2001/ambistory](https://github.com/Seronic2001/ambistory)**.
## Model
- `microsoft/deberta-v3-large` cross-encoder. Segment A is the story with the homonym quoted; segment B is
`homonym: gloss. Example: sentence`.
- Mean pooling, then an ordinal (proportional-odds) head over the 5 ratings. The score is the expected rating.
- Trained on all of train+dev for 6 epochs with KL to the annotator distribution, a within-setup ranking loss and MSE,
story-grouped batches and layer-wise LR decay. The checkpoint was chosen on an inner homonym-grouped split
(`kaggle/e13_train.py --deploy`).
- Two seeds (`model_seed42`, `model_seed43`), stored as fp16 safetensors (828 MB each). The app averages them.
These are custom PyTorch modules, not a `transformers` pipeline. Load them with `load_deployed` from the repo:
```python
from huggingface_hub import snapshot_download
import sys; sys.path[:0] = ["ambistory", "ambistory/kaggle"] # a clone of the GitHub repo
from e13_train import load_deployed, build_inputs
path = snapshot_download("Seronic/ambistory-e13")
model, tok, meta = load_deployed(f"{path}/model_seed42")
```
## Results (official 930-item test set)
| | Spearman ρ | Accuracy within SD |
|---|---|---|
| 2 seeds averaged | 0.681 | 78.0% |
| 2 seeds + within-setup soft-rank blend (as in the app) | **0.702** | **80.1%** |
Accuracy counts a prediction as correct when `|pred − mean| < max(annotator SD, 1)`, as in the official scorer.
These scores were computed after the official evaluation period. The test set was never used for training or model
selection. The cross-validated version of this model (5 folds × 3 seeds) reaches 0.719, and blending it with a
zero-shot Qwen2.5-32B judge reaches 0.811 (see the GitHub README).
## Limitations
Trained only on AmbiStory (English, short constructed stories, 275 homonyms). On CPU it takes about 1.2 s per
story–meaning pair per model, and PyTorch dynamic int8 quantization of the feed-forward layers breaks it (correlation with the
full-precision scores drops to ~0.1), so the weights are served unquantized.
|