---
language: en
base_model: microsoft/deberta-v3-large
pipeline_tag: text-classification
tags:
  - semeval-2026
  - word-sense-disambiguation
  - plausibility
  - cross-encoder
  - deberta-v3
---

# AmbiStory E13: DeBERTa-v3-large plausibility scorer (SemEval 2026 Task 5)

Rates how plausible (1–5) a given sense of a homonym is in a short story, the task of
[SemEval 2026 Task 5 (AmbiStory)](https://github.com/Janosch-Gehring/semeval26-05-scripts).
Code, training scripts, evaluation and the Streamlit app: **[github.com/Seronic2001/ambistory](https://github.com/Seronic2001/ambistory)**.

## Model

- `microsoft/deberta-v3-large` cross-encoder. Segment A is the story with the homonym quoted; segment B is
  `homonym: gloss. Example: sentence`.
- Mean pooling, then an ordinal (proportional-odds) head over the 5 ratings. The score is the expected rating.
- Trained on all of train+dev for 6 epochs with KL to the annotator distribution, a within-setup ranking loss and MSE,
  story-grouped batches and layer-wise LR decay. The checkpoint was chosen on an inner homonym-grouped split
  (`kaggle/e13_train.py --deploy`).
- Two seeds (`model_seed42`, `model_seed43`), stored as fp16 safetensors (828 MB each). The app averages them.

These are custom PyTorch modules, not a `transformers` pipeline. Load them with `load_deployed` from the repo:

```python
from huggingface_hub import snapshot_download
import sys; sys.path[:0] = ["ambistory", "ambistory/kaggle"]        # a clone of the GitHub repo
from e13_train import load_deployed, build_inputs

path = snapshot_download("Seronic/ambistory-e13")
model, tok, meta = load_deployed(f"{path}/model_seed42")
```

## Results (official 930-item test set)

| | Spearman ρ | Accuracy within SD |
|---|---|---|
| 2 seeds averaged | 0.681 | 78.0% |
| 2 seeds + within-setup soft-rank blend (as in the app) | **0.702** | **80.1%** |

Accuracy counts a prediction as correct when `|pred − mean| < max(annotator SD, 1)`, as in the official scorer.
These scores were computed after the official evaluation period. The test set was never used for training or model
selection. The cross-validated version of this model (5 folds × 3 seeds) reaches 0.719, and blending it with a
zero-shot Qwen2.5-32B judge reaches 0.811 (see the GitHub README).

## Limitations

Trained only on AmbiStory (English, short constructed stories, 275 homonyms). On CPU it takes about 1.2 s per
story–meaning pair per model, and PyTorch dynamic int8 quantization of the feed-forward layers breaks it (correlation with the
full-precision scores drops to ~0.1), so the weights are served unquantized.