enes123erman/override-checkpoints

OVERRIDE: detecting revisions in multi-turn dialogue

Users change their minds mid-conversation: "Actually, make it 7 pm at the same place." OVERRIDE labels every word of the new user turn with its revision role:

Label	Meaning	Example
UPDATE_CUE	words that signal a change	Actually, make it
NEW_VALUE	the value the user now wants	7 pm
REFERENCE	a pointer back to an earlier turn	the same place

Eight labels are defined in two BIO layers (VALUE: NEW_VALUE, REFERENCE, OLD_VALUE, KEPT_VALUE; CUE: UPDATE_CUE, CANCEL_CUE, KEEP_CUE, RESTORE_CUE), 18 tags. Results are reported on the three labels frequent enough to measure.

• Code, protocols, all results: https://github.com/eneserman/OVERRIDE
• Live demo (type any sentence): https://override-demo.vercel.app
• Data: https://huggingface.co/datasets/enes123erman/override-dataset

Model

microsoft/deberta-v3-base (184M) with two linear token heads, one for the VALUE layer and one for the CUE layer, fine-tuned on 857 human-labeled user turns. Three seeds (20261029, 20261030, 20261031) are averaged at the softmax level. Selected by grouped 5-fold cross-validation with rules committed before running; the test was opened once.

Files

Path	What
checkpoints/no/final_seed*.pt	selected model: reads only the new user turn
checkpoints/tail60/final_seed*.pt	pre-registered comparison: also reads the last 60 words of earlier turns

Each file is a PyTorch dict with state_dict (weights stored in half precision; load them into float32) and meta (base model, tag lists, history mode, max length). The loading and prediction code is app/server.py and scripts/d5v2_final.py in the GitHub repository:

git clone https://github.com/eneserman/OVERRIDE && cd OVERRIDE
pip install -r requirements.txt
huggingface-cli download enes123erman/override-checkpoints --local-dir .
python app/server.py --checkpoints-no "checkpoints/no/*.pt" --checkpoints-history "checkpoints/tail60/*.pt"

Results (locked test, 363 turns, strict exact-span micro-F1)

System	VALUE F1	CUE F1	S (mean)
Laya zero-shot decision model (JEV alternative)	0.031	0.017	0.024
Surface rules	0.462	0.364	0.413
Lexicon copy	0.526	0.414	0.470
BERT-base (first data version)	0.780	0.644	0.712
CRF	0.758	0.712	0.735
DeBERTa-v3 ensemble, no history (this model, selected)	0.845	0.691	0.768
DeBERTa-v3 ensemble, last 60 words of history (comparison)	0.870	0.757	0.813

• The gain over CRF is +0.033 S with a 95% bootstrap interval of [−0.017, +0.079]: the highest score, not a significant win. CUE F1 rests on 31 test spans.
• History did not help on cross-validation (−0.005) but helps on the locked test (+0.045, [+0.016, +0.078]). The selected model stays the one fixed before the test.
• A tagger that labels nothing gets 87.9% token accuracy and 0 span F1, which is why the metric is strict span F1.

Limitations

Trained on well-spelled English task-oriented dialogue (Google SGD); indirect phrasing without an explicit cue word and misspelled cue words are often missed. The five rare labels have too few examples to be measured.

Team

Team Developers, NLP Project 2: Adnan Burak Özbay, Enes Erman, Mohammed Firas, İlayda Şenel.