Good idea, so we just ran it: the answers don't depend on option position. Same clips, options shuffled, letters mapped back to labels (DuplexJev-32B-Para v1.3, one order per request, no averaging):
- 29 clips; 4-way emotion and 4-way turn state with 15 distinct option orders each, gender with both orders.
- 98.5% of answers are identical across all orders (emotion 686/696, turn 685/696, gender 58/58).
- When the model is confident (โฅ 0.8), zero flips, and the winning label's probability barely moves (median change 0.01 for emotion, 0.04 for turn).
- The few flips all come from 2 genuinely ambiguous clips per question, where the model was only 0.55โ0.67 sure to begin with.
For benchmarks we still average over a few orders, which settles even those borderline clips (29/29 match) at almost no cost, since the orders share the audio prefix. We'll run the same check on the full benchmark set and add it to the repo. Thanks for the suggestion!
