Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
uuhugg 
posted an update 2 days ago
Post
4,732
A retry can succeed and still be unjustified.

RetryBench separates whether an agent’s retry is supported by the visible contract from what happens afterward. After a timeout, success alone does not tell us whether retrying was justified.

Try the interactive demo (English / 中文):
uuhugg/retrybench-demo

Read the findings and reproduction guide:
https://huggingface.co/blog/uuhugg/retrybench-retry-permission

How do you evaluate retries in your agents?

Ran your deterministic core at 5257b75, and src/ is unchanged in today's 3271cbc. All 5 policy rows match the README to the episode.

The table hides something your own framing predicts.
"Always stop" beats "tri-state backoff" on exact state: 36/48 vs 28/48.

That is the schedule mix, not judgment.
3 of the 4 schedules commit the original call, so doing nothing is right 75% of the time by construction.

And scoring.py line 68 folds two things into one bit: exactly one effect, and not unresolved.
12 of tri-state's 20 unresolved episodes already have exactly one effect. The world is correct. The policy just said it could not be sure.
Score the world alone and tri-state is 40/48, above always-stop.

Always-stop's 12 misses are the worst kind for an operator: final action "stop" on fail_before_execution with zero effects. A silent omission.
Tri-state's 8 zero-effect cases at least say "unresolved".

So I count three outcomes, not two: correct, wrong but flagged, wrong and silent.

Would you report "unresolved, state already exact" as its own cell, or is penalizing that honesty the point of the column?

·

The distinction between 'wrong and silent' and 'unresolved' would also be useful around key expiry. Consider the same timeout before and after the original key's retention window: a later successful write should not retroactively justify the expired-key retry. I'd report the decision-time contract beside those outcome cells, so the table does not reward a lucky recovery.

The table already has your expired-key row. It's called Always retry.

In the simulator the dedupe store is a set with no clock (service.py, _commit, unchanged through today's 0c9a53c). Keys never expire, so Stable idempotency is scored with an infinite window.
In policies.py the two rows differ by one boolean. Expire the key and a same-key replay is a plain retry: 12/48 exact, 36 duplicates.

Those 12 are your lucky recovery.
They are the fail_before_execution schedule, where the original never committed. The replay leaned on a key that was gone, and the world still came out exact.

So I'd score two bits, not one: was the retry admissible when decided, and was the state exact when settled.
The 12 sit in the gap between them, and that is the cell worth naming.

Would retention be a per-schedule parameter, or its own contract class next to Stable idempotency?