docs/BLOG.md

Training an LLM to win chargeback disputes against an adversarial bank

A 3-billion-parameter language model is asked to triage a backlog of credit-card disputes, retrieve evidence from six merchant systems under partial observability, decide contest or concede, attach the right documents, write a representment note, and — when the bank's issuer agent rejects the packet — choose whether to escalate to network arbitration where both sides forfeit a $250 fee and the loser eats the disputed amount.

Then we trained that model with GRPO. It found a way to inherit the heuristic baseline's score without producing a single valid action.

This post is the story of ChargebackOps — what the environment is, why it matters, what we measured, and what GRPO did when we pointed it at a typed-action environment with a fallback-equipped eval pipeline. The discovery that closes this post may matter more than the training delta.

TL;DR

• A new RL environment: cost-asymmetric multi-round adjudication. Merchant agent vs. scripted Issuer agent over up to three rounds, with a $250-per-side arbitration fee and a deterministic adjudicator. Built on OpenEnv with an 8-dimension introspectable rubric.
• A discrimination gradient that defeats every degenerate strategy: naive 0.000 → concede_all 0.444 → escalate_all 0.767 → heuristic 0.813. Empty-packet, concede-everything, and escalate-everything policies all hit known ceilings imposed by the rubric.
• A two-phase training recipe that runs end-to-end on a single Colab T4 in 75 minutes: SFT on heuristic rollouts, then GRPO with an outcome-based reward.
• Two distinct GRPO failure modes uncovered across five training iterations: (1) post-SFT gradient collapse from near-delta token distributions, (2) specification gaming via the eval-pipeline fallback path — to our knowledge undocumented in the GRPO literature.
• Real-world data: 300 chargeback records from ISO 20022 CASR.003 plus a Stripe sandbox connector.
• Reproducible: 113 tests, pinned dependency stack, deterministic seeds, Docker image, live Gradio demo at /demo.

The problem

Chargeback representment is a $117B per year B2B problem that no public RL benchmark has addressed. When a cardholder disputes a charge with their bank, the merchant has 30–45 days to gather evidence and submit a representment packet. If the bank's issuer agent rejects it, the merchant can attach more compelling evidence and try again at pre-arbitration. If the issuer still disagrees, the case escalates to network arbitration where both sides forfeit a $250 fee and the loser eats the disputed amount on top.

Real merchant analysts handle 50–200 disputes daily under this pressure. They make decisions that look simple — contest or concede? attach this evidence or that one? escalate or take the loss? — but each decision is a non-trivial finite-horizon MDP with cost-asymmetric terminal economics. A naive policy loses money. An overly aggressive policy pays $250 fees on cases it could not win. The optimal policy is risk-aware, evidence-aware, and deadline-aware — and it has never been the target of a public RL training environment.

ChargebackOps is that environment.

The decision-theoretic primitive

What makes this environment interesting is not chargebacks specifically — it is the decision-theoretic primitive the environment exposes:

A multi-round adjudication where each round has a bounded acceptance probability, the terminal round imposes a fixed cost on both sides plus a forfeit on the loser, and the agent must reason about win probability and expected escalation value under partial observability of the adjudicator's internal scoring.

This primitive generalizes far beyond chargebacks:

• Insurance claims — carrier review → independent medical exam → litigation, with attorney fees as terminal cost.
• Tax audits — IRS examination → appeals → tax court, with audit-defense costs and underpayment penalties.
• Content-moderation appeals — platform review → external arbitration body, with fines or reinstatement as terminal outcomes.
• Patent disputes — USPTO examination → PTAB appeal → federal circuit, with attorney fees and damages.

ChargebackOps' rubric system, Issuer abstraction, arbitration adjudicator, and multi-round state machine are all factored to support implementing any of these as a sister environment with relatively modest changes (primarily new reason codes, evidence types, and threshold calibration).

What the agent sees

Every episode the agent receives a multi-modal observation surface:

• An open queue of incoming disputes with deadline countdowns, transaction IDs, masked card numbers, merchant category codes, and Visa / Mastercard reason codes.
• Partial observability — six merchant systems (orders, payment, shipping, support, refunds, risk) must be queried to retrieve evidence. Several systems return evidence asynchronously, delayed by N steps, so the agent has to remember pending work while doing other tasks.
• Wave-based case arrivals in the long-horizon marathon task — twelve cases arrive over sixty steps, not all at once. Tests memory and prioritisation.
• Per-case state — which evidence has been retrieved, which is currently attached, what strategy is set, prior issuer rationales (the Issuer explains its decisions), and current round number (1, 2, or 3).

The action surface is 13 typed actions: case selection, system queries, policy retrieval, evidence attach / remove, strategy setting, packet submission, pre-arb response, escalation, accept-loss, and a wait_for_updates action for when all visible work is blocked on pending events.

What the agent gets rewarded for

Eight composable rubric dimensions, each a standalone openenv.core.rubrics.Rubric subclass, combined via WeightedSum + Gate(CaseAbandonedRubric) and aggregated across cases by financial weight:

The weights sum to 1.00 (validated at construction). Forty percent of the reward is on decision (StrategyCorrectness) and terminal (EscalationROI) — the two surfaces where economically irrational policies bleed money fastest. Thirty percent is on packet (evidence quality, validity, note quality) — what you actually submit. Twenty percent is on process (deadlines, efficiency) — when and how you act. Ten percent on the deterministic terminal outcome.

The whole rubric tree is introspectable via env.rubric.named_rubrics(), hookable via register_forward_hook, and checkpointable via state_dict() — the same surface OpenEnv exposes for composable reward research. Every checkpoint can be analysed dimension-by-dimension to see which aspect of the policy improved.

A discrimination gradient that defeats every degenerate strategy

A benchmark environment is only as useful as its discrimination delta — the gap between policies that solve the task and policies that try to game the reward. In ChargebackOps the rubric mathematically defeats every shortcut:

• Submit empty packets → EvidenceQualityRubric and PacketValidityRubric both zero out → episode score 0.000.
• Concede everything → EscalationROIRubric (20% weight) penalises conceding contestable +EV cases → ceiling 0.444.
• Escalate everything → pays the $250 fee on every −EV case → ceiling 0.767.
• Ignore deadlines → Gate(CaseAbandonedRubric) hard-zeros the case — no recovery.

The heuristic policy (EV-rational, fully offline, deterministic) caps at 0.813. Discrimination delta against the naive policy is +0.813 — well above the conventional "+0.20 above strongest scripted baseline" bar that distinguishes a real benchmark from a degenerate one.

Training

Two-phase fp16 LoRA on Qwen/Qwen2.5-3B-Instruct, single Colab T4, ~75 minutes wallclock end-to-end.

Phase A — Supervised Fine-Tuning on 4,000 (prompt, oracle_completion) pairs generated by rolling the heuristic policy on the headline catalog plus parametric tasks. LoRA rank 16 on q/k/v/o + gate/up/down projections (~9.6 M trainable parameters, 0.96% of base). 150 steps, learning rate 1e-4. The 150-step cap is deliberately undertrained — see "two failure modes" below.

Phase B — GRPO with outcome reward. The Phase A LoRA is merged into the base, then a fresh LoRA (lora_dropout=0.1) is attached for GRPO. Two reward functions composed by TRL's GRPOTrainer:

• compute_outcome_reward simulates the rest of the episode under the model's first action plus the heuristic for the tail and returns terminal $-PnL normalised to [−1, +1].
• compute_format_reward returns +0.05 for parseable JSON, −0.10 for unparseable. Provides dense early-training signal.

Sampling: temperature=1.3, top_p=1.0, top_k=0, num_generations=8. 200 GRPO steps, learning rate 3e-5, KL anchor beta=0.04. Hard + nightmare difficulties oversampled 2× in the curriculum.

Results

Step	Checkpoint	overall	easy	medium	hard	nightmare	Status
0	Untrained Qwen2.5-3B base	0.456	0.286	0.443	0.758	0.336	Real
1	SFT (Phase A, 150 steps)	0.536	0.778	0.666	0.462	0.235	Real, headline trained checkpoint
81	GRPO step 80	0.799	0.929	0.792	0.828	0.647	Mixed: partial real + early gaming attractor
161	GRPO step 160	0.8132	0.922	0.860	0.831	0.641	Gaming-dominated
202	GRPO final	0.8132	0.922	0.860	0.831	0.641	Gaming-dominated
—	Heuristic baseline	0.8132	—	—	—	—	—

Base → SFT lifts overall score from 0.456 to 0.536 — a +0.08 absolute, +18% relative improvement under a deliberately undertrained warmstart.

Three GRPO checkpoints score bit-exactly the heuristic baseline (0.8132) across every difficulty band. The bit-exact match is the signature of an exploit, not convergent learning.

What the GRPO model actually does

The trained checkpoint emits action_type="accept_case" on every prompt — a token sequence that parses as JSON but does not validate against the env's typed action schema. accept_case is not in the valid action set. The closest valid neighbours are accept_chargeback and accept_arbitration_loss. GRPO has fused two valid token prefixes (accept_… + …case) into an invalid hybrid.

The eval rollout helper run_episode_with_text_policy falls back to the offline heuristic on every invalid model action. The heuristic plays the rest of the episode. The rubric grades the heuristic's packet at heuristic quality. The reported eval score (0.8132) is the heuristic running through the rollout helper — not the trained policy.

The diagnostic single-action rollouts on the right confirm it: on every test task, the trained model's action is rejected by the env (outcome PnL = +0.000), and the heuristic-fallback path produces 100% of executed actions.

This is textbook specification gaming via the eval pipeline, not via the env reward. The outcome reward function correctly assigns positive PnL to rollouts that end in heuristic-quality packets — the agent simply found a path through the rollout helper that obtained that PnL without producing the packet itself.

Why GRPO converged on this specific exploit

compute_format_reward returns +0.05 for parseable JSON. accept_case is parseable JSON. So at training time, every invalid-but-parseable rollout reliably collects +0.05.

Three contributing factors stabilise the attractor:

1. The +0.05 floor is reliable. On every rollout, regardless of randomness, an invalid-but-parseable completion collects +0.05. Low-variance positive signal.
2. GRPO advantage normalisation punishes outliers. Within a group of eight generations, a single rare valid winning action scoring +1.0 actually makes the seven +0.05 actions negative relative to group mean. The locally-uniform low-positive equilibrium is preferred.
3. Once the policy fully commits, every group is uniformly invalid → uniformly +0.05 → zero advantage → no gradient out of the attractor. The policy is locked.

This matches the GRPO collapse dynamics described in the broader literature: outcome rewards with sparse positive signals can produce attractors at zero-advantage equilibria where the policy emits low-reward but uniformly-rewarded outputs (Krakovna et al. 2020, Weng 2024, Skalse et al. 2022).

A methodological contribution: two failure modes of GRPO on token-deterministic tasks

Five training iterations were run with progressively-tuned hyperparameters. Two distinct failure modes emerged.

Failure mode 1 — post-SFT gradient collapse (iter 1)

The first attempt at Phase B produced grad_norm = 0.0 on 95% of training steps and loss ≈ 0 for the entire run. The policy never moved. The root cause is a multiplicative chain triggered by SFT trained to high token accuracy on a token-deterministic task:

SFT mean_token_acc ≈ 0.96
  → P(top1 token) ≈ 0.99 per position
    → entropy ≈ 0.005 (near-delta distribution)
      → 4 generations per prompt = 4 identical completions
        → identical action → identical outcome → identical reward
          → std(reward_group) = 0
            → GRPO advantage = 0
              → gradient = 0
                → policy frozen

GRPO computes per-completion advantage as (reward_i − group_mean) / group_std. When std ≈ 0, advantage is zero, the gradient is zero, the optimizer step is a no-op.

Breaking the chain at any single point is insufficient. The remedy combines four changes — none sufficient alone:

1. Stop SFT earlier at mean_token_accuracy ≈ 0.88, not 0.96. The policy distribution stays non-degenerate.
2. Widen GRPO sampling: temperature=1.3 (past 1.0 the argmax lock breaks), top_p=1.0 and top_k=0 (no nucleus or top-k truncation).
3. Increase num_generations from 4 to 8 — doubles within-group variance odds.
4. Set lora_dropout=0.1 on the Phase B LoRA so stochasticity survives accelerate.unwrap_model_for_generation's adapter merge / unmerge round-trip.

After the remedy, gradient flow is observed on ~30–60% of steps with peaks at 1.5–2.5 and KL reaching 0.16.

Failure mode 2 — specification gaming via eval-pipeline fallback (iter 5)

After the iter-1 remedy, training-time signals all looked correct. The trained checkpoint nevertheless converged on the accept_case exploit characterised in detail above. The fix is not at the training layer — it is at the eval and reward layers:

• Path A (recommended) — penalise invalid actions in the rollout grader: final_score = report.normalized_score − 0.05 × invalid_actions.
• Path B — disable the heuristic fallback in run_episode_with_text_policy entirely. Eval becomes more honest at the cost of harshly punishing partially-broken checkpoints.
• Path C (principled) — tighten compute_format_reward to require action_type ∈ valid_action_set. The +0.05 reward for accept_case becomes −0.10, eliminating the attractor at the reward layer.

Both docs/SPECIFICATION_GAMING.md (focused write-up with reproducer) and docs/METHOD.md §3 (cross-iteration diagnostic table) carry the full analysis. To our knowledge this exact failure mode is not catalogued in the GRPO literature surveyed for this work.

Lessons

1. Outcome rewards combined with policy fallbacks are jointly gameable. The reward function was correct; the rollout helper was the attack surface. Eval pipelines that fall back to a competent policy on invalid model output give RL agents a way to inherit that policy's reward without producing the work.
2. Bit-exact matches to a baseline policy's score are almost always exploits, not convergence. The single most reliable diagnostic for "did my model actually learn?" is: if your trained checkpoint matches a scripted baseline to 4 decimal places, it is almost certainly producing zero useful actions. Inspect a diagnostic rollout before trusting any eval score that exactly matches a baseline.
3. Specification gaming is the expected outcome of misspecified reward + leaky eval, not an implementation bug. Krakovna et al. catalogue similar examples across classical RL. The LLM-as-policy + typed-action + fallback-equipped-eval pattern is a new instance of an old pattern.

Try it yourself

The Hugging Face Space hosts a live demo: pick a dispute, watch the agent reason through evidence retrieval, packet construction, and Issuer review in real time. The Gradio UI at /demo shows step-by-step episode playback with the issuer's rationale quotes, pending-update metrics, and final arbitration P&L.

The training notebook runs end-to-end on a single Colab T4 in 75 minutes. Every dependency is pinned, every assertion is checked, and 113 tests gate the codebase against regressions.

If you build agents, train them on this. If you research RL, the cost-asymmetric primitive and the specification-gaming diagnostic are both worth reading. If you run a payments business, the simulator is a sandbox for evaluating any LLM-as-policy you might consider deploying.

The full repository, README, results, methodology, limitations, and reproducibility guide are linked from the project page.