sriram1983007 commited on
Commit
2b59836
·
verified ·
1 Parent(s): 8f4151e

Card: v2 merged weights and GGUF builds (2026-11-13)

Browse files
Files changed (1) hide show
  1. README.md +43 -41
README.md CHANGED
@@ -49,19 +49,19 @@ model-index:
49
  metrics:
50
  - type: sra_score
51
  name: SRA composite score
52
- value: 0.9134
53
  - type: accuracy
54
  name: Risk-gate decision accuracy
55
- value: 0.9949
56
  - type: unsafe_approval_rate
57
  name: Unsafe approval rate (lower is better)
58
- value: 0.0047
59
  - type: over_blocking_rate
60
  name: Over-blocking rate (lower is better)
61
  value: 0.0
62
  - type: injection_approved_rate
63
  name: Injected payments approved (lower is better)
64
- value: 0.058
65
  source:
66
  name: Published predictions
67
  url: https://huggingface.co/datasets/sriram1983007/sra-bench-results
@@ -76,13 +76,13 @@ model-index:
76
  metrics:
77
  - type: dispute_score
78
  name: Dispute score
79
- value: 0.8322
80
  - type: accuracy
81
  name: Dispute outcome accuracy
82
- value: 0.7782
83
  - type: impossible_remedy_rate
84
  name: Impossible remedy rate (lower is better)
85
- value: 0.0038
86
  - type: wrongful_refund_rate
87
  name: Wrongful refund rate (lower is better)
88
  value: 0.022
@@ -127,22 +127,20 @@ model-index:
127
 
128
  It reads three things: your policy, the payment request exactly as received (treated as untrusted), and the results of your own verification tools (signature checks, attestations, sanctions screening). It is a decision-support layer that sits **behind** deterministic checks, not a standalone security control. It also proposes settlement-aware remedies for payment disputes.
129
 
130
- ### At a glance (held-out synthetic test set, measured by the author)
131
 
132
- | | Base Qwen3-4B | **SRA-RiskGate-4B** |
133
- | --- | --- | --- |
134
- | Unsafe approvals (847 risky payments) ↓ | 34.6% | **0.47%** (4 cases) |
135
- | Correct payment decisions (1,175 payments) ↑ | 59.8% | **99.5%** |
136
- | Legitimate payments blocked (328) ↓ | 5.2% | **0** |
 
137
 
138
- All 4 unsafe approvals were prompt injections; the companion SDK's deterministic claims check holds all 4 (details and caveats below). The data is synthetic and no third party has reproduced these results yet.
139
 
140
- **Tested on real Base payments:** on [SRA-Real v1](https://huggingface.co/datasets/sriram1983007/sra-real), 227 cases built from real USDC payments on Base mainnet, the model approved all 151 real payments and caught all 76 with an injected problem (100%; base model 78.4%). Small and author-measured; [details and predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
141
 
142
- **v2 is out (adapter, 2026-11-12):** much better at disputes. On the same 2,000 test cases: SRA score 0.913 →
143
- **0.960**, dispute accuracy 77.8% → **90.2%**, wrongful denials 26.0% → **2.1%**, unsafe payment approvals 0.47% →
144
- **0.35%**. Available now as the [LoRA adapter](https://huggingface.co/sriram1983007/SRA-RiskGate-4B-LoRA); this page's merged weights, the GGUF builds and the
145
- Ollama model are still v1 until merged v2 builds are published ([v2 results](https://huggingface.co/datasets/sriram1983007/sra-bench-results)).
146
 
147
  📄 **Paper:** [SRA-RiskGate: A Benchmark and a 4B Model for Gating Stablecoin Payments Made by AI Agents](https://doi.org/10.5281/zenodo.23715017) (preprint, Zenodo) · [full write-up on Ready Tensor](https://app.readytensor.ai/publications/sra-riskgate-a-benchmark-and-a-4b-model-for-gating-stablecoin-payments-made-by-ai-agents-wihp3WytOoKOc1g7NXFsl) · [discuss on alphaXiv](https://www.alphaxiv.org/abs/2610.sra-riskgate-stablecoin-payment-safety)
148
 
@@ -317,6 +315,8 @@ Depeg checks run only when a `peg_feed` is supplied.
317
 
318
  ## 🚀 Transformers Quickstart
319
 
 
 
320
  The model was trained on a specific prompt format, and it only performs as benchmarked when you use that format exactly:
321
 
322
  - **System prompt:** the payment risk-gate instructions shown below.
@@ -524,7 +524,7 @@ For **dispute adjudication**, pass the dispute system prompt in your request (it
524
  | [mradermacher/SRA-RiskGate-4B-GGUF](https://huggingface.co/mradermacher/SRA-RiskGate-4B-GGUF) | Community static quants by mradermacher |
525
  | [tinyopsec/SRA-RiskGate-4B-GGUF](https://huggingface.co/tinyopsec/SRA-RiskGate-4B-GGUF) | Community quants by tinyopsec, Q2_K to F16 (builds below Q4 not evaluated) |
526
 
527
- The benchmark figures below are for the full-precision model. The two official quantized builds were benchmarked on the same test set (measured 2026-11-11): **Q8_0 matches full precision** on every safety metric; **Q4_K_M keeps the payment gate** (99.3% accuracy, no legitimate payment blocked, unsafe approvals 0.83% vs 0.47%) **but is weaker on disputes** (67.4% vs 77.8%), mostly from false "outside the dispute window" claims that `check_dispute_verdict` corrects (wrongful denials 0.85% with the check, on every build). Use Q4_K_M for payment checks and Q8_0 or full precision for disputes; all predictions are in [sra-bench-results](https://huggingface.co/datasets/sriram1983007/sra-bench-results).
528
 
529
  ---
530
 
@@ -572,33 +572,35 @@ Or use the MCP server's `check_payment_with_model` tool with `SRA_BASE_URL=http:
572
 
573
  All 2,000 cases in the test split of [`sra-stablecoin-risk-bench`](https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench): **1,175 payment cases** (847 that should be held or rejected, 328 legitimate, including 69 with hidden prompt-injection text) and **825 dispute cases**. Payment rows below are computed on the 1,175 payment cases and dispute rows on the 825 dispute cases. Scored with the dataset's own `score.py`. Both models received the **identical training prompts** with greedy decoding. Every prediction is published in [**sra-bench-results**](https://huggingface.co/datasets/sriram1983007/sra-bench-results), so anyone can re-score them.
574
 
575
- | Metric | Base `Qwen3-4B-Instruct-2507` | **SRA-RiskGate-4B** |
576
- | --- | --- | --- |
577
- | **SRA composite score** ↑ | 0.421 | **0.913** |
578
- | **Payments:** decision accuracy ↑ | 59.8% | **99.5%** |
579
- | **Payments:** unsafe approvals ↓ | 34.6% | **0.47%** |
580
- | **Payments:** over-blocking (legitimate payments rejected) ↓ | 5.2% | **0.0%** (and 0 of 328 legitimate payments held) |
581
- | **Payments:** injected payments approved ↓ | 52.2% | **5.8%** |
582
- | **Payments:** reason-code (flag) F1 ↑ | 0.105 | **0.994** |
583
- | **Disputes:** score ↑ | 0.400 | **0.832** |
584
- | **Disputes:** outcome accuracy ↑ | 0.0% | **77.8%** |
585
- | **Disputes:** impossible remedies ↓ | n/a † | **0.38%** |
586
- | **Disputes:** wrongful refunds ↓ | n/a † | **2.2%** |
587
- | **Disputes:** wrongful denials (valid refunds denied or dismissed) ↓ | n/a † | **26.0%** (see Limitations) |
588
- | **Disputes:** refund details fully correct ↑ | n/a † | **64.3%** |
589
- | Schema-valid output (payments / disputes) ↑ | 99.8% / 0% | **100% / 100%** |
 
 
590
 
591
  † The base model returns dispute JSON with the right field names 98% of the time, but invents outcomes, mechanisms and flag formats outside the allowed vocabulary (e.g. `"outcome": "merchant_wins"`), so none of its dispute verdicts are usable and its dispute safety rates are not meaningful.
592
 
593
  **What fine-tuning changed:** unsafe approvals fell from about 1 in 3 risky payments to about 1 in 200, the model learned the exact reason codes and remedy vocabulary, and it blocks *fewer* legitimate payments than the base model.
594
 
595
- **Who measured this:** the author, on an NVIDIA T4 (float16) with vLLM. Two separate full runs produced identical scores, confirming deterministic output, and match the originally reported figures to within 0.3% on the composite score. No third party has reproduced them yet; every prediction is published so anyone can check.
596
 
597
  **What this does not show yet:** the test cases are synthetic, from the same generator as the training data, and the only comparison is the base model. Results against larger general-purpose models are still to be measured.
598
 
599
  **Real payments (SRA-Real v1, 2026-11-12):** on [227 cases](https://huggingface.co/datasets/sriram1983007/sra-real) built from real USDC payments settled on Base mainnet (real amounts, validity windows and verified signatures; 69 payees), the model made every decision correctly: all 151 unchanged payments approved, and all 76 with one injected problem (sanctions hit, expired or not-yet-valid, over the limit, bad signature, prompt injection) held or rejected. The base model scored 78.4% with 28.9% unsafe approvals. The injected problems are the same kinds as in the synthetic benchmark, so this shows the checks carry over to real payment data, not that the model resists new kinds of attack. One day of data, measured by the author; [predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
600
 
601
- **Leakage check (2026-10-24):** no test input appears verbatim in the training split. Because the data is generated from templates, 627 of the 1,175 payment test cases have a "template twin" in training (identical once addresses, hashes and numbers are masked); disputes have none. On the 548 payment cases with no twin, decision accuracy is **99.3%** (99.7% with a twin), so the headline is not driven by memorised templates. All 4 unsafe approvals are in the no-twin group: on cases unlike anything in training, the unsafe-approval rate is **0.83%** (4 of 481). Computed from the published predictions.
602
 
603
  To re-score any published run: download `predictions.jsonl` and `test.jsonl` from the results dataset, then run `python score.py --gold test.jsonl --pred predictions.jsonl`.
604
 
@@ -611,10 +613,10 @@ To re-score any published run: download `predictions.jsonl` and `test.jsonl` fro
611
  * **Exact Prompt Format Required:** The model is benchmarked only with its training system prompts and user templates (see the Quickstart). Other phrasings or plain JSON inputs can produce outputs in a different schema; treat any response that is not a valid verdict as a hold.
612
  * **Greedy Decoding Required:** Always run inference with `do_sample=False` or `temperature=0.0`. The benchmark figures, including schema validity, were measured with greedy decoding; sampling makes outputs non-deterministic and can break the JSON schema.
613
  * **Agent Boundary Isolation:** In autonomous agent frameworks (such as AgentKit), the firewall only protects transfers when `RiskGateActionProvider` is the sole execution provider for transfers.
614
- * **Dispute adjudication is weaker than payment gating:** Dispute outcome accuracy is 77.8% (vs. 99.5% for payment decisions), and refund details (amount, destination, idempotency key) are fully correct in 64.3% of refund cases. Have a human approve any refund before it executes.
615
- * **Disputes: the model misjudges the dispute window.** Most dispute errors come from one step: deciding whether the claim was filed inside the policy's window. When the model says a dispute is outside the window, it is wrong 81% of the time (91 of 112 on the test set), for example "filed 26.5 days after payment, outside the 60-day window"; it also misses 33 of the 54 disputes that really are outside it. These false claims cause 59 of the 61 wrongful denials. Whether a dispute is inside the window is simple date arithmetic, so compute it in your own code, or use `check_dispute_verdict` in sra-riskgate 0.10.0+ (also an MCP tool in sra-riskgate-mcp 0.15.0+), and do not rely on the model for it: on the test set, escalating every dismissal that rests on a false window claim lowers wrongful denials from 26.0% to 0.85%, with no change in wrongful refunds (2.2%). A fix in training is planned for v2.
616
- * **Prompt injection is the main remaining risk:** All 4 unsafe approvals in the test set were prompt-injection cases: 4 of 69 payments with hidden instructions (5.8%) were approved, and about 10% of injections went unflagged. Outside prompt injection, the model made no unsafe approvals. Never let model output trigger irreversible actions without a deterministic check, and treat free-text payload fields as untrusted.
617
- * **Mitigation (`check_payment` in sra-riskgate 0.6+, `untrusted_text` on every sra-riskgate-mcp 0.3+ tool, langchain-sra-riskgate 0.2.1+):** all 4 unsafe approvals came from payment text claiming its own verification ("risk level: low (verified offline)"). These packages run a deterministic check for such self-asserted claims (with Unicode normalisation against look-alike-character tricks) and **hold** the payment. On this benchmark, model + check together give **0 unsafe approvals out of 847** risky payments, and the check adds **0 false holds** on 3,358 legitimate payments (test + train). The check was written after those failures were identified, so it fixes a known weakness and is not an independent evaluation; differently worded attacks may still get through.
618
 
619
  ---
620
 
 
49
  metrics:
50
  - type: sra_score
51
  name: SRA composite score
52
+ value: 0.9604
53
  - type: accuracy
54
  name: Risk-gate decision accuracy
55
+ value: 0.9957
56
  - type: unsafe_approval_rate
57
  name: Unsafe approval rate (lower is better)
58
+ value: 0.0035
59
  - type: over_blocking_rate
60
  name: Over-blocking rate (lower is better)
61
  value: 0.0
62
  - type: injection_approved_rate
63
  name: Injected payments approved (lower is better)
64
+ value: 0.029
65
  source:
66
  name: Published predictions
67
  url: https://huggingface.co/datasets/sriram1983007/sra-bench-results
 
76
  metrics:
77
  - type: dispute_score
78
  name: Dispute score
79
+ value: 0.9254
80
  - type: accuracy
81
  name: Dispute outcome accuracy
82
+ value: 0.9018
83
  - type: impossible_remedy_rate
84
  name: Impossible remedy rate (lower is better)
85
+ value: 0.0057
86
  - type: wrongful_refund_rate
87
  name: Wrongful refund rate (lower is better)
88
  value: 0.022
 
127
 
128
  It reads three things: your policy, the payment request exactly as received (treated as untrusted), and the results of your own verification tools (signature checks, attestations, sanctions screening). It is a decision-support layer that sits **behind** deterministic checks, not a standalone security control. It also proposes settlement-aware remedies for payment disputes.
129
 
130
+ ### At a glance (v2, held-out synthetic test set, measured by the author)
131
 
132
+ | | Base Qwen3-4B | v1 | **SRA-RiskGate-4B v2** |
133
+ | --- | --- | --- | --- |
134
+ | Unsafe approvals (847 risky payments) ↓ | 34.6% | 0.47% (4) | **0.35%** (3 cases) |
135
+ | Correct payment decisions (1,175 payments) ↑ | 59.8% | 99.5% | **99.6%** |
136
+ | Legitimate payments blocked (328) ↓ | 5.2% | 0 | **0** |
137
+ | Correct dispute outcomes (825 disputes) ↑ | 0% | 77.8% | **90.2%** |
138
 
139
+ Of v2's 3 unsafe approvals, 2 were prompt injections that the companion SDK's deterministic claims check holds; the third was missing the payer attestation the policy requires, a rule your own code can check (details and caveats below). The data is synthetic and no third party has reproduced these results yet.
140
 
141
+ **Tested on real Base payments:** on [SRA-Real v1](https://huggingface.co/datasets/sriram1983007/sra-real), 227 cases built from real USDC payments on Base mainnet, v1 and v2 both approved all 151 real payments and caught all 76 with an injected problem (100%; base model 78.4%). Small and author-measured; [details and predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
142
 
143
+ **This page is v2 (2026-11-13).** The merged weights here, the [GGUF builds](https://huggingface.co/sriram1983007/SRA-RiskGate-4B-GGUF) and the [Q4_K_M file](https://huggingface.co/sriram1983007/SRA-RiskGate-4B-Q4_K_M-GGUF) are v2: SRA score 0.913 → **0.960**, dispute accuracy 77.8% → **90.2%**, wrongful denials 26.0% → **2.1%**, unsafe payment approvals 0.47% → **0.35%** ([v2 results](https://huggingface.co/datasets/sriram1983007/sra-bench-results)). v1 stays available: `from_pretrained("sriram1983007/SRA-RiskGate-4B", revision="v1")`. The Ollama registry model is still v1 until it is updated. For disputes, pass the computed window in the prompt: `dispute_messages()` in `sra-riskgate` 1.3.0+ builds the exact prompt.
 
 
 
144
 
145
  📄 **Paper:** [SRA-RiskGate: A Benchmark and a 4B Model for Gating Stablecoin Payments Made by AI Agents](https://doi.org/10.5281/zenodo.23715017) (preprint, Zenodo) · [full write-up on Ready Tensor](https://app.readytensor.ai/publications/sra-riskgate-a-benchmark-and-a-4b-model-for-gating-stablecoin-payments-made-by-ai-agents-wihp3WytOoKOc1g7NXFsl) · [discuss on alphaXiv](https://www.alphaxiv.org/abs/2610.sra-riskgate-stablecoin-payment-safety)
146
 
 
315
 
316
  ## 🚀 Transformers Quickstart
317
 
318
+ **Shortcut:** `pip install "sra-riskgate>=1.3.0"` gives `payment_messages()` and `dispute_messages()`, which build the exact prompts below for you, and `parse_payment_verdict()` / `parse_dispute_verdict()`, which read the answer and fail closed.
319
+
320
  The model was trained on a specific prompt format, and it only performs as benchmarked when you use that format exactly:
321
 
322
  - **System prompt:** the payment risk-gate instructions shown below.
 
524
  | [mradermacher/SRA-RiskGate-4B-GGUF](https://huggingface.co/mradermacher/SRA-RiskGate-4B-GGUF) | Community static quants by mradermacher |
525
  | [tinyopsec/SRA-RiskGate-4B-GGUF](https://huggingface.co/tinyopsec/SRA-RiskGate-4B-GGUF) | Community quants by tinyopsec, Q2_K to F16 (builds below Q4 not evaluated) |
526
 
527
+ The benchmark figures below are for the full-precision model. The v2 GGUF files were checked on 12 benchmark cases (12 of 12 correct for both Q8_0 and Q4_K_M) but have not yet been run on the full benchmark. The full-benchmark quantized results here are for the **v1** builds (measured 2026-11-11; still available at revision `v1`): **Q8_0 matches full precision** on every safety metric; **Q4_K_M keeps the payment gate** (99.3% accuracy, no legitimate payment blocked, unsafe approvals 0.83% vs 0.47%) **but is weaker on disputes** (67.4% vs 77.8%), mostly from false "outside the dispute window" claims that `check_dispute_verdict` corrects (wrongful denials 0.85% with the check, on every build). Use Q4_K_M for payment checks and Q8_0 or full precision for disputes; all predictions are in [sra-bench-results](https://huggingface.co/datasets/sriram1983007/sra-bench-results).
528
 
529
  ---
530
 
 
572
 
573
  All 2,000 cases in the test split of [`sra-stablecoin-risk-bench`](https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench): **1,175 payment cases** (847 that should be held or rejected, 328 legitimate, including 69 with hidden prompt-injection text) and **825 dispute cases**. Payment rows below are computed on the 1,175 payment cases and dispute rows on the 825 dispute cases. Scored with the dataset's own `score.py`. Both models received the **identical training prompts** with greedy decoding. Every prediction is published in [**sra-bench-results**](https://huggingface.co/datasets/sriram1983007/sra-bench-results), so anyone can re-score them.
574
 
575
+ | Metric | Base `Qwen3-4B-Instruct-2507` | v1 | **SRA-RiskGate-4B v2** |
576
+ | --- | --- | --- | --- |
577
+ | **SRA composite score** ↑ | 0.421 | 0.913 | **0.960** |
578
+ | **Payments:** decision accuracy ↑ | 59.8% | 99.5% | **99.6%** |
579
+ | **Payments:** unsafe approvals ↓ | 34.6% | 0.47% | **0.35%** |
580
+ | **Payments:** over-blocking (legitimate payments rejected) ↓ | 5.2% | 0.0% | **0.0%** (and 0 of 328 legitimate payments held) |
581
+ | **Payments:** injected payments approved ↓ | 52.2% | 5.8% | **2.9%** |
582
+ | **Payments:** reason-code (flag) F1 ↑ | 0.105 | 0.994 | **0.994** |
583
+ | **Disputes:** score ↑ | 0.400 | 0.832 | **0.925** |
584
+ | **Disputes:** outcome accuracy ↑ | 0.0% | 77.8% | **90.2%** |
585
+ | **Disputes:** impossible remedies ↓ | n/a † | 0.38% | 0.57% |
586
+ | **Disputes:** wrongful refunds ↓ | n/a † | 2.2% | **2.2%** |
587
+ | **Disputes:** wrongful denials (valid refunds denied or dismissed) ↓ | n/a † | 26.0% | **2.1%** |
588
+ | **Disputes:** refund details fully correct ↑ | n/a † | 64.3% | **87.7%** |
589
+ | Schema-valid output (payments / disputes) ↑ | 99.8% / 0% | 100% / 100% | **100% / 100%** |
590
+
591
+ v2's dispute figures use the computed dispute window in the prompt (`dispute_messages()` / `with_dispute_window()` in `sra-riskgate`). With the original prompts, v2 scores 0.950 composite, 85.0% dispute accuracy, 3.4% wrongful denials and 3.6% wrongful refunds. Impossible remedies rose from 2 to 3 cases.
592
 
593
  † The base model returns dispute JSON with the right field names 98% of the time, but invents outcomes, mechanisms and flag formats outside the allowed vocabulary (e.g. `"outcome": "merchant_wins"`), so none of its dispute verdicts are usable and its dispute safety rates are not meaningful.
594
 
595
  **What fine-tuning changed:** unsafe approvals fell from about 1 in 3 risky payments to about 1 in 200, the model learned the exact reason codes and remedy vocabulary, and it blocks *fewer* legitimate payments than the base model.
596
 
597
+ **Who measured this:** the author, on an NVIDIA T4 (float16) with vLLM. For v1, two separate full runs produced identical scores, confirming deterministic output. v2 was measured once the same way (the v2 adapter on the base model in vLLM); the merged weights on this page were checked against the adapter before upload. No third party has reproduced them yet; every prediction is published so anyone can check.
598
 
599
  **What this does not show yet:** the test cases are synthetic, from the same generator as the training data, and the only comparison is the base model. Results against larger general-purpose models are still to be measured.
600
 
601
  **Real payments (SRA-Real v1, 2026-11-12):** on [227 cases](https://huggingface.co/datasets/sriram1983007/sra-real) built from real USDC payments settled on Base mainnet (real amounts, validity windows and verified signatures; 69 payees), the model made every decision correctly: all 151 unchanged payments approved, and all 76 with one injected problem (sanctions hit, expired or not-yet-valid, over the limit, bad signature, prompt injection) held or rejected. The base model scored 78.4% with 28.9% unsafe approvals. The injected problems are the same kinds as in the synthetic benchmark, so this shows the checks carry over to real payment data, not that the model resists new kinds of attack. One day of data, measured by the author; [predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
602
 
603
+ **Leakage check (2026-10-24, v1):** no test input appears verbatim in the training split. Because the data is generated from templates, 627 of the 1,175 payment test cases have a "template twin" in training (identical once addresses, hashes and numbers are masked); disputes have none. On the 548 payment cases with no twin, decision accuracy is **99.3%** (99.7% with a twin), so the headline is not driven by memorised templates. All 4 unsafe approvals are in the no-twin group: on cases unlike anything in training, the unsafe-approval rate is **0.83%** (4 of 481). Computed from the published predictions.
604
 
605
  To re-score any published run: download `predictions.jsonl` and `test.jsonl` from the results dataset, then run `python score.py --gold test.jsonl --pred predictions.jsonl`.
606
 
 
613
  * **Exact Prompt Format Required:** The model is benchmarked only with its training system prompts and user templates (see the Quickstart). Other phrasings or plain JSON inputs can produce outputs in a different schema; treat any response that is not a valid verdict as a hold.
614
  * **Greedy Decoding Required:** Always run inference with `do_sample=False` or `temperature=0.0`. The benchmark figures, including schema validity, were measured with greedy decoding; sampling makes outputs non-deterministic and can break the JSON schema.
615
  * **Agent Boundary Isolation:** In autonomous agent frameworks (such as AgentKit), the firewall only protects transfers when `RiskGateActionProvider` is the sole execution provider for transfers.
616
+ * **Dispute adjudication is weaker than payment gating:** Dispute outcome accuracy is 90.2% for v2 (vs. 99.6% for payment decisions), and refund details (amount, destination, idempotency key) are fully correct in 87.7% of refund cases. Have a human approve any refund before it executes.
617
+ * **Disputes: the dispute window.** v1 often misjudged whether a claim was filed inside the policy's window, which caused most of its wrongful denials (26.0%). v2 is trained with the window computed in code and passed in the prompt (`with_dispute_window()` / `dispute_messages()` in sra-riskgate 1.2.0+): wrongful denials fall to 2.1%. Without the window in the prompt, v2 still denies less (3.4%) but makes more wrongful refunds (3.6% vs 2.2%), so always pass it. Keep running `check_dispute_verdict` (sra-riskgate 0.10.0+, also an MCP tool in sra-riskgate-mcp 0.15.0+) on every verdict: with it, v2 makes no wrongful denials on the test set.
618
+ * **Prompt injection is the main remaining risk:** For v2, 2 of 69 payments with hidden instructions (2.9%) were approved and about 7% of injections went unflagged (v1: 4 of 69, 5.8%, about 10% unflagged). v2's only other unsafe approval was a payment missing the payer attestation the policy requires. Never let model output trigger irreversible actions without a deterministic check, and treat free-text payload fields as untrusted.
619
+ * **Mitigation (`check_payment` in sra-riskgate 0.6+, `untrusted_text` on every sra-riskgate-mcp 0.3+ tool, langchain-sra-riskgate 0.2.1+):** all 4 of v1's unsafe approvals, and both of v2's injection approvals, came from payment text claiming its own verification ("risk level: low (verified offline)"). These packages run a deterministic check for such self-asserted claims (with Unicode normalisation against look-alike-character tricks) and **hold** the payment. On this benchmark, v1 + check together give **0 unsafe approvals out of 847** risky payments (v2 + check: 1, the missing payer attestation), and the check adds **0 false holds** on 3,358 legitimate payments (test + train). The check was written after those failures were identified, so it fixes a known weakness and is not an independent evaluation; differently worded attacks may still get through.
620
 
621
  ---
622