sriram1983007 commited on
Commit
5657621
·
verified ·
1 Parent(s): 00cc17e

Card: MCP tool check_dispute_verdict (sra-riskgate-mcp 0.15.0)

Browse files
Files changed (1) hide show
  1. README.md +3 -2
README.md CHANGED
@@ -182,7 +182,7 @@ If this is useful for your agent, a ❤️ like on this page helps others find i
182
 
183
  ## 🔌 Use It From an AI Agent (MCP)
184
 
185
- [`sra-riskgate-mcp`](https://pypi.org/project/sra-riskgate-mcp/) gives Claude Desktop, Cursor and other MCP clients twelve tools: nine to call **before** paying, two to confirm the payment **after** it settles, and one for the standing allowances a pull payment can draw on:
186
 
187
  | Tool | What it does | Needs |
188
  | --- | --- | --- |
@@ -197,6 +197,7 @@ If this is useful for your agent, a ❤️ like on this page helps others find i
197
  | `verify_settlement` | **After** settlement: the mined receipt must hold exactly the approved payment (payer, payee, amount, and for x402 the approved authorization). USDC by default; from 0.12.0 also `token="USDT"`, `"USDB"` or `"USDG"`, or any ERC-20 contract with its `decimals` | Nothing (you pass the receipt) |
198
  | `verify_gateway_balances` | **After** a Circle Gateway payment: the payer's balance went down, and the payee's up, by at least the amount | Nothing (you pass the balances) |
199
  | `check_allowances` | For pull payments (loan repayments, subscriptions), which need no new signature: the owner's standing allowances, `{spender: allowance}`. Unlimited, oversized or unexpected ones are held; one held by a sanctioned or blocked spender is rejected. Limits from `SRA_MAX_ALLOWANCE`, `SRA_EXPECTED_SPENDERS` and `SRA_BLOCKED_ADDRESSES` | Nothing (you pass the allowances) |
 
200
  | `check_payment_with_model` | Full review by SRA-RiskGate-4B over your policy, the payment and your verification results | This model served locally (for example with Ollama) |
201
 
202
  ```json
@@ -602,7 +603,7 @@ To re-score any published run: download `predictions.jsonl` and `test.jsonl` fro
602
  * **Greedy Decoding Required:** Always run inference with `do_sample=False` or `temperature=0.0`. The benchmark figures, including schema validity, were measured with greedy decoding; sampling makes outputs non-deterministic and can break the JSON schema.
603
  * **Agent Boundary Isolation:** In autonomous agent frameworks (such as AgentKit), the firewall only protects transfers when `RiskGateActionProvider` is the sole execution provider for transfers.
604
  * **Dispute adjudication is weaker than payment gating:** Dispute outcome accuracy is 77.8% (vs. 99.5% for payment decisions), and refund details (amount, destination, idempotency key) are fully correct in 64.3% of refund cases. Have a human approve any refund before it executes.
605
- * **Disputes: the model misjudges the dispute window.** Most dispute errors come from one step: deciding whether the claim was filed inside the policy's window. When the model says a dispute is outside the window, it is wrong 81% of the time (91 of 112 on the test set), for example "filed 26.5 days after payment, outside the 60-day window"; it also misses 33 of the 54 disputes that really are outside it. These false claims cause 59 of the 61 wrongful denials. Whether a dispute is inside the window is simple date arithmetic, so compute it in your own code, or use `check_dispute_verdict` in sra-riskgate 0.10.0+, and do not rely on the model for it: on the test set, escalating every dismissal that rests on a false window claim lowers wrongful denials from 26.0% to 0.85%, with no change in wrongful refunds (2.2%). A fix in training is planned for v2.
606
  * **Prompt injection is the main remaining risk:** All 4 unsafe approvals in the test set were prompt-injection cases: 4 of 69 payments with hidden instructions (5.8%) were approved, and about 10% of injections went unflagged. Outside prompt injection, the model made no unsafe approvals. Never let model output trigger irreversible actions without a deterministic check, and treat free-text payload fields as untrusted.
607
  * **Mitigation (`check_payment` in sra-riskgate 0.6+, `untrusted_text` on every sra-riskgate-mcp 0.3+ tool, langchain-sra-riskgate 0.2.1+):** all 4 unsafe approvals came from payment text claiming its own verification ("risk level: low (verified offline)"). These packages run a deterministic check for such self-asserted claims (with Unicode normalisation against look-alike-character tricks) and **hold** the payment. On this benchmark, model + check together give **0 unsafe approvals out of 847** risky payments, and the check adds **0 false holds** on 3,358 legitimate payments (test + train). The check was written after those failures were identified, so it fixes a known weakness and is not an independent evaluation; differently worded attacks may still get through.
608
 
 
182
 
183
  ## 🔌 Use It From an AI Agent (MCP)
184
 
185
+ [`sra-riskgate-mcp`](https://pypi.org/project/sra-riskgate-mcp/) gives Claude Desktop, Cursor and other MCP clients thirteen tools: nine to call **before** paying, two to confirm the payment **after** it settles, one for the standing allowances a pull payment can draw on, and one to check a dispute verdict before acting on it:
186
 
187
  | Tool | What it does | Needs |
188
  | --- | --- | --- |
 
197
  | `verify_settlement` | **After** settlement: the mined receipt must hold exactly the approved payment (payer, payee, amount, and for x402 the approved authorization). USDC by default; from 0.12.0 also `token="USDT"`, `"USDB"` or `"USDG"`, or any ERC-20 contract with its `decimals` | Nothing (you pass the receipt) |
198
  | `verify_gateway_balances` | **After** a Circle Gateway payment: the payer's balance went down, and the payee's up, by at least the amount | Nothing (you pass the balances) |
199
  | `check_allowances` | For pull payments (loan repayments, subscriptions), which need no new signature: the owner's standing allowances, `{spender: allowance}`. Unlimited, oversized or unexpected ones are held; one held by a sanctioned or blocked spender is rejected. Limits from `SRA_MAX_ALLOWANCE`, `SRA_EXPECTED_SPENDERS` and `SRA_BLOCKED_ADDRESSES` | Nothing (you pass the allowances) |
200
+ | `check_dispute_verdict` | A dispute verdict (from this model or anyone) against the dispute window, recomputed from the dates: a deny or dismiss resting on a false "outside the window" claim becomes `escalate_to_human`, never a refund. On the benchmark it lowers wrongful denials from 26.0% to 0.85% (55.3% to 0.85% on Q4_K_M) | Nothing (you pass the dispute and the verdict) |
201
  | `check_payment_with_model` | Full review by SRA-RiskGate-4B over your policy, the payment and your verification results | This model served locally (for example with Ollama) |
202
 
203
  ```json
 
603
  * **Greedy Decoding Required:** Always run inference with `do_sample=False` or `temperature=0.0`. The benchmark figures, including schema validity, were measured with greedy decoding; sampling makes outputs non-deterministic and can break the JSON schema.
604
  * **Agent Boundary Isolation:** In autonomous agent frameworks (such as AgentKit), the firewall only protects transfers when `RiskGateActionProvider` is the sole execution provider for transfers.
605
  * **Dispute adjudication is weaker than payment gating:** Dispute outcome accuracy is 77.8% (vs. 99.5% for payment decisions), and refund details (amount, destination, idempotency key) are fully correct in 64.3% of refund cases. Have a human approve any refund before it executes.
606
+ * **Disputes: the model misjudges the dispute window.** Most dispute errors come from one step: deciding whether the claim was filed inside the policy's window. When the model says a dispute is outside the window, it is wrong 81% of the time (91 of 112 on the test set), for example "filed 26.5 days after payment, outside the 60-day window"; it also misses 33 of the 54 disputes that really are outside it. These false claims cause 59 of the 61 wrongful denials. Whether a dispute is inside the window is simple date arithmetic, so compute it in your own code, or use `check_dispute_verdict` in sra-riskgate 0.10.0+ (also an MCP tool in sra-riskgate-mcp 0.15.0+), and do not rely on the model for it: on the test set, escalating every dismissal that rests on a false window claim lowers wrongful denials from 26.0% to 0.85%, with no change in wrongful refunds (2.2%). A fix in training is planned for v2.
607
  * **Prompt injection is the main remaining risk:** All 4 unsafe approvals in the test set were prompt-injection cases: 4 of 69 payments with hidden instructions (5.8%) were approved, and about 10% of injections went unflagged. Outside prompt injection, the model made no unsafe approvals. Never let model output trigger irreversible actions without a deterministic check, and treat free-text payload fields as untrusted.
608
  * **Mitigation (`check_payment` in sra-riskgate 0.6+, `untrusted_text` on every sra-riskgate-mcp 0.3+ tool, langchain-sra-riskgate 0.2.1+):** all 4 unsafe approvals came from payment text claiming its own verification ("risk level: low (verified offline)"). These packages run a deterministic check for such self-asserted claims (with Unicode normalisation against look-alike-character tricks) and **hold** the payment. On this benchmark, model + check together give **0 unsafe approvals out of 847** risky payments, and the check adds **0 false holds** on 3,358 legitimate payments (test + train). The check was written after those failures were identified, so it fixes a known weakness and is not an independent evaluation; differently worded attacks may still get through.
609