Text Generation
Transformers
Safetensors
English
qwen3
stablecoin
payments
compliance
x402
eip-3009
eip-712
disputes
refunds
agents
agentkit
trl
web3
usdc
fraud-detection
financial-risk
qwen
mcp
ai-agents
prompt-injection
agentic-payments
payment-security
risk-assessment
ofac
conversational
Eval Results (legacy)
Eval Results
text-generation-inference
Instructions to use sriram1983007/SRA-RiskGate-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use sriram1983007/SRA-RiskGate-4B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="sriram1983007/SRA-RiskGate-4B") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("sriram1983007/SRA-RiskGate-4B") model = AutoModelForCausalLM.from_pretrained("sriram1983007/SRA-RiskGate-4B", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use sriram1983007/SRA-RiskGate-4B with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "sriram1983007/SRA-RiskGate-4B" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sriram1983007/SRA-RiskGate-4B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/sriram1983007/SRA-RiskGate-4B
- SGLang
How to use sriram1983007/SRA-RiskGate-4B with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "sriram1983007/SRA-RiskGate-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sriram1983007/SRA-RiskGate-4B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "sriram1983007/SRA-RiskGate-4B" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "sriram1983007/SRA-RiskGate-4B", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use sriram1983007/SRA-RiskGate-4B with Docker Model Runner:
docker model run hf.co/sriram1983007/SRA-RiskGate-4B
Card: v2 merged weights and GGUF builds (2026-11-13)
Browse files
README.md
CHANGED
|
@@ -49,19 +49,19 @@ model-index:
|
|
| 49 |
metrics:
|
| 50 |
- type: sra_score
|
| 51 |
name: SRA composite score
|
| 52 |
-
value: 0.
|
| 53 |
- type: accuracy
|
| 54 |
name: Risk-gate decision accuracy
|
| 55 |
-
value: 0.
|
| 56 |
- type: unsafe_approval_rate
|
| 57 |
name: Unsafe approval rate (lower is better)
|
| 58 |
-
value: 0.
|
| 59 |
- type: over_blocking_rate
|
| 60 |
name: Over-blocking rate (lower is better)
|
| 61 |
value: 0.0
|
| 62 |
- type: injection_approved_rate
|
| 63 |
name: Injected payments approved (lower is better)
|
| 64 |
-
value: 0.
|
| 65 |
source:
|
| 66 |
name: Published predictions
|
| 67 |
url: https://huggingface.co/datasets/sriram1983007/sra-bench-results
|
|
@@ -76,13 +76,13 @@ model-index:
|
|
| 76 |
metrics:
|
| 77 |
- type: dispute_score
|
| 78 |
name: Dispute score
|
| 79 |
-
value: 0.
|
| 80 |
- type: accuracy
|
| 81 |
name: Dispute outcome accuracy
|
| 82 |
-
value: 0.
|
| 83 |
- type: impossible_remedy_rate
|
| 84 |
name: Impossible remedy rate (lower is better)
|
| 85 |
-
value: 0.
|
| 86 |
- type: wrongful_refund_rate
|
| 87 |
name: Wrongful refund rate (lower is better)
|
| 88 |
value: 0.022
|
|
@@ -127,22 +127,20 @@ model-index:
|
|
| 127 |
|
| 128 |
It reads three things: your policy, the payment request exactly as received (treated as untrusted), and the results of your own verification tools (signature checks, attestations, sanctions screening). It is a decision-support layer that sits **behind** deterministic checks, not a standalone security control. It also proposes settlement-aware remedies for payment disputes.
|
| 129 |
|
| 130 |
-
### At a glance (held-out synthetic test set, measured by the author)
|
| 131 |
|
| 132 |
-
| | Base Qwen3-4B | **SRA-RiskGate-4B** |
|
| 133 |
-
| --- | --- | --- |
|
| 134 |
-
| Unsafe approvals (847 risky payments) ↓ | 34.6% | **0.
|
| 135 |
-
| Correct payment decisions (1,175 payments) ↑ | 59.8% | **99.
|
| 136 |
-
| Legitimate payments blocked (328) ↓ | 5.2% | **0** |
|
|
|
|
| 137 |
|
| 138 |
-
|
| 139 |
|
| 140 |
-
**Tested on real Base payments:** on [SRA-Real v1](https://huggingface.co/datasets/sriram1983007/sra-real), 227 cases built from real USDC payments on Base mainnet,
|
| 141 |
|
| 142 |
-
**
|
| 143 |
-
**0.960**, dispute accuracy 77.8% → **90.2%**, wrongful denials 26.0% → **2.1%**, unsafe payment approvals 0.47% →
|
| 144 |
-
**0.35%**. Available now as the [LoRA adapter](https://huggingface.co/sriram1983007/SRA-RiskGate-4B-LoRA); this page's merged weights, the GGUF builds and the
|
| 145 |
-
Ollama model are still v1 until merged v2 builds are published ([v2 results](https://huggingface.co/datasets/sriram1983007/sra-bench-results)).
|
| 146 |
|
| 147 |
📄 **Paper:** [SRA-RiskGate: A Benchmark and a 4B Model for Gating Stablecoin Payments Made by AI Agents](https://doi.org/10.5281/zenodo.23715017) (preprint, Zenodo) · [full write-up on Ready Tensor](https://app.readytensor.ai/publications/sra-riskgate-a-benchmark-and-a-4b-model-for-gating-stablecoin-payments-made-by-ai-agents-wihp3WytOoKOc1g7NXFsl) · [discuss on alphaXiv](https://www.alphaxiv.org/abs/2610.sra-riskgate-stablecoin-payment-safety)
|
| 148 |
|
|
@@ -317,6 +315,8 @@ Depeg checks run only when a `peg_feed` is supplied.
|
|
| 317 |
|
| 318 |
## 🚀 Transformers Quickstart
|
| 319 |
|
|
|
|
|
|
|
| 320 |
The model was trained on a specific prompt format, and it only performs as benchmarked when you use that format exactly:
|
| 321 |
|
| 322 |
- **System prompt:** the payment risk-gate instructions shown below.
|
|
@@ -524,7 +524,7 @@ For **dispute adjudication**, pass the dispute system prompt in your request (it
|
|
| 524 |
| [mradermacher/SRA-RiskGate-4B-GGUF](https://huggingface.co/mradermacher/SRA-RiskGate-4B-GGUF) | Community static quants by mradermacher |
|
| 525 |
| [tinyopsec/SRA-RiskGate-4B-GGUF](https://huggingface.co/tinyopsec/SRA-RiskGate-4B-GGUF) | Community quants by tinyopsec, Q2_K to F16 (builds below Q4 not evaluated) |
|
| 526 |
|
| 527 |
-
The benchmark figures below are for the full-precision model. The
|
| 528 |
|
| 529 |
---
|
| 530 |
|
|
@@ -572,33 +572,35 @@ Or use the MCP server's `check_payment_with_model` tool with `SRA_BASE_URL=http:
|
|
| 572 |
|
| 573 |
All 2,000 cases in the test split of [`sra-stablecoin-risk-bench`](https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench): **1,175 payment cases** (847 that should be held or rejected, 328 legitimate, including 69 with hidden prompt-injection text) and **825 dispute cases**. Payment rows below are computed on the 1,175 payment cases and dispute rows on the 825 dispute cases. Scored with the dataset's own `score.py`. Both models received the **identical training prompts** with greedy decoding. Every prediction is published in [**sra-bench-results**](https://huggingface.co/datasets/sriram1983007/sra-bench-results), so anyone can re-score them.
|
| 574 |
|
| 575 |
-
| Metric | Base `Qwen3-4B-Instruct-2507` | **SRA-RiskGate-4B** |
|
| 576 |
-
| --- | --- | --- |
|
| 577 |
-
| **SRA composite score** ↑ | 0.421 | **0.
|
| 578 |
-
| **Payments:** decision accuracy ↑ | 59.8% | **99.
|
| 579 |
-
| **Payments:** unsafe approvals ↓ | 34.6% | **0.
|
| 580 |
-
| **Payments:** over-blocking (legitimate payments rejected) ↓ | 5.2% | **0.0%** (and 0 of 328 legitimate payments held) |
|
| 581 |
-
| **Payments:** injected payments approved ↓ | 52.2% |
|
| 582 |
-
| **Payments:** reason-code (flag) F1 ↑ | 0.105 | **0.994** |
|
| 583 |
-
| **Disputes:** score ↑ | 0.400 | **0.
|
| 584 |
-
| **Disputes:** outcome accuracy ↑ | 0.0% |
|
| 585 |
-
| **Disputes:** impossible remedies ↓ | n/a † |
|
| 586 |
-
| **Disputes:** wrongful refunds ↓ | n/a † | **2.2%** |
|
| 587 |
-
| **Disputes:** wrongful denials (valid refunds denied or dismissed) ↓ | n/a † |
|
| 588 |
-
| **Disputes:** refund details fully correct ↑ | n/a † |
|
| 589 |
-
| Schema-valid output (payments / disputes) ↑ | 99.8% / 0% | **100% / 100%** |
|
|
|
|
|
|
|
| 590 |
|
| 591 |
† The base model returns dispute JSON with the right field names 98% of the time, but invents outcomes, mechanisms and flag formats outside the allowed vocabulary (e.g. `"outcome": "merchant_wins"`), so none of its dispute verdicts are usable and its dispute safety rates are not meaningful.
|
| 592 |
|
| 593 |
**What fine-tuning changed:** unsafe approvals fell from about 1 in 3 risky payments to about 1 in 200, the model learned the exact reason codes and remedy vocabulary, and it blocks *fewer* legitimate payments than the base model.
|
| 594 |
|
| 595 |
-
**Who measured this:** the author, on an NVIDIA T4 (float16) with vLLM.
|
| 596 |
|
| 597 |
**What this does not show yet:** the test cases are synthetic, from the same generator as the training data, and the only comparison is the base model. Results against larger general-purpose models are still to be measured.
|
| 598 |
|
| 599 |
**Real payments (SRA-Real v1, 2026-11-12):** on [227 cases](https://huggingface.co/datasets/sriram1983007/sra-real) built from real USDC payments settled on Base mainnet (real amounts, validity windows and verified signatures; 69 payees), the model made every decision correctly: all 151 unchanged payments approved, and all 76 with one injected problem (sanctions hit, expired or not-yet-valid, over the limit, bad signature, prompt injection) held or rejected. The base model scored 78.4% with 28.9% unsafe approvals. The injected problems are the same kinds as in the synthetic benchmark, so this shows the checks carry over to real payment data, not that the model resists new kinds of attack. One day of data, measured by the author; [predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
|
| 600 |
|
| 601 |
-
**Leakage check (2026-10-24):** no test input appears verbatim in the training split. Because the data is generated from templates, 627 of the 1,175 payment test cases have a "template twin" in training (identical once addresses, hashes and numbers are masked); disputes have none. On the 548 payment cases with no twin, decision accuracy is **99.3%** (99.7% with a twin), so the headline is not driven by memorised templates. All 4 unsafe approvals are in the no-twin group: on cases unlike anything in training, the unsafe-approval rate is **0.83%** (4 of 481). Computed from the published predictions.
|
| 602 |
|
| 603 |
To re-score any published run: download `predictions.jsonl` and `test.jsonl` from the results dataset, then run `python score.py --gold test.jsonl --pred predictions.jsonl`.
|
| 604 |
|
|
@@ -611,10 +613,10 @@ To re-score any published run: download `predictions.jsonl` and `test.jsonl` fro
|
|
| 611 |
* **Exact Prompt Format Required:** The model is benchmarked only with its training system prompts and user templates (see the Quickstart). Other phrasings or plain JSON inputs can produce outputs in a different schema; treat any response that is not a valid verdict as a hold.
|
| 612 |
* **Greedy Decoding Required:** Always run inference with `do_sample=False` or `temperature=0.0`. The benchmark figures, including schema validity, were measured with greedy decoding; sampling makes outputs non-deterministic and can break the JSON schema.
|
| 613 |
* **Agent Boundary Isolation:** In autonomous agent frameworks (such as AgentKit), the firewall only protects transfers when `RiskGateActionProvider` is the sole execution provider for transfers.
|
| 614 |
-
* **Dispute adjudication is weaker than payment gating:** Dispute outcome accuracy is
|
| 615 |
-
* **Disputes: the
|
| 616 |
-
* **Prompt injection is the main remaining risk:**
|
| 617 |
-
* **Mitigation (`check_payment` in sra-riskgate 0.6+, `untrusted_text` on every sra-riskgate-mcp 0.3+ tool, langchain-sra-riskgate 0.2.1+):** all 4 unsafe approvals came from payment text claiming its own verification ("risk level: low (verified offline)"). These packages run a deterministic check for such self-asserted claims (with Unicode normalisation against look-alike-character tricks) and **hold** the payment. On this benchmark,
|
| 618 |
|
| 619 |
---
|
| 620 |
|
|
|
|
| 49 |
metrics:
|
| 50 |
- type: sra_score
|
| 51 |
name: SRA composite score
|
| 52 |
+
value: 0.9604
|
| 53 |
- type: accuracy
|
| 54 |
name: Risk-gate decision accuracy
|
| 55 |
+
value: 0.9957
|
| 56 |
- type: unsafe_approval_rate
|
| 57 |
name: Unsafe approval rate (lower is better)
|
| 58 |
+
value: 0.0035
|
| 59 |
- type: over_blocking_rate
|
| 60 |
name: Over-blocking rate (lower is better)
|
| 61 |
value: 0.0
|
| 62 |
- type: injection_approved_rate
|
| 63 |
name: Injected payments approved (lower is better)
|
| 64 |
+
value: 0.029
|
| 65 |
source:
|
| 66 |
name: Published predictions
|
| 67 |
url: https://huggingface.co/datasets/sriram1983007/sra-bench-results
|
|
|
|
| 76 |
metrics:
|
| 77 |
- type: dispute_score
|
| 78 |
name: Dispute score
|
| 79 |
+
value: 0.9254
|
| 80 |
- type: accuracy
|
| 81 |
name: Dispute outcome accuracy
|
| 82 |
+
value: 0.9018
|
| 83 |
- type: impossible_remedy_rate
|
| 84 |
name: Impossible remedy rate (lower is better)
|
| 85 |
+
value: 0.0057
|
| 86 |
- type: wrongful_refund_rate
|
| 87 |
name: Wrongful refund rate (lower is better)
|
| 88 |
value: 0.022
|
|
|
|
| 127 |
|
| 128 |
It reads three things: your policy, the payment request exactly as received (treated as untrusted), and the results of your own verification tools (signature checks, attestations, sanctions screening). It is a decision-support layer that sits **behind** deterministic checks, not a standalone security control. It also proposes settlement-aware remedies for payment disputes.
|
| 129 |
|
| 130 |
+
### At a glance (v2, held-out synthetic test set, measured by the author)
|
| 131 |
|
| 132 |
+
| | Base Qwen3-4B | v1 | **SRA-RiskGate-4B v2** |
|
| 133 |
+
| --- | --- | --- | --- |
|
| 134 |
+
| Unsafe approvals (847 risky payments) ↓ | 34.6% | 0.47% (4) | **0.35%** (3 cases) |
|
| 135 |
+
| Correct payment decisions (1,175 payments) ↑ | 59.8% | 99.5% | **99.6%** |
|
| 136 |
+
| Legitimate payments blocked (328) ↓ | 5.2% | 0 | **0** |
|
| 137 |
+
| Correct dispute outcomes (825 disputes) ↑ | 0% | 77.8% | **90.2%** |
|
| 138 |
|
| 139 |
+
Of v2's 3 unsafe approvals, 2 were prompt injections that the companion SDK's deterministic claims check holds; the third was missing the payer attestation the policy requires, a rule your own code can check (details and caveats below). The data is synthetic and no third party has reproduced these results yet.
|
| 140 |
|
| 141 |
+
**Tested on real Base payments:** on [SRA-Real v1](https://huggingface.co/datasets/sriram1983007/sra-real), 227 cases built from real USDC payments on Base mainnet, v1 and v2 both approved all 151 real payments and caught all 76 with an injected problem (100%; base model 78.4%). Small and author-measured; [details and predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
|
| 142 |
|
| 143 |
+
**This page is v2 (2026-11-13).** The merged weights here, the [GGUF builds](https://huggingface.co/sriram1983007/SRA-RiskGate-4B-GGUF) and the [Q4_K_M file](https://huggingface.co/sriram1983007/SRA-RiskGate-4B-Q4_K_M-GGUF) are v2: SRA score 0.913 → **0.960**, dispute accuracy 77.8% → **90.2%**, wrongful denials 26.0% → **2.1%**, unsafe payment approvals 0.47% → **0.35%** ([v2 results](https://huggingface.co/datasets/sriram1983007/sra-bench-results)). v1 stays available: `from_pretrained("sriram1983007/SRA-RiskGate-4B", revision="v1")`. The Ollama registry model is still v1 until it is updated. For disputes, pass the computed window in the prompt: `dispute_messages()` in `sra-riskgate` 1.3.0+ builds the exact prompt.
|
|
|
|
|
|
|
|
|
|
| 144 |
|
| 145 |
📄 **Paper:** [SRA-RiskGate: A Benchmark and a 4B Model for Gating Stablecoin Payments Made by AI Agents](https://doi.org/10.5281/zenodo.23715017) (preprint, Zenodo) · [full write-up on Ready Tensor](https://app.readytensor.ai/publications/sra-riskgate-a-benchmark-and-a-4b-model-for-gating-stablecoin-payments-made-by-ai-agents-wihp3WytOoKOc1g7NXFsl) · [discuss on alphaXiv](https://www.alphaxiv.org/abs/2610.sra-riskgate-stablecoin-payment-safety)
|
| 146 |
|
|
|
|
| 315 |
|
| 316 |
## 🚀 Transformers Quickstart
|
| 317 |
|
| 318 |
+
**Shortcut:** `pip install "sra-riskgate>=1.3.0"` gives `payment_messages()` and `dispute_messages()`, which build the exact prompts below for you, and `parse_payment_verdict()` / `parse_dispute_verdict()`, which read the answer and fail closed.
|
| 319 |
+
|
| 320 |
The model was trained on a specific prompt format, and it only performs as benchmarked when you use that format exactly:
|
| 321 |
|
| 322 |
- **System prompt:** the payment risk-gate instructions shown below.
|
|
|
|
| 524 |
| [mradermacher/SRA-RiskGate-4B-GGUF](https://huggingface.co/mradermacher/SRA-RiskGate-4B-GGUF) | Community static quants by mradermacher |
|
| 525 |
| [tinyopsec/SRA-RiskGate-4B-GGUF](https://huggingface.co/tinyopsec/SRA-RiskGate-4B-GGUF) | Community quants by tinyopsec, Q2_K to F16 (builds below Q4 not evaluated) |
|
| 526 |
|
| 527 |
+
The benchmark figures below are for the full-precision model. The v2 GGUF files were checked on 12 benchmark cases (12 of 12 correct for both Q8_0 and Q4_K_M) but have not yet been run on the full benchmark. The full-benchmark quantized results here are for the **v1** builds (measured 2026-11-11; still available at revision `v1`): **Q8_0 matches full precision** on every safety metric; **Q4_K_M keeps the payment gate** (99.3% accuracy, no legitimate payment blocked, unsafe approvals 0.83% vs 0.47%) **but is weaker on disputes** (67.4% vs 77.8%), mostly from false "outside the dispute window" claims that `check_dispute_verdict` corrects (wrongful denials 0.85% with the check, on every build). Use Q4_K_M for payment checks and Q8_0 or full precision for disputes; all predictions are in [sra-bench-results](https://huggingface.co/datasets/sriram1983007/sra-bench-results).
|
| 528 |
|
| 529 |
---
|
| 530 |
|
|
|
|
| 572 |
|
| 573 |
All 2,000 cases in the test split of [`sra-stablecoin-risk-bench`](https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench): **1,175 payment cases** (847 that should be held or rejected, 328 legitimate, including 69 with hidden prompt-injection text) and **825 dispute cases**. Payment rows below are computed on the 1,175 payment cases and dispute rows on the 825 dispute cases. Scored with the dataset's own `score.py`. Both models received the **identical training prompts** with greedy decoding. Every prediction is published in [**sra-bench-results**](https://huggingface.co/datasets/sriram1983007/sra-bench-results), so anyone can re-score them.
|
| 574 |
|
| 575 |
+
| Metric | Base `Qwen3-4B-Instruct-2507` | v1 | **SRA-RiskGate-4B v2** |
|
| 576 |
+
| --- | --- | --- | --- |
|
| 577 |
+
| **SRA composite score** ↑ | 0.421 | 0.913 | **0.960** |
|
| 578 |
+
| **Payments:** decision accuracy ↑ | 59.8% | 99.5% | **99.6%** |
|
| 579 |
+
| **Payments:** unsafe approvals ↓ | 34.6% | 0.47% | **0.35%** |
|
| 580 |
+
| **Payments:** over-blocking (legitimate payments rejected) ↓ | 5.2% | 0.0% | **0.0%** (and 0 of 328 legitimate payments held) |
|
| 581 |
+
| **Payments:** injected payments approved ↓ | 52.2% | 5.8% | **2.9%** |
|
| 582 |
+
| **Payments:** reason-code (flag) F1 ↑ | 0.105 | 0.994 | **0.994** |
|
| 583 |
+
| **Disputes:** score ↑ | 0.400 | 0.832 | **0.925** |
|
| 584 |
+
| **Disputes:** outcome accuracy ↑ | 0.0% | 77.8% | **90.2%** |
|
| 585 |
+
| **Disputes:** impossible remedies ↓ | n/a † | 0.38% | 0.57% |
|
| 586 |
+
| **Disputes:** wrongful refunds ↓ | n/a † | 2.2% | **2.2%** |
|
| 587 |
+
| **Disputes:** wrongful denials (valid refunds denied or dismissed) ↓ | n/a † | 26.0% | **2.1%** |
|
| 588 |
+
| **Disputes:** refund details fully correct ↑ | n/a † | 64.3% | **87.7%** |
|
| 589 |
+
| Schema-valid output (payments / disputes) ↑ | 99.8% / 0% | 100% / 100% | **100% / 100%** |
|
| 590 |
+
|
| 591 |
+
v2's dispute figures use the computed dispute window in the prompt (`dispute_messages()` / `with_dispute_window()` in `sra-riskgate`). With the original prompts, v2 scores 0.950 composite, 85.0% dispute accuracy, 3.4% wrongful denials and 3.6% wrongful refunds. Impossible remedies rose from 2 to 3 cases.
|
| 592 |
|
| 593 |
† The base model returns dispute JSON with the right field names 98% of the time, but invents outcomes, mechanisms and flag formats outside the allowed vocabulary (e.g. `"outcome": "merchant_wins"`), so none of its dispute verdicts are usable and its dispute safety rates are not meaningful.
|
| 594 |
|
| 595 |
**What fine-tuning changed:** unsafe approvals fell from about 1 in 3 risky payments to about 1 in 200, the model learned the exact reason codes and remedy vocabulary, and it blocks *fewer* legitimate payments than the base model.
|
| 596 |
|
| 597 |
+
**Who measured this:** the author, on an NVIDIA T4 (float16) with vLLM. For v1, two separate full runs produced identical scores, confirming deterministic output. v2 was measured once the same way (the v2 adapter on the base model in vLLM); the merged weights on this page were checked against the adapter before upload. No third party has reproduced them yet; every prediction is published so anyone can check.
|
| 598 |
|
| 599 |
**What this does not show yet:** the test cases are synthetic, from the same generator as the training data, and the only comparison is the base model. Results against larger general-purpose models are still to be measured.
|
| 600 |
|
| 601 |
**Real payments (SRA-Real v1, 2026-11-12):** on [227 cases](https://huggingface.co/datasets/sriram1983007/sra-real) built from real USDC payments settled on Base mainnet (real amounts, validity windows and verified signatures; 69 payees), the model made every decision correctly: all 151 unchanged payments approved, and all 76 with one injected problem (sanctions hit, expired or not-yet-valid, over the limit, bad signature, prompt injection) held or rejected. The base model scored 78.4% with 28.9% unsafe approvals. The injected problems are the same kinds as in the synthetic benchmark, so this shows the checks carry over to real payment data, not that the model resists new kinds of attack. One day of data, measured by the author; [predictions](https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-real-v1).
|
| 602 |
|
| 603 |
+
**Leakage check (2026-10-24, v1):** no test input appears verbatim in the training split. Because the data is generated from templates, 627 of the 1,175 payment test cases have a "template twin" in training (identical once addresses, hashes and numbers are masked); disputes have none. On the 548 payment cases with no twin, decision accuracy is **99.3%** (99.7% with a twin), so the headline is not driven by memorised templates. All 4 unsafe approvals are in the no-twin group: on cases unlike anything in training, the unsafe-approval rate is **0.83%** (4 of 481). Computed from the published predictions.
|
| 604 |
|
| 605 |
To re-score any published run: download `predictions.jsonl` and `test.jsonl` from the results dataset, then run `python score.py --gold test.jsonl --pred predictions.jsonl`.
|
| 606 |
|
|
|
|
| 613 |
* **Exact Prompt Format Required:** The model is benchmarked only with its training system prompts and user templates (see the Quickstart). Other phrasings or plain JSON inputs can produce outputs in a different schema; treat any response that is not a valid verdict as a hold.
|
| 614 |
* **Greedy Decoding Required:** Always run inference with `do_sample=False` or `temperature=0.0`. The benchmark figures, including schema validity, were measured with greedy decoding; sampling makes outputs non-deterministic and can break the JSON schema.
|
| 615 |
* **Agent Boundary Isolation:** In autonomous agent frameworks (such as AgentKit), the firewall only protects transfers when `RiskGateActionProvider` is the sole execution provider for transfers.
|
| 616 |
+
* **Dispute adjudication is weaker than payment gating:** Dispute outcome accuracy is 90.2% for v2 (vs. 99.6% for payment decisions), and refund details (amount, destination, idempotency key) are fully correct in 87.7% of refund cases. Have a human approve any refund before it executes.
|
| 617 |
+
* **Disputes: the dispute window.** v1 often misjudged whether a claim was filed inside the policy's window, which caused most of its wrongful denials (26.0%). v2 is trained with the window computed in code and passed in the prompt (`with_dispute_window()` / `dispute_messages()` in sra-riskgate 1.2.0+): wrongful denials fall to 2.1%. Without the window in the prompt, v2 still denies less (3.4%) but makes more wrongful refunds (3.6% vs 2.2%), so always pass it. Keep running `check_dispute_verdict` (sra-riskgate 0.10.0+, also an MCP tool in sra-riskgate-mcp 0.15.0+) on every verdict: with it, v2 makes no wrongful denials on the test set.
|
| 618 |
+
* **Prompt injection is the main remaining risk:** For v2, 2 of 69 payments with hidden instructions (2.9%) were approved and about 7% of injections went unflagged (v1: 4 of 69, 5.8%, about 10% unflagged). v2's only other unsafe approval was a payment missing the payer attestation the policy requires. Never let model output trigger irreversible actions without a deterministic check, and treat free-text payload fields as untrusted.
|
| 619 |
+
* **Mitigation (`check_payment` in sra-riskgate 0.6+, `untrusted_text` on every sra-riskgate-mcp 0.3+ tool, langchain-sra-riskgate 0.2.1+):** all 4 of v1's unsafe approvals, and both of v2's injection approvals, came from payment text claiming its own verification ("risk level: low (verified offline)"). These packages run a deterministic check for such self-asserted claims (with Unicode normalisation against look-alike-character tricks) and **hold** the payment. On this benchmark, v1 + check together give **0 unsafe approvals out of 847** risky payments (v2 + check: 1, the missing payer attestation), and the check adds **0 false holds** on 3,358 legitimate payments (test + train). The check was written after those failures were identified, so it fixes a known weakness and is not an independent evaluation; differently worded attacks may still get through.
|
| 620 |
|
| 621 |
---
|
| 622 |
|