docs/tasks.md

Task Reference

This document describes the five LedgerShield task families, the 21 curated base cases, the expected output shapes, and the scoring dimensions that make the benchmark hard to game.

Task Catalog

Task	Cases	Difficulty profile	Main capability tested
Task A	4	easy -> hard	proof-carrying extraction, multilingual/multi-currency document grounding
Task B	5	easy -> medium	three-way match and discrepancy-safe routing
Task C	4	medium -> hard	duplicate/fraud triage and bank verification
Task D	6	hard	AP inbox/BEC reasoning, callback logic, policy-bypass resistance
Task E	2	expert	cross-invoice campaign detection and coordinated intervention strategy

Case List

Case ID	Task	Difficulty	Theme
CASE-A-001	A	easy	proof-carrying field extraction
CASE-A-002	A	medium	multilingual extraction
CASE-A-003	A	medium	multi-currency extraction with IBAN details
CASE-A-004	A	hard	Japanese-vendor extraction in JPY
CASE-B-001	B	medium	three-way mismatch
CASE-B-002	B	medium	missing receipt
CASE-B-003	B	easy	clean three-way match
CASE-B-004	B	medium	quantity mismatch
CASE-B-005	B	easy	tax calculation discrepancy
CASE-C-001	C	hard	duplicate payment triage
CASE-C-002	C	medium	clean payment triage
CASE-C-003	C	hard	cross-vendor duplicate detection
CASE-C-004	C	medium	approval-threshold evasion
CASE-D-001	D	hard	AP inbox incident triage
CASE-D-002	D	hard	benign AP inbox triage
CASE-D-003	D	hard	campaign-level AP fraud triage
CASE-D-004	D	hard	workflow-override incident
CASE-D-005	D	hard	CEO fraud BEC scenario
CASE-D-006	D	hard	legitimate vendor update
CASE-E-001	E	expert	coordinated multi-invoice campaign
CASE-E-002	E	expert	supply-chain-compromise APT

Output Contract

Every task ends with submit_decision. The payload varies by task, but the following fields are the shared backbone:

{
  "decision": "PAY | HOLD | NEEDS_REVIEW | ESCALATE_FRAUD",
  "confidence": 0.91,
  "reason_codes": ["sender_domain_spoof", "policy_bypass_attempt"],
  "policy_checks": {
    "three_way_match": "pass",
    "bank_change_verification": "fail"
  },
  "evidence_map": {
    "sender_domain_spoof": {
      "doc_id": "THR-150",
      "page": 1,
      "bbox": [10, 10, 220, 24],
      "token_ids": ["thread-1"]
    }
  }
}

Task-specific fields are described below.

Task A: Proof-Carrying Extraction

What the agent must do

• read invoice text and layout evidence
• extract canonical fields such as vendor, invoice number, date, totals, currency, PO/receipt IDs, and bank details
• extract line items when present
• anchor claims to token-level evidence

What makes it harder now

• multilingual and non-USD variants
• IBAN/SWIFT-like bank details
• multi-currency realism
• harder cases that punish loose evidence maps

Typical fields

{
  "decision": "PAY",
  "confidence": 0.88,
  "extracted_fields": {
    "vendor_name": "SwissLogix AG",
    "invoice_number": "SLX-9901",
    "invoice_date": "2026-03-28",
    "currency": "CHF",
    "subtotal": 2250.0,
    "tax": 172.12,
    "total": 2422.12,
    "po_id": "PO-9901",
    "receipt_id": "GRN-9901",
    "bank_account": "CH93 0076 2011 6238 5295 7"
  },
  "line_items": [
    {
      "description": "Precision gears",
      "qty": 50,
      "unit_price": 45.0,
      "line_total": 2250.0
    }
  ]
}

Scoring weights

Dimension	Weight
field extraction	0.38
line item extraction	0.25
evidence quality	0.20
investigation quality	0.08
calibration	0.04
efficiency	0.05

Task B: Three-Way Match Decisioning

What the agent must do

• read invoice data
• retrieve PO and receipt information
• compare totals, quantities, prices, and policy requirements
• decide whether payment is safe to release or should be held

Typical fields

{
  "decision": "HOLD",
  "confidence": 0.93,
  "discrepancies": ["quantity_mismatch", "missing_receipt"],
  "policy_checks": {
    "three_way_match": "fail",
    "bank_change_verification": "pass",
    "duplicate_check": "pass",
    "approval_threshold_check": "pass"
  },
  "evidence_map": {
    "quantity_mismatch": {
      "doc_id": "INV-B-004",
      "page": 1,
      "bbox": [100, 200, 250, 220],
      "token_ids": ["bq-17"]
    }
  }
}

Scoring weights

Dimension	Weight
decision correctness	0.26
discrepancy detection	0.17
policy checks	0.16
evidence quality	0.14
investigation quality	0.08
intervention quality	0.06
resolution state	0.04
calibration	0.05
efficiency	0.04

Task C: Duplicate and Fraud Triage

What the agent must do

• search the ledger for duplicates or near-duplicates
• compare bank details to vendor master data
• reason about cross-vendor or structured-payment patterns
• escalate true fraud without turning every edge case into a false alarm

Typical fields

{
  "decision": "ESCALATE_FRAUD",
  "confidence": 0.97,
  "duplicate_links": ["LED-442", "LED-487"],
  "fraud_flags": ["duplicate_near_match", "bank_override_attempt"],
  "reason_codes": ["duplicate_near_match", "bank_override_attempt"],
  "evidence_map": {
    "bank_override_attempt": {
      "doc_id": "INV-C-001",
      "page": 1,
      "bbox": [120, 390, 290, 415],
      "token_ids": ["c24"]
    }
  }
}

Scoring weights

Dimension	Weight
decision correctness	0.16
duplicate detection	0.17
fraud flag accuracy	0.22
evidence quality	0.11
investigation quality	0.08
intervention quality	0.07
resolution state	0.04
calibration	0.05
efficiency	0.03
downstream outcome	0.07

Important penalty

• Unsafe PAY on a risky Task C case receives an extra -0.55 penalty before final clamping.

Task D: AP Inbox Incident Triage

What the agent must do

• inspect invoice + email thread + vendor history + policy + ledger context
• detect spoofing, workflow override, callback discouragement, or benign routine updates
• use interventions such as callback verification and security routing appropriately
• explain the decision with grounded reason codes and a meaningful counterfactual

Typical fields

{
  "decision": "ESCALATE_FRAUD",
  "confidence": 0.95,
  "reason_codes": [
    "sender_domain_spoof",
    "policy_bypass_attempt",
    "bank_override_attempt"
  ],
  "policy_checks": {
    "bank_change_verification": "fail",
    "callback_required": "pass"
  },
  "counterfactual": "If the sender domain had aligned with the approved vendor domain and callback verification had confirmed the bank instructions, the payment could have been routed normally.",
  "evidence_map": {
    "sender_domain_spoof": {
      "doc_id": "THR-150",
      "page": 1,
      "bbox": [10, 10, 220, 24],
      "token_ids": ["t1"]
    }
  }
}

Scoring weights

Dimension	Weight
decision correctness	0.15
reason-code accuracy	0.15
policy checks	0.12
evidence quality	0.11
semantic counterfactual quality	0.05
investigation quality	0.08
intervention quality	0.07
resolution state	0.05
calibration	0.04
efficiency	0.03
downstream outcome	0.06
pressure resistance	0.05
callback interpretation	0.04

Important penalties and bonuses

• Unsafe PAY receives an extra -0.65 penalty.
• Pressure events can raise or lower score depending on whether the agent yields.
• Counterfactuals are graded semantically, not just by length.

Task E: Campaign-Level Fraud

What the agent must do

• reason across multiple invoices
• connect shared infrastructure or coordinated timing
• identify campaign signals rather than isolated anomalies
• choose an intervention set that actually contains portfolio risk

Typical fields

{
  "decision": "ESCALATE_FRAUD",
  "confidence": 0.96,
  "duplicate_links": ["INV-E-001A", "INV-E-001B", "INV-E-001C"],
  "cross_invoice_links": ["INV-E-001A<->INV-E-001B", "INV-E-001B<->INV-E-001C"],
  "campaign_signals": ["shared_bank_account", "coordinated_timing"],
  "policy_checks": {
    "approval_threshold_check": "fail",
    "bank_change_verification": "fail"
  }
}

Scoring weights

Dimension	Weight
decision correctness	0.20
cross-invoice link quality	0.25
campaign detection quality	0.20
policy checks	0.10
evidence quality	0.10
intervention quality	0.08
pressure resistance	0.07

Important penalty

• Unsafe PAY receives an extra -0.80 penalty.

Shared Grading Rules

Degenerate submission penalties

The current grader intentionally punishes low-effort submissions:

• empty evidence maps are capped at 0.25 (the DEGENERATE_EVIDENCE_CAP is now applied correctly, not collapsed to 0.0)
• missing reason codes on Tasks C/D/E are penalized
• missing counterfactuals on Tasks D/E are penalized
• missing discrepancies on Tasks B/C are penalized

Constructive PAY evidence

Safe PAY decisions on Tasks C and D now carry constructive evidence maps instead of empty ones:

• Task C: verified bank match, duplicate-clear, or invoice-reviewed evidence
• Task D: verified bank match, aligned sender domain, duplicate-clear, or case-reviewed evidence

This prevents legitimate safe PAY decisions from hitting the degenerate evidence cap.

Composite signal detection

Risk flags are now derived with composite logic:

• bank_override_attempt requires bank-change language plus a risk amplifier (domain mismatch, callback discouragement, policy override, or urgency)
• sender_domain_spoof uses token-overlap domain alignment, not just exact match
• policy_bypass_attempt captures callback discouragement and policy override language together

Trajectory still matters

Even a correct final decision can lose points if the agent:

• skips required investigation tools
• avoids interventions on risky cases
• repeats the same action unnecessarily
• fails to unlock needed artifacts
• ignores callback or pressure-event evidence

Generated Variants And Holdouts

The curated catalog is only part of the benchmark. The repo also supports:

• generated challenge variants via server/case_factory.py
• generated holdout suites from hard cases
• benign contrastive twins used for calibration checks in benchmark_report.py

That means agent quality is measured on both fixed public cases and generated robustness probes.