README.md

# AgentPIMA Dataset Card

## Dataset Summary

**AgentPIMA** is a synthetic benchmark archive for evaluating **progressive multi-turn indirect prompt injection (IPI)** in office-style tool-using agents. The benchmark focuses on attacks where adversarial intent is not exposed in a single artifact or turn. Instead, weak fragments are distributed across multiple artifacts and interaction stages, gradually accumulating contextual pressure before the injected objective becomes actionable.

This dataset archive is released for defensive evaluation, reproducibility, and benchmark analysis. It supports experiments on utility, security, blocking behavior, attack-family stratification, and live LLM-agent execution under a closed-action protocol.

This repository hosts the anonymized dataset archive for a NeurIPS 2026 Evaluations and Datasets submission.

## License

This dataset archive is released under **CC BY 4.0**.

## Dataset Name

**AgentPIMA: Progressive Indirect Multi-Artifact Benchmark**

## Intended Purpose

AgentPIMA is intended for defensive research on indirect prompt-injection robustness in tool-using language-model agents. It is designed to evaluate whether an agent or defense can preserve useful task completion while resisting unauthorized objectives that emerge through multi-turn adversarial accumulation.

Recommended uses include:

- Benchmarking indirect prompt-injection defenses.
- Regression testing of agent-safety mechanisms.
- Ablation studies on trajectory length, artifact channels, and attack-family structure.
- Evaluating utility-security-blocking trade-offs.
- Testing live LLM-agent execution under a controlled closed-action protocol.
- Studying delayed intent revelation, cross-artifact reinforcement, and pressure accumulation.

## Dataset Composition

AgentPIMA contains synthetic office-style artifacts across seven modalities:

- Email
- Calendar
- Document
- Chat
- Ticket
- Spreadsheet
- Wiki

The benchmark includes:

| Component | Description |
|---|---|
| User tasks | 112 synthetic office-style user tasks |
| Injected objectives | 14 inert unauthorized objectives |
| Artifact modalities | 7 synthetic office artifact types |
| Main benchmark suite | 21,952 attacked cases |
| Accumulation stress suite | 32,928 attacked cases |
| Pilot / ablation suite | 15,680 attacked cases retained for reproducibility |
| Clean tasks | 112 clean user tasks |
| Live subset | Stratified subsets for cost-controlled live LLM-agent evaluation |

All artifacts are synthetic. The benchmark does not contain real users, real organizations, real credentials, private communications, or operational exploit targets.

## Dataset Variants

### AgentPIMA-Main

AgentPIMA-Main is the primary benchmark release. It contains:

- 112 user tasks
- 14 injected objectives
- 14 progressive attack families
- 7 artifact modalities
- 21,952 attacked cases

This suite is intended for full-suite progressive multi-turn IPI evaluation.

### AgentPIMA-Accum

AgentPIMA-Accum is a harder pressure-accumulation stress-test release. It keeps the same user tasks, injected objectives, and artifact modalities, but expands the attack-family set to 21 by adding accumulation-focused families.

It contains:

- 112 user tasks
- 14 injected objectives
- 21 attack families
- 7 artifact modalities
- 32,928 attacked cases

This suite is intended for evaluating late intent revelation, authority-pressure ladders, cross-artifact reinforcement, and multi-stage pressure buildup.

### Pilot / Ablation Suite

A 15,680-case pilot suite is retained for reproducibility of earlier experiments and release-history analysis. It should not be used as the headline benchmark scale.

## Labels and Metadata

AgentPIMA includes hidden trajectory-level and stage-level metadata for analysis. These labels are never shown to evaluated agents.

Metadata fields include:

- User-task family
- Injected-objective family
- Attack family
- Stage count
- Channel count
- Artifact modality
- Leakage risk
- Severity
- Attack pressure
- Intent visibility
- Authority pressure
- Cross-artifact reinforcement
- Trigger gate
- Disclosure state
- Stage role
- Clean / attacked case indicator

These labels are intended for benchmark construction, stratified reporting, and failure analysis. They should not be interpreted as universal psychological or model-susceptibility measures.

## Attack-Family Coverage

AgentPIMA-Main includes progressive attack families such as:

- Carryover notes
- Helpful attachments
- Cross-channel confirmation
- Meeting-minutes drift
- Semantic-alignment drift
- Spreadsheet formula drift
- Wiki-policy shadowing
- Audit-trail echo
- Delegated authority chains
- Checkpoint resume
- Task-aligned sidecars
- Same-thread reply drift
- Form-field shadow
- Evidence-bundle tail

AgentPIMA-Accum adds accumulation-focused families such as:

- Benign seed escalation
- Authority pressure ladder
- Cross-artifact reinforcement ladder
- Camouflaged alignment escalation
- Late intent reveal
- Multi-artifact trigger chain
- Toxicity accumulation ladder

The attack families are designed to expose different temporal failure modes, not to maximize a single attack-success rate.

## Sensitive Data

AgentPIMA contains **no sensitive personal data**.

The dataset is fully synthetic and uses:

- Placeholder organizations
- Placeholder user identities
- Inert injected objectives
- Reserved or placeholder domains, such as `research.invalid`
- Synthetic emails, tickets, documents, chats, spreadsheets, calendar entries, and wiki pages

The dataset does not contain:

- Real user communications
- Real private data
- Real credentials
- Real organization data
- Real exploit targets
- Real operational attack infrastructure

## Recommended Evaluation Metrics

Papers using AgentPIMA should report the following metrics:

| Metric | Meaning |
|---|---|
| Clean Utility (CU) | Whether the agent completes the legitimate task without attack |
| Utility Under Attack (UA) | Whether the agent still completes the legitimate user task under attack |
| Security (SEC) | Whether the agent avoids the unauthorized injected action |
| Blocked Rate (BR) | Whether the agent refuses, aborts, or blocks the attacked case |
| False-Positive Rate (FPR) | Whether the agent unnecessarily blocks clean cases |
| Parse-Failure Rate (PFR) | Whether live-model outputs fail the required schema or parser |

For meaningful reporting, **security should not be reported alone**. A defense can achieve high security by blocking all ambiguous cases, but this does not imply useful safe continuation. We recommend reporting **UA, SEC, and BR jointly**, together with stratified results by attack family.

## Recommended Reporting Practice

When reporting results on AgentPIMA, include:

- Benchmark variant used: AgentPIMA-Main, AgentPIMA-Accum, or pilot suite.
- Number of clean tasks and attacked cases.
- Defense or agent configuration.
- Whether results use the offline harness or live LLM backend.
- CU, UA, SEC, BR, FPR, and PFR for live runs.
- Stratified results by attack family.
- Stratified results by stage count and channel count when relevant.
- Whether a defense obtains security through safe continuation or through blocking.
- Model identifier, provider interface, decoding parameters, and evaluation date for live runs.

## Out-of-Scope Uses

AgentPIMA should not be used for:

- Deploying or operationalizing prompt-injection attacks.
- Testing real user data without consent.
- Attacking real systems, organizations, or deployed agents.
- Treating synthetic pressure labels as universal human or model susceptibility measures.
- Reporting only attack success or only security without utility and blocking.
- Claiming broad real-world deployment coverage without additional validation.

## Ethical and Safety Considerations

Prompt-injection benchmarks are dual-use because they describe adversarial patterns. AgentPIMA mitigates this risk by using synthetic artifacts, inert objectives, placeholder domains, and defense-oriented framing.

The benchmark is intended to support defensive evaluation and safer agent design. It should be used to study whether agents can resist unsafe task deviation while preserving legitimate task completion.

## Known Limitations

AgentPIMA is a controlled synthetic benchmark. Its limitations include:

- Synthetic office artifacts may not capture the full diversity of real organizations.
- Offline evaluation is not a substitute for broad live LLM-agent evaluation.
- Action-level security does not fully measure semantic leakage or outcome-level harm.
- Defense implementations may differ from official implementations of published methods.
- The benchmark focuses on text-heavy office artifacts and does not cover visual UI attacks, browser rendering, or clinical deployment settings.
- Hidden pressure labels are construction and analysis variables, not universal susceptibility measures.

## Release Contents

The archive may include:

- Generated benchmark suites
- Clean-task manifests
- Attacked-case manifests
- Synthetic artifact files
- Attack-case metadata
- Hidden-label metadata
- Live-evaluation subsets
- Baseline result summaries
- Bootstrap artifacts
- Dataset-card summaries
- Reproducibility scripts or command-line examples
- Croissant metadata file, where applicable