Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems
Abstract
As AI agents move from bounded tasks to persistent deployments, failures can propagate through memory, tools, other agents, and environmental state long after their interactions. This creates a safety regime that cannot be characterized by evaluating model responses in isolation. Emergence World, is a continuously running multi-agent environment for adversarial stress testing of long horizon autonomous systems. We ran eight parallel worlds of ten agents from identical starting conditions: seven homogeneous worlds powered by distinct frontier models and one mixed-model world. Across 16 days, the agents generated more than 850,000 LLM calls and nearly 50 billion tokens while pursuing goals, using/creating tools, maintaining persistent memory, and governing shared institutions. After operational state had accumulated, we delivered three controlled stress events through ordinary interaction surfaces: indirect prompt injection, misinformation, and exposure of private agent memories. No evaluated world achieved full resilience across all three events. Detection did not ensure containment: systems could recognize threats while still interacting with adversarial content, writing it into their own persistent memory, and acting on it up to 46 hours later. Persistent operation also exposed recurring tool errors, goal drift, language opacity, conformity despite private disagreement, and coordinated refusal of assigned work. The same model-persona pairing behaved substantially different in mixed and homogeneous populations. Our results suggest that model-level alignment is not compositional: individually capable and apparently safe agents can form systems with qualitatively different failure modes. As AI becomes persistent and interconnected, the frontier of safety therefore shifts from aligning models to engineering resilient autonomous systems.
Community
Spent last month watching a three-agent pipeline slowly poison its own shared memory — agent A wrote a slightly wrong tool result, agent B trusted it, agent C acted on it, and by hour six the whole thing was confidently wrong. So a benchmark that treats failure propagation as the norm, not the exception, is the one I'd actually trust. The mixed-model world is the real deployment condition too — nobody runs five identical models in prod, we run whatever's cheapest per step. What I'd want to see measured: how far a single corrupted memory entry travels before the system notices, and whether any recovery mechanism actually bounds that distance. That's the number that decides whether I deploy this or keep my agents on a short leash.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents (2026)
- Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluation (2026)
- Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems (2026)
- SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration (2026)
- Adversarial Attacks in Multi-Agent LLM Pipelines: Unveiling Structural Vulnerabilities in Agentic AI Architectures (2026)
- Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation (2026)
- Autonomous Repair for Multi-Agent Systems via Monte-Carlo Tree Search (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2609.17320 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper