course_notes/multi-agent.md

# ---
# comments: True
# description: Multi-agent reinforcement learning basics, covering cooperative, competitive, and mixed settings, CTDE paradigms (MADDPG/MAPPO), value decomposition (VDN/QMIX), non-stationarity, and credit assignment.
# ---

# Multi-Agent RL

Multi-agent RL (MARL) studies multiple learners interacting in a shared environment. Interactions can be **cooperative**, **competitive**, or **mixed**, leading to non-stationary dynamics from each agent’s perspective.

## Problem Settings
- **Fully cooperative**: agents share a team reward (e.g., SMAC).
- **Fully competitive / zero-sum**: agents optimize opposing returns (e.g., two-player games).
- **General-sum / mixed**: partially aligned interests (e.g., traffic control, resource allocation).
- **Communication**: explicit messages may be allowed or learned implicitly via actions.

## Core Challenges
- **Non-stationarity**: other agents’ policies change during learning, breaking the Markov property.
- **Credit assignment**: decomposing team rewards to individuals.
- **Scalability**: joint action spaces grow exponentially.
- **Partial observability**: each agent sees only local observations.

## Centralized Training, Decentralized Execution (CTDE)
Train with access to global information, execute with local observations:
- **MADDPG**: deterministic actors per agent; centralized critics conditioned on joint observations/actions.
- **MAPPO**: PPO with centralized value functions; decentralized stochastic policies.
- **HATRPO/HAPPO**: trust-region variants with hierarchical or per-agent constraints.

## Value Decomposition
Factor team Q-values into per-agent terms to enable decentralized policies:
- **VDN**: \(Q_{tot} = \sum_i Q_i\).
- **QMIX**: monotonic mixing network ensures \(\arg\max Q_{tot}\) aligns with individual argmaxes.
- **QTRAN/QPLEX**: relax monotonicity for greater expressiveness.

## Exploration and Stability
- **Role randomization** and **parameter noise** help avoid symmetric equilibria.
- **Population-based training / self-play**: maintain opponent pools to avoid overfitting; crucial in competitive settings.
- **Entropy regularization** or intrinsic bonuses can mitigate coordination failures.

## Communication and Coordination
- Learned communication channels (e.g., CommNet, TarMAC) allow agents to share latent messages.
- **Differentiable inter-agent communication**: end-to-end training with attention or graph neural networks to propagate information.
- Regularize bandwidth or use discrete bottlenecks to prevent overfitting to dense messaging.

## Evaluation
- Report both **team return** and **individual consistency** across seeds and partner policies.
- In competitive settings, evaluate against held-out opponents and measure exploitability or NashConv.
- Measure **generalization**: performance with unseen partners or different numbers of agents.

## Practical Tips
- Start with CTDE baselines (MAPPO for discrete/stochastic, MADDPG for continuous actions).
- Normalize observations per agent; share parameters when agents are homogeneous.
- Use replay buffers with fingerprints (episode step, policy version) to reduce non-stationarity.
- Clip or bound gradient norms; tune mixing network capacity to avoid overfitting.

## References
- Lowe et al. (2017) MADDPG; Rashid et al. (2018) QMIX; Sunehag et al. (2017) VDN; Yu et al. (2021) MAPPO; Vinyals et al. (2019) AlphaStar for large-scale self-play.