File size: 40,450 Bytes
50f38bd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 | ---
title: Reinforcement Learning Graphical Representations
date: 2026-04-08
category: Reinforcement Learning
description: A comprehensive gallery of 230 standard RL components and their graphical presentations.
---
# Reinforcement Learning Graphical Representations
This repository contains a full set of 230 visualizations representing foundational concepts, algorithms, and advanced topics in Reinforcement Learning.
| Category | Component | Illustration | Details | Context |
|----------|-----------|--------------|---------|---------|
| **MDP & Environment** | **Agent-Environment Interaction Loop** |  | Core cycle: observation of state → selection of action → environment transition → receipt of reward + next state | All RL algorithms |
| **MDP & Environment** | **Markov Decision Process (MDP) Tuple** |  | (S, A, P, R, γ) with transition dynamics and reward function | s,a) and R(s,a,s′)) |
| **MDP & Environment** | **State Transition Graph** |  | Full probabilistic transitions between discrete states | Gridworld, Taxi, Cliff Walking |
| **MDP & Environment** | **Trajectory / Episode Sequence** |  | Sequence of (s₀, a₀, r₁, s₁, …, s_T) | Monte Carlo, episodic tasks |
| **MDP & Environment** | **Continuous State/Action Space Visualization** |  | High-dimensional spaces (e.g., robot joints, pixel inputs) | Continuous-control tasks (MuJoCo, PyBullet) |
| **MDP & Environment** | **Reward Function / Landscape** |  | Scalar reward as function of state/action | All algorithms; especially reward shaping |
| **MDP & Environment** | **Discount Factor (γ) Effect** |  | How future rewards are weighted | All discounted MDPs |
| **Value & Policy** | **State-Value Function V(s)** |  | Expected return from state s under policy π | Value-based methods |
| **Value & Policy** | **Action-Value Function Q(s,a)** |  | Expected return from state-action pair | Q-learning family |
| **Value & Policy** | **Policy π(s) or π(a\** |  | s) | Arrow overlays on grid (optimal policy), probability bar charts, or softmax heatmaps |
| **Value & Policy** | **Advantage Function A(s,a)** |  | Q(s,a) – V(s) | A2C, PPO, SAC, TD3 |
| **Value & Policy** | **Optimal Value Function V* / Q*** |  | Solution to Bellman optimality | Value iteration, Q-learning |
| **Dynamic Programming** | **Policy Evaluation Backup** |  | Iterative update of V using Bellman expectation | Policy iteration |
| **Dynamic Programming** | **Policy Improvement** |  | Greedy policy update over Q | Policy iteration |
| **Dynamic Programming** | **Value Iteration Backup** |  | Update using Bellman optimality | Value iteration |
| **Dynamic Programming** | **Policy Iteration Full Cycle** |  | Evaluation → Improvement loop | Classic DP methods |
| **Monte Carlo** | **Monte Carlo Backup** |  | Update using full episode return G_t | First-visit / every-visit MC |
| **Monte Carlo** | **Monte Carlo Tree (MCTS)** |  | Search tree with selection, expansion, simulation, backprop | AlphaGo, AlphaZero |
| **Monte Carlo** | **Importance Sampling Ratio** |  | Off-policy correction ρ = π(a\ | s) |
| **Temporal Difference** | **TD(0) Backup** |  | Bootstrapped update using R + γV(s′) | TD learning |
| **Temporal Difference** | **Bootstrapping (general)** |  | Using estimated future value instead of full return | All TD methods |
| **Temporal Difference** | **n-step TD Backup** |  | Multi-step return G_t^{(n)} | n-step TD, TD(λ) |
| **Temporal Difference** | **TD(λ) & Eligibility Traces** |  | Decaying trace z_t for credit assignment | TD(λ), SARSA(λ), Q(λ) |
| **Temporal Difference** | **SARSA Update** |  | On-policy TD control | SARSA |
| **Temporal Difference** | **Q-Learning Update** |  | Off-policy TD control | Q-learning, Deep Q-Network |
| **Temporal Difference** | **Expected SARSA** |  | Expectation over next action under policy | Expected SARSA |
| **Temporal Difference** | **Double Q-Learning / Double DQN** |  | Two separate Q estimators to reduce overestimation | Double DQN, TD3 |
| **Temporal Difference** | **Dueling DQN Architecture** |  | Separate streams for state value V(s) and advantage A(s,a) | Dueling DQN |
| **Temporal Difference** | **Prioritized Experience Replay** |  | Importance sampling of transitions by TD error | Prioritized DQN, Rainbow |
| **Temporal Difference** | **Rainbow DQN Components** |  | All extensions combined (Double, Dueling, PER, etc.) | Rainbow DQN |
| **Function Approximation** | **Linear Function Approximation** |  | Feature vector φ(s) → wᵀφ(s) | Tabular → linear FA |
| **Function Approximation** | **Neural Network Layers (MLP, CNN, RNN, Transformer)** |  | Full deep network for value/policy | DQN, A3C, PPO, Decision Transformer |
| **Function Approximation** | **Computation Graph / Backpropagation Flow** |  | Gradient flow through network | All deep RL |
| **Function Approximation** | **Target Network** |  | Frozen copy of Q-network for stability | DQN, DDQN, SAC, TD3 |
| **Policy Gradients** | **Policy Gradient Theorem** |  | ∇_θ J(θ) = E[∇_θ log π(a\ | Flow diagram from reward → log-prob → gradient |
| **Policy Gradients** | **REINFORCE Update** |  | Monte-Carlo policy gradient | REINFORCE |
| **Policy Gradients** | **Baseline / Advantage Subtraction** |  | Subtract b(s) to reduce variance | All modern PG |
| **Policy Gradients** | **Trust Region (TRPO)** |  | KL-divergence constraint on policy update | TRPO |
| **Policy Gradients** | **Proximal Policy Optimization (PPO)** |  | Clipped surrogate objective | PPO, PPO-Clip |
| **Actor-Critic** | **Actor-Critic Architecture** |  | Separate or shared actor (policy) + critic (value) networks | A2C, A3C, SAC, TD3 |
| **Actor-Critic** | **Advantage Actor-Critic (A2C/A3C)** |  | Synchronous/asynchronous multi-worker | A2C/A3C |
| **Actor-Critic** | **Soft Actor-Critic (SAC)** |  | Entropy-regularized policy + twin critics | SAC |
| **Actor-Critic** | **Twin Delayed DDPG (TD3)** |  | Twin critics + delayed policy + target smoothing | TD3 |
| **Exploration** | **ε-Greedy Strategy** |  | Probability ε of random action | DQN family |
| **Exploration** | **Softmax / Boltzmann Exploration** |  | Temperature τ in softmax | Softmax policies |
| **Exploration** | **Upper Confidence Bound (UCB)** |  | Optimism in face of uncertainty | UCB1, bandits |
| **Exploration** | **Intrinsic Motivation / Curiosity** |  | Prediction error as intrinsic reward | ICM, RND, Curiosity-driven RL |
| **Exploration** | **Entropy Regularization** |  | Bonus term αH(π) | SAC, maximum-entropy RL |
| **Hierarchical RL** | **Options Framework** |  | High-level policy over options (temporally extended actions) | Option-Critic |
| **Hierarchical RL** | **Feudal Networks / Hierarchical Actor-Critic** |  | Manager-worker hierarchy | Feudal RL |
| **Hierarchical RL** | **Skill Discovery** |  | Unsupervised emergence of reusable skills | DIAYN, VALOR |
| **Model-Based RL** | **Learned Dynamics Model** |  | ˆP(s′\ | Separate model network diagram (often RNN or transformer) |
| **Model-Based RL** | **Model-Based Planning** |  | Rollouts inside learned model | MuZero, DreamerV3 |
| **Model-Based RL** | **Imagination-Augmented Agents (I2A)** |  | Imagination module + policy | I2A |
| **Offline RL** | **Offline Dataset** |  | Fixed batch of trajectories | BC, CQL, IQL |
| **Offline RL** | **Conservative Q-Learning (CQL)** |  | Penalty on out-of-distribution actions | CQL |
| **Multi-Agent RL** | **Multi-Agent Interaction Graph** |  | Agents communicating or competing | MARL, MADDPG |
| **Multi-Agent RL** | **Centralized Training Decentralized Execution (CTDE)** |  | Shared critic during training | QMIX, VDN, MADDPG |
| **Multi-Agent RL** | **Cooperative / Competitive Payoff Matrix** |  | Joint reward for multiple agents | Prisoner's Dilemma, multi-agent gridworlds |
| **Inverse RL / IRL** | **Reward Inference** |  | Infer reward from expert demonstrations | IRL, GAIL |
| **Inverse RL / IRL** | **Generative Adversarial Imitation Learning (GAIL)** |  | Discriminator vs. policy generator | GAIL, AIRL |
| **Meta-RL** | **Meta-RL Architecture** |  | Outer loop (meta-policy) + inner loop (task adaptation) | MAML for RL, RL² |
| **Meta-RL** | **Task Distribution Visualization** |  | Multiple MDPs sampled from meta-distribution | Meta-RL benchmarks |
| **Advanced / Misc** | **Experience Replay Buffer** |  | Stored (s,a,r,s′,done) tuples | DQN and all off-policy deep RL |
| **Advanced / Misc** | **State Visitation / Occupancy Measure** |  | Frequency of visiting each state | All algorithms (analysis) |
| **Advanced / Misc** | **Learning Curve** |  | Average episodic return vs. episodes / steps | Standard performance reporting |
| **Advanced / Misc** | **Regret / Cumulative Regret** |  | Sub-optimality accumulated | Bandits and online RL |
| **Advanced / Misc** | **Attention Mechanisms (Transformers in RL)** |  | Attention weights | Decision Transformer, Trajectory Transformer |
| **Advanced / Misc** | **Diffusion Policy** |  | Denoising diffusion process for action generation | Diffusion-RL policies |
| **Advanced / Misc** | **Graph Neural Networks for RL** |  | Node/edge message passing | Graph RL, relational RL |
| **Advanced / Misc** | **World Model / Latent Space** |  | Encoder-decoder dynamics in latent space | Dreamer, PlaNet |
| **Advanced / Misc** | **Convergence Analysis Plots** |  | Error / value change over iterations | DP, TD, value iteration |
| **Advanced / Misc** | **RL Algorithm Taxonomy** |  | Comprehensive classification of algorithms | All RL |
| **Advanced / Misc** | **Probabilistic Graphical Model (RL as Inference)** |  | Formalizing RL as probabilistic inference | Control as Inference, MaxEnt RL |
| **Value & Policy** | **Distributional RL (C51 / Categorical)** |  | Representing return as a probability distribution | C51, QR-DQN, IQN |
| **Exploration** | **Hindsight Experience Replay (HER)** |  | Learning from failures by relabeling goals | Sparse reward robotics, HER |
| **Model-Based RL** | **Dyna-Q Architecture** |  | Integration of real experience and model-based planning | Dyna-Q, Dyna-2 |
| **Function Approximation** | **Noisy Networks (Parameter Noise)** |  | Stochastic weights for exploration | Noisy DQN, Rainbow |
| **Exploration** | **Intrinsic Curiosity Module (ICM)** |  | Reward based on prediction error | Curiosity-driven exploration, ICM |
| **Temporal Difference** | **V-trace (IMPALA)** |  | Asynchronous off-policy importance sampling | IMPALA, V-trace |
| **Multi-Agent RL** | **QMIX Mixing Network** |  | Monotonic value function factorization | QMIX, VDN |
| **Advanced / Misc** | **Saliency Maps / Attention on State** |  | Visualizing what the agent "sees" or prioritizes | Interpretability, Atari RL |
| **Exploration** | **Action Selection Noise (OU vs Gaussian)** |  | Temporal correlation in exploration noise | DDPG, TD3 |
| **Advanced / Misc** | **t-SNE / UMAP State Embeddings** |  | Dimension reduction of high-dim neural states | Interpretability, SRL |
| **Advanced / Misc** | **Loss Landscape Visualization** |  | Optimization surface geometry | Training stability analysis |
| **Advanced / Misc** | **Success Rate vs Steps** |  | Percentage of successful episodes | Goal-conditioned RL, Robotics |
| **Advanced / Misc** | **Hyperparameter Sensitivity Heatmap** |  | Performance across parameter grids | Hyperparameter tuning |
| **Dynamics** | **Action Persistence (Frame Skipping)** |  | Temporal abstraction by repeating actions | Atari RL, Robotics |
| **Model-Based RL** | **MuZero Dynamics Search Tree** |  | Planning with learned transition and value functions | MuZero, Gumbel MuZero |
| **Deep RL** | **Policy Distillation** |  | Compressing knowledge from teacher to student | Kickstarting, multitask learning |
| **Transformers** | **Decision Transformer Token Sequence** |  | Sequential modeling of RL as a translation task | Decision Transformer, TT |
| **Advanced / Misc** | **Performance Profiles (rliable)** |  | Robust aggregate performance metrics | Reliable RL evaluation |
| **Safety RL** | **Safety Shielding / Barrier Functions** |  | Hard constraints on the action space | Constrained MDPs, Safe RL |
| **Training** | **Automated Curriculum Learning** |  | Progressively increasing task difficulty | Curriculum RL, ALP-GMM |
| **Sim-to-Real** | **Domain Randomization** |  | Generalizing across environment variations | Robotics, Sim-to-Real |
| **Alignment** | **RL with Human Feedback (RLHF)** |  | Aligning agents with human preferences | ChatGPT, InstructGPT |
| **Neuro-inspired RL** | **Successor Representation (SR)** |  | Predictive state representations | SR-Dyna, Neuro-RL |
| **Inverse RL / IRL** | **Maximum Entropy IRL** |  | Probability distribution over trajectories | MaxEnt IRL, Ziebart |
| **Theory** | **Information Bottleneck** |  | Mutual information $I(S;Z)$ and $I(Z;A)$ balance | VIB-RL, Information Theory |
| **Evolutionary RL** | **Evolutionary Strategies Population** |  | Population-based parameter search | OpenAI-ES, Salimans |
| **Safety RL** | **Control Barrier Functions (CBF)** |  | Set-theoretic safety guarantees | CBF-RL, Control Theory |
| **Exploration** | **Count-based Exploration Heatmap** |  | Visitation frequency and intrinsic bonus | MBIE-EB, RND |
| **Exploration** | **Thompson Sampling Posteriors** |  | Direct uncertainty-based action selection | Bandits, Bayesian RL |
| **Multi-Agent RL** | **Adversarial RL Interaction** |  | Competition between protaganist and antagonist | Robust RL, RARL |
| **Hierarchical RL** | **Hierarchical Subgoal Trajectory** |  | Decomposing long-horizon tasks | Subgoal RL, HIRO |
| **Offline RL** | **Offline Action Distribution Shift** |  | Mismatch between dataset and current policy | CQL, IQL, D4RL |
| **Exploration** | **Random Network Distillation (RND)** |  | Prediction error as intrinsic reward | RND, OpenAI |
| **Offline RL** | **Batch-Constrained Q-learning (BCQ)** |  | Constraining actions to behavior dataset | BCQ, Fujimoto |
| **Training** | **Population-Based Training (PBT)** |  | Evolutionary hyperparameter optimization | PBT, DeepMind |
| **Deep RL** | **Recurrent State Flow (DRQN/R2D2)** |  | Temporal dependency in state-action value | DRQN, R2D2 |
| **Theory** | **Belief State in POMDPs** |  | Probability distribution over hidden states | POMDPs, Belief Space |
| **Multi-Objective RL** | **Multi-Objective Pareto Front** |  | Balancing conflicting reward signals | MORL, Pareto Optimal |
| **Theory** | **Differential Value (Average Reward RL)** |  | Values relative to average gain | Average Reward RL, Mahadevan |
| **Infrastructure** | **Distributed RL Cluster (Ray/RLLib)** |  | Parallelizing experience collection | Ray, RLLib, Ape-X |
| **Evolutionary RL** | **Neuroevolution Topology Evolution** |  | Evolving neural network architectures | NEAT, HyperNEAT |
| **Continual RL** | **Elastic Weight Consolidation (EWC)** |  | Preventing catastrophic forgetting | EWC, Kirkpatric |
| **Theory** | **Successor Features (SF)** |  | Generalizing predictive representations | SF-Dyna, Barreto |
| **Safety** | **Adversarial State Noise (Perception)** |  | Attacks on agent observation space | Adversarial RL, Huang |
| **Imitation Learning** | **Behavioral Cloning (Imitation)** |  | Direct supervised learning from experts | BC, DAGGER |
| **Relational RL** | **Relational Graph State Representation** |  | Modeling objects and their relations | Relational MDPs, BoxWorld |
| **Quantum RL** | **Quantum RL Circuit (PQC)** |  | Gate-based quantum policy networks | Quantum RL, PQC |
| **Symbolic RL** | **Symbolic Policy Tree** |  | Policies as mathematical expressions | Symbolic RL, GP |
| **Control** | **Differentiable Physics Gradient Flow** |  | Gradient-based planning through simulators | Brax, Isaac Gym |
| **Multi-Agent RL** | **MARL Communication Channel** |  | Information exchange between agents | CommNet, DIAL |
| **Safety** | **Lagrangian Constraint Landscape** |  | Constrained optimization boundaries | Constrained RL, CPO |
| **Hierarchical RL** | **MAXQ Task Hierarchy** |  | Recursive task decomposition | MAXQ, Dietterich |
| **Agentic AI** | **ReAct Agentic Cycle** |  | Reasoning-Action loops for LLMs | ReAct, Agentic LLM |
| **Bio-inspired RL** | **Synaptic Plasticity RL** |  | Hebbian-style synaptic weight updates | Hebbian RL, STDP |
| **Control** | **Guided Policy Search (GPS)** |  | Distilling trajectories into a policy | GPS, Levine |
| **Robotics** | **Sim-to-Real Jitter & Latency** |  | Temporal robustness in transfer | Sim-to-Real, Robustness |
| **Policy Gradients** | **Deterministic Policy Gradient (DDPG) Flow** |  | Gradient flow for deterministic policies | DDPG |
| **Model-Based RL** | **Dreamer Latent Imagination** |  | Learning and planning in latent space | Dreamer (V1-V3) |
| **Deep RL** | **UNREAL Auxiliary Tasks** |  | Learning from non-reward signals | UNREAL, A3C extension |
| **Offline RL** | **Implicit Q-Learning (IQL) Expectile** |  | In-sample learning via expectile regression | IQL |
| **Model-Based RL** | **Prioritized Sweeping** |  | Planning prioritized by TD error | Sutton & Barto classic MBRL |
| **Imitation Learning** | **DAgger Expert Loop** |  | Training on expert labels in agent-visited states | DAgger |
| **Representation** | **Self-Predictive Representations (SPR)** |  | Consistency between predicted and target latents | SPR, sample-efficient RL |
| **Multi-Agent RL** | **Joint Action Space** |  | Cartesian product of individual actions | MARL theory, Game Theory |
| **Multi-Agent RL** | **Dec-POMDP Formal Model** |  | Decentralized partially observable MDP | Multi-agent coordination |
| **Theory** | **Bisimulation Metric** |  | State equivalence based on transitions/rewards | State abstraction, bisimulation theory |
| **Theory** | **Potential-Based Reward Shaping** |  | Reward transformation preserving optimal policy | Sutton & Barto, Ng et al. |
| **Training** | **Transfer RL: Source to Target** |  | Reusing knowledge across different MDPs | Transfer Learning, Distillation |
| **Deep RL** | **Multi-Task Backbone Arch** |  | Single agent learning multiple tasks | Multi-task RL, IMPALA |
| **Bandits** | **Contextual Bandit Pipeline** |  | Decision making given context but no transitions | Personalization, Ad-tech |
| **Theory** | **Theoretical Regret Bounds** |  | Analytical performance guarantees | Online Learning, Bandits |
| **Value-based** | **Soft Q Boltzmann Probabilities** |  | Probabilistic action selection from Q-values | s) \propto \exp(Q/\tau)$ |
| **Robotics** | **Autonomous Driving RL Pipeline** |  | End-to-end or modular driving stack | Wayve, Tesla, Comma.ai |
| **Policy** | **Policy action gradient comparison** |  | Comparison of gradient derivation types | PG Theorem vs DPG Theorem |
| **Inverse RL / IRL** | **IRL: Feature Expectation Matching** |  | Comparing expert vs learner feature visitor frequency | \mu(\pi^*) - \mu(\pi) |
| **Imitation Learning** | **Apprenticeship Learning Loop** |  | Training to match expert performance via reward inference | Apprenticeship Learning |
| **Theory** | **Active Inference Loop** |  | Agents minimizing surprise (free energy) | Free Energy Principle, Friston |
| **Theory** | **Bellman Residual Landscape** |  | Training surface of the Bellman error | TD learning, fitted Q-iteration |
| **Model-Based RL** | **Plan-to-Explore Uncertainty Map** |  | Systematic exploration in learned world models | Plan-to-Explore, Sekar et al. |
| **Safety RL** | **Robust RL Uncertainty Set** |  | Optimizing for the worst-case environment transition | Robust MDPs, minimax RL |
| **Training** | **HPO Bayesian Opt Cycle** |  | Automating hyperparameter selection with GP | Hyperparameter Optimization |
| **Applied RL** | **Slate RL Recommendation** |  | Optimizing list/slate of items for users | Recommender Systems, Ie et al. |
| **Multi-Agent RL** | **Fictitious Play Interaction** |  | Belief-based learning in games | Game Theory, Brown (1951) |
| **Conceptual** | **Universal RL Framework Diagram** |  | High-level summary of RL components | All RL |
| **Offline RL** | **Offline Density Ratio Estimator** |  | Estimating $w(s,a)$ for off-policy data | Importance Sampling, Offline RL |
| **Continual RL** | **Continual Task Interference Heatmap** |  | Measuring negative transfer between tasks | Lifelong Learning, EWC |
| **Safety RL** | **Lyapunov Stability Safe Set** |  | Invariant sets for safe control | Lyapunov RL, Chow et al. |
| **Applied RL** | **Molecular RL (Atom Coordinates)** |  | RL for molecular design/protein folding | Chemistry RL, AlphaFold-style |
| **Architecture** | **MoE Multi-task Architecture** |  | Scaling models with mixture of experts | MoE-RL, Sparsity |
| **Direct Policy Search** | **CMA-ES Policy Search** |  | Evolutionary strategy for policy weights | ES for RL, Salimans |
| **Alignment** | **Elo Rating Preference Plot** |  | Measuring agent strength over time | AlphaZero, League training |
| **Explainable RL** | **Explainable RL (SHAP Attribution)** |  | Local attribution of features to agent actions | Interpretability, SHAP/LIME |
| **Meta-RL** | **PEARL Context Encoder** |  | Learning latent task representations | PEARL, Rakelly et al. |
| **Applied RL** | **Medical RL Therapy Pipeline** |  | Personalized medicine and dosing | Healthcare RL, ICU Sepsis |
| **Applied RL** | **Supply Chain RL Pipeline** |  | Optimizing stock levels and orders | Logistics, Inventory Management |
| **Robotics** | **Sim-to-Real SysID Loop** |  | Closing the reality gap via parameter estimation | System Identification, Robotics |
| **Architecture** | **Transformer World Model** |  | Sequence-to-sequence dynamics modeling | DreamerV3, Transframer |
| **Applied RL** | **Network Traffic RL** |  | Optimizing data packet routing in graphs | Networking, Traffic Engineering |
| **Training** | **RLHF: PPO with Reference Policy** |  | Ensuring RL fine-tuning doesn't drift too far | InstructGPT, Llama 2/3 |
| **Multi-Agent RL** | **PSRO Meta-Game Update** |  | Reaching Nash equilibrium in large games | PSRO, Lanctot et al. |
| **Multi-Agent RL** | **DIAL: Differentiable Comm** |  | End-to-end learning of communication protocols | DIAL, Foerster et al. |
| **Batch RL** | **Fitted Q-Iteration Loop** |  | Data-driven iteration with a supervised regressor | Ernst et al. (2005) |
| **Safety RL** | **CMDP Feasible Region** |  | Constrained optimization within a safety budget | Constrained MDPs, Altman |
| **Control** | **MPC vs RL Planning** |  | Comparison of control paradigms | Control Theory vs RL |
| **AutoML** | **Learning to Optimize (L2O)** |  | Using RL to learn an optimization update rule | L2O, Li & Malik |
| **Applied RL** | **Smart Grid RL Management** |  | Optimizing energy supply and demand | Energy RL, Smart Grids |
| **Applied RL** | **Quantum State Tomography RL** |  | RL for quantum state estimation | Quantum RL, Neural Tomography |
| **Applied RL** | **RL for Chip Placement** |  | Placing components on silicon grids | Google Chip Placement |
| **Applied RL** | **RL Compiler Optimization (MLGO)** |  | Inlining and sizing in compilers | MLGO, LLVM |
| **Applied RL** | **RL for Theorem Proving** |  | Automated reasoning and proof search | LeanRL, AlphaProof |
| **Modern RL** | **Diffusion-QL Offline RL** |  | Policy as reverse diffusion process | s,k)$ with noise injection |
| **Principles** | **Fairness-reward Pareto Frontier** |  | Balancing equity and returns | Fair RL, Jabbari et al. |
| **Principles** | **Differentially Private RL** |  | Privacy-preserving training | DP-RL, Agarwal et al. |
| **Applied RL** | **Smart Agriculture RL** |  | Optimizing crop yield and resources | Precision Agriculture |
| **Applied RL** | **Climate Mitigation RL (Grid)** |  | Environmental control policies | ClimateRL, Carbon Control |
| **Applied RL** | **AI Education (Knowledge Tracing)** |  | Personalized learning paths | ITS, Bayesian Knowledge Tracing |
| **Modern RL** | **Decision SDE Flow** |  | RL in continuous stochastic systems | Neural SDEs, Control |
| **Control** | **Differentiable physics (Brax)** |  | Gradients through simulators | Brax, PhysX, MuJoCo |
| **Applied RL** | **Wireless Beamforming RL** |  | Optimizing antenna signal directions | 5G/6G Networking |
| **Applied RL** | **Quantum Error Correction RL** |  | Correcting noise in quantum circuits | Quantum Computing RL |
| **Multi-Agent RL** | **Mean Field RL Interaction** |  | Large population agent dynamics | MF-RL, Yang et al. |
| **HRL** | **Goal-GAN Curriculum** |  | Automatic goal generation | Goal-GAN, Florensa et al. |
| **Modern RL** | **JEPA: Predictive Architecture** |  | LeCun's world model framework | JEPA, I-JEPA |
| **Offline RL** | **CQL Value Penalty Landscape** |  | Conservatism in value functions | CQL, Kumar et al. |
| **Applied RL** | **Causal RL** |  | Causal Inverse RL Graph | DAG with $S, A, R$ and latent $U$ |
| **Quantum RL** | **VQE-RL Optimization** |  | Quantum circuit param tuning | VQE, Quantum RL |
| **Applied RL** | **De-novo Drug Discovery RL** |  | Generating optimized lead molecules | Drug Discovery, Molecule RL |
| **Applied RL** | **Traffic Signal Coordination RL** |  | Multi-intersection coordination | IntelliLight, PressLight |
| **Applied RL** | **Mars Rover Pathfinding RL** |  | Navigation on rough terrain | Space RL, Mars Rover |
| **Applied RL** | **Sports Player Movement RL** |  | Predicting/Optimizing player actions | Sports Analytics, Ghosting |
| **Applied RL** | **Cryptography Attack RL** |  | Searching for keys/vulnerabilities | Crypto-RL, Learning to Attack |
| **Applied RL** | **Humanitarian Resource RL** |  | Disaster response allocation | AI for Good, Resource RL |
| **Applied RL** | **Video Compression RL (RD)** |  | Optimizing bit-rate vs distortion | Learned Video Compression |
| **Applied RL** | **Kubernetes Auto-scaling RL** |  | Cloud resource management | Cloud RL, K8s Scaling |
| **Applied RL** | **Fluid Dynamics Flow Control RL** |  | Airfoil/Turbulence control | Aero-RL, Flow Control |
| **Applied RL** | **Structural Optimization RL** |  | Topology/Material design | Structural RL, Topology Opt |
| **Applied RL** | **Human Decision Modeling** |  | Prospect Theory in RL | Behavioral RL, Prospect Theory |
| **Applied RL** | **Semantic Parsing RL** |  | Language to Logic transformation | Semantic Parsing, Seq2Seq-RL |
| **Applied RL** | **Music Melody RL** |  | Reward-based melody generation | Music-RL, Magenta |
| **Applied RL** | **Plasma Fusion Control RL** |  | Magnetic control of Tokamaks | DeepMind Fusion, Tokamak RL |
| **Applied RL** | **Carbon Capture RL cycle** |  | Adsorption/Desorption optimization | Carbon Capture, Green RL |
| **Applied RL** | **Swarm Robotics RL** |  | Decentralized swarm coordination | Swarm-RL, Multi-Robot |
| **Applied RL** | **Legal Compliance RL Game** |  | Regulatory games | Legal-RL, RegTech |
| **Physics RL** | **Physics-Informed RL (PINN)** |  | Constraint-based RL loss | PINN-RL, SciML |
| **Modern RL** | **Neuro-Symbolic RL** |  | Combining logic and neural nets | Neuro-Symbolic, Logic RL |
| **Applied RL** | **DeFi Liquidity Pool RL** |  | Yield farming/Liquidity balancing | DeFi-RL, AMM Optimization |
| **Neuro RL** | **Dopamine Reward Prediction Error** |  | Biological RL signal curves | Neuroscience-RL, Wolfram |
| **Robotics** | **Proprioceptive Sensory-Motor RL** |  | Low-level joint control | Proprioceptive RL, Unitree |
| **Applied RL** | **AR Object Placement RL** |  | AR visual overlay optimization | AR-RL, Visual Overlay |
| **Reco RL** | **Sequential Bundle RL** |  | Recommendation item grouping | Bundle-RL, E-commerce |
| **Theoretical** | **Online Gradient Descent vs RL** |  | Gradient-based learning comparison | Online Learning, Regret |
| **Modern RL** | **Active Learning: Query RL** |  | Query-based sample selection | Active-RL, Query Opt |
| **Modern RL** | **Federated RL global Aggregator** |  | Privacy-preserving distributed RL | Federated-RL, FedAvg-RL |
| **Conceptual** | **Ultimate Universal RL Mastery Diagram** |  | Final summary of 230 items | Absolute Mastery Milestone |
|