ai/OPTIMIZATION_IDEAS.md

AI Training Optimization Roadmap

This document outlines potential strategies to further accelerate training throughput, focusing on optimizations that require significant refactoring or architectural changes.

1. GPU-Resident Environment (The "Isaac Gym" Approach)

Impact: High (Potential 5-10x speedup for large batches) Difficulty: High

Currently, the VectorEnv runs on CPU (Numba), and observations are copied to the GPU for the Policy Network. This CPU -> GPU transfer becomes a bottleneck at high throughputs (e.g., >100k SPS).

• Proposal: Port the entire logic in ai/vector_env.py and engine/game/fast_logic.py to Numba CUDA or CuPy.
• Result: The environment state remains on the GPU. step() returns a GPU tensor directly, which is fed into the Policy Network without transfer.
• Challenges: requires rewriting Numba CPU kernels to Numba CUDA kernels (handling thread divergence, shared memory, etc.).
• Status: [FEASIBILITY ANALYSIS COMPLETE]. See ai/GPU_MIGRATION_GUIDE.md and ai/cuda_proof_of_concept.py for the architectural blueprint.

2. Pure Numba Adapter & Zero-Copy Interface

Impact: Medium (10-20% speedup) Difficulty: Medium

The VectorEnvAdapter currently performs some Python-level logic in step_wait (reward calculation, array copying, info dictionary construction).

• Proposal: Move the reward calculation (delta_scores * 50 - 5) and "Auto-Reset" logic into the Numba VectorGameState class.
• Result: step_wait becomes a thin wrapper that just returns views of the underlying Numba arrays.
• Refinement: Use the __array_interface__ or blind pointer passing to avoid any numpy array allocation overhead in Python.

3. Observation Compression & Quantization

Impact: Medium (Reduced memory bandwidth, larger batch sizes) Difficulty: Low/Medium

The observation space is 8192 floats (float32). This is 32KB per environment per step. For 256 envs, that's 8MB per step.

• Proposal: Most features are binary (0/1) or small integers.
  • Return observations as uint8 or float16.
  • Use a custom SB3 FeaturesExtractor to cast to float32 only inside the GPU network.
• Benefit: Reduces memory bandwidth between CPU and GPU by 4x (float32 -> uint8).

4. Incremental Action Masking

Impact: Low/Medium Difficulty: Medium

compute_action_masks scans the entire hand every step.

• Proposal: Maintain the action mask as part of the persistent state.
  • Only update the mask when the state changes (e.g., Card Played, Energy Charged).
  • Most steps (e.g., Opponent Turn simulation) might not change the Agent's legal actions if the Agent is waiting? (Actually, Agent acts every step in this setup).
  • Optimization: If a card was illegal last step and state hasn't changed relevantly (e.g. energy), it's still illegal. This is hard to prove correct.

5. Opponent Distillation / Caching

Impact: Medium (Depends on Opponent Complexity) Difficulty: High

If we move to smarter opponents (e.g., MCTS or Neural Net based), step_opponent_vectorized will become the bottleneck.

• Proposal:
  • Distillation: Train a tiny decision tree or small MLP to mimic the smart opponent and run it via Numba inference.
  • Caching: Pre-calculate opponent moves for common states? (Input space too large).

6. Asynchronous Environment Stepping (Pipelining)

Impact: Medium Difficulty: Medium

While the GPU is performing the Forward/Backward pass (Policy Update), the CPU is idle.

• Proposal: Run VectorEnv.step() in a separate thread/process while the GPU trains on the previous batch.
• Note: SB3's SubprocVecEnv tries this, but IPC overhead kills it. We need a Threaded Numba environment (releasing GIL) to do this efficiently in one process. Numba's @njit(nogil=True) enables this.

7. Memory Layout Optimization (AoS vs SoA)

Impact: Low/Medium Difficulty: High (Refactor hell)

Current layout mixes Structure of Arrays (SoA) and Arrays of Structures (AoS).

• Proposal: Ensure all hot arrays (batch_global_ctx, batch_scores) are contiguous in memory for the exact access pattern used by step_vectorized.
• Check: Access batch_global_ctx[i, :] vs batch_global_ctx[:, k]. Numba prefers loop-invariant access.