repro-bundle/docs/setup.md

# Paper's numerical experiment setup (Appendix D)

MDP: S = {0,1,2}; N = 6 state-action pairs, order:
(0,stay),(0,leave),(1,stay),(1,leave),(2,left),(2,right)

Reward r = (0.001, 0.5, 0.001, 0.5, 1, 1)

True transition P (rows = state-action, cols = next state 0,1,2):
  (0,stay):  [1,   0,   0  ]
  (0,leave): [0.4, 0,   0.6]
  (1,stay):  [0,   1,   0  ]
  (1,leave): [0,   0.4, 0.6]
  (2,left):  [0.4, 0.4, 0.2]
  (2,right): [0.2, 0.2, 0.6]

Settings: γ = 0.5, ε = 0.05, initial dist q = (0.4, 0.4, 0.2). Vary samples T from 100 to 16000.

Inaccurate prediction P̂ (OpPMD-NAC):
  [0,1,0],[0,1,0],[1,0,0],[1,0,0],[1,0,0],[0,1,0]   ("Dist(P,P̂) ~ 1" per paper)

Three algorithms compared:
1. OpPMD-AC:  Algorithm 1 with accurate prediction P̂ = P
2. OpPMD-NAC: Algorithm 1 with inaccurate P̂ above
3. SMD-DMDP-JINSID: Jin & Sidford [2020] Algorithm 1 (no prediction) — baseline

Metrics: Figure 1 = Duality Gap GAP(μ̄, v̄); Figure 2 = value function.
Optimal policy: leave (state 0), leave (state 1), right (state 2).