File size: 3,128 Bytes
c62b9e8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 | {
"schema_version": 1,
"title": "Reproduction: Finite-time Convergence Analysis of Actor-Critic with Evolving Reward",
"emoji": "🎯",
"space_id": "snaykey/repro-actor-critic-evolving",
"paper": {
"openreview_id": "hocdjAEAIm"
},
"tags": [
"icml2026-repro",
"paper-hocdjAEAIm"
],
"updated_at": "2026-08-02T00:00:00+00:00",
"root": {
"slug": "index",
"title": "Reproduction: Finite-time Convergence Analysis of Actor-Critic with Evolving Reward",
"file": "pages/index.md",
"children": [
{
"slug": "claim-1-thm46-finite-time-bounds",
"title": "Theorem 4.6 establishes finite-time bounds for single-timescale actor-critic with evolving rewards, with actor error G_T and critic error W_T each bounded by O(1/√T) + O(F_T·√T) + O(√(F_T/T)) + O(ε), where F_T measures cumulative reward-parameter variation over T iterations (Section 4.2).",
"file": "pages/claim-1-thm46-finite-time-bounds/page.md",
"children": []
},
{
"slug": "claim-2-cor47-gradient-reward-rate",
"title": "Corollary 4.7 shows that when the reward parameter is updated via gradient-based steps with step size η^φ_t = c_φ/t and bounded gradients, F_T = O(1/T), so the bounds reduce to G_T = O(1/√T) + O(ε) and W_T = O(1/√T) + O(ε), matching the best-known rate for static-reward actor-critic (Section 4.2).",
"file": "pages/claim-2-cor47-gradient-reward-rate/page.md",
"children": []
},
{
"slug": "claim-3-prop48-distribution-mismatch",
"title": "Proposition 4.8 bounds the distribution mismatch under Markovian sampling without dependence on the mixing time, improving prior convergence rates by a factor of log²T in the static-reward case (Section 4.3).",
"file": "pages/claim-3-prop48-distribution-mismatch/page.md",
"children": []
},
{
"slug": "claim-4-assumption41-sufficient-exploration",
"title": "Assumption 4.1 (sufficient exploration) requires the matrix A_θ to be negative definite with singular values bounded above by -λ, underpinning the actor-critic convergence analysis (Section 4, Assumption 4.1).",
"file": "pages/claim-4-assumption41-sufficient-exploration/page.md",
"children": []
},
{
"slug": "claim-5-algorithm1-single-timescale",
"title": "Algorithm 1 defines a single-timescale actor-critic scheme with step sizes η^θ_t = c_θ/√t and η^ω_t = c_ω/√t for policy and critic updates, alongside an arbitrary update rule for the evolving reward parameter φ_t (Section 3.3, Algorithm 1).",
"file": "pages/claim-5-algorithm1-single-timescale/page.md",
"children": []
},
{
"slug": "executive-summary",
"title": "executive-summary",
"file": "pages/executive-summary/page.md",
"children": []
},
{
"slug": "conclusion",
"title": "conclusion",
"file": "pages/conclusion/page.md",
"children": []
}
]
}
} |