31,700.0 37,150.0 31,530.0 35,520.0 Beam Rider 9 363.9 16,926.5 8,627.5 13,772.8 12,164.0 23,384.2 30,276.5 Berzerk 18 123.7 2,630.4 585.6 1,225.4 1,472.6 1,305.6 3,409.0 Bowling 6 23.1 160.7 50.4 68.1 65.5 47.9 46.7 Boxing 18 0.1 12.1 88.0 91.6 99.4 95.6 98.9 Breakout 4 1.7 30.5 385.5 418.5 345.3 373.9 366.0 Centipede 18 2,090.9 12,017.0 4,657.7 5,409.4 7,561.4 4,463.2 7,687.5 Chopper Command 18 811.0 7,387.8 6,126.0 5,809.0 11,215.0 8,600.0 13,185.0 Crazy Climber 9 10,780.5 35,829.4 110,763.0 117,282.0 143,570.0 141,161.0 162,224.0 Defender 18 2,874.5 18,688.9 23,633.0 35,338.5 42,214.0 31,286.5 41,324.5 Demon Attack 6 152.1 1,971.0 12,149.4 58,044.2 60,813.3 71,846.4 72,878.6 Double Dunk 18 -18.6 -16.4 -6.6 -5.5 0.1 18.5 -12.5 Enduro 9 0.0 860.5 729.0 1,211.8 2,258.2 2,093.0 2,306.4 Fi


18 -18.6 -16.4 -6.6 -5.5 0.1 18.5 -12.5 Enduro 9 0.0 860.5 729.0 1,211.8 2,258.2 2,093.0 2,306.4 Fishing Derby 18 -91.7 -38.7 -4.9 15.5 46.4 39.5 41.3 Freeway 3 0.0 29.6 30.8 33.3 0.0 33.7 33.0 Frostbite 18 65.2 4,334.7 797.4 1,683.3 4,672.8 4,380.1 7,413.0 Gopher 8 257.6 2,412.5 8,777.4 14,840.8 15,718.4 32,487.2 104,368.2 Gravitar 18 173.0 3,351.4 473.0 412.0 588.0 548.5 238.0 H.E.R.O. 18 1,027.0 30,826.4 20,437.8 20,130.2 20,818.2 23,037.7 21,036.5 Ice Hockey 18 -11.2 0.9 -1.9 -2.7 0.5 1.3 -0.4 James Bond 18 29.0 302.8 768.5 1,358.0 1,312.5 5,148.0 812.0 Kangaroo 18 52.0 3,035.0 7,259.0 12,992.0 14,854.0 16,200.0 1,792.0 Krull 18 1,598.0 2,665.5 8,422.3 7,920.5 11,451.9 9,728.0 10,374.4 Kung-Fu Master 14 258.5 22,736.3 26,059.0 29,710.0 34,294.0 39,581.0 48,375.0 Montezuma’s Revenge 18


Kung-Fu Master 14 258.5 22,736.3 26,059.0 29,710.0 34,294.0 39,581.0 48,375.0 Montezuma’s Revenge 18 0.0 4,753.3 0.0 0.0 0.0 0.0 0.0 Ms. Pac-Man 9 307.3 6,951.6 3,085.6 2,711.4 6,283.5 6,518.7 3,327.3 Name This Game 6 2,292.3 8,049.0 8,207.8 10,616.0 11,971.1 12,270.5 15,572.5 Phoenix 8 761.4 7,242.6 8,485.2 12,252.5 23,092.2 18,992.7 70,324.3 Pitfall! 18 -229.4 6,463.7 -286.1 -29.9 0.0 -356.5 0.0 Pong 3 -20.7 14.6 19.5 20.9 21.0 20.6 20.9 Private Eye 18 24.9 69,571.3 146.7 129.7 103.0 200.0 206.0 Q*Bert 6 163.9 13,455.0 13,117.3 15,088.5 19,220.3 16,256.5 18,760.3 River Raid 18 1,338.5 17,118.0 7,377.6 14,884.5 21,162.6 14,522.3 20,607.6 Road Runner 18 11.5 7,845.0 39,544.0 44,127.0 69,524.0 57,608.0 62,151.0 Robotank 18 2.2 11.9 63.9 65.1 65.3 62.6 27.5 Seaquest 18 68.4 42,054.7 5,860.6


0 57,608.0 62,151.0 Robotank 18 2.2 11.9 63.9 65.1 65.3 62.6 27.5 Seaquest 18 68.4 42,054.7 5,860.6 16,452.7 50,254.2 26,357.8 931.6 Skiing 3 -17,098.1 -4,336.9 -13,062.3 -9,021.8 -8,857.4 -9,996.9 -19,949.9 Solaris 18 1,236.3 12,326.7 3,482.8 3,067.8 2,250.8 4,309.0 133.4 Space Invaders 6 148.0 1,668.7 1,692.3 2,525.5 6,427.3 2,865.8 15,311.5 Star Gunner 18 664.0 10,250.0 54,282.0 60,142.0 89,238.0 63,302.0 125,117.0 Surround 5 -10.0 6.5 -5.6 -2.9 4.4 8.9 1.2 Tennis 18 -23.8 -8.3 12.2 -22.8 5.1 0.0 0.0 Time Pilot 10 3,568.0 5,229.2 4,870.0 8,339.0 11,666.0 9,197.0 7,553.0 Tutankham 8 11.4 167.6 68.1 218.4 211.4 204.6 245.9 Up and Down 6 533.4 11,693.2 9,989.9 22,972.2 44,939.6 16,154.1 33,879.1 Venture 18 0.0 1,187.5 163.0 98.0 497.0 54.0 48.0 Video Pinball 9 16,256.9 17,667.9 196,760.4 3


79.1 Venture 18 0.0 1,187.5 163.0 98.0 497.0 54.0 48.0 Video Pinball 9 16,256.9 17,667.9 196,760.4 309,941.9 98,209.5 282,007.3 479,197.0 Wizard Of Wor 10 563.5 4,756.5 2,704.0 7,492.0 7,855.0 4,802.0 12,352.0 Yars’ Revenge 18 3,092.9 54,576.9 18,098.9 11,712.6 49,622.1 11,357.0 69,618.1 Zaxxon 18 32.5 9,173.3 5,363.0 10,163.0 12,944.0 10,469.0 13,886.0


Dueling Network Architectures for Deep Reinforcement Learning Table 3. Raw scores across all games. Starting with Human starts. GAMES NO. ACTIONS RANDOM HUMAN DQN DDQN DUEL PRIOR. PRIOR. DUEL. Alien 18 128.3 6,371.3 634.0 1,033.4 1,486.5 1,334.7 823.7 Amidar 10 11.8 1,540.4 178.4 169.1 172.7 129.1 238.4 Assault 7 166.9 628.9 3,489.3 6,060.8 3,994.8 6,548.9 10,950.6 Asterix 9 164.5 7,536.0 3,170.5 16,837.0 15,840.0 22,484.5 364,200.0 Asteroids 14 871.3 36,517.3 1,458.7 1,193.2 2,035.4 1,745.1 1,021.9 Atlantis 4 13,463.0 26,575.0 292,491.0 319,688.0 445,360.0 330,647.0 423,252.0 Bank Heist 18 21.7 644.5 312.7 886.0 1,129.3 876.6 1,004.6 Battle Zone 18 3,560.0 33,030.0 23,750.0 24,740.0 31,320.0 25,520.0 30,650.0 Beam Rider 9 254.6 14,961.0 9,743.2 17,417.2 14,591.3 31,181.3 37,412.2 Berzerk


0 25,520.0 30,650.0 Beam Rider 9 254.6 14,961.0 9,743.2 17,417.2 14,591.3 31,181.3 37,412.2 Berzerk 18 196.1 2,237.5 493.4 1,011.1 910.6 865.9 2,178.6 Bowling 6 35.2 146.5 56.5 69.6 65.7 52.0 50.4 Boxing 18 -1.5 9.6 70.3 73.5 77.3 72.3 79.2 Breakout 4 1.6 27.9 354.5 368.9 411.6 343.0 354.6 Centipede 18 1,925.5 10,321.9 3,973.9 3,853.5 4,881.0 3,489.1 5,570.2 Chopper Command 18 644.0 8,930.0 5,017.0 3,495.0 3,784.0 4,635.0 8,058.0 Crazy Climber 9 9,337.0 32,667.0 98,128.0 113,782.0 124,566.0 127,512.0 127,853.0 Defender 18 1,965.5 14,296.0 15,917.5 27,510.0 33,996.0 23,666.5 34,415.0 Demon Attack 6 208.3 3,442.8 12,550.7 69,803.4 56,322.8 61,277.5 73,371.3 Double Dunk 18 -16.0 -14.4 -6.0 -0.3 -0.8 16.0 -10.7 Enduro 9 -81.8 740.2 626.7 1,216.6 2,077.4 1,831.0 2,223.9 Fishing Derby 18 -77.1 5


-0.8 16.0 -10.7 Enduro 9 -81.8 740.2 626.7 1,216.6 2,077.4 1,831.0 2,223.9 Fishing Derby 18 -77.1 5.1 -1.6 3.2 -4.1 9.8 17.0 Freeway 3 0.1 25.6 26.9 28.8 0.2 28.9 28.2 Frostbite 18 66.4 4,202.8 496.1 1,448.1 2,332.4 3,510.0 4,038.4 Gopher 8 250.0 2,311.0 8,190.4 15,253.0 20,051.4 34,858.8 105,148.4 Gravitar 18 245.5 3,116.0 298.0 200.5 297.0 269.5 167.0 H.E.R.O. 18 1,580.3 25,839.4 14,992.9 14,892.5 15,207.9 20,889.9 15,459.2 Ice Hockey 18 -9.7 0.5 -1.6 -2.5 -1.3 -0.2 0.5 James Bond 18 33.5 368.5 697.5 573.0 835.5 3,961.0 585.0 Kangaroo 18 100.0 2,739.0 4,496.0 11,204.0 10,334.0 12,185.0 861.0 Krull 18 1,151.9 2,109.1 6,206.0 6,796.1 8,051.6 6,872.8 7,658.6 Kung-Fu Master 14 304.0 20,786.8 20,882.0 30,207.0 24,288.0 31,676.0 37,484.0 Montezuma’s Revenge 18 25.0 4,182.0 47.0 42.0 22.0 51.0


20,882.0 30,207.0 24,288.0 31,676.0 37,484.0 Montezuma’s Revenge 18 25.0 4,182.0 47.0 42.0 22.0 51.0 24.0 Ms. Pac-Man 9 197.8 15,375.0 1,092.3 1,241.3 2,250.6 1,865.9 1,007.8 Name This Game 6 1,747.8 6,796.0 6,738.8 8,960.3 11,185.1 10,497.6 13,637.9 Phoenix 8 1,134.4 6,686.2 7,484.8 12,366.5 20,410.5 16,903.6 63,597.0 Pitfall! 18 -348.8 5,998.9 -113.2 -186.7 -46.9 -427.0 -243.6 Pong 3 -18.0 15.5 18.0 19.1 18.8 18.9 18.4 Private Eye 18 662.8 64,169.1 207.9 -575.5 292.6 670.7 1,277.6 Q*Bert 6 183.0 12,085.0 9,271.5 11,020.8 14,175.8 9,944.0 14,063.0 River Raid 18 588.3 14,382.2 4,748.5 10,838.4 16,569.4 11,807.2 16,496.8 Road Runner 18 200.0 6,878.0 35,215.0 43,156.0 58,549.0 52,264.0 54,630.0 Robotank 18 2.4 8.9 58.7 59.1 62.0 56.2 24.7 Seaquest 18 215.5 40,425.8 4,216.7 14,498.0 37,361.6


0 Robotank 18 2.4 8.9 58.7 59.1 62.0 56.2 24.7 Seaquest 18 215.5 40,425.8 4,216.7 14,498.0 37,361.6 25,463.7 1,431.2 Skiing 3 -15,287.4 -3,686.6 -12,142.1 -11,490.4 -11,928.0 -10,169.1 -18,955.8 Solaris 18 2,047.2 11,032.6 1,295.4 810.0 1,768.4 2,272.8 280.6 Space Invaders 6 182.6 1,464.9 1,293.8 2,628.7 5,993.1 3,912.1 8,978.0 Star Gunner 18 697.0 9,528.0 52,970.0 58,365.0 90,804.0 61,582.0 127,073.0 Surround 5 -9.7 5.4 -6.0 1.9 4.0 5.9 -0.2 Tennis 18 -21.4 -6.7 11.1 -7.8 4.4 -5.3 -13.2 Time Pilot 10 3,273.0 5,650.0 4,786.0 6,608.0 6,601.0 5,963.0 4,871.0 Tutankham 8 12.7 138.3 45.6 92.2 48.0 56.9 108.6 Up and Down 6 707.2 9,896.1 8,038.5 19,086.9 24,759.2 12,157.4 22,681.3 Venture 18 18.0 1,039.0 136.0 21.0 200.0 94.0 29.0 Video Pinball 9 20,452.0 15,641.1 154,414.1 367,823.7 110,976.2 2


1,039.0 136.0 21.0 200.0 94.0 29.0 Video Pinball 9 20,452.0 15,641.1 154,414.1 367,823.7 110,976.2 295,972.8 447,408.6 Wizard Of Wor 10 804.0 4,556.0 1,609.0 6,201.0 7,054.0 5,727.0 10,471.0 Yars’ Revenge 18 1,476.9 47,135.2 4,577.5 6,270.6 25,976.5 4,687.4 58,145.9 Zaxxon 18 475.0 8,443.0 4,412.0 8,593.0 10,164.0 9,474.0 11,320.0


Dueling Network Architectures for Deep Reinforcement Learning Table 4. Normalized scores across all games. Starting with 30 no-op actions. GAMES DQN DDQN DUEL PRIOR. PRIOR. DUEL. Alien 20.2% 51.0% 61.4% 57.6% 53.8% Amidar 56.7% 104.3% 137.1% 107.0% 133.7% Assault 781.0% 995.1% 846.5% 1433.7% 2166.0% Asterix 50.0% 206.8% 337.4% 377.6% 4520.1% Asteroids 1.4% 0.0% 4.5% 4.1% 1.0% Atlantis 1651.2% 576.1% 2285.3% 2129.3% 2366.9% Bank Heist 59.7% 137.6% 216.2% 140.8% 201.5% Battle Zone 79.1% 84.2% 99.9% 83.8% 95.2% Beam Rider 49.9% 81.0% 71.2% 139.0% 180.6% Berzerk 18.4% 44.0% 53.8% 47.2% 131.1% Bowling 19.8% 32.7% 30.8% 18.0% 17.1% Boxing 732.5% 762.1% 827.1% 795.5% 823.1% Breakout 1334.5% 1449.2% 1194.5% 1294.3% 1266.6% Centipede 25.9% 33.4% 55.1% 23.9% 56.4% Chopper Command 80.8% 76.0% 158.2%


1194.5% 1294.3% 1266.6% Centipede 25.9% 33.4% 55.1% 23.9% 56.4% Chopper Command 80.8% 76.0% 158.2% 118.4% 188.1% Crazy Climber 399.1% 425.2% 530.1% 520.5% 604.6% Defender 131.3% 205.3% 248.8% 179.7% 243.1% Demon Attack 659.6% 3182.8% 3335.0% 3941.6% 3998.3% Double Dunk 557.7% 607.9% 866.5% 1723.3% 280.5% Enduro 84.7% 140.8% 262.4% 243.2% 268.0% Fishing Derby 163.8% 202.4% 260.7% 247.7% 251.1% Freeway 104.0% 112.5% 0.1% 114.0% 111.3% Frostbite 17.1% 37.9% 107.9% 101.1% 172.1% Gopher 395.4% 676.7% 717.5% 1495.6% 4831.3% Gravitar 9.4% 7.5% 13.1% 11.8% 2.0% H.E.R.O. 65.1% 64.1% 66.4% 73.9% 67.1% Ice Hockey 76.9% 70.0% 96.4% 103.2% 89.6% James Bond 270.1% 485.4% 468.8% 1869.8% 286.0% Kangaroo 241.6% 433.8% 496.2% 541.3% 58.3% Krull 639.3% 592.3% 923.1% 761.6% 822.2% Kung-Fu Master 114.8% 131.0


1.6% 433.8% 496.2% 541.3% 58.3% Krull 639.3% 592.3% 923.1% 761.6% 822.2% Kung-Fu Master 114.8% 131.0% 151.4% 174.9% 214.1% Montezuma’s Revenge 0.0% 0.0% 0.0% 0.0% 0.0% Ms. Pac-Man 41.8% 36.2% 89.9% 93.5% 45.5% Name This Game 102.8% 144.6% 168.1% 173.3% 230.7% Phoenix 119.2% 177.3% 344.5% 281.3% 1073.3% Pitfall! -0.8% 3.0% 3.4% -1.9% 3.4% Pong 114.0% 117.8% 118.2% 117.1% 118.0% Private Eye 0.2% 0.2% 0.1% 0.3% 0.3% Q*Bert 97.5% 112.3% 143.4% 121.1% 139.9% River Raid 38.3% 85.8% 125.6% 83.6% 122.1% Road Runner 504.7% 563.2% 887.4% 735.3% 793.3% Robotank 631.5% 643.7% 645.1% 617.5% 259.5% Seaquest 13.8% 39.0% 119.5% 62.6% 2.1% Skiing 31.6% 63.3% 64.6% 55.6% -22.3% Solaris 20.3% 16.5% 9.1% 27.7% -9.9% Space Invaders 101.6% 156.3% 412.9% 178.7% 997.2% Star Gunner 559.3% 620.5% 924.0% 653.4% 1298


-9.9% Space Invaders 101.6% 156.3% 412.9% 178.7% 997.2% Star Gunner 559.3% 620.5% 924.0% 653.4% 1298.3% Surround 26.5% 43.2% 86.9% 114.6% 67.6% Tennis 231.3% 6.8% 186.2% 153.2% 153.2% Time Pilot 78.4% 287.2% 487.5% 338.9% 239.9% Tutankham 36.3% 132.5% 128.1% 123.7% 150.1% Up and Down 84.7% 201.1% 397.9% 140.0% 298.8% Venture 13.7% 8.3% 41.9% 4.5% 4.0% Video Pinball 1113.7% 1754.3% 555.9% 1596.2% 2712.2% Wizard Of Wor 51.0% 165.2% 173.9% 101.1% 281.1% Yars’ Revenge 29.1% 16.7% 90.4% 16.1% 129.2% Zaxxon 58.3% 110.8% 141.3% 114.2% 151.6%


Dueling Network Architectures for Deep Reinforcement Learning Table 5. Normalized scores across all games. Starting with Human Starts. GAMES DQN DDQN DUEL PRIOR. PRIOR. DUEL. Alien 8.1% 14.5% 21.8% 19.3% 11.1% Amidar 10.9% 10.3% 10.5% 7.7% 14.8% Assault 719.2% 1275.9% 828.6% 1381.5% 2334.4% Asterix 40.8% 226.2% 212.7% 302.8% 4938.4% Asteroids 1.6% 0.9% 3.3% 2.5% 0.4% Atlantis 2128.0% 2335.5% 3293.9% 2419.0% 3125.3% Bank Heist 46.7% 138.8% 177.8% 137.3% 157.8% Battle Zone 68.5% 71.9% 94.2% 74.5% 91.9% Beam Rider 64.5% 116.7% 97.5% 210.3% 252.7% Berzerk 14.6% 39.9% 35.0% 32.8% 97.1% Bowling 19.2% 30.9% 27.5% 15.1% 13.7% Boxing 648.2% 677.0% 711.2% 666.7% 728.5% Breakout 1341.9% 1396.7% 1559.0% 1298.3% 1342.4% Centipede 24.4% 23.0% 35.2% 18.6% 43.4% Chopper Command 52.8% 34.4% 37.9% 48.2% 89.


1298.3% 1342.4% Centipede 24.4% 23.0% 35.2% 18.6% 43.4% Chopper Command 52.8% 34.4% 37.9% 48.2% 89.5% Crazy Climber 380.6% 447.7% 493.9% 506.5% 508.0% Defender 113.2% 207.2% 259.8% 176.0% 263.2% Demon Attack 381.6% 2151.6% 1734.8% 1888.0% 2261.9% Double Dunk 622.5% 982.5% 948.7% 1998.7% 328.7% Enduro 86.2% 158.0% 262.7% 232.7% 280.5% Fishing Derby 91.8% 97.7% 88.8% 105.7% 114.5% Freeway 105.1% 112.5% 0.6% 113.1% 110.2% Frostbite 10.4% 33.4% 54.8% 83.2% 96.0% Gopher 385.3% 727.9% 960.8% 1679.2% 5089.7% Gravitar 1.8% -1.6% 1.8% 0.8% -2.7% H.E.R.O. 55.3% 54.9% 56.2% 79.6% 57.2% Ice Hockey 79.2% 70.8% 82.4% 92.5% 99.4% James Bond 198.2% 161.0% 239.4% 1172.4% 164.6% Kangaroo 166.6% 420.8% 387.8% 457.9% 28.8% Krull 528.0% 589.7% 720.8% 597.7% 679.8% Kung-Fu Master 100.5% 146.0% 117.1% 153.2% 18


457.9% 28.8% Krull 528.0% 589.7% 720.8% 597.7% 679.8% Kung-Fu Master 100.5% 146.0% 117.1% 153.2% 181.5% Montezuma’s Revenge 0.5% 0.4% -0.1% 0.6% -0.0% Ms. Pac-Man 5.9% 6.9% 13.5% 11.0% 5.3% Name This Game 98.9% 142.9% 186.9% 173.3% 235.5% Phoenix 114.4% 202.3% 347.2% 284.0% 1125.1% Pitfall! 3.7% 2.6% 4.8% -1.2% 1.7% Pong 107.6% 110.9% 110.0% 110.1% 108.6% Private Eye -0.7% -1.9% -0.6% 0.0% 1.0% Q*Bert 76.4% 91.1% 117.6% 82.0% 116.6% River Raid 30.2% 74.3% 115.9% 81.3% 115.3% Road Runner 524.3% 643.2% 873.7% 779.6% 815.1% Robotank 863.3% 868.7% 913.3% 824.4% 341.1% Seaquest 10.0% 35.5% 92.4% 62.8% 3.0% Skiing 27.1% 32.7% 29.0% 44.1% -31.6% Solaris -8.4% -13.8% -3.1% 2.5% -19.7% Space Invaders 86.7% 190.8% 453.1% 290.8% 685.9% Star Gunner 591.9% 653.0% 1020.3% 689.4% 1431.0% Surround 24.7%


s 86.7% 190.8% 453.1% 290.8% 685.9% Star Gunner 591.9% 653.0% 1020.3% 689.4% 1431.0% Surround 24.7% 76.9% 91.1% 103.2% 62.9% Tennis 220.8% 92.1% 175.0% 109.6% 55.6% Time Pilot 63.7% 140.3% 140.0% 113.2% 67.2% Tutankham 26.2% 63.3% 28.1% 35.2% 76.4% Up and Down 79.8% 200.0% 261.8% 124.6% 239.1% Venture 11.6% 0.3% 17.8% 7.4% 1.1% Video Pinball 987.2% 2351.6% 709.5% 1892.3% 2860.5% Wizard Of Wor 21.5% 143.8% 166.6% 131.2% 257.6% Yars’ Revenge 6.8% 10.5% 53.7% 7.0% 124.1% Zaxxon 49.4% 101.9% 121.6% 112.9% 136.1% Mean 219.6% 332.9% 343.8% 386.7% 567.0% Median 68.5% 110.9% 117.1% 112.9% 115.3%


Asynchronous Methods for Deep Reinforcement Learning Volodymyr Mnih1 VMNIH@GOOGLE.COM Adrià Puigdomènech Badia1 ADRIAP@GOOGLE.COM Mehdi Mirza1,2 MIRZAMOM@IRO.UMONTREAL.CA Alex Graves1 GRAVESA@GOOGLE.COM Tim Harley1 THARLEY@GOOGLE.COM Timothy P. Lillicrap1 COUNTZERO@GOOGLE.COM David Silver1 DAVIDSILVER@GOOGLE.COM Koray Kavukcuoglu 1 KORAYK@GOOGLE.COM 1 Google DeepMind 2 Montreal Institute for Learning Algorithms (MILA), University of Montreal Abstract We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training a


nt learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input. 1. Introduction Deep neural networks provide rich representations that can enable reinforcement learning (RL) algorithms to perform effectively.


rich representations that can enable reinforcement learning (RL) algorithms to perform effectively. However, it was previously thought that the combination of simple online RL algorithms with deep neural networks was fundamentally unstable. Instead, a variety of solutions have been proposed to stabilize the algorithm (Riedmiller, 2005; Mnih et al., 2013; 2015; Van Hasselt et al., 2015; Schulman et al., 2015a). These approaches share a common idea: the sequence of observed data encountered by an online RL agent is non-stationary, and onProceedings of the 33 rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48. Copyright 2016 by the author(s). line RL updates are strongly correlated.


16. JMLR: W&CP volume 48. Copyright 2016 by the author(s). line RL updates are strongly correlated. By storing the agent’s data in an experience replay memory, the data can be batched (Riedmiller, 2005; Schulman et al., 2015a) or randomly sampled (Mnih et al., 2013; 2015; Van Hasselt et al., 2015) from different time-steps. Aggregating over memory in this way reduces non-stationarity and decorrelates updates, but at the same time limits the methods to off-policy reinforcement learning algorithms. Deep RL algorithms based on experience replay have achieved unprecedented success in challenging domains such as Atari 2600. However, experience replay has several drawbacks: it uses more memory and computation per real interaction; and it requires off-policy learning algorithms that can update fr


computation per real interaction; and it requires off-policy learning algorithms that can update from data generated by an older policy. In this paper we provide a very different paradigm for deep reinforcement learning. Instead of experience replay, we asynchronously execute multiple agents in parallel, on multiple instances of the environment. This parallelism also decorrelates the agents’ data into a more stationary process, since at any given time-step the parallel agents will be experiencing a variety of different states. This simple idea enables a much larger spectrum of fundamental on-policy RL algorithms, such as Sarsa, n-step methods, and actorcritic methods, as well as off-policy RL algorithms such as Q-learning, to be applied robustly and effectively using deep neural networks.


RL algorithms such as Q-learning, to be applied robustly and effectively using deep neural networks. Our parallel reinforcement learning paradigm also offers practical beneﬁts. Whereas previous approaches to deep reinforcement learning rely heavily on specialized hardware such as GPUs (Mnih et al., 2015; Van Hasselt et al., 2015; Schaul et al., 2015) or massively distributed architectures (Nair et al., 2015), our experiments run on a single machine with a standard multi-core CPU. When applied to a variety of Atari 2600 domains, on many games asynchronous reinforcement learning achieves better results, in far less arXiv:1602.01783v2 [cs.LG] 16 Jun 2016


Asynchronous Methods for Deep Reinforcement Learning time than previous GPU-based algorithms, using far less resource than massively distributed approaches. The best of the proposed methods, asynchronous advantage actorcritic (A3C), also mastered a variety of continuous motor control tasks as well as learned general strategies for exploring 3D mazes purely from visual inputs. We believe that the success of A3C on both 2D and 3D games, discrete and continuous action spaces, as well as its ability to train feedforward and recurrent agents makes it the most general and successful reinforcement learning agent to date. 2. Related Work The General Reinforcement Learning Architecture (Gorila) of (Nair et al., 2015) performs asynchronous training of reinforcement learning agents in a distributed s


air et al., 2015) performs asynchronous training of reinforcement learning agents in a distributed setting. In Gorila, each process contains an actor that acts in its own copy of the environment, a separate replay memory, and a learner that samples data from the replay memory and computes gradients of the DQN loss (Mnih et al., 2015) with respect to the policy parameters. The gradients are asynchronously sent to a central parameter server which updates a central copy of the model. The updated policy parameters are sent to the actor-learners at ﬁxed intervals. By using 100 separate actor-learner processes and 30 parameter server instances, a total of 130 machines, Gorila was able to significantly outperform DQN over 49 Atari games.


nces, a total of 130 machines, Gorila was able to significantly outperform DQN over 49 Atari games. On many games Gorila reached the score achieved by DQN over 20 times faster than DQN. We also note that a similar way of parallelizing DQN was proposed by (Chavez et al., 2015). In earlier work, (Li & Schuurmans, 2011) applied the Map Reduce framework to parallelizing batch reinforcement learning methods with linear function approximation. Parallelism was used to speed up large matrix operations but not to parallelize the collection of experience or stabilize learning. (Grounds & Kudenko, 2008) proposed a parallel version of the Sarsa algorithm that uses multiple separate actor-learners to accelerate training.


l version of the Sarsa algorithm that uses multiple separate actor-learners to accelerate training. Each actorlearner learns separately and periodically sends updates to weights that have changed signiﬁcantly to the other learners using peer-to-peer communication. (Tsitsiklis, 1994) studied convergence properties of Qlearning in the asynchronous optimization setting. These results show that Q-learning is still guaranteed to converge when some of the information is outdated as long as outdated information is always eventually discarded and several other technical assumptions are satisﬁed. Even earlier, (Bertsekas, 1982) studied the related problem of distributed dynamic programming. Another related area of work is in evolutionary methods, which are often straightforward to parallelize by di


elated area of work is in evolutionary methods, which are often straightforward to parallelize by distributing ﬁtness evaluations over multiple machines or threads (Tomassini, 1999). Such parallel evolutionary approaches have recently been applied to some visual reinforcement learning tasks. In one example, (Koutník et al., 2014) evolved convolutional neural network controllers for the TORCS driving simulator by performing ﬁtness evaluations on 8 CPU cores in parallel. 3. Reinforcement Learning Background We consider the standard reinforcement learning setting where an agent interacts with an environment E over a number of discrete time steps. At each time step t, the agent receives a state st and selects an action at from some set of possible actions A according to its policy π, where π i


st and selects an action at from some set of possible actions A according to its policy π, where π is a mapping from states st to actions at. In return, the agent receives the next state st+1 and receives a scalar reward rt. The process continues until the agent reaches a terminal state after which the process restarts. The return Rt = P∞ k=0 γkrt+k is the total accumulated return from time step t with discount factor γ ∈(0, 1]. The goal of the agent is to maximize the expected return from each state st. The action value Qπ(s, a) = E [Rt|st = s, a] is the expected return for selecting action a in state s and following policy π. The optimal value function Q∗(s, a) = maxπ Qπ(s, a) gives the maximum action value for state s and action a achievable by any policy.


) = maxπ Qπ(s, a) gives the maximum action value for state s and action a achievable by any policy. Similarly, the value of state s under policy π is deﬁned as V π(s) = E [Rt|st = s] and is simply the expected return for following policy π from state s. In value-based model-free reinforcement learning methods, the action value function is represented using a function approximator, such as a neural network. Let Q(s, a; θ) be an approximate action-value function with parameters θ. The updates to θ can be derived from a variety of reinforcement learning algorithms. One example of such an algorithm is Q-learning, which aims to directly approximate the optimal action value function: Q∗(s, a) ≈Q(s, a; θ).


arning, which aims to directly approximate the optimal action value function: Q∗(s, a) ≈Q(s, a; θ). In one-step Q-learning, the parameters θ of the action value function Q(s, a; θ) are learned by iteratively minimizing a sequence of loss functions, where the ith loss function deﬁned as Li(θi) = E  r + γ max a′ Q(s′, a′; θi−1) −Q(s, a; θi) 2 where s′ is the state encountered after state s. We refer to the above method as one-step Q-learning because it updates the action value Q(s, a) toward the onestep return r + γ maxa′ Q(s′, a′; θ). One drawback of using one-step methods is that obtaining a reward r only directly affects the value of the state action pair s, a that led to the reward. The values of other state action pairs are affected only indirectly through the updated value Q(s, a).


values of other state action pairs are affected only indirectly through the updated value Q(s, a). This can make the learning process slow since many updates are required the propagate a reward to the relevant preceding states and actions.


Asynchronous Methods for Deep Reinforcement Learning One way of propagating rewards faster is by using nstep returns (Watkins, 1989; Peng & Williams, 1996). In n-step Q-learning, Q(s, a) is updated toward the nstep return deﬁned as rt + γrt+1 + · · · + γn−1rt+n−1 + maxa γnQ(st+n, a). This results in a single reward r directly affecting the values of n preceding state action pairs. This makes the process of propagating rewards to relevant state-action pairs potentially much more efﬁcient. In contrast to value-based methods, policy-based modelfree methods directly parameterize the policy π(a|s; θ) and update the parameters θ by performing, typically approximate, gradient ascent on E[Rt]. One example of such a method is the REINFORCE family of algorithms due to Williams (1992).


n E[Rt]. One example of such a method is the REINFORCE family of algorithms due to Williams (1992). Standard REINFORCE updates the policy parameters θ in the direction ∇θ log π(at|st; θ)Rt, which is an unbiased estimate of ∇θE[Rt]. It is possible to reduce the variance of this estimate while keeping it unbiased by subtracting a learned function of the state bt(st), known as a baseline (Williams, 1992), from the return. The resulting gradient is ∇θ log π(at|st; θ) (Rt −bt(st)). A learned estimate of the value function is commonly used as the baseline bt(st) ≈V π(st) leading to a much lower variance estimate of the policy gradient. When an approximate value function is used as the baseline, the quantity Rt −bt used to scale the policy gradient can be seen as an estimate of the advantage of a


e quantity Rt −bt used to scale the policy gradient can be seen as an estimate of the advantage of action at in state st, or A(at, st) = Q(at, st)−V (st), because Rt is an estimate of Qπ(at, st) and bt is an estimate of V π(st). This approach can be viewed as an actor-critic architecture where the policy π is the actor and the baseline bt is the critic (Sutton & Barto, 1998; Degris et al., 2012). 4. Asynchronous RL Framework We now present multi-threaded asynchronous variants of one-step Sarsa, one-step Q-learning, n-step Q-learning, and advantage actor-critic. The aim in designing these methods was to ﬁnd RL algorithms that can train deep neural network policies reliably and without large resource requirements.


ithms that can train deep neural network policies reliably and without large resource requirements. While the underlying RL methods are quite different, with actor-critic being an on-policy policy search method and Q-learning being an off-policy value-based method, we use two main ideas to make all four algorithms practical given our design goal. First, we use asynchronous actor-learners, similarly to the Gorila framework (Nair et al., 2015), but instead of using separate machines and a parameter server, we use multiple CPU threads on a single machine. Keeping the learners on a single machine removes the communication costs of sending gradients and parameters and enables us to use Hogwild! (Recht et al., 2011) style updates for training.


ents and parameters and enables us to use Hogwild! (Recht et al., 2011) style updates for training. Second, we make the observation that multiple actorsAlgorithm 1 Asynchronous one-step Q-learning - pseudocode for each actor-learner thread. // Assume global shared θ, θ−, and counter T = 0. Initialize thread step counter t ←0 Initialize target network weights θ−←θ Initialize network gradients dθ ←0 Get initial state s repeat Take action a with ϵ-greedy policy based on Q(s, a; θ) Receive new state s′ and reward r y =  r for terminal s′ r + γ maxa′ Q(s′, a′; θ−) for non-terminal s′ Accumulate gradients wrt θ: dθ ←dθ + ∂(y−Q(s,a;θ))2 ∂θ s = s′ T ←T + 1 and t ←t + 1 if T mod Itarget == 0 then Update the target network θ−←θ end if if t mod IAsyncUpdate == 0 or s is terminal then Perform asynchr


date the target network θ−←θ end if if t mod IAsyncUpdate == 0 or s is terminal then Perform asynchronous update of θ using dθ. Clear gradients dθ ←0. end if until T > Tmax learners running in parallel are likely to be exploring different parts of the environment. Moreover, one can explicitly use different exploration policies in each actor-learner to maximize this diversity. By running different exploration policies in different threads, the overall changes being made to the parameters by multiple actor-learners applying online updates in parallel are likely to be less correlated in time than a single agent applying online updates. Hence, we do not use a replay memory and rely on parallel actors employing different exploration policies to perform the stabilizing role undertaken by experie


ctors employing different exploration policies to perform the stabilizing role undertaken by experience replay in the DQN training algorithm. In addition to stabilizing learning, using multiple parallel actor-learners has multiple practical beneﬁts. First, we obtain a reduction in training time that is roughly linear in the number of parallel actor-learners. Second, since we no longer rely on experience replay for stabilizing learning we are able to use on-policy reinforcement learning methods such as Sarsa and actor-critic to train neural networks in a stable way. We now describe our variants of one-step Qlearning, one-step Sarsa, n-step Q-learning and advantage actor-critic. Asynchronous one-step Q-learning: Pseudocode for our variant of Q-learning, which we call Asynchronous onestep Q-l


ne-step Q-learning: Pseudocode for our variant of Q-learning, which we call Asynchronous onestep Q-learning, is shown in Algorithm 1. Each thread interacts with its own copy of the environment and at each step computes a gradient of the Q-learning loss. We use a shared and slowly changing target network in computing the Q-learning loss, as was proposed in the DQN training method. We also accumulate gradients over multiple timesteps before they are applied, which is similar to us


Asynchronous Methods for Deep Reinforcement Learning ing minibatches. This reduces the chances of multiple actor learners overwriting each other’s updates. Accumulating updates over several steps also provides some ability to trade off computational efﬁciency for data efﬁciency. Finally, we found that giving each thread a different exploration policy helps improve robustness. Adding diversity to exploration in this manner also generally improves performance through better exploration. While there are many possible ways of making the exploration policies differ we experiment with using ϵ-greedy exploration with ϵ periodically sampled from some distribution by each thread. Asynchronous one-step Sarsa: The asynchronous onestep Sarsa algorithm is the same as asynchronous one-step Q-learning as


p Sarsa: The asynchronous onestep Sarsa algorithm is the same as asynchronous one-step Q-learning as given in Algorithm 1 except that it uses a different target value for Q(s, a). The target value used by one-step Sarsa is r + γQ(s′, a′; θ−) where a′ is the action taken in state s′ (Rummery & Niranjan, 1994; Sutton & Barto, 1998). We again use a target network and updates accumulated over multiple timesteps to stabilize learning. Asynchronous n-step Q-learning: Pseudocode for our variant of multi-step Q-learning is shown in Supplementary Algorithm S2. The algorithm is somewhat unusual because it operates in the forward view by explicitly computing nstep returns, as opposed to the more common backward view used by techniques like eligibility traces (Sutton & Barto, 1998).


to the more common backward view used by techniques like eligibility traces (Sutton & Barto, 1998). We found that using the forward view is easier when training neural networks with momentum-based methods and backpropagation through time. In order to compute a single update, the algorithm ﬁrst selects actions using its exploration policy for up to tmax steps or until a terminal state is reached. This process results in the agent receiving up to tmax rewards from the environment since its last update. The algorithm then computes gradients for n-step Q-learning updates for each of the state-action pairs encountered since the last update. Each n-step update uses the longest possible n-step return resulting in a one-step update for the last state, a two-step update for the second last state, a


rn resulting in a one-step update for the last state, a two-step update for the second last state, and so on for a total of up to tmax updates. The accumulated updates are applied in a single gradient step. Asynchronous advantage actor-critic: The algorithm, which we call asynchronous advantage actor-critic (A3C), maintains a policy π(at|st; θ) and an estimate of the value function V (st; θv). Like our variant of n-step Q-learning, our variant of actor-critic also operates in the forward view and uses the same mix of n-step returns to update both the policy and the value-function. The policy and the value function are updated after every tmax actions or when a terminal state is reached. The update performed by the algorithm can be seen as ∇θ′ log π(at|st; θ′)A(st, at; θ, θv) where A(st, at


update performed by the algorithm can be seen as ∇θ′ log π(at|st; θ′)A(st, at; θ, θv) where A(st, at; θ, θv) is an estimate of the advantage function given by Pk−1 i=0 γirt+i + γkV (st+k; θv) −V (st; θv), where k can vary from state to state and is upper-bounded by tmax. The pseudocode for the algorithm is presented in Supplementary Algorithm S3. As with the value-based methods we rely on parallel actorlearners and accumulated updates for improving training stability. Note that while the parameters θ of the policy and θv of the value function are shown as being separate for generality, we always share some of the parameters in practice. We typically use a convolutional neural network that has one softmax output for the policy π(at|st; θ) and one linear output for the value function V (st;


s one softmax output for the policy π(at|st; θ) and one linear output for the value function V (st; θv), with all non-output layers shared. We also found that adding the entropy of the policy π to the objective function improved exploration by discouraging premature convergence to suboptimal deterministic policies. This technique was originally proposed by (Williams & Peng, 1991), who found that it was particularly helpful on tasks requiring hierarchical behavior. The gradient of the full objective function including the entropy regularization term with respect to the policy parameters takes the form ∇θ′ log π(at|st; θ′)(Rt −V (st; θv)) + β∇θ′H(π(st; θ′)), where H is the entropy. The hyperparameter β controls the strength of the entropy regularization term.


re H is the entropy. The hyperparameter β controls the strength of the entropy regularization term. Optimization: We investigated three different optimization algorithms in our asynchronous framework – SGD with momentum, RMSProp (Tieleman & Hinton, 2012) without shared statistics, and RMSProp with shared statistics. We used the standard non-centered RMSProp update given by g = αg + (1 −α)∆θ2 and θ ←θ −η ∆θ √g + ϵ, (1) where all operations are performed elementwise. A comparison on a subset of Atari 2600 games showed that a variant of RMSProp where statistics g are shared across threads is considerably more robust than the other two methods. Full details of the methods and comparisons are included in Supplementary Section 7. 5.


wo methods. Full details of the methods and comparisons are included in Supplementary Section 7. 5. Experiments We use four different platforms for assessing the properties of the proposed framework. We perform most of our experiments using the Arcade Learning Environment (Bellemare et al., 2012), which provides a simulator for Atari 2600 games. This is one of the most commonly used benchmark environments for RL algorithms. We use the Atari domain to compare against state of the art results (Van Hasselt et al., 2015; Wang et al., 2015; Schaul et al., 2015; Nair et al., 2015; Mnih et al., 2015), as well as to carry out a detailed stability and scalability analysis of the proposed methods. We performed further comparisons using the TORCS 3D car racing simulator (Wymann et al., 2013).


ds. We performed further comparisons using the TORCS 3D car racing simulator (Wymann et al., 2013). We also use


Asynchronous Methods for Deep Reinforcement Learning 0 2 4 6 8 10 12 14 Training time (hours) 0 2000 4000 6000 8000 10000 12000 14000 16000 Score Beamrider DQN 1-step Q 1-step SARSA n-step Q A3C 0 2 4 6 8 10 12 14 Training time (hours) 0 100 200 300 400 500 600 Score Breakout DQN 1-step Q 1-step SARSA n-step Q A3C 0 2 4 6 8 10 12 14 Training time (hours) 30 20 10 0 10 20 30 Score Pong DQN 1-step Q 1-step SARSA n-step Q A3C 0 2 4 6 8 10 12 14 Training time (hours) 0 2000 4000 6000 8000 10000 12000 Score Q*bert DQN 1-step Q 1-step SARSA n-step Q A3C 0 2 4 6 8 10 12 14 Training time (hours) 0 200 400 600 800 1000 1200 1400 1600 Score Space Invaders DQN 1-step Q 1-step SARSA n-step Q A3C Figure 1. Learning speed comparison for DQN and the new asynchronous algorithms on ﬁve Atari 2600 games.


e 1. Learning speed comparison for DQN and the new asynchronous algorithms on ﬁve Atari 2600 games. DQN was trained on a single Nvidia K40 GPU while the asynchronous methods were trained using 16 CPU cores. The plots are averaged over 5 runs. In the case of DQN the runs were for different seeds with ﬁxed hyperparameters. For asynchronous methods we average over the best 5 models from 50 experiments with learning rates sampled from LogUniform(10−4, 10−2) and all other hyperparameters ﬁxed. two additional domains to evaluate only the A3C algorithm – Mujoco and Labyrinth. MuJoCo (Todorov, 2015) is a physics simulator for evaluating agents on continuous motor control tasks with contact dynamics. Labyrinth is a new 3D environment where the agent must learn to ﬁnd rewards in randomly generated m


Labyrinth is a new 3D environment where the agent must learn to ﬁnd rewards in randomly generated mazes from a visual input. The precise details of our experimental setup can be found in Supplementary Section 8. 5.1. Atari 2600 Games We ﬁrst present results on a subset of Atari 2600 games to demonstrate the training speed of the new methods. Figure 1 compares the learning speed of the DQN algorithm trained on an Nvidia K40 GPU with the asynchronous methods trained using 16 CPU cores on ﬁve Atari 2600 games. The results show that all four asynchronous methods we presented can successfully train neural network controllers on the Atari domain. The asynchronous methods tend to learn faster than DQN, with signiﬁcantly faster learning on some games, while training on only 16 CPU cores.


ter than DQN, with signiﬁcantly faster learning on some games, while training on only 16 CPU cores. Additionally, the results suggest that n-step methods learn faster than one-step methods on some games. Overall, the policy-based advantage actor-critic method signiﬁcantly outperforms all three value-based methods. We then evaluated asynchronous advantage actor-critic on 57 Atari games. In order to compare with the state of the art in Atari game playing, we largely followed the training and evaluation protocol of (Van Hasselt et al., 2015). Speciﬁcally, we tuned hyperparameters (learning rate and amount of gradient norm clipping) using a search on six Atari games (Beamrider, Breakout, Pong, Q*bert, Seaquest and Space Invaders) and then ﬁxed all hyperparameters for all 57 games.


out, Pong, Q*bert, Seaquest and Space Invaders) and then ﬁxed all hyperparameters for all 57 games. We trained both a feedforward agent with the same architecture as (Mnih et al., 2015; Nair et al., 2015; Van Hasselt et al., 2015) as well as a recurrent agent with an additional 256 LSTM cells after the ﬁnal hidden layer. We additionally used the ﬁnal network weights for evaluation to make the results more comparable to the original results Method Training Time Mean Median DQN 8 days on GPU 121.9% 47.5% Gorila 4 days, 100 machines 215.2% 71.3% D-DQN 8 days on GPU 332.9% 110.9% Dueling D-DQN 8 days on GPU 343.8% 117.1% Prioritized DQN 8 days on GPU 463.6% 127.6% A3C, FF 1 day on CPU 344.1% 68.2% A3C, FF 4 days on CPU 496.8% 116.6% A3C, LSTM 4 days on CPU 623.0% 112.6% Table 1.


CPU 344.1% 68.2% A3C, FF 4 days on CPU 496.8% 116.6% A3C, LSTM 4 days on CPU 623.0% 112.6% Table 1. Mean and median human-normalized scores on 57 Atari games using the human starts evaluation metric. Supplementary Table SS3 shows the raw scores for all games. from (Bellemare et al., 2012). We trained our agents for four days using 16 CPU cores, while the other agents were trained for 8 to 10 days on Nvidia K40 GPUs. Table 1 shows the average and median human-normalized scores obtained by our agents trained by asynchronous advantage actor-critic (A3C) as well as the current state-of-the art. Supplementary Table S3 shows the scores on all games. A3C signiﬁcantly improves on state-of-the-art the average score over 57 games in half the training time of the other methods while using only 16 CPU


e average score over 57 games in half the training time of the other methods while using only 16 CPU cores and no GPU. Furthermore, after just one day of training, A3C matches the average human normalized score of Dueling Double DQN and almost reaches the median human normalized score of Gorila. We note that many of the improvements that are presented in Double DQN (Van Hasselt et al., 2015) and Dueling Double DQN (Wang et al., 2015) can be incorporated to 1-step Q and n-step Q methods presented in this work with similar potential improvements. 5.2. TORCS Car Racing Simulator We also compared the four asynchronous methods on the TORCS 3D car racing game (Wymann et al., 2013). TORCS not only has more realistic graphics than Atari 2600 games, but also requires the agent to learn the dynamics


has more realistic graphics than Atari 2600 games, but also requires the agent to learn the dynamics of the car it is controlling. At each step, an agent received only a visual input in the form of an RGB image


Asynchronous Methods for Deep Reinforcement Learning of the current frame as well as a reward proportional to the agent’s velocity along the center of the track at the agent’s current position. We used the same neural network architecture as the one used in the Atari experiments speciﬁed in Supplementary Section 8. We performed experiments using four different settings – the agent controlling a slow car with and without opponent bots, and the agent controlling a fast car with and without opponent bots. Full results can be found in Supplementary Figure S6. A3C was the best performing agent, reaching between roughly 75% and 90% of the score obtained by a human tester on all four game conﬁgurations in about 12 hours of training.


the score obtained by a human tester on all four game conﬁgurations in about 12 hours of training. A video showing the learned driving behavior of the A3C agent can be found at https://youtu.be/0xo1Ldx3L5Q. 5.3. Continuous Action Control Using the MuJoCo Physics Simulator We also examined a set of tasks where the action space is continuous. In particular, we looked at a set of rigid body physics domains with contact dynamics where the tasks include many examples of manipulation and locomotion. These tasks were simulated using the Mujoco physics engine. We evaluated only the asynchronous advantage actor-critic algorithm since, unlike the value-based methods, it is easily extended to continuous actions.


ritic algorithm since, unlike the value-based methods, it is easily extended to continuous actions. In all problems, using either the physical state or pixels as input, Asynchronous Advantage-Critic found good solutions in less than 24 hours of training and typically in under a few hours. Some successful policies learned by our agent can be seen in the following video https://youtu.be/ Ajjc08-iPx8. Further details about this experiment can be found in Supplementary Section 9. 5.4. Labyrinth We performed an additional set of experiments with A3C on a new 3D environment called Labyrinth. The speciﬁc task we considered involved the agent learning to ﬁnd rewards in randomly generated mazes. At the beginning of each episode the agent was placed in a new randomly generated maze consisting of roo


he beginning of each episode the agent was placed in a new randomly generated maze consisting of rooms and corridors. Each maze contained two types of objects that the agent was rewarded for ﬁnding – apples and portals. Picking up an apple led to a reward of 1. Entering a portal led to a reward of 10 after which the agent was respawned in a new random location in the maze and all previously collected apples were regenerated. An episode terminated after 60 seconds after which a new episode would begin. The aim of the agent is to collect as many points as possible in the time limit and the optimal strategy involves ﬁrst ﬁnding the portal and then repeatedly going back to it after each respawn. This task is much more challenging than the TORCS driving domain because the agent is faced with a


This task is much more challenging than the TORCS driving domain because the agent is faced with a new maze in each episode and must learn a general strategy for exploring random mazes. Number of threads Method 1 2 4 8 16 1-step Q 1.0 3.0 6.3 13.3 24.1 1-step SARSA 1.0 2.8 5.9 13.1 22.1 n-step Q 1.0 2.7 5.9 10.7 17.2 A3C 1.0 2.1 3.7 6.9 12.5 Table 2. The average training speedup for each method and number of threads averaged over seven Atari games. To compute the training speed-up on a single game we measured the time to required reach a ﬁxed reference score using each method and number of threads. The speedup from using n threads on a game was deﬁned as the time required to reach a ﬁxed reference score using one thread divided the time required to reach the reference score using n thread


ference score using one thread divided the time required to reach the reference score using n threads. The table shows the speedups averaged over seven Atari games (Beamrider, Breakout, Enduro, Pong, Q*bert, Seaquest, and Space Invaders). We trained an A3C LSTM agent on this task using only 84 × 84 RGB images as input. The ﬁnal average score of around 50 indicates that the agent learned a reasonable strategy for exploring random 3D maxes using only a visual input. A video showing one of the agents exploring previously unseen mazes is included at https: //youtu.be/nMR5mjCFZCw. 5.5. Scalability and Data Efﬁciency We analyzed the effectiveness of our proposed framework by looking at how the training time and data efﬁciency changes with the number of parallel actor-learners.


ing at how the training time and data efﬁciency changes with the number of parallel actor-learners. When using multiple workers in parallel and updating a shared model, one would expect that in an ideal case, for a given task and algorithm, the number of training steps to achieve a certain score would remain the same with varying numbers of workers. Therefore, the advantage would be solely due to the ability of the system to consume more data in the same amount of wall clock time and possibly improved exploration. Table 2 shows the training speed-up achieved by using increasing numbers of parallel actor-learners averaged over seven Atari games. These results show that all four methods achieve substantial speedups from using multiple worker threads, with 16 threads leading to at least an or


e substantial speedups from using multiple worker threads, with 16 threads leading to at least an order of magnitude speedup. This conﬁrms that our proposed framework scales well with the number of parallel workers, making efﬁcient use of resources. Somewhat surprisingly, asynchronous one-step Q-learning and Sarsa algorithms exhibit superlinear speedups that cannot be explained by purely computational gains. We observe that one-step methods (one-step Q and one-step Sarsa) often require less data to achieve a particular score when using more parallel actor-learners. We believe this is due to positive effect of multiple threads to reduce the bias in one-step methods. These effects are shown more clearly in Figure 3, which shows plots of the average score against the total number of training


re clearly in Figure 3, which shows plots of the average score against the total number of training frames for different


Asynchronous Methods for Deep Reinforcement Learning 10-4 10-3 10-2 Learning rate 2000 0 2000 4000 6000 8000 10000 12000 14000 16000 Score A3C, Beamrider 10-4 10-3 10-2 Learning rate 200 0 200 400 600 800 1000 Score A3C, Breakout 10-4 10-3 10-2 Learning rate 30 20 10 0 10 20 30 Score A3C, Pong 10-4 10-3 10-2 Learning rate 2000 0 2000 4000 6000 8000 10000 12000 Score A3C, Q*bert 10-4 10-3 10-2 Learning rate 0 200 400 600 800 1000 1200 1400 Score A3C, Space Invaders Figure 2. Scatter plots of scores obtained by asynchronous advantage actor-critic on ﬁve games (Beamrider, Breakout, Pong, Q*bert, Space Invaders) for 50 different learning rates and random initializations. On each game, there is a wide range of learning rates for which all random initializations acheive good scores.


, there is a wide range of learning rates for which all random initializations acheive good scores. This shows that A3C is quite robust to learning rates and initial random weights. numbers of actor-learners and training methods on ﬁve Atari games, and Figure 4, which shows plots of the average score against wall-clock time. 5.6. Robustness and Stability Finally, we analyzed the stability and robustness of the four proposed asynchronous algorithms. For each of the four algorithms we trained models on ﬁve games (Breakout, Beamrider, Pong, Q*bert, Space Invaders) using 50 different learning rates and random initializations. Figure 2 shows scatter plots of the resulting scores for A3C, while Supplementary Figure S11 shows plots for the other three methods.


e resulting scores for A3C, while Supplementary Figure S11 shows plots for the other three methods. There is usually a range of learning rates for each method and game combination that leads to good scores, indicating that all methods are quite robust to the choice of learning rate and random initialization. The fact that there are virtually no points with scores of 0 in regions with good learning rates indicates that the methods are stable and do not collapse or diverge once they are learning. 6. Conclusions and Discussion We have presented asynchronous versions of four standard reinforcement learning algorithms and showed that they are able to train neural network controllers on a variety of domains in a stable manner.


that they are able to train neural network controllers on a variety of domains in a stable manner. Our results show that in our proposed framework stable training of neural networks through reinforcement learning is possible with both valuebased and policy-based methods, off-policy as well as onpolicy methods, and in discrete as well as continuous domains. When trained on the Atari domain using 16 CPU cores, the proposed asynchronous algorithms train faster than DQN trained on an Nvidia K40 GPU, with A3C surpassing the current state-of-the-art in half the training time. One of our main ﬁndings is that using parallel actorlearners to update a shared model had a stabilizing effect on the learning process of the three value-based methods we considered.


el had a stabilizing effect on the learning process of the three value-based methods we considered. While this shows that stable online Q-learning is possible without experience replay, which was used for this purpose in DQN, it does not mean that experience replay is not useful. Incorporating experience replay into the asynchronous reinforcement learning framework could substantially improve the data efﬁciency of these methods by reusing old data. This could in turn lead to much faster training times in domains like TORCS where interacting with the environment is more expensive than updating the model for the architecture we used. Combining other existing reinforcement learning methods or recent advances in deep reinforcement learning with our asynchronous framework presents many possibil


ecent advances in deep reinforcement learning with our asynchronous framework presents many possibilities for immediate improvements to the methods we presented. While our n-step methods operate in the forward view (Sutton & Barto, 1998) by using corrected n-step returns directly as targets, it has been more common to use the backward view to implicitly combine different returns through eligibility traces (Watkins, 1989; Sutton & Barto, 1998; Peng & Williams, 1996). The asynchronous advantage actor-critic method could be potentially improved by using other ways of estimating the advantage function, such as generalized advantage estimation of (Schulman et al., 2015b). All of the value-based methods we investigated could beneﬁt from different ways of reducing overestimation bias of Q-values


ethods we investigated could beneﬁt from different ways of reducing overestimation bias of Q-values (Van Hasselt et al., 2015; Bellemare et al., 2016). Yet another, more speculative, direction is to try and combine the recent work on true online temporal difference methods (van Seijen et al., 2015) with nonlinear function approximation. In addition to these algorithmic improvements, a number of complementary improvements to the neural network architecture are possible. The dueling architecture of (Wang et al., 2015) has been shown to produce more accurate estimates of Q-values by including separate streams for the state value and advantage in the network. The spatial softmax proposed by (Levine et al., 2015) could improve both value-based and policy-based methods by making it easier for th


ine et al., 2015) could improve both value-based and policy-based methods by making it easier for the network to represent feature coordinates. ACKNOWLEDGMENTS We thank Thomas Degris, Remi Munos, Marc Lanctot, Sasha Vezhnevets and Joseph Modayil for many helpful discussions, suggestions and comments on the paper. We also thank the DeepMind evaluation team for setting up the environments used to evaluate the agents in the paper.


Asynchronous Methods for Deep Reinforcement Learning 0 10 20 30 40 Training epochs 0 2000 4000 6000 8000 10000 Score Beamrider 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 10 20 30 40 Training epochs 0 50 100 150 200 250 300 350 Score Breakout 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 10 20 30 40 Training epochs 25 20 15 10 5 0 5 10 15 20 Score Pong 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 10 20 30 40 Training epochs 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Score Q*bert 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 10 20 30 40 Training epochs 100 200


1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 10 20 30 40 Training epochs 100 200 300 400 500 600 700 800 Score Space Invaders 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 10 20 30 40 Training epochs 0 2000 4000 6000 8000 10000 12000 Score Beamrider n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 10 20 30 40 Training epochs 0 50 100 150 200 250 300 350 Score Breakout n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 10 20 30 40 Training epochs 25 20 15 10 5 0 5 10 15 20 Score Pong n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 10 20 30 40 Training epochs


2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 10 20 30 40 Training epochs 0 1000 2000 3000 4000 5000 6000 Score Q*bert n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 10 20 30 40 Training epochs 100 200 300 400 500 600 700 800 Score Space Invaders n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 10 20 30 40 Training epochs 0 2000 4000 6000 8000 10000 12000 14000 16000 Score Beamrider A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 10 20 30 40 Training epochs 0 100 200 300 400 500 600 700 800 Score Breakout A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 10 20 30 40 Training epochs 30 20 10 0 10 20


threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 10 20 30 40 Training epochs 30 20 10 0 10 20 30 Score Pong A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 10 20 30 40 Training epochs 0 2000 4000 6000 8000 10000 12000 Score Q*bert A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 10 20 30 40 Training epochs 0 200 400 600 800 1000 1200 1400 Score Space Invaders A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads Figure 3. Data efﬁciency comparison of different numbers of actor-learners for three asynchronous methods on ﬁve Atari games. The x-axis shows the total number of training epochs where an epoch corresponds to four million frames (across all threads). The y-axis shows the average score.


epoch corresponds to four million frames (across all threads). The y-axis shows the average score. Each curve shows the average over the three best learning rates. Single step methods show increased data efﬁciency from more parallel workers. Results for Sarsa are shown in Supplementary Figure S9. 0 2 4 6 8 10 12 14 Training time (hours) 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Score Beamrider 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 50 100 150 200 250 300 Score Breakout 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 25 20 15 10 5 0 5 10 15 20 Score Pong 1-step Q, 1 threads 1-step Q, 2 thr


2 14 Training time (hours) 25 20 15 10 5 0 5 10 15 20 Score Pong 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 500 1000 1500 2000 2500 3000 3500 4000 Score Q*bert 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 100 200 300 400 500 600 700 800 Score Space Invaders 1-step Q, 1 threads 1-step Q, 2 threads 1-step Q, 4 threads 1-step Q, 8 threads 1-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 2000 4000 6000 8000 10000 12000 Score Beamrider n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 50 100 1


threads n-step Q, 8 threads n-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 50 100 150 200 250 300 350 Score Breakout n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 25 20 15 10 5 0 5 10 15 20 Score Pong n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Score Q*bert n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 100 200 300 400 500 600 700 800 Score Space Invaders n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q,


Invaders n-step Q, 1 threads n-step Q, 2 threads n-step Q, 4 threads n-step Q, 8 threads n-step Q, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 2000 4000 6000 8000 10000 12000 14000 16000 Score Beamrider A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 100 200 300 400 500 600 Score Breakout A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 30 20 10 0 10 20 30 Score Pong A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 2000 4000 6000 8000 10000 12000 Score Q*bert A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 2 4 6 8 10 12 14 Training time (hours)


3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 200 400 600 800 1000 1200 1400 1600 Score Space Invaders A3C, 1 threads A3C, 2 threads A3C, 4 threads A3C, 8 threads A3C, 16 threads Figure 4. Training speed comparison of different numbers of actor-learners on ﬁve Atari games. The x-axis shows training time in hours while the y-axis shows the average score. Each curve shows the average over the three best learning rates. All asynchronous methods show signiﬁcant speedups from using greater numbers of parallel actor-learners. Results for Sarsa are shown in Supplementary Figure S10.


Asynchronous Methods for Deep Reinforcement Learning References Bellemare, Marc G, Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning environment: An evaluation platform for general agents. Journal of Artiﬁcial Intelligence Research, 2012. Bellemare, Marc G., Ostrovski, Georg, Guez, Arthur, Thomas, Philip S., and Munos, Rémi. Increasing the action gap: New operators for reinforcement learning. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence, 2016. Bertsekas, Dimitri P. Distributed dynamic programming. Automatic Control, IEEE Transactions on, 27(3):610– 616, 1982. Chavez, Kevin, Ong, Hao Yi, and Hong, Augustus. Distributed deep q-learning. Technical report, Stanford University, June 2015. Degris, Thomas, Pilarski, Patrick M, and Sutton, Richard S.


report, Stanford University, June 2015. Degris, Thomas, Pilarski, Patrick M, and Sutton, Richard S. Model-free reinforcement learning with continuous action in practice. In American Control Conference (ACC), 2012, pp. 2177–2182. IEEE, 2012. Grounds, Matthew and Kudenko, Daniel. Parallel reinforcement learning with linear function approximation. In Proceedings of the 5th, 6th and 7th European Conference on Adaptive and Learning Agents and Multi-agent Systems: Adaptation and Multi-agent Learning, pp. 60– 74. Springer-Verlag, 2008. Koutník, Jan, Schmidhuber, Jürgen, and Gomez, Faustino. Evolving deep unsupervised convolutional networks for vision-based reinforcement learning. In Proceedings of the 2014 conference on Genetic and evolutionary computation, pp. 541–548. ACM, 2014.


Proceedings of the 2014 conference on Genetic and evolutionary computation, pp. 541–548. ACM, 2014. Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel, Pieter. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015. Li, Yuxi and Schuurmans, Dale. Mapreduce for parallel reinforcement learning. In Recent Advances in Reinforcement Learning - 9th European Workshop, EWRL 2011, Athens, Greece, September 9-11, 2011, Revised Selected Papers, pp. 309–320, 2011. Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.


, Daan. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Graves, Alex, Antonoglou, Ioannis, Wierstra, Daan, and Riedmiller, Martin. Playing atari with deep reinforcement learning. In NIPS Deep Learning Workshop. 2013. Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A., Veness, Joel, Bellemare, Marc G., Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K., Ostrovski, Georg, Petersen, Stig, Beattie, Charles, Sadik, Amir, Antonoglou, Ioannis, King, Helen, Kumaran, Dharshan, Wierstra, Daan, Legg, Shane, and Hassabis, Demis. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 02 2015. URL http://dx.doi.org/10.1038/nature14236.


nforcement learning. Nature, 518(7540):529–533, 02 2015. URL http://dx.doi.org/10.1038/nature14236. Nair, Arun, Srinivasan, Praveen, Blackwell, Sam, Alcicek, Cagdas, Fearon, Rory, Maria, Alessandro De, Panneershelvam, Vedavyas, Suleyman, Mustafa, Beattie, Charles, Petersen, Stig, Legg, Shane, Mnih, Volodymyr, Kavukcuoglu, Koray, and Silver, David. Massively parallel methods for deep reinforcement learning. In ICML Deep Learning Workshop. 2015. Peng, Jing and Williams, Ronald J. Incremental multi-step q-learning. Machine Learning, 22(1-3):283–290, 1996. Recht, Benjamin, Re, Christopher, Wright, Stephen, and Niu, Feng. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems, pp. 693–701, 2011. Riedmiller, Martin.


scent. In Advances in Neural Information Processing Systems, pp. 693–701, 2011. Riedmiller, Martin. Neural ﬁtted q iteration–ﬁrst experiences with a data efﬁcient neural reinforcement learning method. In Machine Learning: ECML 2005, pp. 317– 328. Springer Berlin Heidelberg, 2005. Rummery, Gavin A and Niranjan, Mahesan. On-line qlearning using connectionist systems. 1994. Schaul, Tom, Quan, John, Antonoglou, Ioannis, and Silver, David. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015. Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan, Michael I, and Abbeel, Pieter. Trust region policy optimization. In International Conference on Machine Learning (ICML), 2015a. Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael, and Abbeel, Pieter.


ICML), 2015a. Schulman, John, Moritz, Philipp, Levine, Sergey, Jordan, Michael, and Abbeel, Pieter. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015b. Sutton, R. and Barto, A. Reinforcement Learning: an Introduction. MIT Press, 1998. Tieleman, Tijmen and Hinton, Geoffrey. Lecture 6.5rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 4, 2012. Todorov, E. MuJoCo: Modeling, Simulation and Visualization of Multi-Joint Dynamics with Contact (ed 1.0). Roboti Publishing, 2015.


Asynchronous Methods for Deep Reinforcement Learning Tomassini, Marco. Parallel and distributed evolutionary algorithms: A review. Technical report, 1999. Tsitsiklis, John N. Asynchronous stochastic approximation and q-learning. Machine Learning, 16(3):185–202, 1994. Van Hasselt, Hado, Guez, Arthur, and Silver, David. Deep reinforcement learning with double q-learning. arXiv preprint arXiv:1509.06461, 2015. van Seijen, H., Rupam Mahmood, A., Pilarski, P. M., Machado, M. C., and Sutton, R. S. True Online Temporal-Difference Learning. ArXiv e-prints, December 2015. Wang, Z., de Freitas, N., and Lanctot, M. Dueling Network Architectures for Deep Reinforcement Learning. ArXiv e-prints, November 2015. Watkins, Christopher John Cornish Hellaby. Learning from delayed rewards.


e-prints, November 2015. Watkins, Christopher John Cornish Hellaby. Learning from delayed rewards. PhD thesis, University of Cambridge England, 1989. Williams, R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229–256, 1992. Williams, Ronald J and Peng, Jing. Function optimization using connectionist reinforcement learning algorithms. Connection Science, 3(3):241–268, 1991. Wymann, B., EspiÃl’, E., Guionneau, C., Dimitrakakis, C., Coulom, R., and Sumner, A. Torcs: The open racing car simulator, v1.3.5, 2013.


Supplementary Material for "Asynchronous Methods for Deep Reinforcement Learning" June 17, 2016 7. Optimization Details We investigated two different optimization algorithms with our asynchronous framework – stochastic gradient descent and RMSProp. Our implementations of these algorithms do not use any locking in order to maximize throughput when using a large number of threads. Momentum SGD: The implementation of SGD in an asynchronous setting is relatively straightforward and well studied (Recht et al., 2011). Let θ be the parameter vector that is shared across all threads and let ∆θi be the accumulated gradients of the loss with respect to parameters θ computed by thread number i. Each thread i independently applies the standard momentum SGD update mi = αmi + (1 −α)∆θi followed by θ ←θ


read i independently applies the standard momentum SGD update mi = αmi + (1 −α)∆θi followed by θ ←θ −ηmi with learning rate η, momentum α and without any locks. Note that in this setting, each thread maintains its own separate gradient and momentum vector. RMSProp: While RMSProp (Tieleman & Hinton, 2012) has been widely used in the deep learning literature, it has not been extensively studied in the asynchronous optimization setting. The standard non-centered RMSProp update is given by g = αg + (1 −α)∆θ2 (S2) θ ←θ −η ∆θ √g + ϵ, (S3) where all operations are performed elementwise. In order to apply RMSProp in the asynchronous optimization setting one must decide whether the moving average of elementwise squared gradients g is shared or per-thread.


must decide whether the moving average of elementwise squared gradients g is shared or per-thread. We experimented with two versions of the algorithm. In one version, which we refer to as RMSProp, each thread maintains its own g shown in Equation S2. In the other version, which we call Shared RMSProp, the vector g is shared among threads and is updated asynchronously and without locking. Sharing statistics among threads also reduces memory requirements by using one fewer copy of the parameter vector per thread. We compared these three asynchronous optimization algorithms in terms of their sensitivity to different learning rates and random network initializations. Figure S5 shows a comparison of the methods for two different reinforcement learning methods (Async n-step Q and Async Advantag


n of the methods for two different reinforcement learning methods (Async n-step Q and Async Advantage Actor-Critic) on four different games (Breakout, Beamrider, Seaquest and Space Invaders). Each curve shows the scores for 50 experiments that correspond to 50 different random learning rates and initializations. The x-axis shows the rank of the model after sorting in descending order by ﬁnal average score and the y-axis shows the ﬁnal average score achieved by the corresponding model. In this representation, the algorithm that performs better would achieve higher maximum rewards on the y-axis and the algorithm that is most robust would have its slope closest to horizontal, thus maximizing the area under the curve.


s most robust would have its slope closest to horizontal, thus maximizing the area under the curve. RMSProp with shared statistics tends to be more robust than RMSProp with per-thread statistics, which is in turn more robust than Momentum SGD.


Asynchronous Methods for Deep Reinforcement Learning 8. Experimental Setup The experiments performed on a subset of Atari games (Figures 1, 3, 4 and Table 2) as well as the TORCS experiments (Figure S6) used the following setup. Each experiment used 16 actor-learner threads running on a single machine and no GPUs. All methods performed updates after every 5 actions (tmax = 5 and IUpdate = 5) and shared RMSProp was used for optimization. The three asynchronous value-based methods used a shared target network that was updated every 40000 frames. The Atari experiments used the same input preprocessing as (Mnih et al., 2015) and an action repeat of 4. The agents used the network architecture from (Mnih et al., 2013).


2015) and an action repeat of 4. The agents used the network architecture from (Mnih et al., 2013). The network used a convolutional layer with 16 ﬁlters of size 8 × 8 with stride 4, followed by a convolutional layer with with 32 ﬁlters of size 4 × 4 with stride 2, followed by a fully connected layer with 256 hidden units. All three hidden layers were followed by a rectiﬁer nonlinearity. The value-based methods had a single linear output unit for each action representing the action-value. The model used by actor-critic agents had two set of outputs – a softmax output with one entry per action representing the probability of selecting the action, and a single linear output representing the value function. All experiments used a discount of γ = 0.99 and an RMSProp decay factor of α = 0.99.


alue function. All experiments used a discount of γ = 0.99 and an RMSProp decay factor of α = 0.99. The value based methods sampled the exploration rate ϵ from a distribution taking three values ϵ1, ϵ2, ϵ3 with probabilities 0.4, 0.3, 0.3. The values of ϵ1, ϵ2, ϵ3 were annealed from 1 to 0.1, 0.01, 0.5 respectively over the ﬁrst four million frames. Advantage actor-critic used entropy regularization with a weight β = 0.01 for all Atari and TORCS experiments. We performed a set of 50 experiments for ﬁve Atari games and every TORCS level, each using a different random initialization and initial learning rate. The initial learning rate was sampled from a LogUniform(10−4, 10−2) distribution and annealed to 0 over the course of training.


s sampled from a LogUniform(10−4, 10−2) distribution and annealed to 0 over the course of training. Note that in comparisons to prior work (Tables 1 and S3) we followed standard evaluation protocol and used ﬁxed hyperparameters. 9. Continuous Action Control Using the MuJoCo Physics Simulator To apply the asynchronous advantage actor-critic algorithm to the Mujoco tasks the necessary setup is nearly identical to that used in the discrete action domains, so here we enumerate only the differences required for the continuous action domains. The essential elements for many of the tasks (i.e. the physics models and task objectives) are near identical to the tasks examined in (Lillicrap et al., 2015).


s models and task objectives) are near identical to the tasks examined in (Lillicrap et al., 2015). However, the rewards and thus performance are not comparable for most of the tasks due to changes made by the developers of Mujoco which altered the contact model. For all the domains we attempted to learn the task using the physical state as input. The physical state consisted of the joint positions and velocities as well as the target position if the task required a target. In addition, for three of the tasks (pendulum, pointmass2D, and gripper) we also examined training directly from RGB pixel inputs. In the low dimensional physical state case, the inputs are mapped to a hidden state using one hidden layer with 200 ReLU units.


cal state case, the inputs are mapped to a hidden state using one hidden layer with 200 ReLU units. In the cases where we used pixels, the input was passed through two layers of spatial convolutions without any non-linearity or pooling. In either case, the output of the encoder layers were fed to a single layer of 128 LSTM cells. The most important difference in the architecture is in the the output layer of the policy network. Unlike the discrete action domain where the action output is a Softmax, here the two outputs of the policy network are two real number vectors which we treat as the mean vector µ and scalar variance σ2 of a multidimensional normal distribution with a spherical covariance.


tor µ and scalar variance σ2 of a multidimensional normal distribution with a spherical covariance. To act, the input is passed through the model to the output layer where we sample from the normal distribution determined by µ and σ2. In practice, µ is modeled by a linear layer and σ2 by a SoftPlus operation, log(1 + exp(x)), as the activation computed as a function of the output of a linear layer. In our experiments with continuous control problems the networks for policy network and value network do not share any parameters, though this detail is unlikely to be crucial. Finally, since the episodes were typically at most several hundred time steps long, we did not use any bootstrapping in the policy or value function updates and batched each episode into a single update.


ootstrapping in the policy or value function updates and batched each episode into a single update. As in the discrete action case, we included an entropy cost which encouraged exploration. In the continuous


Asynchronous Methods for Deep Reinforcement Learning case the we used a cost on the differential entropy of the normal distribution deﬁned by the output of the actor network, −1 2(log(2πσ2) + 1), we used a constant multiplier of 10−4 for this cost across all of the tasks examined. The asynchronous advantage actor-critic algorithm ﬁnds solutions for all the domains. Figure S8 shows learning curves against wall-clock time, and demonstrates that most of the domains from states can be solved within a few hours. All of the experiments, including those done from pixel based observations, were run on CPU. Even in the case of solving the domains directly from pixel inputs we found that it was possible to reliably discover solutions within 24 hours.


tly from pixel inputs we found that it was possible to reliably discover solutions within 24 hours. Figure S7 shows scatter plots of the top scores against the sampled learning rates. In most of the domains there is large range of learning rates that consistently achieve good performance on the task. Algorithm S2 Asynchronous n-step Q-learning - pseudocode for each actor-learner thread. // Assume global shared parameter vector θ. // Assume global shared target parameter vector θ−. // Assume global shared counter T = 0. Initialize thread step counter t ←1 Initialize target network parameters θ−←θ Initialize thread-speciﬁc parameters θ′ = θ Initialize network gradients dθ ←0 repeat Clear gradients dθ ←0 Synchronize thread-speciﬁc parameters θ′ = θ tstart = t Get state st repeat Take action a


ents dθ ←0 Synchronize thread-speciﬁc parameters θ′ = θ tstart = t Get state st repeat Take action at according to the ϵ-greedy policy based on Q(st, a; θ′) Receive reward rt and new state st+1 t ←t + 1 T ←T + 1 until terminal st or t −tstart == tmax R =  0 for terminal st maxa Q(st, a; θ−) for non-terminal st for i ∈{t −1, . . . , tstart} do R ←ri + γR Accumulate gradients wrt θ′: dθ ←dθ + ∂(R−Q(si,ai;θ′)) 2 ∂θ′ end for Perform asynchronous update of θ using dθ. if T mod Itarget == 0 then θ−←θ end if until T > Tmax


Asynchronous Methods for Deep Reinforcement Learning Algorithm S3 Asynchronous advantage actor-critic - pseudocode for each actor-learner thread. // Assume global shared parameter vectors θ and θv and global shared counter T = 0 // Assume thread-speciﬁc parameter vectors θ′ and θ′ v Initialize thread step counter t ←1 repeat Reset gradients: dθ ←0 and dθv ←0. Synchronize thread-speciﬁc parameters θ′ = θ and θ′ v = θv tstart = t Get state st repeat Perform at according to policy π(at|st; θ′) Receive reward rt and new state st+1 t ←t + 1 T ←T + 1 until terminal st or t −tstart == tmax R =  0 for terminal st V (st, θ′ v) for non-terminal st// Bootstrap from last state for i ∈{t −1, . . . , tstart} do R ←ri + γR Accumulate gradients wrt θ′: dθ ←dθ + ∇θ′ log π(ai|si; θ′)(R −V (si; θ′ v)) Accum


art} do R ←ri + γR Accumulate gradients wrt θ′: dθ ←dθ + ∇θ′ log π(ai|si; θ′)(R −V (si; θ′ v)) Accumulate gradients wrt θ′ v: dθv ←dθv + ∂(R −V (si; θ′ v))2/∂θ′ v end for Perform asynchronous update of θ using dθ and of θv using dθv. until T > Tmax


Asynchronous Methods for Deep Reinforcement Learning 10 20 30 40 50 Model Rank 0 50 100 150 200 250 300 350 400 Score Breakout n-step Q, SGD n-step Q, RMSProp n-step Q, Shared RMSProp 10 20 30 40 50 Model Rank 0 5000 10000 15000 20000 25000 Score Beamrider n-step Q, SGD n-step Q, RMSProp n-step Q, Shared RMSProp 10 20 30 40 50 Model Rank 0 1000 2000 3000 4000 5000 6000 Score Seaquest n-step Q, SGD n-step Q, RMSProp n-step Q, Shared RMSProp 10 20 30 40 50 Model Rank 0 200 400 600 800 1000 1200 1400 1600 1800 Score Space Invaders n-step Q, SGD n-step Q, RMSProp n-step Q, Shared RMSProp 10 20 30 40 50 Model Rank 0 100 200 300 400 500 600 700 800 900 Score Breakout A3C, SGD A3C, RMSProp A3C, Shared RMSProp 10 20 30 40 50 Model Rank 0 5000 10000 15000 20000 25000 Score Beamrider A3C, SGD A3C, R


red RMSProp 10 20 30 40 50 Model Rank 0 5000 10000 15000 20000 25000 Score Beamrider A3C, SGD A3C, RMSProp A3C, Shared RMSProp 10 20 30 40 50 Model Rank 200 400 600 800 1000 1200 1400 1600 1800 Score Seaquest A3C, SGD A3C, RMSProp A3C, Shared RMSProp 10 20 30 40 50 Model Rank 0 500 1000 1500 2000 2500 3000 3500 4000 Score Space Invaders A3C, SGD A3C, RMSProp A3C, Shared RMSProp Figure S5. Comparison of three different optimization methods (Momentum SGD, RMSProp, Shared RMSProp) tested using two different algorithms (Async n-step Q and Async Advantage Actor-Critic) on four different Atari games (Breakout, Beamrider, Seaquest and Space Invaders). Each curve shows the ﬁnal scores for 50 experiments sorted in descending order that covers a search over 50 random initializations and learning rat


ents sorted in descending order that covers a search over 50 random initializations and learning rates. The top row shows results using Async n-step Q algorithm and bottom row shows results with Async Advantage Actor-Critic. Each individual graph shows results for one of the four games and three different optimization methods. Shared RMSProp tends to be more robust to different learning rates and random initializations than Momentum SGD and RMSProp without sharing. 0 10 20 30 40 Training time (hours) 1000 0 1000 2000 3000 4000 5000 Score Slow car, no bots Async 1-step Q Async SARSA Async n-step Q Async actor-critic Human tester 0 10 20 30 40 Training time (hours) 1000 0 1000 2000 3000 4000 5000 Score Slow car, bots Async 1-step Q Async SARSA Async n-step Q Async actor-critic Human tester 0


000 Score Slow car, bots Async 1-step Q Async SARSA Async n-step Q Async actor-critic Human tester 0 10 20 30 40 Training time (hours) 1000 0 1000 2000 3000 4000 5000 6000 Score Fast car, no bots Async 1-step Q Async SARSA Async n-step Q Async actor-critic Human tester 0 10 20 30 40 Training time (hours) 1000 0 1000 2000 3000 4000 5000 6000 Score Fast car, bots Async 1-step Q Async SARSA Async n-step Q Async actor-critic Human tester Figure S6. Comparison of algorithms on the TORCS car racing simulator. Four different conﬁgurations of car speed and opponent presence or absence are shown. In each plot, all four algorithms (one-step Q, one-step Sarsa, n-step Q and Advantage Actor-Critic) are compared on score vs training time in wall clock hours.


a, n-step Q and Advantage Actor-Critic) are compared on score vs training time in wall clock hours. Multi-step algorithms achieve better policies much faster than one-step algorithms on all four levels. The curves show averages over the 5 best runs from 50 experiments with learning rates sampled from LogUniform(10−4, 10−2) and all other hyperparameters ﬁxed.


Asynchronous Methods for Deep Reinforcement Learning Figure S7. Performance for the Mujoco continuous action domains. Scatter plot of the best score obtained against learning rates sampled from LogUniform(10−5, 10−1). For nearly all of the tasks there is a wide range of learning rates that lead to good performance on the task.


Asynchronous Methods for Deep Reinforcement Learning Figure S8. Score per episode vs wall-clock time plots for the Mujoco domains. Each plot shows error bars for the top 5 experiments. 0 10 20 30 40 Training epochs 0 2000 4000 6000 8000 10000 12000 Score Beamrider 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 10 20 30 40 Training epochs 0 50 100 150 200 250 300 350 Score Breakout 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 10 20 30 40 Training epochs 25 20 15 10 5 0 5 10 15 20 Score Pong 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 10 20 30 40 Training epochs


tep SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 10 20 30 40 Training epochs 0 500 1000 1500 2000 2500 3000 3500 4000 4500 Score Q*bert 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 10 20 30 40 Training epochs 100 200 300 400 500 600 700 800 900 Score Space Invaders 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads Figure S9. Data efﬁciency comparison of different numbers of actor-learners one-step Sarsa on ﬁve Atari games. The x-axis shows the total number of training epochs where an epoch corresponds to four million frames (across all threads). The y-axis shows the average score.


epoch corresponds to four million frames (across all threads). The y-axis shows the average score. Each curve shows the average of the three best performing agents from a search over 50 random learning rates. Sarsa shows increased data efﬁciency with increased numbers of parallel workers.


Asynchronous Methods for Deep Reinforcement Learning 0 2 4 6 8 10 12 14 Training time (hours) 0 2000 4000 6000 8000 10000 12000 Score Beamrider 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 50 100 150 200 250 300 350 Score Breakout 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 25 20 15 10 5 0 5 10 15 20 Score Pong 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 0 500 1000 1500 2000 2500 3000 3500 Score Q*bert 1-step SARSA, 1 threads 1-step SARSA, 2


ime (hours) 0 500 1000 1500 2000 2500 3000 3500 Score Q*bert 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads 0 2 4 6 8 10 12 14 Training time (hours) 100 200 300 400 500 600 700 800 Score Space Invaders 1-step SARSA, 1 threads 1-step SARSA, 2 threads 1-step SARSA, 4 threads 1-step SARSA, 8 threads 1-step SARSA, 16 threads Figure S10. Training speed comparison of different numbers of actor-learners for all one-step Sarsa on ﬁve Atari games. The x-axis shows training time in hours while the y-axis shows the average score. Each curve shows the average of the three best performing agents from a search over 50 random learning rates. Sarsa shows signiﬁcant speedups from using greater numbers of parallel actor-learners.


rning rates. Sarsa shows signiﬁcant speedups from using greater numbers of parallel actor-learners. 10-4 10-3 10-2 Learning rate 0 2000 4000 6000 8000 10000 12000 Score 1-step Q, Beamrider 10-4 10-3 10-2 Learning rate 50 0 50 100 150 200 250 300 350 400 Score 1-step Q, Breakout 10-4 10-3 10-2 Learning rate 30 20 10 0 10 20 30 Score 1-step Q, Pong 10-4 10-3 10-2 Learning rate 1000 0 1000 2000 3000 4000 5000 Score 1-step Q, Q*bert 10-4 10-3 10-2 Learning rate 100 200 300 400 500 600 700 800 Score 1-step Q, Space Invaders 10-4 10-3 10-2 Learning rate 2000 0 2000 4000 6000 8000 10000 12000 14000 Score 1-step SARSA, Beamrider 10-4 10-3 10-2 Learning rate 50 0 50 100 150 200 250 300 350 400 Score 1-step SARSA, Breakout 10-4 10-3 10-2 Learning rate 25 20 15 10 5 0 5 10 15 20 Score 1-step SARSA, P


1-step SARSA, Breakout 10-4 10-3 10-2 Learning rate 25 20 15 10 5 0 5 10 15 20 Score 1-step SARSA, Pong 10-4 10-3 10-2 Learning rate 1000 0 1000 2000 3000 4000 5000 Score 1-step SARSA, Q*bert 10-4 10-3 10-2 Learning rate 100 200 300 400 500 600 700 800 900 Score 1-step SARSA, Space Invaders 10-4 10-3 10-2 Learning rate 2000 0 2000 4000 6000 8000 10000 12000 14000 16000 Score n-step Q, Beamrider 10-4 10-3 10-2 Learning rate 50 0 50 100 150 200 250 300 350 400 Score n-step Q, Breakout 10-4 10-3 10-2 Learning rate 30 20 10 0 10 20 30 Score n-step Q, Pong 10-4 10-3 10-2 Learning rate 1000 0 1000 2000 3000 4000 5000 Score n-step Q, Q*bert 10-4 10-3 10-2 Learning rate 300 400 500 600 700 800 900 1000 Score n-step Q, Space Invaders Figure S11.


10-3 10-2 Learning rate 300 400 500 600 700 800 900 1000 Score n-step Q, Space Invaders Figure S11. Scatter plots of scores obtained by one-step Q, one-step Sarsa, and n-step Q on ﬁve games (Beamrider, Breakout, Pong, Q*bert, Space Invaders) for 50 different learning rates and random initializations. All algorithms exhibit some level of robustness to the choice of learning rate.


Asynchronous Methods for Deep Reinforcement Learning Game DQN Gorila Double Dueling Prioritized A3C FF, 1 day A3C FF A3C LSTM Alien 570.2 813.5 1033.4 1486.5 900.5 182.1 518.4 945.3 Amidar 133.4 189.2 169.1 172.7 218.4 283.9 263.9 173.0 Assault 3332.3 1195.8 6060.8 3994.8 7748.5 3746.1 5474.9 14497.9 Asterix 124.5 3324.7 16837.0 15840.0 31907.5 6723.0 22140.5 17244.5 Asteroids 697.1 933.6 1193.2 2035.4 1654.0 3009.4 4474.5 5093.1 Atlantis 76108.0 629166.5 319688.0 445360.0 593642.0 772392.0 911091.0 875822.0 Bank Heist 176.3 399.4 886.0 1129.3 816.8 946.0 970.1 932.8 Battle Zone 17560.0 19938.0 24740.0 31320.0 29100.0 11340.0 12950.0 20760.0 Beam Rider 8672.4 3822.1 17417.2 14591.3 26172.7 13235.9 22707.9 24622.2 Berzerk 1011.1 910.6 1165.6 1433.4 817.9 862.2 Bowling 41.2 54.0 69.6 65.7 65


3235.9 22707.9 24622.2 Berzerk 1011.1 910.6 1165.6 1433.4 817.9 862.2 Bowling 41.2 54.0 69.6 65.7 65.8 36.2 35.1 41.8 Boxing 25.8 74.2 73.5 77.3 68.6 33.7 59.8 37.3 Breakout 303.9 313.0 368.9 411.6 371.6 551.6 681.9 766.8 Centipede 3773.1 6296.9 3853.5 4881.0 3421.9 3306.5 3755.8 1997.0 Chopper Comman 3046.0 3191.8 3495.0 3784.0 6604.0 4669.0 7021.0 10150.0 Crazy Climber 50992.0 65451.0 113782.0 124566.0 131086.0 101624.0 112646.0 138518.0 Defender 27510.0 33996.0 21093.5 36242.5 56533.0 233021.5 Demon Attack 12835.2 14880.1 69803.4 56322.8 73185.8 84997.5 113308.4 115201.9 Double Dunk -21.6 -11.3 -0.3 -0.8 2.7 0.1 -0.1 0.1 Enduro 475.6 71.0 1216.6 2077.4 1884.4 -82.2 -82.5 -82.5 Fishing Derby -2.3 4.6 3.2 -4.1 9.2 13.6 18.8 22.6 Freeway 25.8 10.2 28.8 0.2 27.9 0.1 0.1 0.1 Frostbite 157.4


by -2.3 4.6 3.2 -4.1 9.2 13.6 18.8 22.6 Freeway 25.8 10.2 28.8 0.2 27.9 0.1 0.1 0.1 Frostbite 157.4 426.6 1448.1 2332.4 2930.2 180.1 190.5 197.6 Gopher 2731.8 4373.0 15253.0 20051.4 57783.8 8442.8 10022.8 17106.8 Gravitar 216.5 538.4 200.5 297.0 218.0 269.5 303.5 320.0 H.E.R.O. 12952.5 8963.4 14892.5 15207.9 20506.4 28765.8 32464.1 28889.5 Ice Hockey -3.8 -1.7 -2.5 -1.3 -1.0 -4.7 -2.8 -1.7 James Bond 348.5 444.0 573.0 835.5 3511.5 351.5 541.0 613.0 Kangaroo 2696.0 1431.0 11204.0 10334.0 10241.0 106.0 94.0 125.0 Krull 3864.0 6363.1 6796.1 8051.6 7406.5 8066.6 5560.0 5911.4 Kung-Fu Master 11875.0 20620.0 30207.0 24288.0 31244.0 3046.0 28819.0 40835.0 Montezuma’s Revenge 50.0 84.0 42.0 22.0 13.0 53.0 67.0 41.0 Ms.


88.0 31244.0 3046.0 28819.0 40835.0 Montezuma’s Revenge 50.0 84.0 42.0 22.0 13.0 53.0 67.0 41.0 Ms. Pacman 763.5 1263.0 1241.3 2250.6 1824.6 594.4 653.7 850.7 Name This Game 5439.9 9238.5 8960.3 11185.1 11836.1 5614.0 10476.1 12093.7 Phoenix 12366.5 20410.5 27430.1 28181.8 52894.1 74786.7 Pit Fall -186.7 -46.9 -14.8 -123.0 -78.5 -135.7 Pong 16.2 16.7 19.1 18.8 18.9 11.4 5.6 10.7 Private Eye 298.2 2598.6 -575.5 292.6 179.0 194.4 206.9 421.1 Q*Bert 4589.8 7089.8 11020.8 14175.8 11277.0 13752.3 15148.8 21307.5 River Raid 4065.3 5310.3 10838.4 16569.4 18184.4 10001.2 12201.8 6591.9 Road Runner 9264.0 43079.8 43156.0 58549.0 56990.0 31769.0 34216.0 73949.0 Robotank 58.5 61.8 59.1 62.0 55.4 2.3 32.8 2.6 Seaquest 2793.9 10145.9 14498.0 37361.6 39096.7 2300.2 2355.4 1326.1 Skiing -11490.4 -11928.0


.8 2.6 Seaquest 2793.9 10145.9 14498.0 37361.6 39096.7 2300.2 2355.4 1326.1 Skiing -11490.4 -11928.0 -10852.8 -13700.0 -10911.1 -14863.8 Solaris 810.0 1768.4 2238.2 1884.8 1956.0 1936.4 Space Invaders 1449.7 1183.3 2628.7 5993.1 9063.0 2214.7 15730.5 23846.0 Star Gunner 34081.0 14919.2 58365.0 90804.0 51959.0 64393.0 138218.0 164766.0 Surround 1.9 4.0 -0.9 -9.6 -9.7 -8.3 Tennis -2.3 -0.7 -7.8 4.4 -2.0 -10.2 -6.3 -6.4 Time Pilot 5640.0 8267.8 6608.0 6601.0 7448.0 5825.0 12679.0 27202.0 Tutankham 32.4 118.5 92.2 48.0 33.6 26.1 156.3 144.2 Up and Down 3311.3 8747.7 19086.9 24759.2 29443.7 54525.4 74705.7 105728.7 Venture 54.0 523.4 21.0 200.0 244.0 19.0 23.0 25.0 Video Pinball 20228.1 112093.4 367823.7 110976.2 374886.9 185852.6 331628.1 470310.5 Wizard of Wor 246.0 10431.0 6201.0 7054.0 7451


367823.7 110976.2 374886.9 185852.6 331628.1 470310.5 Wizard of Wor 246.0 10431.0 6201.0 7054.0 7451.0 5278.0 17244.0 18082.0 Yars Revenge 6270.6 25976.5 5965.1 7270.8 7157.5 5615.5 Zaxxon 831.0 6159.4 8593.0 10164.0 9501.0 2659.0 24622.0 23519.0 Table S3. Raw scores for the human start condition (30 minutes emulator time). DQN scores taken from (Nair et al., 2015). Double DQN scores taken from (Van Hasselt et al., 2015), Dueling scores from (Wang et al., 2015) and Prioritized scores taken from (Schaul et al., 2015)


Proximal Policy Optimization Algorithms John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov OpenAI {joschu, filip, prafulla, alec, oleg}@openai.com Abstract We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a “surrogate” objective function using stochastic gradient ascent. Whereas standard policy gradient methods perform one gradient update per data sample, we propose a novel objective function that enables multiple epochs of minibatch updates. The new methods, which we call proximal policy optimization (PPO), have some of the beneﬁts of trust region policy optimization (TRPO), but they are much simpler to implement, more general, and have better


on policy optimization (TRPO), but they are much simpler to implement, more general, and have better sample complexity (empirically). Our experiments test PPO on a collection of benchmark tasks, including simulated robotic locomotion and Atari game playing, and we show that PPO outperforms other online policy gradient methods, and overall strikes a favorable balance between sample complexity, simplicity, and wall-time. 1 Introduction In recent years, several diﬀerent approaches have been proposed for reinforcement learning with neural network function approximators. The leading contenders are deep Q-learning [Mni+15], “vanilla” policy gradient methods [Mni+16], and trust region / natural policy gradient methods [Sch+15b].


la” policy gradient methods [Mni+16], and trust region / natural policy gradient methods [Sch+15b]. However, there is room for improvement in developing a method that is scalable (to large models and parallel implementations), data eﬃcient, and robust (i.e., successful on a variety of problems without hyperparameter tuning). Q-learning (with function approximation) fails on many simple problems1 and is poorly understood, vanilla policy gradient methods have poor data eﬃency and robustness; and trust region policy optimization (TRPO) is relatively complicated, and is not compatible with architectures that include noise (such as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks).


as dropout) or parameter sharing (between the policy and value function, or with auxiliary tasks). This paper seeks to improve the current state of aﬀairs by introducing an algorithm that attains the data eﬃciency and reliable performance of TRPO, while using only ﬁrst-order optimization. We propose a novel objective with clipped probability ratios, which forms a pessimistic estimate (i.e., lower bound) of the performance of the policy. To optimize policies, we alternate between sampling data from the policy and performing several epochs of optimization on the sampled data. Our experiments compare the performance of various diﬀerent versions of the surrogate objective, and ﬁnd that the version with the clipped probability ratios performs best.


he surrogate objective, and ﬁnd that the version with the clipped probability ratios performs best. We also compare PPO to several previous algorithms from the literature. On continuous control tasks, it performs better than the algorithms we compare against. On Atari, it performs signiﬁcantly better (in terms of sample complexity) than A2C and similarly to ACER though it is much simpler. 1While DQN works well on game environments like the Arcade Learning Environment [Bel+15] with discrete action spaces, it has not been demonstrated to perform well on continuous control benchmarks such as those in OpenAI Gym [Bro+16] and described by Duan et al. [Dua+16]. 1 arXiv:1707.06347v2 [cs.LG] 28 Aug 2017


2 Background: Policy Optimization 2.1 Policy Gradient Methods Policy gradient methods work by computing an estimator of the policy gradient and plugging it into a stochastic gradient ascent algorithm. The most commonly used gradient estimator has the form ˆg = ˆEt h ∇θ log πθ(at | st) ˆAt i (1) where πθ is a stochastic policy and ˆAt is an estimator of the advantage function at timestep t. Here, the expectation ˆEt[. . .] indicates the empirical average over a ﬁnite batch of samples, in an algorithm that alternates between sampling and optimization. Implementations that use automatic diﬀerentiation software work by constructing an objective function whose gradient is the policy gradient estimator; the estimator ˆg is obtained by diﬀerentiating the objective LPG(θ) = ˆEt h log πθ(at | st) ˆ


mator; the estimator ˆg is obtained by diﬀerentiating the objective LPG(θ) = ˆEt h log πθ(at | st) ˆAt i . (2) While it is appealing to perform multiple steps of optimization on this loss LPG using the same trajectory, doing so is not well-justiﬁed, and empirically it often leads to destructively large policy updates (see Section 6.1; results are not shown but were similar or worse than the “no clipping or penalty” setting). 2.2 Trust Region Methods In TRPO [Sch+15b], an objective function (the “surrogate” objective) is maximized subject to a constraint on the size of the policy update. Speciﬁcally, maximize θ ˆEt  πθ(at | st) πθold(at | st) ˆAt  (3) subject to ˆEt[KL[πθold(· | st), πθ(· | st)]] ≤δ. (4) Here, θold is the vector of policy parameters before the update.