It seems that DQN performs the worst if trained for 1e6 timesteps. But it did train quicker, taking about 17 min, as opposed to 20-22.

A2C trained on LunarLander-v2 for 1e6 timesteps

Trained on my local machine.

The initial run of the lander, after training for 1M timestamps.