Hugging Face's logo

MattStammers
/
appo-atari_alien

Reinforcement Learning
sample-factory
TensorBoard
deep-reinforcement-learning
Eval Results (legacy)
Model card Files
xet
Metrics Community
MattStammers commited on
Commit
a01f551
·
1 Parent(s): 3696df7

Upload folder using huggingface_hub

Browse files
Files changed (12) hide show
  1. .gitattributes +2 -0
  2. .summary/0/events.out.tfevents.1696695192.rhmmedcatt-proliant-ml350-gen10 +3 -0
  3. .summary/1/events.out.tfevents.1696695192.rhmmedcatt-proliant-ml350-gen10 +3 -0
  4. README.md +166 -0
  5. checkpoint_p0/best_000000704_180224_reward_8.280.pth +3 -0
  6. checkpoint_p0/checkpoint_000000736_188416.pth +3 -0
  7. checkpoint_p1/best_000000704_180224_reward_9.140.pth +3 -0
  8. checkpoint_p1/checkpoint_000000736_188416.pth +3 -0
  9. config.json +150 -0
  10. git.diff +3 -0
  11. replay.mp4 +3 -0
  12. sf_log.txt +329 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ git.diff filter=lfs diff=lfs merge=lfs -text
37
+ replay.mp4 filter=lfs diff=lfs merge=lfs -text
.summary/0/events.out.tfevents.1696695192.rhmmedcatt-proliant-ml350-gen10 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76e533d0e3683b948066c125d04827b3cb1abce4db4736d79b313f69fbe85f29
3
+ size 34442
.summary/1/events.out.tfevents.1696695192.rhmmedcatt-proliant-ml350-gen10 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:111afc691d0285ab4f17068f84f22ec52a2b8ed69cfbb4e812d3be0393e5fcf0
3
+ size 26544
README.md ADDED
@@ -0,0 +1,166 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: sample-factory
3
+ tags:
4
+ - deep-reinforcement-learning
5
+ - reinforcement-learning
6
+ - sample-factory
7
+ model-index:
8
+ - name: APPO
9
+ results:
10
+ - task:
11
+ type: reinforcement-learning
12
+ name: reinforcement-learning
13
+ dataset:
14
+ name: atari_alien
15
+ type: atari_alien
16
+ metrics:
17
+ - type: mean_reward
18
+ value: 354.00 +/- 198.66
19
+ name: mean_reward
20
+ verified: false
21
+ ---
22
+
23
+ A(n) **APPO** model trained on the **atari_alien** environment.
24
+
25
+ This model was trained using Sample-Factory 2.0: https://github.com/alex-petrenko/sample-factory.
26
+ Documentation for how to use Sample-Factory can be found at https://www.samplefactory.dev/
27
+
28
+
29
+ ## Downloading the model
30
+
31
+ After installing Sample-Factory, download the model with:
32
+ ```
33
+ python -m sample_factory.huggingface.load_from_hub -r MattStammers/APPO-atari_alien
34
+ ```
35
+
36
+
37
+ ## About the Model
38
+
39
+ This model as with all the others in the benchmarks was trained asynchronously un-seeded to 10 million steps for the purposes of setting a sample factory async baseline for this model on this environment.
40
+
41
+ The aim is to reach state of the art (SOTA) performance on each atari environment. I will flag the models with SOTA when they reach at or near these levels.
42
+
43
+ The hyperparameters used in the model are the sample-factory defaults which are:
44
+ ```
45
+ hyperparameters = {
46
+ "device": "gpu",
47
+ "seed": null,
48
+ "num_policies": 2,
49
+ "async_rl": false,
50
+ "serial_mode": false,
51
+ "batched_sampling": false,
52
+ "num_batches_to_accumulate": 2,
53
+ "worker_num_splits": 1,
54
+ "policy_workers_per_policy": 2,
55
+ "max_policy_lag": 1000,
56
+ "num_workers": 16,
57
+ "num_envs_per_worker": 2,
58
+ "batch_size": 256,
59
+ "num_batches_per_epoch": 8,
60
+ "num_epochs": 4,
61
+ "rollout": 128,
62
+ "recurrence": 1,
63
+ "shuffle_minibatches": false,
64
+ "gamma": 0.99,
65
+ "reward_scale": 1.0,
66
+ "reward_clip": 1000.0,
67
+ "value_bootstrap": false,
68
+ "normalize_returns": true,
69
+ "exploration_loss_coeff": 0.01,
70
+ "value_loss_coeff": 0.5,
71
+ "kl_loss_coeff": 0.0,
72
+ "exploration_loss": "entropy",
73
+ "gae_lambda": 0.95,
74
+ "ppo_clip_ratio": 0.1,
75
+ "ppo_clip_value": 1.0,
76
+ "with_vtrace": false,
77
+ "vtrace_rho": 1.0,
78
+ "vtrace_c": 1.0,
79
+ "optimizer": "adam",
80
+ "adam_eps": 1e-05,
81
+ "adam_beta1": 0.9,
82
+ "adam_beta2": 0.999,
83
+ "max_grad_norm": 0.5,
84
+ "learning_rate": 0.00025,
85
+ "lr_schedule": "linear_decay",
86
+ "lr_schedule_kl_threshold": 0.008,
87
+ "lr_adaptive_min": 1e-06,
88
+ "lr_adaptive_max": 0.01,
89
+ "obs_subtract_mean": 0.0,
90
+ "obs_scale": 255.0,
91
+ "normalize_input": true,
92
+ "normalize_input_keys": null,
93
+ "decorrelate_experience_max_seconds": 0,
94
+ "decorrelate_envs_on_one_worker": true,
95
+ "actor_worker_gpus": [],
96
+ "set_workers_cpu_affinity": true,
97
+ "force_envs_single_thread": false,
98
+ "default_niceness": 0,
99
+ "log_to_file": true,
100
+ "experiment_summaries_interval": 3,
101
+ "flush_summaries_interval": 30,
102
+ "stats_avg": 100,
103
+ "summaries_use_frameskip": true,
104
+ "heartbeat_interval": 20,
105
+ "heartbeat_reporting_interval": 180,
106
+ "train_for_env_steps": 100000000,
107
+ "train_for_seconds": 10000000000,
108
+ "save_every_sec": 120,
109
+ "keep_checkpoints": 2,
110
+ "load_checkpoint_kind": "latest",
111
+ "save_milestones_sec": -1,
112
+ "save_best_every_sec": 5,
113
+ "save_best_metric": "reward",
114
+ "save_best_after": 100000,
115
+ "benchmark": false,
116
+ "encoder_mlp_layers": [
117
+ 512,
118
+ 512
119
+ ],
120
+ "encoder_conv_architecture": "convnet_atari",
121
+ "encoder_conv_mlp_layers": [
122
+ 512
123
+ ],
124
+ "use_rnn": false,
125
+ "rnn_size": 512,
126
+ "rnn_type": "gru",
127
+ "rnn_num_layers": 1,
128
+ "decoder_mlp_layers": [],
129
+ "nonlinearity": "relu",
130
+ "policy_initialization": "orthogonal",
131
+ "policy_init_gain": 1.0,
132
+ "actor_critic_share_weights": true,
133
+ "adaptive_stddev": false,
134
+ "continuous_tanh_scale": 0.0,
135
+ "initial_stddev": 1.0,
136
+ "use_env_info_cache": false,
137
+ "env_gpu_actions": false,
138
+ "env_gpu_observations": true,
139
+ "env_frameskip": 4,
140
+ "env_framestack": 4
141
+ }
142
+
143
+ ```
144
+
145
+
146
+
147
+ ## Using the model
148
+
149
+ To run the model after download, use the `enjoy` script corresponding to this environment:
150
+ ```
151
+ python -m sf_examples.atari.enjoy_atari --algo=APPO --env=atari_alien --train_dir=./train_dir --experiment=APPO-atari_alien
152
+ ```
153
+
154
+
155
+ You can also upload models to the Hugging Face Hub using the same script with the `--push_to_hub` flag.
156
+ See https://www.samplefactory.dev/10-huggingface/huggingface/ for more details
157
+
158
+ ## Training with this model
159
+
160
+ To continue training with this model, use the `train` script corresponding to this environment:
161
+ ```
162
+ python -m sf_examples.atari.train_atari --algo=APPO --env=atari_alien --train_dir=./train_dir --experiment=APPO-atari_alien --restart_behavior=resume --train_for_env_steps=10000000000
163
+ ```
164
+
165
+ Note, you may have to adjust `--train_for_env_steps` to a suitably high number as the experiment will resume at the number of steps it concluded at.
166
+
checkpoint_p0/best_000000704_180224_reward_8.280.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d0ff3643b1575a50e37154f05fbe4cf50c2fa18dea5d14d91652759abdb30973
3
+ size 20795763
checkpoint_p0/checkpoint_000000736_188416.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48760daf9ab56b3bfae3293e5170aa90a637aa62e3f1bbccac10e370f2e8ce0c
3
+ size 20796099
checkpoint_p1/best_000000704_180224_reward_9.140.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4723f8a479fb15df121a1fa434e5f2a33e9f7aff82b9ba4d816842f2128368a9
3
+ size 20795763
checkpoint_p1/checkpoint_000000736_188416.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d3dc43a8b38f07195a6a4893e48d7953ce4f3611f5d3194b909877e52a1986d
3
+ size 20796099
config.json ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "help": false,
3
+ "algo": "APPO",
4
+ "env": "atari_alien",
5
+ "experiment": "atari_alien_APPO",
6
+ "train_dir": "./train_atari",
7
+ "restart_behavior": "restart",
8
+ "device": "gpu",
9
+ "seed": 1234,
10
+ "num_policies": 2,
11
+ "async_rl": false,
12
+ "serial_mode": false,
13
+ "batched_sampling": false,
14
+ "num_batches_to_accumulate": 2,
15
+ "worker_num_splits": 1,
16
+ "policy_workers_per_policy": 1,
17
+ "max_policy_lag": 1000,
18
+ "num_workers": 16,
19
+ "num_envs_per_worker": 2,
20
+ "batch_size": 256,
21
+ "num_batches_per_epoch": 8,
22
+ "num_epochs": 4,
23
+ "rollout": 128,
24
+ "recurrence": 1,
25
+ "shuffle_minibatches": false,
26
+ "gamma": 0.99,
27
+ "reward_scale": 1.0,
28
+ "reward_clip": 1000.0,
29
+ "value_bootstrap": false,
30
+ "normalize_returns": true,
31
+ "exploration_loss_coeff": 0.01,
32
+ "value_loss_coeff": 0.5,
33
+ "kl_loss_coeff": 0.0,
34
+ "exploration_loss": "entropy",
35
+ "gae_lambda": 0.95,
36
+ "ppo_clip_ratio": 0.1,
37
+ "ppo_clip_value": 1.0,
38
+ "with_vtrace": false,
39
+ "vtrace_rho": 1.0,
40
+ "vtrace_c": 1.0,
41
+ "optimizer": "adam",
42
+ "adam_eps": 1e-05,
43
+ "adam_beta1": 0.9,
44
+ "adam_beta2": 0.999,
45
+ "max_grad_norm": 0.5,
46
+ "learning_rate": 0.00025,
47
+ "lr_schedule": "linear_decay",
48
+ "lr_schedule_kl_threshold": 0.008,
49
+ "lr_adaptive_min": 1e-06,
50
+ "lr_adaptive_max": 0.01,
51
+ "obs_subtract_mean": 0.0,
52
+ "obs_scale": 255.0,
53
+ "normalize_input": true,
54
+ "normalize_input_keys": null,
55
+ "decorrelate_experience_max_seconds": 0,
56
+ "decorrelate_envs_on_one_worker": true,
57
+ "actor_worker_gpus": [],
58
+ "set_workers_cpu_affinity": true,
59
+ "force_envs_single_thread": false,
60
+ "default_niceness": 0,
61
+ "log_to_file": true,
62
+ "experiment_summaries_interval": 3,
63
+ "flush_summaries_interval": 30,
64
+ "stats_avg": 100,
65
+ "summaries_use_frameskip": true,
66
+ "heartbeat_interval": 20,
67
+ "heartbeat_reporting_interval": 180,
68
+ "train_for_env_steps": 100000000,
69
+ "train_for_seconds": 10000000000,
70
+ "save_every_sec": 120,
71
+ "keep_checkpoints": 2,
72
+ "load_checkpoint_kind": "latest",
73
+ "save_milestones_sec": -1,
74
+ "save_best_every_sec": 5,
75
+ "save_best_metric": "reward",
76
+ "save_best_after": 100000,
77
+ "benchmark": false,
78
+ "encoder_mlp_layers": [
79
+ 512,
80
+ 512
81
+ ],
82
+ "encoder_conv_architecture": "convnet_atari",
83
+ "encoder_conv_mlp_layers": [
84
+ 512
85
+ ],
86
+ "use_rnn": false,
87
+ "rnn_size": 512,
88
+ "rnn_type": "gru",
89
+ "rnn_num_layers": 1,
90
+ "decoder_mlp_layers": [],
91
+ "nonlinearity": "relu",
92
+ "policy_initialization": "orthogonal",
93
+ "policy_init_gain": 1.0,
94
+ "actor_critic_share_weights": true,
95
+ "adaptive_stddev": false,
96
+ "continuous_tanh_scale": 0.0,
97
+ "initial_stddev": 1.0,
98
+ "use_env_info_cache": false,
99
+ "env_gpu_actions": false,
100
+ "env_gpu_observations": true,
101
+ "env_frameskip": 4,
102
+ "env_framestack": 4,
103
+ "pixel_format": "CHW",
104
+ "use_record_episode_statistics": true,
105
+ "with_wandb": true,
106
+ "wandb_user": "matt-stammers",
107
+ "wandb_project": "atari_APPO",
108
+ "wandb_group": "atari_alien",
109
+ "wandb_job_type": "SF",
110
+ "wandb_tags": [
111
+ "atari"
112
+ ],
113
+ "with_pbt": false,
114
+ "pbt_mix_policies_in_one_env": true,
115
+ "pbt_period_env_steps": 5000000,
116
+ "pbt_start_mutation": 20000000,
117
+ "pbt_replace_fraction": 0.3,
118
+ "pbt_mutation_rate": 0.15,
119
+ "pbt_replace_reward_gap": 0.1,
120
+ "pbt_replace_reward_gap_absolute": 1e-06,
121
+ "pbt_optimize_gamma": false,
122
+ "pbt_target_objective": "true_objective",
123
+ "pbt_perturb_min": 1.1,
124
+ "pbt_perturb_max": 1.5,
125
+ "command_line": "--algo=APPO --env=atari_alien --experiment=atari_alien_APPO --num_policies=2 --restart_behavior=restart --train_dir=./train_atari --train_for_env_steps=100000000 --seed=1234 --num_workers=16 --num_envs_per_worker=2 --num_batches_per_epoch=8 --with_wandb=true --wandb_user=matt-stammers --wandb_project=atari_APPO --wandb_group=atari_alien --wandb_job_type=SF --wandb_tags=atari",
126
+ "cli_args": {
127
+ "algo": "APPO",
128
+ "env": "atari_alien",
129
+ "experiment": "atari_alien_APPO",
130
+ "train_dir": "./train_atari",
131
+ "restart_behavior": "restart",
132
+ "seed": 1234,
133
+ "num_policies": 2,
134
+ "num_workers": 16,
135
+ "num_envs_per_worker": 2,
136
+ "num_batches_per_epoch": 8,
137
+ "train_for_env_steps": 100000000,
138
+ "with_wandb": true,
139
+ "wandb_user": "matt-stammers",
140
+ "wandb_project": "atari_APPO",
141
+ "wandb_group": "atari_alien",
142
+ "wandb_job_type": "SF",
143
+ "wandb_tags": [
144
+ "atari"
145
+ ]
146
+ },
147
+ "git_hash": "5fff97c2f535da5987d358cdbe6927cccd43621e",
148
+ "git_repo_name": "not a git repository",
149
+ "wandb_unique_id": "atari_alien_APPO_20231007_171310_311603"
150
+ }
git.diff ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3357904f421d3f4924836316b1741bf64d5dd0e807d5e80ac07059b4c52a7008
3
+ size 14426734
replay.mp4 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bf812b396ea7f2c4b30ed70292246de2db54668f1689c039176f868a9586f02c
3
+ size 1479645
sf_log.txt ADDED
@@ -0,0 +1,329 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [2023-10-07 17:13:16,813][67527] Saving configuration to ./train_atari/atari_alien_APPO/config.json...
2
+ [2023-10-07 17:13:17,130][67527] Rollout worker 0 uses device cpu
3
+ [2023-10-07 17:13:17,131][67527] Rollout worker 1 uses device cpu
4
+ [2023-10-07 17:13:17,132][67527] Rollout worker 2 uses device cpu
5
+ [2023-10-07 17:13:17,132][67527] Rollout worker 3 uses device cpu
6
+ [2023-10-07 17:13:17,133][67527] Rollout worker 4 uses device cpu
7
+ [2023-10-07 17:13:17,133][67527] Rollout worker 5 uses device cpu
8
+ [2023-10-07 17:13:17,134][67527] Rollout worker 6 uses device cpu
9
+ [2023-10-07 17:13:17,135][67527] Rollout worker 7 uses device cpu
10
+ [2023-10-07 17:13:17,135][67527] Rollout worker 8 uses device cpu
11
+ [2023-10-07 17:13:17,136][67527] Rollout worker 9 uses device cpu
12
+ [2023-10-07 17:13:17,136][67527] Rollout worker 10 uses device cpu
13
+ [2023-10-07 17:13:17,137][67527] Rollout worker 11 uses device cpu
14
+ [2023-10-07 17:13:17,137][67527] Rollout worker 12 uses device cpu
15
+ [2023-10-07 17:13:17,138][67527] Rollout worker 13 uses device cpu
16
+ [2023-10-07 17:13:17,138][67527] Rollout worker 14 uses device cpu
17
+ [2023-10-07 17:13:17,139][67527] Rollout worker 15 uses device cpu
18
+ [2023-10-07 17:13:17,139][67527] In synchronous mode, we only accumulate one batch. Setting num_batches_to_accumulate to 1
19
+ [2023-10-07 17:13:17,233][67527] Using GPUs [0] for process 0 (actually maps to GPUs [0])
20
+ [2023-10-07 17:13:17,234][67527] InferenceWorker_p0-w0: min num requests: 2
21
+ [2023-10-07 17:13:17,237][67527] Using GPUs [1] for process 1 (actually maps to GPUs [1])
22
+ [2023-10-07 17:13:17,237][67527] InferenceWorker_p1-w0: min num requests: 2
23
+ [2023-10-07 17:13:17,282][67527] Starting all processes...
24
+ [2023-10-07 17:13:17,283][67527] Starting process learner_proc0
25
+ [2023-10-07 17:13:18,970][67527] Starting process learner_proc1
26
+ [2023-10-07 17:13:18,974][68242] Using GPUs [0] for process 0 (actually maps to GPUs [0])
27
+ [2023-10-07 17:13:18,974][68242] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0
28
+ [2023-10-07 17:13:18,992][68242] Num visible devices: 1
29
+ [2023-10-07 17:13:19,014][68242] Setting fixed seed 1234
30
+ [2023-10-07 17:13:19,016][68242] Using GPUs [0] for process 0 (actually maps to GPUs [0])
31
+ [2023-10-07 17:13:19,016][68242] Initializing actor-critic model on device cuda:0
32
+ [2023-10-07 17:13:19,017][68242] RunningMeanStd input shape: (4, 84, 84)
33
+ [2023-10-07 17:13:19,018][68242] RunningMeanStd input shape: (1,)
34
+ [2023-10-07 17:13:19,036][68242] ConvEncoder: input_channels=4
35
+ [2023-10-07 17:13:19,211][68242] Conv encoder output size: 512
36
+ [2023-10-07 17:13:19,213][68242] Created Actor Critic model with architecture:
37
+ [2023-10-07 17:13:19,213][68242] ActorCriticSharedWeights(
38
+ (obs_normalizer): ObservationNormalizer(
39
+ (running_mean_std): RunningMeanStdDictInPlace(
40
+ (running_mean_std): ModuleDict(
41
+ (obs): RunningMeanStdInPlace()
42
+ )
43
+ )
44
+ )
45
+ (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace)
46
+ (encoder): MultiInputEncoder(
47
+ (encoders): ModuleDict(
48
+ (obs): ConvEncoder(
49
+ (enc): RecursiveScriptModule(
50
+ original_name=ConvEncoderImpl
51
+ (conv_head): RecursiveScriptModule(
52
+ original_name=Sequential
53
+ (0): RecursiveScriptModule(original_name=Conv2d)
54
+ (1): RecursiveScriptModule(original_name=ReLU)
55
+ (2): RecursiveScriptModule(original_name=Conv2d)
56
+ (3): RecursiveScriptModule(original_name=ReLU)
57
+ (4): RecursiveScriptModule(original_name=Conv2d)
58
+ (5): RecursiveScriptModule(original_name=ReLU)
59
+ )
60
+ (mlp_layers): RecursiveScriptModule(
61
+ original_name=Sequential
62
+ (0): RecursiveScriptModule(original_name=Linear)
63
+ (1): RecursiveScriptModule(original_name=ReLU)
64
+ )
65
+ )
66
+ )
67
+ )
68
+ )
69
+ (core): ModelCoreIdentity()
70
+ (decoder): MlpDecoder(
71
+ (mlp): Identity()
72
+ )
73
+ (critic_linear): Linear(in_features=512, out_features=1, bias=True)
74
+ (action_parameterization): ActionParameterizationDefault(
75
+ (distribution_linear): Linear(in_features=512, out_features=18, bias=True)
76
+ )
77
+ )
78
+ [2023-10-07 17:13:19,747][68242] Using optimizer <class 'torch.optim.adam.Adam'>
79
+ [2023-10-07 17:13:19,747][68242] No checkpoints found
80
+ [2023-10-07 17:13:19,747][68242] Did not load from checkpoint, starting from scratch!
81
+ [2023-10-07 17:13:19,747][68242] Initialized policy 0 weights for model version 0
82
+ [2023-10-07 17:13:19,749][68242] LearnerWorker_p0 finished initialization!
83
+ [2023-10-07 17:13:19,749][68242] Using GPUs [0] for process 0 (actually maps to GPUs [0])
84
+ [2023-10-07 17:13:20,730][67527] Starting all processes...
85
+ [2023-10-07 17:13:20,735][68341] Using GPUs [1] for process 1 (actually maps to GPUs [1])
86
+ [2023-10-07 17:13:20,736][68341] Set environment var CUDA_VISIBLE_DEVICES to '1' (GPU indices [1]) for learning process 1
87
+ [2023-10-07 17:13:20,737][67527] Starting process inference_proc0-0
88
+ [2023-10-07 17:13:20,737][67527] Starting process inference_proc1-0
89
+ [2023-10-07 17:13:20,738][67527] Starting process rollout_proc0
90
+ [2023-10-07 17:13:20,738][67527] Starting process rollout_proc1
91
+ [2023-10-07 17:13:20,738][67527] Starting process rollout_proc2
92
+ [2023-10-07 17:13:20,755][68341] Num visible devices: 1
93
+ [2023-10-07 17:13:20,738][67527] Starting process rollout_proc3
94
+ [2023-10-07 17:13:20,739][67527] Starting process rollout_proc4
95
+ [2023-10-07 17:13:20,777][68341] Setting fixed seed 1234
96
+ [2023-10-07 17:13:20,741][67527] Starting process rollout_proc5
97
+ [2023-10-07 17:13:20,779][68341] Using GPUs [0] for process 1 (actually maps to GPUs [1])
98
+ [2023-10-07 17:13:20,779][68341] Initializing actor-critic model on device cuda:0
99
+ [2023-10-07 17:13:20,743][67527] Starting process rollout_proc6
100
+ [2023-10-07 17:13:20,779][68341] RunningMeanStd input shape: (4, 84, 84)
101
+ [2023-10-07 17:13:20,744][67527] Starting process rollout_proc7
102
+ [2023-10-07 17:13:20,780][68341] RunningMeanStd input shape: (1,)
103
+ [2023-10-07 17:13:20,747][67527] Starting process rollout_proc8
104
+ [2023-10-07 17:13:20,749][67527] Starting process rollout_proc9
105
+ [2023-10-07 17:13:20,752][67527] Starting process rollout_proc10
106
+ [2023-10-07 17:13:20,754][67527] Starting process rollout_proc11
107
+ [2023-10-07 17:13:20,755][67527] Starting process rollout_proc12
108
+ [2023-10-07 17:13:20,757][67527] Starting process rollout_proc13
109
+ [2023-10-07 17:13:20,800][68341] ConvEncoder: input_channels=4
110
+ [2023-10-07 17:13:21,270][68341] Conv encoder output size: 512
111
+ [2023-10-07 17:13:21,273][68341] Created Actor Critic model with architecture:
112
+ [2023-10-07 17:13:21,273][68341] ActorCriticSharedWeights(
113
+ (obs_normalizer): ObservationNormalizer(
114
+ (running_mean_std): RunningMeanStdDictInPlace(
115
+ (running_mean_std): ModuleDict(
116
+ (obs): RunningMeanStdInPlace()
117
+ )
118
+ )
119
+ )
120
+ (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace)
121
+ (encoder): MultiInputEncoder(
122
+ (encoders): ModuleDict(
123
+ (obs): ConvEncoder(
124
+ (enc): RecursiveScriptModule(
125
+ original_name=ConvEncoderImpl
126
+ (conv_head): RecursiveScriptModule(
127
+ original_name=Sequential
128
+ (0): RecursiveScriptModule(original_name=Conv2d)
129
+ (1): RecursiveScriptModule(original_name=ReLU)
130
+ (2): RecursiveScriptModule(original_name=Conv2d)
131
+ (3): RecursiveScriptModule(original_name=ReLU)
132
+ (4): RecursiveScriptModule(original_name=Conv2d)
133
+ (5): RecursiveScriptModule(original_name=ReLU)
134
+ )
135
+ (mlp_layers): RecursiveScriptModule(
136
+ original_name=Sequential
137
+ (0): RecursiveScriptModule(original_name=Linear)
138
+ (1): RecursiveScriptModule(original_name=ReLU)
139
+ )
140
+ )
141
+ )
142
+ )
143
+ )
144
+ (core): ModelCoreIdentity()
145
+ (decoder): MlpDecoder(
146
+ (mlp): Identity()
147
+ )
148
+ (critic_linear): Linear(in_features=512, out_features=1, bias=True)
149
+ (action_parameterization): ActionParameterizationDefault(
150
+ (distribution_linear): Linear(in_features=512, out_features=18, bias=True)
151
+ )
152
+ )
153
+ [2023-10-07 17:13:21,871][68341] Using optimizer <class 'torch.optim.adam.Adam'>
154
+ [2023-10-07 17:13:21,872][68341] No checkpoints found
155
+ [2023-10-07 17:13:21,872][68341] Did not load from checkpoint, starting from scratch!
156
+ [2023-10-07 17:13:21,872][68341] Initialized policy 1 weights for model version 0
157
+ [2023-10-07 17:13:21,874][68341] LearnerWorker_p1 finished initialization!
158
+ [2023-10-07 17:13:21,874][68341] Using GPUs [0] for process 1 (actually maps to GPUs [1])
159
+ [2023-10-07 17:13:22,965][67527] Starting process rollout_proc14
160
+ [2023-10-07 17:13:22,969][68536] Worker 7 uses CPU cores [14, 15]
161
+ [2023-10-07 17:13:22,992][67527] Starting process rollout_proc15
162
+ [2023-10-07 17:13:22,997][68540] Worker 11 uses CPU cores [22, 23]
163
+ [2023-10-07 17:13:23,032][68533] Worker 4 uses CPU cores [8, 9]
164
+ [2023-10-07 17:13:23,108][68541] Worker 12 uses CPU cores [24, 25]
165
+ [2023-10-07 17:13:23,152][68537] Worker 8 uses CPU cores [16, 17]
166
+ [2023-10-07 17:13:23,281][68530] Worker 2 uses CPU cores [4, 5]
167
+ [2023-10-07 17:13:23,308][68498] Worker 1 uses CPU cores [2, 3]
168
+ [2023-10-07 17:13:23,312][68497] Worker 0 uses CPU cores [0, 1]
169
+ [2023-10-07 17:13:23,324][68531] Worker 3 uses CPU cores [6, 7]
170
+ [2023-10-07 17:13:23,325][68535] Worker 6 uses CPU cores [12, 13]
171
+ [2023-10-07 17:13:23,325][68538] Worker 9 uses CPU cores [18, 19]
172
+ [2023-10-07 17:13:23,437][68539] Worker 10 uses CPU cores [20, 21]
173
+ [2023-10-07 17:13:23,482][68491] Using GPUs [0] for process 0 (actually maps to GPUs [0])
174
+ [2023-10-07 17:13:23,482][68491] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0
175
+ [2023-10-07 17:13:23,500][68491] Num visible devices: 1
176
+ [2023-10-07 17:13:23,697][68496] Using GPUs [1] for process 1 (actually maps to GPUs [1])
177
+ [2023-10-07 17:13:23,697][68496] Set environment var CUDA_VISIBLE_DEVICES to '1' (GPU indices [1]) for inference process 1
178
+ [2023-10-07 17:13:23,702][68534] Worker 5 uses CPU cores [10, 11]
179
+ [2023-10-07 17:13:23,715][68496] Num visible devices: 1
180
+ [2023-10-07 17:13:23,838][68542] Worker 13 uses CPU cores [26, 27]
181
+ [2023-10-07 17:13:24,136][68491] RunningMeanStd input shape: (4, 84, 84)
182
+ [2023-10-07 17:13:24,137][68491] RunningMeanStd input shape: (1,)
183
+ [2023-10-07 17:13:24,148][68491] ConvEncoder: input_channels=4
184
+ [2023-10-07 17:13:24,251][68491] Conv encoder output size: 512
185
+ [2023-10-07 17:13:24,337][68496] RunningMeanStd input shape: (4, 84, 84)
186
+ [2023-10-07 17:13:24,338][68496] RunningMeanStd input shape: (1,)
187
+ [2023-10-07 17:13:24,349][68496] ConvEncoder: input_channels=4
188
+ [2023-10-07 17:13:24,450][68496] Conv encoder output size: 512
189
+ [2023-10-07 17:13:24,966][69313] Worker 14 uses CPU cores [28, 29]
190
+ [2023-10-07 17:13:25,058][67527] Inference worker 0-0 is ready!
191
+ [2023-10-07 17:13:25,059][69345] Worker 15 uses CPU cores [30, 31]
192
+ [2023-10-07 17:13:25,059][67527] Inference worker 1-0 is ready!
193
+ [2023-10-07 17:13:25,060][67527] All inference workers are ready! Signal rollout workers to start!
194
+ [2023-10-07 17:13:25,061][67527] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan, 1: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
195
+ [2023-10-07 17:13:25,762][68541] Decorrelating experience for 0 frames...
196
+ [2023-10-07 17:13:25,766][68497] Decorrelating experience for 0 frames...
197
+ [2023-10-07 17:13:25,780][68533] Decorrelating experience for 0 frames...
198
+ [2023-10-07 17:13:25,783][68537] Decorrelating experience for 0 frames...
199
+ [2023-10-07 17:13:25,784][68536] Decorrelating experience for 0 frames...
200
+ [2023-10-07 17:13:25,853][69345] Decorrelating experience for 0 frames...
201
+ [2023-10-07 17:13:25,981][68541] Decorrelating experience for 128 frames...
202
+ [2023-10-07 17:13:25,982][68497] Decorrelating experience for 128 frames...
203
+ [2023-10-07 17:13:26,003][68533] Decorrelating experience for 128 frames...
204
+ [2023-10-07 17:13:26,007][68537] Decorrelating experience for 128 frames...
205
+ [2023-10-07 17:13:26,008][68536] Decorrelating experience for 128 frames...
206
+ [2023-10-07 17:13:26,075][69345] Decorrelating experience for 128 frames...
207
+ [2023-10-07 17:13:26,089][68542] Decorrelating experience for 0 frames...
208
+ [2023-10-07 17:13:26,105][68534] Decorrelating experience for 0 frames...
209
+ [2023-10-07 17:13:26,278][68535] Decorrelating experience for 0 frames...
210
+ [2023-10-07 17:13:26,306][68542] Decorrelating experience for 128 frames...
211
+ [2023-10-07 17:13:26,318][68530] Decorrelating experience for 0 frames...
212
+ [2023-10-07 17:13:26,321][68539] Decorrelating experience for 0 frames...
213
+ [2023-10-07 17:13:26,325][68534] Decorrelating experience for 128 frames...
214
+ [2023-10-07 17:13:26,326][68540] Decorrelating experience for 0 frames...
215
+ [2023-10-07 17:13:26,327][68498] Decorrelating experience for 0 frames...
216
+ [2023-10-07 17:13:26,329][68538] Decorrelating experience for 0 frames...
217
+ [2023-10-07 17:13:26,333][68531] Decorrelating experience for 0 frames...
218
+ [2023-10-07 17:13:26,421][69313] Decorrelating experience for 0 frames...
219
+ [2023-10-07 17:13:26,681][68535] Decorrelating experience for 128 frames...
220
+ [2023-10-07 17:13:26,734][68539] Decorrelating experience for 128 frames...
221
+ [2023-10-07 17:13:26,735][68540] Decorrelating experience for 128 frames...
222
+ [2023-10-07 17:13:26,735][68530] Decorrelating experience for 128 frames...
223
+ [2023-10-07 17:13:26,740][68531] Decorrelating experience for 128 frames...
224
+ [2023-10-07 17:13:26,740][68498] Decorrelating experience for 128 frames...
225
+ [2023-10-07 17:13:26,745][68538] Decorrelating experience for 128 frames...
226
+ [2023-10-07 17:13:26,828][69313] Decorrelating experience for 128 frames...
227
+ [2023-10-07 17:13:27,914][67527] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 184.4, 1: 326.4. Samples: 1457. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0)
228
+ [2023-10-07 17:13:27,915][67527] Avg episode reward: [(1, '1.750')]
229
+ [2023-10-07 17:13:32,914][67527] Fps is (10 sec: 2086.5, 60 sec: 2086.5, 300 sec: 2086.5). Total num frames: 16384. Throughput: 0: 260.8, 1: 260.8. Samples: 4096. Policy #0 lag: (min: 21.0, avg: 21.0, max: 21.0)
230
+ [2023-10-07 17:13:32,915][67527] Avg episode reward: [(0, '8.417'), (1, '3.278')]
231
+ [2023-10-07 17:13:37,221][67527] Heartbeat connected on Batcher_0
232
+ [2023-10-07 17:13:37,224][67527] Heartbeat connected on LearnerWorker_p0
233
+ [2023-10-07 17:13:37,227][67527] Heartbeat connected on Batcher_1
234
+ [2023-10-07 17:13:37,229][67527] Heartbeat connected on LearnerWorker_p1
235
+ [2023-10-07 17:13:37,236][67527] Heartbeat connected on InferenceWorker_p0-w0
236
+ [2023-10-07 17:13:37,239][67527] Heartbeat connected on InferenceWorker_p1-w0
237
+ [2023-10-07 17:13:37,244][67527] Heartbeat connected on RolloutWorker_w0
238
+ [2023-10-07 17:13:37,245][67527] Heartbeat connected on RolloutWorker_w1
239
+ [2023-10-07 17:13:37,251][67527] Heartbeat connected on RolloutWorker_w4
240
+ [2023-10-07 17:13:37,252][67527] Heartbeat connected on RolloutWorker_w2
241
+ [2023-10-07 17:13:37,252][67527] Heartbeat connected on RolloutWorker_w3
242
+ [2023-10-07 17:13:37,256][67527] Heartbeat connected on RolloutWorker_w5
243
+ [2023-10-07 17:13:37,258][67527] Heartbeat connected on RolloutWorker_w6
244
+ [2023-10-07 17:13:37,265][67527] Heartbeat connected on RolloutWorker_w9
245
+ [2023-10-07 17:13:37,267][67527] Heartbeat connected on RolloutWorker_w7
246
+ [2023-10-07 17:13:37,267][67527] Heartbeat connected on RolloutWorker_w8
247
+ [2023-10-07 17:13:37,273][67527] Heartbeat connected on RolloutWorker_w10
248
+ [2023-10-07 17:13:37,276][67527] Heartbeat connected on RolloutWorker_w11
249
+ [2023-10-07 17:13:37,278][67527] Heartbeat connected on RolloutWorker_w12
250
+ [2023-10-07 17:13:37,280][67527] Heartbeat connected on RolloutWorker_w13
251
+ [2023-10-07 17:13:37,280][67527] Heartbeat connected on RolloutWorker_w14
252
+ [2023-10-07 17:13:37,288][67527] Heartbeat connected on RolloutWorker_w15
253
+ [2023-10-07 17:13:37,914][67527] Fps is (10 sec: 6553.6, 60 sec: 5099.1, 300 sec: 5099.1). Total num frames: 65536. Throughput: 0: 599.0, 1: 627.0. Samples: 15757. Policy #0 lag: (min: 22.0, avg: 22.0, max: 22.0)
254
+ [2023-10-07 17:13:37,915][67527] Avg episode reward: [(0, '7.379'), (1, '4.000')]
255
+ [2023-10-07 17:13:42,914][67527] Fps is (10 sec: 9830.4, 60 sec: 6424.2, 300 sec: 6424.2). Total num frames: 114688. Throughput: 0: 803.0, 1: 813.4. Samples: 28857. Policy #0 lag: (min: 31.0, avg: 31.0, max: 31.0)
256
+ [2023-10-07 17:13:42,915][67527] Avg episode reward: [(0, '7.439'), (1, '5.736')]
257
+ [2023-10-07 17:13:47,404][68496] Updated weights for policy 1, policy_version 320 (0.0019)
258
+ [2023-10-07 17:13:47,404][68491] Updated weights for policy 0, policy_version 320 (0.0018)
259
+ [2023-10-07 17:13:47,914][67527] Fps is (10 sec: 9830.5, 60 sec: 7169.5, 300 sec: 7169.5). Total num frames: 163840. Throughput: 0: 806.6, 1: 806.6. Samples: 36864. Policy #0 lag: (min: 31.0, avg: 31.0, max: 31.0)
260
+ [2023-10-07 17:13:47,915][67527] Avg episode reward: [(0, '7.338'), (1, '5.790')]
261
+ [2023-10-07 17:13:52,914][67527] Fps is (10 sec: 9830.3, 60 sec: 7647.1, 300 sec: 7647.1). Total num frames: 212992. Throughput: 0: 915.7, 1: 929.6. Samples: 51396. Policy #0 lag: (min: 31.0, avg: 31.0, max: 31.0)
262
+ [2023-10-07 17:13:52,915][67527] Avg episode reward: [(0, '7.480'), (1, '6.820')]
263
+ [2023-10-07 17:13:52,916][68242] Saving new best policy, reward=7.480!
264
+ [2023-10-07 17:13:52,916][68341] Saving new best policy, reward=6.820!
265
+ [2023-10-07 17:13:57,914][67527] Fps is (10 sec: 9830.3, 60 sec: 7979.4, 300 sec: 7979.4). Total num frames: 262144. Throughput: 0: 997.4, 1: 997.4. Samples: 65536. Policy #0 lag: (min: 26.0, avg: 26.0, max: 26.0)
266
+ [2023-10-07 17:13:57,915][67527] Avg episode reward: [(0, '7.480'), (1, '7.570')]
267
+ [2023-10-07 17:13:57,921][68341] Saving new best policy, reward=7.570!
268
+ [2023-10-07 17:14:02,914][67527] Fps is (10 sec: 9830.7, 60 sec: 8224.0, 300 sec: 8224.0). Total num frames: 311296. Throughput: 0: 967.7, 1: 973.8. Samples: 73490. Policy #0 lag: (min: 31.0, avg: 31.0, max: 31.0)
269
+ [2023-10-07 17:14:02,914][67527] Avg episode reward: [(0, '7.720'), (1, '7.960')]
270
+ [2023-10-07 17:14:02,915][68341] Saving new best policy, reward=7.960!
271
+ [2023-10-07 17:14:02,915][68242] Saving new best policy, reward=7.720!
272
+ [2023-10-07 17:14:04,271][68496] Updated weights for policy 1, policy_version 640 (0.0017)
273
+ [2023-10-07 17:14:04,272][68491] Updated weights for policy 0, policy_version 640 (0.0017)
274
+ [2023-10-07 17:14:07,914][67527] Fps is (10 sec: 9830.6, 60 sec: 8411.4, 300 sec: 8411.4). Total num frames: 360448. Throughput: 0: 1020.5, 1: 1028.6. Samples: 87807. Policy #0 lag: (min: 15.0, avg: 15.0, max: 15.0)
275
+ [2023-10-07 17:14:07,914][67527] Avg episode reward: [(0, '8.280'), (1, '9.140')]
276
+ [2023-10-07 17:14:07,915][68242] Saving new best policy, reward=8.280!
277
+ [2023-10-07 17:14:07,915][68341] Saving new best policy, reward=9.140!
278
+ [2023-10-07 17:14:10,532][67527] Keyboard interrupt detected in the event loop EvtLoop [Runner_EvtLoop, process=main process 67527], exiting...
279
+ [2023-10-07 17:14:10,534][68542] Stopping RolloutWorker_w13...
280
+ [2023-10-07 17:14:10,534][68534] Stopping RolloutWorker_w5...
281
+ [2023-10-07 17:14:10,534][68531] Stopping RolloutWorker_w3...
282
+ [2023-10-07 17:14:10,534][69313] Stopping RolloutWorker_w14...
283
+ [2023-10-07 17:14:10,534][68540] Stopping RolloutWorker_w11...
284
+ [2023-10-07 17:14:10,534][68537] Stopping RolloutWorker_w8...
285
+ [2023-10-07 17:14:10,534][68533] Stopping RolloutWorker_w4...
286
+ [2023-10-07 17:14:10,534][68538] Stopping RolloutWorker_w9...
287
+ [2023-10-07 17:14:10,534][68542] Loop rollout_proc13_evt_loop terminating...
288
+ [2023-10-07 17:14:10,534][68534] Loop rollout_proc5_evt_loop terminating...
289
+ [2023-10-07 17:14:10,534][68540] Loop rollout_proc11_evt_loop terminating...
290
+ [2023-10-07 17:14:10,534][68531] Loop rollout_proc3_evt_loop terminating...
291
+ [2023-10-07 17:14:10,534][68536] Stopping RolloutWorker_w7...
292
+ [2023-10-07 17:14:10,534][69313] Loop rollout_proc14_evt_loop terminating...
293
+ [2023-10-07 17:14:10,534][68535] Stopping RolloutWorker_w6...
294
+ [2023-10-07 17:14:10,534][68539] Stopping RolloutWorker_w10...
295
+ [2023-10-07 17:14:10,534][68530] Stopping RolloutWorker_w2...
296
+ [2023-10-07 17:14:10,534][68537] Loop rollout_proc8_evt_loop terminating...
297
+ [2023-10-07 17:14:10,534][68498] Stopping RolloutWorker_w1...
298
+ [2023-10-07 17:14:10,534][68533] Loop rollout_proc4_evt_loop terminating...
299
+ [2023-10-07 17:14:10,534][68536] Loop rollout_proc7_evt_loop terminating...
300
+ [2023-10-07 17:14:10,534][69345] Stopping RolloutWorker_w15...
301
+ [2023-10-07 17:14:10,534][68535] Loop rollout_proc6_evt_loop terminating...
302
+ [2023-10-07 17:14:10,534][68539] Loop rollout_proc10_evt_loop terminating...
303
+ [2023-10-07 17:14:10,534][68497] Stopping RolloutWorker_w0...
304
+ [2023-10-07 17:14:10,534][68541] Stopping RolloutWorker_w12...
305
+ [2023-10-07 17:14:10,534][68538] Loop rollout_proc9_evt_loop terminating...
306
+ [2023-10-07 17:14:10,534][68242] Stopping Batcher_0...
307
+ [2023-10-07 17:14:10,534][68341] Stopping Batcher_1...
308
+ [2023-10-07 17:14:10,534][67527] Runner profile tree view:
309
+ main_loop: 53.2518
310
+ [2023-10-07 17:14:10,535][69345] Loop rollout_proc15_evt_loop terminating...
311
+ [2023-10-07 17:14:10,535][68530] Loop rollout_proc2_evt_loop terminating...
312
+ [2023-10-07 17:14:10,535][68498] Loop rollout_proc1_evt_loop terminating...
313
+ [2023-10-07 17:14:10,535][68497] Loop rollout_proc0_evt_loop terminating...
314
+ [2023-10-07 17:14:10,535][68541] Loop rollout_proc12_evt_loop terminating...
315
+ [2023-10-07 17:14:10,535][68242] Loop batcher_evt_loop terminating...
316
+ [2023-10-07 17:14:10,535][67527] Collected {0: 188416, 1: 188416}, FPS: 7076.4
317
+ [2023-10-07 17:14:10,536][68341] Loop batcher_evt_loop terminating...
318
+ [2023-10-07 17:14:10,536][68242] Saving ./train_atari/atari_alien_APPO/checkpoint_p0/checkpoint_000000736_188416.pth...
319
+ [2023-10-07 17:14:10,537][68341] Saving ./train_atari/atari_alien_APPO/checkpoint_p1/checkpoint_000000736_188416.pth...
320
+ [2023-10-07 17:14:10,568][68242] Stopping LearnerWorker_p0...
321
+ [2023-10-07 17:14:10,568][68242] Loop learner_proc0_evt_loop terminating...
322
+ [2023-10-07 17:14:10,574][68341] Stopping LearnerWorker_p1...
323
+ [2023-10-07 17:14:10,574][68341] Loop learner_proc1_evt_loop terminating...
324
+ [2023-10-07 17:14:10,582][68491] Weights refcount: 2 0
325
+ [2023-10-07 17:14:10,583][68496] Weights refcount: 2 0
326
+ [2023-10-07 17:14:10,584][68491] Stopping InferenceWorker_p0-w0...
327
+ [2023-10-07 17:14:10,584][68491] Loop inference_proc0-0_evt_loop terminating...
328
+ [2023-10-07 17:14:10,585][68496] Stopping InferenceWorker_p1-w0...
329
+ [2023-10-07 17:14:10,585][68496] Loop inference_proc1-0_evt_loop terminating...