| [2025-02-23 13:36:27,706][00318] Saving configuration to /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/config.json... | |
| [2025-02-23 13:36:27,710][00318] Rollout worker 0 uses device cpu | |
| [2025-02-23 13:36:27,710][00318] Rollout worker 1 uses device cpu | |
| [2025-02-23 13:36:27,711][00318] Rollout worker 2 uses device cpu | |
| [2025-02-23 13:36:27,713][00318] Rollout worker 3 uses device cpu | |
| [2025-02-23 13:36:27,714][00318] Rollout worker 4 uses device cpu | |
| [2025-02-23 13:36:27,715][00318] Rollout worker 5 uses device cpu | |
| [2025-02-23 13:36:27,716][00318] Rollout worker 6 uses device cpu | |
| [2025-02-23 13:36:27,717][00318] Rollout worker 7 uses device cpu | |
| [2025-02-23 13:36:27,719][00318] Rollout worker 8 uses device cpu | |
| [2025-02-23 13:36:27,720][00318] Rollout worker 9 uses device cpu | |
| [2025-02-23 13:36:27,721][00318] Rollout worker 10 uses device cpu | |
| [2025-02-23 13:36:27,722][00318] Rollout worker 11 uses device cpu | |
| [2025-02-23 13:36:27,722][00318] Rollout worker 12 uses device cpu | |
| [2025-02-23 13:36:27,723][00318] Rollout worker 13 uses device cpu | |
| [2025-02-23 13:36:27,724][00318] Rollout worker 14 uses device cpu | |
| [2025-02-23 13:36:27,725][00318] Rollout worker 15 uses device cpu | |
| [2025-02-23 13:36:27,725][00318] Rollout worker 16 uses device cpu | |
| [2025-02-23 13:36:27,726][00318] Rollout worker 17 uses device cpu | |
| [2025-02-23 13:36:27,727][00318] Rollout worker 18 uses device cpu | |
| [2025-02-23 13:36:27,727][00318] Rollout worker 19 uses device cpu | |
| [2025-02-23 13:36:28,489][00318] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-02-23 13:36:28,490][00318] InferenceWorker_p0-w0: min num requests: 6 | |
| [2025-02-23 13:36:28,571][00318] Starting all processes... | |
| [2025-02-23 13:36:28,572][00318] Starting process learner_proc0 | |
| [2025-02-23 13:36:28,639][00318] Starting all processes... | |
| [2025-02-23 13:36:28,649][00318] Starting process inference_proc0-0 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc0 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc1 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc2 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc3 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc4 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc5 | |
| [2025-02-23 13:36:28,650][00318] Starting process rollout_proc6 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc7 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc8 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc9 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc10 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc11 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc12 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc13 | |
| [2025-02-23 13:36:28,651][00318] Starting process rollout_proc14 | |
| [2025-02-23 13:36:29,315][00318] Starting process rollout_proc15 | |
| [2025-02-23 13:36:29,336][00318] Starting process rollout_proc16 | |
| [2025-02-23 13:36:29,381][00318] Starting process rollout_proc17 | |
| [2025-02-23 13:36:29,412][00318] Starting process rollout_proc18 | |
| [2025-02-23 13:36:29,453][00318] Starting process rollout_proc19 | |
| [2025-02-23 13:37:03,982][02489] Worker 12 uses CPU cores [0] | |
| [2025-02-23 13:37:04,062][02456] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-02-23 13:37:04,083][02456] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for learning process 0 | |
| [2025-02-23 13:37:04,102][02490] Worker 13 uses CPU cores [1] | |
| [2025-02-23 13:37:04,282][02485] Worker 8 uses CPU cores [0] | |
| [2025-02-23 13:37:04,352][02456] Num visible devices: 1 | |
| [2025-02-23 13:37:04,392][02479] Worker 2 uses CPU cores [0] | |
| [2025-02-23 13:37:04,455][02456] Setting fixed seed 3333 | |
| [2025-02-23 13:37:04,456][00318] Heartbeat connected on Batcher_0 | |
| [2025-02-23 13:37:04,463][02456] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-02-23 13:37:04,464][02456] Initializing actor-critic model on device cuda:0 | |
| [2025-02-23 13:37:04,469][02456] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-02-23 13:37:04,504][00318] Heartbeat connected on RolloutWorker_w12 | |
| [2025-02-23 13:37:04,501][02456] RunningMeanStd input shape: (1,) | |
| [2025-02-23 13:37:04,578][00318] Heartbeat connected on RolloutWorker_w8 | |
| [2025-02-23 13:37:04,608][00318] Heartbeat connected on RolloutWorker_w2 | |
| [2025-02-23 13:37:04,705][00318] Heartbeat connected on RolloutWorker_w13 | |
| [2025-02-23 13:37:04,789][02456] ConvEncoder: input_channels=3 | |
| [2025-02-23 13:37:04,941][02481] Worker 4 uses CPU cores [0] | |
| [2025-02-23 13:37:05,118][02477] Worker 0 uses CPU cores [0] | |
| [2025-02-23 13:37:05,218][02482] Worker 5 uses CPU cores [1] | |
| [2025-02-23 13:37:05,350][02484] Worker 7 uses CPU cores [1] | |
| [2025-02-23 13:37:05,417][00318] Heartbeat connected on RolloutWorker_w4 | |
| [2025-02-23 13:37:05,478][00318] Heartbeat connected on RolloutWorker_w0 | |
| [2025-02-23 13:37:05,646][02498] Worker 19 uses CPU cores [1] | |
| [2025-02-23 13:37:05,638][02478] Worker 1 uses CPU cores [1] | |
| [2025-02-23 13:37:05,658][02476] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-02-23 13:37:05,658][02476] Set environment var CUDA_VISIBLE_DEVICES to '0' (GPU indices [0]) for inference process 0 | |
| [2025-02-23 13:37:05,673][00318] Heartbeat connected on RolloutWorker_w5 | |
| [2025-02-23 13:37:05,686][02487] Worker 11 uses CPU cores [1] | |
| [2025-02-23 13:37:05,892][00318] Heartbeat connected on RolloutWorker_w7 | |
| [2025-02-23 13:37:05,902][02491] Worker 14 uses CPU cores [0] | |
| [2025-02-23 13:37:05,905][02476] Num visible devices: 1 | |
| [2025-02-23 13:37:05,909][02483] Worker 6 uses CPU cores [0] | |
| [2025-02-23 13:37:05,948][00318] Heartbeat connected on InferenceWorker_p0-w0 | |
| [2025-02-23 13:37:05,965][00318] Heartbeat connected on RolloutWorker_w1 | |
| [2025-02-23 13:37:05,981][00318] Heartbeat connected on RolloutWorker_w19 | |
| [2025-02-23 13:37:05,996][02495] Worker 18 uses CPU cores [0] | |
| [2025-02-23 13:37:06,032][02480] Worker 3 uses CPU cores [1] | |
| [2025-02-23 13:37:06,045][02493] Worker 16 uses CPU cores [0] | |
| [2025-02-23 13:37:06,048][00318] Heartbeat connected on RolloutWorker_w11 | |
| [2025-02-23 13:37:06,053][02488] Worker 10 uses CPU cores [0] | |
| [2025-02-23 13:37:06,056][00318] Heartbeat connected on RolloutWorker_w3 | |
| [2025-02-23 13:37:06,074][00318] Heartbeat connected on RolloutWorker_w14 | |
| [2025-02-23 13:37:06,083][00318] Heartbeat connected on RolloutWorker_w6 | |
| [2025-02-23 13:37:06,130][00318] Heartbeat connected on RolloutWorker_w18 | |
| [2025-02-23 13:37:06,131][02486] Worker 9 uses CPU cores [1] | |
| [2025-02-23 13:37:06,156][00318] Heartbeat connected on RolloutWorker_w9 | |
| [2025-02-23 13:37:06,182][00318] Heartbeat connected on RolloutWorker_w16 | |
| [2025-02-23 13:37:06,183][00318] Heartbeat connected on RolloutWorker_w10 | |
| [2025-02-23 13:37:06,219][02492] Worker 15 uses CPU cores [1] | |
| [2025-02-23 13:37:06,234][00318] Heartbeat connected on RolloutWorker_w15 | |
| [2025-02-23 13:37:06,255][02494] Worker 17 uses CPU cores [1] | |
| [2025-02-23 13:37:06,263][00318] Heartbeat connected on RolloutWorker_w17 | |
| [2025-02-23 13:37:06,269][02456] Conv encoder output size: 512 | |
| [2025-02-23 13:37:06,270][02456] Policy head output size: 512 | |
| [2025-02-23 13:37:06,347][02456] Created Actor Critic model with architecture: | |
| [2025-02-23 13:37:06,347][02456] ActorCriticSharedWeights( | |
| (obs_normalizer): ObservationNormalizer( | |
| (running_mean_std): RunningMeanStdDictInPlace( | |
| (running_mean_std): ModuleDict( | |
| (obs): RunningMeanStdInPlace() | |
| ) | |
| ) | |
| ) | |
| (returns_normalizer): RecursiveScriptModule(original_name=RunningMeanStdInPlace) | |
| (encoder): VizdoomEncoder( | |
| (basic_encoder): ConvEncoder( | |
| (enc): RecursiveScriptModule( | |
| original_name=ConvEncoderImpl | |
| (conv_head): RecursiveScriptModule( | |
| original_name=Sequential | |
| (0): RecursiveScriptModule(original_name=Conv2d) | |
| (1): RecursiveScriptModule(original_name=ReLU) | |
| (2): RecursiveScriptModule(original_name=Conv2d) | |
| (3): RecursiveScriptModule(original_name=ReLU) | |
| (4): RecursiveScriptModule(original_name=Conv2d) | |
| (5): RecursiveScriptModule(original_name=ReLU) | |
| ) | |
| (mlp_layers): RecursiveScriptModule( | |
| original_name=Sequential | |
| (0): RecursiveScriptModule(original_name=Linear) | |
| (1): RecursiveScriptModule(original_name=ReLU) | |
| ) | |
| ) | |
| ) | |
| ) | |
| (core): ModelCoreRNN( | |
| (core): LSTM(512, 512) | |
| ) | |
| (decoder): MlpDecoder( | |
| (mlp): Identity() | |
| ) | |
| (critic_linear): Linear(in_features=512, out_features=1, bias=True) | |
| (action_parameterization): ActionParameterizationDefault( | |
| (distribution_linear): Linear(in_features=512, out_features=11, bias=True) | |
| ) | |
| ) | |
| [2025-02-23 13:37:06,771][02456] Using optimizer <class 'torch.optim.adam.Adam'> | |
| [2025-02-23 13:37:10,917][02456] No checkpoints found | |
| [2025-02-23 13:37:10,917][02456] Did not load from checkpoint, starting from scratch! | |
| [2025-02-23 13:37:10,917][02456] Initialized policy 0 weights for model version 0 | |
| [2025-02-23 13:37:10,920][02456] Using GPUs [0] for process 0 (actually maps to GPUs [0]) | |
| [2025-02-23 13:37:10,927][02456] LearnerWorker_p0 finished initialization! | |
| [2025-02-23 13:37:10,928][00318] Heartbeat connected on LearnerWorker_p0 | |
| [2025-02-23 13:37:11,217][02476] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-02-23 13:37:11,220][02476] RunningMeanStd input shape: (1,) | |
| [2025-02-23 13:37:11,242][02476] ConvEncoder: input_channels=3 | |
| [2025-02-23 13:37:11,349][02476] Conv encoder output size: 512 | |
| [2025-02-23 13:37:11,349][02476] Policy head output size: 512 | |
| [2025-02-23 13:37:11,389][00318] Inference worker 0-0 is ready! | |
| [2025-02-23 13:37:11,390][00318] All inference workers are ready! Signal rollout workers to start! | |
| [2025-02-23 13:37:12,049][02477] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,051][02490] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,063][02493] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,077][02480] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,117][02486] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,139][02484] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,112][02482] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,164][02498] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,154][02479] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,167][02487] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,156][02488] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,166][02478] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,189][02492] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,172][02495] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,181][02485] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,203][02494] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,211][02491] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,208][02489] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,228][02481] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:12,389][02483] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 13:37:13,314][00318] Fps is (10 sec: nan, 60 sec: nan, 300 sec: nan). Total num frames: 0. Throughput: 0: nan. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:16,119][02490] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:16,120][02498] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:16,118][02479] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:16,122][02480] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:16,120][02477] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:16,122][02484] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:16,119][02493] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:17,327][02478] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:17,356][02492] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:17,393][02498] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:18,182][02489] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:18,188][02477] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:18,193][02493] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:18,197][02495] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:18,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:19,622][02490] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:19,616][02494] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:19,730][02498] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:20,030][02492] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:20,063][02488] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:21,120][02495] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:21,141][02491] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:21,139][02481] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:21,867][02486] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:22,402][02493] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:22,413][02477] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:22,867][02485] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:22,898][02488] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:23,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:23,376][02490] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:23,387][02498] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:23,390][02491] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:23,756][02479] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:24,182][02485] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:24,696][02477] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:24,698][02486] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:24,711][02478] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:24,976][02492] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:25,277][02480] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:25,279][02494] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:25,382][02481] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:26,025][02491] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:26,673][02492] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:26,849][02479] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:26,851][02489] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:26,857][02495] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:26,908][02484] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:26,915][02498] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:27,074][02494] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:27,489][02485] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:28,028][02491] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:28,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:28,565][02483] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:28,699][02495] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:29,020][02486] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:29,024][02480] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:29,080][02490] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:29,094][02487] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:29,172][02484] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:30,096][02479] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:30,560][02478] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:30,715][02480] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:30,866][02490] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:31,030][02483] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:31,041][02493] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:31,829][02488] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:31,890][02481] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:31,940][02477] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:32,091][02495] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:32,152][02484] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:33,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:34,254][02479] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:34,629][02482] Another process currently holds the lock /tmp/sf2_root/doom_008.lockfile, attempt: 1 | |
| [2025-02-23 13:37:35,126][02498] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:35,141][02480] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:35,393][02490] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:35,963][02483] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:36,146][02491] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:36,385][02494] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:36,575][02478] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:36,701][02487] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:38,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:38,346][02485] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:38,522][02484] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:38,979][02477] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:39,000][02480] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:39,012][02479] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:39,076][02492] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:39,092][02495] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:39,585][02487] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:39,627][02494] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:40,814][02491] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:41,297][02482] Decorrelating experience for 0 frames... | |
| [2025-02-23 13:37:41,627][02490] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:41,629][02478] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:41,757][02483] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:41,808][02487] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:41,863][02481] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:43,259][02485] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:43,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:43,593][02493] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:43,597][02477] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:43,598][02484] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:43,816][02480] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:43,887][02486] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:44,125][02489] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:44,139][02478] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:45,210][02481] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:45,212][02488] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:45,654][02491] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:45,983][02498] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:46,441][02482] Decorrelating experience for 32 frames... | |
| [2025-02-23 13:37:46,688][02494] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:46,851][02489] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:47,229][02488] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:47,330][02480] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:47,679][02479] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:47,822][02486] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:48,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:48,600][02485] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:49,505][02483] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:49,855][02482] Decorrelating experience for 64 frames... | |
| [2025-02-23 13:37:50,009][02484] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:50,884][02488] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:51,161][02498] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:51,501][02481] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:51,591][02492] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:52,518][02480] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:37:52,826][02493] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:53,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:54,041][02495] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:54,391][02489] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:37:54,659][02478] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:55,394][02483] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:37:55,685][02484] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:56,062][02490] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:57,145][02482] Decorrelating experience for 96 frames... | |
| [2025-02-23 13:37:58,135][02477] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:58,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:37:59,025][02491] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:59,397][02479] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:37:59,602][02488] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:59,694][02485] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:37:59,998][02478] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:00,172][02492] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:00,816][02498] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:00,921][02484] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:01,173][02486] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:38:01,991][02483] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:02,146][02490] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:02,171][02493] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:03,013][02495] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:03,136][02494] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:03,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:03,528][02488] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:03,810][02479] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:04,048][02491] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:04,084][02487] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:38:05,074][02486] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:05,219][02478] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:05,446][02477] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:05,569][02481] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:06,058][02490] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:06,367][02480] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:06,620][02493] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:06,862][02498] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:07,207][02483] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:08,065][02494] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:08,115][02488] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:08,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:08,378][02491] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:08,397][02484] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:08,951][02485] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:09,253][02478] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:09,322][02495] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:11,570][02498] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:11,760][02483] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:11,799][02493] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:11,998][02490] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:12,046][02482] Decorrelating experience for 128 frames... | |
| [2025-02-23 13:38:12,709][02494] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:12,983][02492] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:13,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:13,727][02480] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:13,923][02479] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:14,272][02491] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:15,820][02495] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:16,556][02485] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:16,721][02481] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:17,142][02478] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:18,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:18,318][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000000_0.pth... | |
| [2025-02-23 13:38:18,877][02489] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:38:19,130][02482] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:38:20,597][02492] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:20,608][02494] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:21,627][02479] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:22,152][02493] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:22,622][02484] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:23,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:23,412][02488] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:23,682][02480] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:24,560][02483] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:24,582][02487] Another process currently holds the lock /tmp/sf2_root/doom_005.lockfile, attempt: 1 | |
| [2025-02-23 13:38:24,831][02481] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:26,026][02491] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:26,606][02478] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:26,722][02477] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:27,160][02486] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:27,218][02492] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:27,708][02498] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:28,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 0.0. Samples: 0. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:28,664][02483] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:28,791][02494] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:29,612][02484] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:30,926][02488] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:32,701][02489] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:33,235][02482] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:33,257][02479] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:33,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 18.0. Samples: 810. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:33,315][00318] Avg episode reward: [(0, '-0.825')] | |
| [2025-02-23 13:38:33,952][02493] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:34,022][02491] Worker 14, sleep for 0.700 sec to decorrelate experience collection | |
| [2025-02-23 13:38:34,724][02491] Worker 14 awakens! | |
| [2025-02-23 13:38:35,116][02485] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:35,460][02480] Worker 3, sleep for 0.150 sec to decorrelate experience collection | |
| [2025-02-23 13:38:35,611][02480] Worker 3 awakens! | |
| [2025-02-23 13:38:36,392][02478] Worker 1, sleep for 0.050 sec to decorrelate experience collection | |
| [2025-02-23 13:38:36,443][02478] Worker 1 awakens! | |
| [2025-02-23 13:38:36,929][02486] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:38,255][02495] Another process currently holds the lock /tmp/sf2_root/doom_004.lockfile, attempt: 1 | |
| [2025-02-23 13:38:38,315][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 31.5. Samples: 1416. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:38,316][00318] Avg episode reward: [(0, '-0.721')] | |
| [2025-02-23 13:38:38,657][02498] Worker 19, sleep for 0.950 sec to decorrelate experience collection | |
| [2025-02-23 13:38:38,992][02483] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:39,127][02490] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:39,608][02498] Worker 19 awakens! | |
| [2025-02-23 13:38:40,158][02487] Decorrelating experience for 160 frames... | |
| [2025-02-23 13:38:40,333][02484] Worker 7, sleep for 0.350 sec to decorrelate experience collection | |
| [2025-02-23 13:38:40,540][02492] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:40,718][02484] Worker 7 awakens! | |
| [2025-02-23 13:38:41,299][02494] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:43,314][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 72.9. Samples: 3282. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:43,319][00318] Avg episode reward: [(0, '-0.675')] | |
| [2025-02-23 13:38:43,712][02477] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:44,476][02479] Worker 2, sleep for 0.100 sec to decorrelate experience collection | |
| [2025-02-23 13:38:44,581][02479] Worker 2 awakens! | |
| [2025-02-23 13:38:44,957][02489] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:45,448][02493] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:46,698][02488] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:46,759][02495] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:47,667][02485] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:38:48,315][00318] Fps is (10 sec: 0.0, 60 sec: 0.0, 300 sec: 0.0). Total num frames: 0. Throughput: 0: 117.6. Samples: 5292. Policy #0 lag: (min: -1.0, avg: -1.0, max: -1.0) | |
| [2025-02-23 13:38:48,318][00318] Avg episode reward: [(0, '-0.622')] | |
| [2025-02-23 13:38:48,392][02481] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:49,153][02456] Signal inference workers to stop experience collection... | |
| [2025-02-23 13:38:49,201][02476] InferenceWorker_p0-w0: stopping experience collection | |
| [2025-02-23 13:38:49,945][02456] Signal inference workers to resume experience collection... | |
| [2025-02-23 13:38:49,946][02476] InferenceWorker_p0-w0: resuming experience collection | |
| [2025-02-23 13:38:50,604][02483] Worker 6, sleep for 0.300 sec to decorrelate experience collection | |
| [2025-02-23 13:38:50,939][02483] Worker 6 awakens! | |
| [2025-02-23 13:38:52,141][02490] Worker 13, sleep for 0.650 sec to decorrelate experience collection | |
| [2025-02-23 13:38:52,792][02490] Worker 13 awakens! | |
| [2025-02-23 13:38:52,925][02492] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 409.6, 300 sec: 245.8). Total num frames: 24576. Throughput: 0: 140.8. Samples: 6336. Policy #0 lag: (min: 0.0, avg: 1.7, max: 2.0) | |
| [2025-02-23 13:38:53,318][00318] Avg episode reward: [(0, '-0.580')] | |
| [2025-02-23 13:38:53,405][02482] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:38:53,475][02486] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:38:54,840][02494] Worker 17, sleep for 0.850 sec to decorrelate experience collection | |
| [2025-02-23 13:38:55,190][02489] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:38:55,718][02494] Worker 17 awakens! | |
| [2025-02-23 13:38:55,884][02493] Worker 16, sleep for 0.800 sec to decorrelate experience collection | |
| [2025-02-23 13:38:56,110][02487] Decorrelating experience for 192 frames... | |
| [2025-02-23 13:38:56,616][02488] Worker 10, sleep for 0.500 sec to decorrelate experience collection | |
| [2025-02-23 13:38:56,698][02493] Worker 16 awakens! | |
| [2025-02-23 13:38:56,906][02495] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:57,116][02488] Worker 10 awakens! | |
| [2025-02-23 13:38:58,206][02485] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:38:58,314][00318] Fps is (10 sec: 3277.0, 60 sec: 546.1, 300 sec: 312.1). Total num frames: 32768. Throughput: 0: 225.1. Samples: 10128. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) | |
| [2025-02-23 13:38:58,316][00318] Avg episode reward: [(0, '-0.445')] | |
| [2025-02-23 13:39:03,315][00318] Fps is (10 sec: 1638.2, 60 sec: 682.7, 300 sec: 372.4). Total num frames: 40960. Throughput: 0: 286.0. Samples: 12870. Policy #0 lag: (min: 0.0, avg: 1.1, max: 3.0) | |
| [2025-02-23 13:39:03,321][00318] Avg episode reward: [(0, '-0.363')] | |
| [2025-02-23 13:39:04,177][02477] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:39:05,366][02481] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:39:08,123][02482] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:39:08,314][00318] Fps is (10 sec: 1638.4, 60 sec: 819.2, 300 sec: 427.4). Total num frames: 49152. Throughput: 0: 321.5. Samples: 14466. Policy #0 lag: (min: 0.0, avg: 0.9, max: 3.0) | |
| [2025-02-23 13:39:08,318][00318] Avg episode reward: [(0, '-0.410')] | |
| [2025-02-23 13:39:08,853][02486] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:39:08,948][02492] Worker 15, sleep for 0.750 sec to decorrelate experience collection | |
| [2025-02-23 13:39:09,698][02492] Worker 15 awakens! | |
| [2025-02-23 13:39:11,483][02495] Worker 18, sleep for 0.900 sec to decorrelate experience collection | |
| [2025-02-23 13:39:12,406][02495] Worker 18 awakens! | |
| [2025-02-23 13:39:12,536][02485] Worker 8, sleep for 0.400 sec to decorrelate experience collection | |
| [2025-02-23 13:39:12,956][02485] Worker 8 awakens! | |
| [2025-02-23 13:39:13,314][00318] Fps is (10 sec: 2457.9, 60 sec: 1092.3, 300 sec: 546.1). Total num frames: 65536. Throughput: 0: 399.5. Samples: 17976. Policy #0 lag: (min: 0.0, avg: 1.2, max: 3.0) | |
| [2025-02-23 13:39:13,315][00318] Avg episode reward: [(0, '-0.309')] | |
| [2025-02-23 13:39:13,713][02489] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:39:16,145][02487] Decorrelating experience for 224 frames... | |
| [2025-02-23 13:39:17,253][02476] Updated weights for policy 0, policy_version 10 (0.0014) | |
| [2025-02-23 13:39:17,943][02482] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:39:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 1365.3, 300 sec: 655.4). Total num frames: 81920. Throughput: 0: 505.7. Samples: 23568. Policy #0 lag: (min: 0.0, avg: 1.5, max: 3.0) | |
| [2025-02-23 13:39:18,317][00318] Avg episode reward: [(0, '-0.046')] | |
| [2025-02-23 13:39:18,427][02481] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:39:19,823][02486] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:39:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 1638.4, 300 sec: 756.2). Total num frames: 98304. Throughput: 0: 547.9. Samples: 26070. Policy #0 lag: (min: 0.0, avg: 1.3, max: 3.0) | |
| [2025-02-23 13:39:23,315][00318] Avg episode reward: [(0, '-0.003')] | |
| [2025-02-23 13:39:27,224][02487] Decorrelating experience for 256 frames... | |
| [2025-02-23 13:39:28,314][00318] Fps is (10 sec: 2457.6, 60 sec: 1774.9, 300 sec: 788.9). Total num frames: 106496. Throughput: 0: 587.7. Samples: 29730. Policy #0 lag: (min: 0.0, avg: 1.6, max: 3.0) | |
| [2025-02-23 13:39:28,315][00318] Avg episode reward: [(0, '0.349')] | |
| [2025-02-23 13:39:28,324][02456] Saving new best policy, reward=0.349! | |
| [2025-02-23 13:39:31,614][02489] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:39:33,314][00318] Fps is (10 sec: 2457.6, 60 sec: 2048.0, 300 sec: 877.7). Total num frames: 122880. Throughput: 0: 628.9. Samples: 33594. Policy #0 lag: (min: 0.0, avg: 1.2, max: 4.0) | |
| [2025-02-23 13:39:33,318][00318] Avg episode reward: [(0, '0.505')] | |
| [2025-02-23 13:39:33,321][02456] Saving new best policy, reward=0.505! | |
| [2025-02-23 13:39:34,053][02481] Worker 4, sleep for 0.200 sec to decorrelate experience collection | |
| [2025-02-23 13:39:34,189][02486] Worker 9, sleep for 0.450 sec to decorrelate experience collection | |
| [2025-02-23 13:39:34,285][02481] Worker 4 awakens! | |
| [2025-02-23 13:39:34,661][02486] Worker 9 awakens! | |
| [2025-02-23 13:39:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 2321.1, 300 sec: 960.4). Total num frames: 139264. Throughput: 0: 669.5. Samples: 36462. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 13:39:38,316][00318] Avg episode reward: [(0, '0.748')] | |
| [2025-02-23 13:39:38,320][02456] Saving new best policy, reward=0.748! | |
| [2025-02-23 13:39:39,133][02482] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:39:39,661][02487] Decorrelating experience for 288 frames... | |
| [2025-02-23 13:39:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 2594.1, 300 sec: 1037.7). Total num frames: 155648. Throughput: 0: 698.4. Samples: 41556. Policy #0 lag: (min: 0.0, avg: 1.4, max: 4.0) | |
| [2025-02-23 13:39:43,316][00318] Avg episode reward: [(0, '0.544')] | |
| [2025-02-23 13:39:46,740][02476] Updated weights for policy 0, policy_version 20 (0.0014) | |
| [2025-02-23 13:39:48,315][00318] Fps is (10 sec: 2457.4, 60 sec: 2730.7, 300 sec: 1057.0). Total num frames: 163840. Throughput: 0: 726.4. Samples: 45558. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 13:39:48,318][00318] Avg episode reward: [(0, '0.735')] | |
| [2025-02-23 13:39:50,214][02489] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:39:53,317][00318] Fps is (10 sec: 2457.0, 60 sec: 2594.0, 300 sec: 1126.4). Total num frames: 180224. Throughput: 0: 730.9. Samples: 47358. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:39:53,318][00318] Avg episode reward: [(0, '0.476')] | |
| [2025-02-23 13:39:55,893][02487] Decorrelating experience for 320 frames... | |
| [2025-02-23 13:39:55,914][02482] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:39:58,314][00318] Fps is (10 sec: 2457.7, 60 sec: 2594.1, 300 sec: 1141.9). Total num frames: 188416. Throughput: 0: 727.6. Samples: 50718. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:39:58,316][00318] Avg episode reward: [(0, '1.078')] | |
| [2025-02-23 13:39:58,460][02456] Saving new best policy, reward=1.078! | |
| [2025-02-23 13:40:03,314][00318] Fps is (10 sec: 3277.7, 60 sec: 2867.3, 300 sec: 1252.9). Total num frames: 212992. Throughput: 0: 738.7. Samples: 56808. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:40:03,316][00318] Avg episode reward: [(0, '0.873')] | |
| [2025-02-23 13:40:04,064][02489] Worker 12, sleep for 0.600 sec to decorrelate experience collection | |
| [2025-02-23 13:40:04,680][02489] Worker 12 awakens! | |
| [2025-02-23 13:40:07,103][02482] Worker 5, sleep for 0.250 sec to decorrelate experience collection | |
| [2025-02-23 13:40:07,353][02482] Worker 5 awakens! | |
| [2025-02-23 13:40:07,945][02487] Decorrelating experience for 352 frames... | |
| [2025-02-23 13:40:08,316][00318] Fps is (10 sec: 4095.5, 60 sec: 3003.7, 300 sec: 1310.7). Total num frames: 229376. Throughput: 0: 755.7. Samples: 60078. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:40:08,321][00318] Avg episode reward: [(0, '1.058')] | |
| [2025-02-23 13:40:13,316][00318] Fps is (10 sec: 3276.2, 60 sec: 3003.6, 300 sec: 1365.3). Total num frames: 245760. Throughput: 0: 765.6. Samples: 64182. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:40:13,318][00318] Avg episode reward: [(0, '1.098')] | |
| [2025-02-23 13:40:13,327][02456] Saving new best policy, reward=1.098! | |
| [2025-02-23 13:40:13,340][02476] Updated weights for policy 0, policy_version 30 (0.0013) | |
| [2025-02-23 13:40:18,314][00318] Fps is (10 sec: 2458.0, 60 sec: 2867.2, 300 sec: 1372.7). Total num frames: 253952. Throughput: 0: 766.3. Samples: 68076. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 13:40:18,316][00318] Avg episode reward: [(0, '1.096')] | |
| [2025-02-23 13:40:18,322][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000031_253952.pth... | |
| [2025-02-23 13:40:22,503][02487] Worker 11, sleep for 0.550 sec to decorrelate experience collection | |
| [2025-02-23 13:40:23,054][02487] Worker 11 awakens! | |
| [2025-02-23 13:40:23,314][00318] Fps is (10 sec: 3277.4, 60 sec: 3003.7, 300 sec: 1465.9). Total num frames: 278528. Throughput: 0: 750.1. Samples: 70218. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:40:23,315][00318] Avg episode reward: [(0, '1.237')] | |
| [2025-02-23 13:40:23,316][02456] Saving new best policy, reward=1.237! | |
| [2025-02-23 13:40:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3140.3, 300 sec: 1512.4). Total num frames: 294912. Throughput: 0: 783.3. Samples: 76806. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:40:28,315][00318] Avg episode reward: [(0, '1.258')] | |
| [2025-02-23 13:40:28,322][02456] Saving new best policy, reward=1.258! | |
| [2025-02-23 13:40:33,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3140.3, 300 sec: 1556.5). Total num frames: 311296. Throughput: 0: 821.6. Samples: 82530. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:40:33,315][00318] Avg episode reward: [(0, '1.348')] | |
| [2025-02-23 13:40:33,317][02456] Saving new best policy, reward=1.348! | |
| [2025-02-23 13:40:38,237][02476] Updated weights for policy 0, policy_version 40 (0.0013) | |
| [2025-02-23 13:40:38,316][00318] Fps is (10 sec: 3276.2, 60 sec: 3140.2, 300 sec: 1598.4). Total num frames: 327680. Throughput: 0: 825.6. Samples: 84510. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:40:38,325][00318] Avg episode reward: [(0, '1.483')] | |
| [2025-02-23 13:40:38,328][02456] Saving new best policy, reward=1.483! | |
| [2025-02-23 13:40:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3140.3, 300 sec: 1638.4). Total num frames: 344064. Throughput: 0: 842.1. Samples: 88614. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:40:43,320][00318] Avg episode reward: [(0, '1.588')] | |
| [2025-02-23 13:40:43,322][02456] Saving new best policy, reward=1.588! | |
| [2025-02-23 13:40:48,314][00318] Fps is (10 sec: 2458.0, 60 sec: 3140.3, 300 sec: 1638.4). Total num frames: 352256. Throughput: 0: 818.1. Samples: 93624. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:40:48,319][00318] Avg episode reward: [(0, '1.226')] | |
| [2025-02-23 13:40:53,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3276.9, 300 sec: 1712.9). Total num frames: 376832. Throughput: 0: 818.6. Samples: 96912. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:40:53,319][00318] Avg episode reward: [(0, '1.752')] | |
| [2025-02-23 13:40:53,381][02456] Saving new best policy, reward=1.752! | |
| [2025-02-23 13:40:58,317][00318] Fps is (10 sec: 4913.9, 60 sec: 3549.7, 300 sec: 1784.0). Total num frames: 401408. Throughput: 0: 858.8. Samples: 102828. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:40:58,318][00318] Avg episode reward: [(0, '1.735')] | |
| [2025-02-23 13:41:01,562][02476] Updated weights for policy 0, policy_version 50 (0.0013) | |
| [2025-02-23 13:41:03,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3276.8, 300 sec: 1780.9). Total num frames: 409600. Throughput: 0: 865.9. Samples: 107040. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:41:03,315][00318] Avg episode reward: [(0, '1.861')] | |
| [2025-02-23 13:41:03,317][02456] Saving new best policy, reward=1.861! | |
| [2025-02-23 13:41:08,314][00318] Fps is (10 sec: 2458.2, 60 sec: 3276.9, 300 sec: 1812.7). Total num frames: 425984. Throughput: 0: 863.1. Samples: 109056. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 13:41:08,315][00318] Avg episode reward: [(0, '1.586')] | |
| [2025-02-23 13:41:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.9, 300 sec: 1843.2). Total num frames: 442368. Throughput: 0: 839.9. Samples: 114600. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:41:13,315][00318] Avg episode reward: [(0, '1.931')] | |
| [2025-02-23 13:41:13,319][02456] Saving new best policy, reward=1.931! | |
| [2025-02-23 13:41:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 1905.9). Total num frames: 466944. Throughput: 0: 835.5. Samples: 120126. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:41:18,324][00318] Avg episode reward: [(0, '1.673')] | |
| [2025-02-23 13:41:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 1900.5). Total num frames: 475136. Throughput: 0: 843.1. Samples: 122448. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:41:23,316][00318] Avg episode reward: [(0, '1.677')] | |
| [2025-02-23 13:41:26,447][02476] Updated weights for policy 0, policy_version 60 (0.0023) | |
| [2025-02-23 13:41:28,314][00318] Fps is (10 sec: 2457.5, 60 sec: 3276.8, 300 sec: 1927.5). Total num frames: 491520. Throughput: 0: 842.4. Samples: 126522. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:41:28,318][00318] Avg episode reward: [(0, '1.843')] | |
| [2025-02-23 13:41:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 1953.5). Total num frames: 507904. Throughput: 0: 831.5. Samples: 131040. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:41:33,317][00318] Avg episode reward: [(0, '1.975')] | |
| [2025-02-23 13:41:33,320][02456] Saving new best policy, reward=1.975! | |
| [2025-02-23 13:41:38,314][00318] Fps is (10 sec: 4096.2, 60 sec: 3413.4, 300 sec: 2009.4). Total num frames: 532480. Throughput: 0: 829.7. Samples: 134250. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:41:38,316][00318] Avg episode reward: [(0, '2.260')] | |
| [2025-02-23 13:41:38,321][02456] Saving new best policy, reward=2.260! | |
| [2025-02-23 13:41:43,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3413.3, 300 sec: 2032.8). Total num frames: 548864. Throughput: 0: 842.6. Samples: 140742. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:41:43,325][00318] Avg episode reward: [(0, '1.879')] | |
| [2025-02-23 13:41:48,317][00318] Fps is (10 sec: 2457.0, 60 sec: 3413.2, 300 sec: 2025.6). Total num frames: 557056. Throughput: 0: 840.5. Samples: 144864. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:41:48,321][00318] Avg episode reward: [(0, '2.113')] | |
| [2025-02-23 13:41:51,551][02476] Updated weights for policy 0, policy_version 70 (0.0013) | |
| [2025-02-23 13:41:53,314][00318] Fps is (10 sec: 2457.8, 60 sec: 3276.8, 300 sec: 2048.0). Total num frames: 573440. Throughput: 0: 842.8. Samples: 146982. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:41:53,316][00318] Avg episode reward: [(0, '2.108')] | |
| [2025-02-23 13:41:58,314][00318] Fps is (10 sec: 3277.7, 60 sec: 3140.4, 300 sec: 2069.6). Total num frames: 589824. Throughput: 0: 810.4. Samples: 151068. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:41:58,315][00318] Avg episode reward: [(0, '2.357')] | |
| [2025-02-23 13:41:58,323][02456] Saving new best policy, reward=2.357! | |
| [2025-02-23 13:42:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 2118.6). Total num frames: 614400. Throughput: 0: 831.5. Samples: 157542. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:42:03,315][00318] Avg episode reward: [(0, '2.163')] | |
| [2025-02-23 13:42:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 2138.3). Total num frames: 630784. Throughput: 0: 852.8. Samples: 160824. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:42:08,315][00318] Avg episode reward: [(0, '2.291')] | |
| [2025-02-23 13:42:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 2193.8). Total num frames: 647168. Throughput: 0: 864.4. Samples: 165420. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:42:13,316][00318] Avg episode reward: [(0, '2.264')] | |
| [2025-02-23 13:42:15,015][02476] Updated weights for policy 0, policy_version 80 (0.0016) | |
| [2025-02-23 13:42:18,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.3, 300 sec: 2221.6). Total num frames: 655360. Throughput: 0: 852.8. Samples: 169416. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:42:18,316][00318] Avg episode reward: [(0, '1.995')] | |
| [2025-02-23 13:42:18,328][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000080_655360.pth... | |
| [2025-02-23 13:42:18,685][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000000_0.pth | |
| [2025-02-23 13:42:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 2304.9). Total num frames: 679936. Throughput: 0: 826.9. Samples: 171462. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:42:23,316][00318] Avg episode reward: [(0, '2.407')] | |
| [2025-02-23 13:42:23,319][02456] Saving new best policy, reward=2.407! | |
| [2025-02-23 13:42:28,314][00318] Fps is (10 sec: 4095.9, 60 sec: 3413.3, 300 sec: 2360.4). Total num frames: 696320. Throughput: 0: 818.3. Samples: 177564. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:42:28,317][00318] Avg episode reward: [(0, '2.534')] | |
| [2025-02-23 13:42:28,325][02456] Saving new best policy, reward=2.534! | |
| [2025-02-23 13:42:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 2415.9). Total num frames: 712704. Throughput: 0: 857.8. Samples: 183462. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:42:33,316][00318] Avg episode reward: [(0, '3.100')] | |
| [2025-02-23 13:42:33,321][02456] Saving new best policy, reward=3.100! | |
| [2025-02-23 13:42:38,317][00318] Fps is (10 sec: 3275.9, 60 sec: 3276.6, 300 sec: 2471.5). Total num frames: 729088. Throughput: 0: 853.9. Samples: 185412. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:42:38,318][00318] Avg episode reward: [(0, '2.636')] | |
| [2025-02-23 13:42:39,365][02476] Updated weights for policy 0, policy_version 90 (0.0015) | |
| [2025-02-23 13:42:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 2527.0). Total num frames: 745472. Throughput: 0: 854.8. Samples: 189534. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:42:43,324][00318] Avg episode reward: [(0, '2.766')] | |
| [2025-02-23 13:42:48,314][00318] Fps is (10 sec: 3277.8, 60 sec: 3413.5, 300 sec: 2582.6). Total num frames: 761856. Throughput: 0: 827.2. Samples: 194766. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:42:48,315][00318] Avg episode reward: [(0, '3.111')] | |
| [2025-02-23 13:42:48,320][02456] Saving new best policy, reward=3.111! | |
| [2025-02-23 13:42:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 2638.1). Total num frames: 778240. Throughput: 0: 812.7. Samples: 197394. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:42:53,317][00318] Avg episode reward: [(0, '3.080')] | |
| [2025-02-23 13:42:58,317][00318] Fps is (10 sec: 4094.6, 60 sec: 3549.7, 300 sec: 2721.4). Total num frames: 802816. Throughput: 0: 834.7. Samples: 202986. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:42:58,320][00318] Avg episode reward: [(0, '2.739')] | |
| [2025-02-23 13:43:03,317][00318] Fps is (10 sec: 3275.9, 60 sec: 3276.6, 300 sec: 2749.2). Total num frames: 811008. Throughput: 0: 837.7. Samples: 207114. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:43:03,318][00318] Avg episode reward: [(0, '2.499')] | |
| [2025-02-23 13:43:05,069][02476] Updated weights for policy 0, policy_version 100 (0.0038) | |
| [2025-02-23 13:43:08,314][00318] Fps is (10 sec: 2458.4, 60 sec: 3276.8, 300 sec: 2804.7). Total num frames: 827392. Throughput: 0: 838.3. Samples: 209184. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:43:08,316][00318] Avg episode reward: [(0, '2.957')] | |
| [2025-02-23 13:43:13,314][00318] Fps is (10 sec: 3277.7, 60 sec: 3276.8, 300 sec: 2860.3). Total num frames: 843776. Throughput: 0: 829.1. Samples: 214872. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:43:13,315][00318] Avg episode reward: [(0, '2.760')] | |
| [2025-02-23 13:43:18,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3549.8, 300 sec: 2943.6). Total num frames: 868352. Throughput: 0: 843.9. Samples: 221436. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 13:43:18,317][00318] Avg episode reward: [(0, '2.910')] | |
| [2025-02-23 13:43:23,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3413.3, 300 sec: 2999.1). Total num frames: 884736. Throughput: 0: 845.8. Samples: 223470. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:43:23,320][00318] Avg episode reward: [(0, '2.769')] | |
| [2025-02-23 13:43:28,314][00318] Fps is (10 sec: 2457.8, 60 sec: 3276.8, 300 sec: 3026.9). Total num frames: 892928. Throughput: 0: 845.5. Samples: 227580. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:43:28,320][00318] Avg episode reward: [(0, '2.692')] | |
| [2025-02-23 13:43:28,734][02476] Updated weights for policy 0, policy_version 110 (0.0013) | |
| [2025-02-23 13:43:33,317][00318] Fps is (10 sec: 2457.1, 60 sec: 3276.7, 300 sec: 3082.4). Total num frames: 909312. Throughput: 0: 824.9. Samples: 231888. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:43:33,318][00318] Avg episode reward: [(0, '2.923')] | |
| [2025-02-23 13:43:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.5, 300 sec: 3165.7). Total num frames: 933888. Throughput: 0: 838.9. Samples: 235146. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:43:38,321][00318] Avg episode reward: [(0, '3.127')] | |
| [2025-02-23 13:43:38,327][02456] Saving new best policy, reward=3.127! | |
| [2025-02-23 13:43:43,314][00318] Fps is (10 sec: 4916.5, 60 sec: 3549.9, 300 sec: 3249.0). Total num frames: 958464. Throughput: 0: 859.9. Samples: 241680. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:43:43,321][00318] Avg episode reward: [(0, '3.067')] | |
| [2025-02-23 13:43:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3193.5). Total num frames: 966656. Throughput: 0: 868.7. Samples: 246204. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:43:48,318][00318] Avg episode reward: [(0, '3.226')] | |
| [2025-02-23 13:43:48,336][02456] Saving new best policy, reward=3.226! | |
| [2025-02-23 13:43:53,124][02476] Updated weights for policy 0, policy_version 120 (0.0029) | |
| [2025-02-23 13:43:53,316][00318] Fps is (10 sec: 2457.1, 60 sec: 3413.2, 300 sec: 3221.2). Total num frames: 983040. Throughput: 0: 865.0. Samples: 248112. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:43:53,317][00318] Avg episode reward: [(0, '2.805')] | |
| [2025-02-23 13:43:58,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.4, 300 sec: 3221.3). Total num frames: 991232. Throughput: 0: 827.9. Samples: 252126. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:43:58,319][00318] Avg episode reward: [(0, '3.140')] | |
| [2025-02-23 13:44:03,314][00318] Fps is (10 sec: 3277.4, 60 sec: 3413.5, 300 sec: 3276.8). Total num frames: 1015808. Throughput: 0: 828.4. Samples: 258714. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:44:03,315][00318] Avg episode reward: [(0, '3.268')] | |
| [2025-02-23 13:44:03,317][02456] Saving new best policy, reward=3.268! | |
| [2025-02-23 13:44:08,317][00318] Fps is (10 sec: 4913.7, 60 sec: 3549.7, 300 sec: 3304.5). Total num frames: 1040384. Throughput: 0: 854.4. Samples: 261918. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:44:08,324][00318] Avg episode reward: [(0, '3.403')] | |
| [2025-02-23 13:44:08,329][02456] Saving new best policy, reward=3.403! | |
| [2025-02-23 13:44:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3276.8). Total num frames: 1048576. Throughput: 0: 864.0. Samples: 266460. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:44:13,317][00318] Avg episode reward: [(0, '3.193')] | |
| [2025-02-23 13:44:17,733][02476] Updated weights for policy 0, policy_version 130 (0.0019) | |
| [2025-02-23 13:44:18,314][00318] Fps is (10 sec: 2458.4, 60 sec: 3276.8, 300 sec: 3276.8). Total num frames: 1064960. Throughput: 0: 863.1. Samples: 270726. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:44:18,322][00318] Avg episode reward: [(0, '3.063')] | |
| [2025-02-23 13:44:18,334][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000130_1064960.pth... | |
| [2025-02-23 13:44:18,780][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000031_253952.pth | |
| [2025-02-23 13:44:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3304.6). Total num frames: 1081344. Throughput: 0: 838.4. Samples: 272874. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:44:23,322][00318] Avg episode reward: [(0, '3.463')] | |
| [2025-02-23 13:44:23,323][02456] Saving new best policy, reward=3.463! | |
| [2025-02-23 13:44:28,314][00318] Fps is (10 sec: 4095.8, 60 sec: 3549.8, 300 sec: 3332.3). Total num frames: 1105920. Throughput: 0: 825.2. Samples: 278814. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:44:28,318][00318] Avg episode reward: [(0, '3.410')] | |
| [2025-02-23 13:44:33,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3332.3). Total num frames: 1122304. Throughput: 0: 851.5. Samples: 284520. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:44:33,319][00318] Avg episode reward: [(0, '3.206')] | |
| [2025-02-23 13:44:38,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3413.3, 300 sec: 3332.3). Total num frames: 1138688. Throughput: 0: 856.2. Samples: 286638. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:44:38,317][00318] Avg episode reward: [(0, '3.410')] | |
| [2025-02-23 13:44:41,897][02476] Updated weights for policy 0, policy_version 140 (0.0013) | |
| [2025-02-23 13:44:43,314][00318] Fps is (10 sec: 2457.5, 60 sec: 3140.2, 300 sec: 3332.3). Total num frames: 1146880. Throughput: 0: 860.8. Samples: 290862. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:44:43,321][00318] Avg episode reward: [(0, '3.403')] | |
| [2025-02-23 13:44:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3360.1). Total num frames: 1171456. Throughput: 0: 844.1. Samples: 296700. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:44:48,315][00318] Avg episode reward: [(0, '3.316')] | |
| [2025-02-23 13:44:53,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 1187840. Throughput: 0: 846.9. Samples: 300024. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:44:53,320][00318] Avg episode reward: [(0, '3.590')] | |
| [2025-02-23 13:44:53,323][02456] Saving new best policy, reward=3.590! | |
| [2025-02-23 13:44:58,319][00318] Fps is (10 sec: 3275.0, 60 sec: 3549.6, 300 sec: 3360.0). Total num frames: 1204224. Throughput: 0: 850.3. Samples: 304728. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:44:58,321][00318] Avg episode reward: [(0, '3.171')] | |
| [2025-02-23 13:45:03,316][00318] Fps is (10 sec: 2457.1, 60 sec: 3276.7, 300 sec: 3332.3). Total num frames: 1212416. Throughput: 0: 849.4. Samples: 308952. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:45:03,320][00318] Avg episode reward: [(0, '3.431')] | |
| [2025-02-23 13:45:05,681][02476] Updated weights for policy 0, policy_version 150 (0.0014) | |
| [2025-02-23 13:45:08,314][00318] Fps is (10 sec: 2458.9, 60 sec: 3140.4, 300 sec: 3332.4). Total num frames: 1228800. Throughput: 0: 846.1. Samples: 310950. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:45:08,315][00318] Avg episode reward: [(0, '3.866')] | |
| [2025-02-23 13:45:08,321][02456] Saving new best policy, reward=3.866! | |
| [2025-02-23 13:45:13,314][00318] Fps is (10 sec: 4096.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 1253376. Throughput: 0: 838.7. Samples: 316554. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:45:13,315][00318] Avg episode reward: [(0, '3.348')] | |
| [2025-02-23 13:45:18,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 1277952. Throughput: 0: 856.3. Samples: 323052. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:45:18,317][00318] Avg episode reward: [(0, '3.289')] | |
| [2025-02-23 13:45:23,318][00318] Fps is (10 sec: 3275.6, 60 sec: 3413.1, 300 sec: 3360.1). Total num frames: 1286144. Throughput: 0: 856.7. Samples: 325194. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:45:23,319][00318] Avg episode reward: [(0, '3.602')] | |
| [2025-02-23 13:45:28,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 1302528. Throughput: 0: 851.2. Samples: 329166. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:45:28,315][00318] Avg episode reward: [(0, '3.604')] | |
| [2025-02-23 13:45:30,750][02476] Updated weights for policy 0, policy_version 160 (0.0016) | |
| [2025-02-23 13:45:33,314][00318] Fps is (10 sec: 3278.0, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 1318912. Throughput: 0: 822.3. Samples: 333702. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:45:33,318][00318] Avg episode reward: [(0, '3.498')] | |
| [2025-02-23 13:45:38,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 1335296. Throughput: 0: 822.9. Samples: 337056. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:45:38,318][00318] Avg episode reward: [(0, '3.568')] | |
| [2025-02-23 13:45:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 1359872. Throughput: 0: 866.0. Samples: 343692. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:45:43,317][00318] Avg episode reward: [(0, '3.325')] | |
| [2025-02-23 13:45:48,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 1376256. Throughput: 0: 866.4. Samples: 347940. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:45:48,319][00318] Avg episode reward: [(0, '3.629')] | |
| [2025-02-23 13:45:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3332.4). Total num frames: 1384448. Throughput: 0: 868.1. Samples: 350016. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:45:53,315][00318] Avg episode reward: [(0, '3.958')] | |
| [2025-02-23 13:45:53,319][02456] Saving new best policy, reward=3.958! | |
| [2025-02-23 13:45:54,091][02476] Updated weights for policy 0, policy_version 170 (0.0018) | |
| [2025-02-23 13:45:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.6, 300 sec: 3387.9). Total num frames: 1409024. Throughput: 0: 833.5. Samples: 354060. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:45:58,316][00318] Avg episode reward: [(0, '3.271')] | |
| [2025-02-23 13:46:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3387.9). Total num frames: 1425408. Throughput: 0: 835.3. Samples: 360642. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:46:03,315][00318] Avg episode reward: [(0, '3.909')] | |
| [2025-02-23 13:46:08,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3549.8, 300 sec: 3387.9). Total num frames: 1441792. Throughput: 0: 860.7. Samples: 363924. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:46:08,316][00318] Avg episode reward: [(0, '4.009')] | |
| [2025-02-23 13:46:08,326][02456] Saving new best policy, reward=4.009! | |
| [2025-02-23 13:46:13,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3413.2, 300 sec: 3360.1). Total num frames: 1458176. Throughput: 0: 867.2. Samples: 368190. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:46:13,317][00318] Avg episode reward: [(0, '3.648')] | |
| [2025-02-23 13:46:18,315][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.2, 300 sec: 3360.1). Total num frames: 1466368. Throughput: 0: 859.9. Samples: 372396. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:46:18,318][00318] Avg episode reward: [(0, '3.532')] | |
| [2025-02-23 13:46:18,416][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000180_1474560.pth... | |
| [2025-02-23 13:46:18,420][02476] Updated weights for policy 0, policy_version 180 (0.0038) | |
| [2025-02-23 13:46:18,795][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000080_655360.pth | |
| [2025-02-23 13:46:23,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3413.5, 300 sec: 3387.9). Total num frames: 1490944. Throughput: 0: 841.1. Samples: 374904. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:46:23,317][00318] Avg episode reward: [(0, '3.732')] | |
| [2025-02-23 13:46:28,314][00318] Fps is (10 sec: 4915.5, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 1515520. Throughput: 0: 828.4. Samples: 380970. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:46:28,315][00318] Avg episode reward: [(0, '3.685')] | |
| [2025-02-23 13:46:33,315][00318] Fps is (10 sec: 4095.4, 60 sec: 3549.8, 300 sec: 3387.9). Total num frames: 1531904. Throughput: 0: 848.5. Samples: 386124. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:46:33,317][00318] Avg episode reward: [(0, '3.208')] | |
| [2025-02-23 13:46:38,316][00318] Fps is (10 sec: 2457.2, 60 sec: 3413.3, 300 sec: 3360.1). Total num frames: 1540096. Throughput: 0: 848.9. Samples: 388218. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 13:46:38,317][00318] Avg episode reward: [(0, '3.509')] | |
| [2025-02-23 13:46:42,626][02476] Updated weights for policy 0, policy_version 190 (0.0013) | |
| [2025-02-23 13:46:43,314][00318] Fps is (10 sec: 2457.9, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 1556480. Throughput: 0: 851.7. Samples: 392388. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:46:43,318][00318] Avg episode reward: [(0, '3.903')] | |
| [2025-02-23 13:46:48,314][00318] Fps is (10 sec: 3277.4, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 1572864. Throughput: 0: 845.2. Samples: 398676. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:46:48,316][00318] Avg episode reward: [(0, '3.909')] | |
| [2025-02-23 13:46:53,322][00318] Fps is (10 sec: 4092.9, 60 sec: 3549.4, 300 sec: 3415.6). Total num frames: 1597440. Throughput: 0: 845.5. Samples: 401976. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:46:53,325][00318] Avg episode reward: [(0, '3.901')] | |
| [2025-02-23 13:46:58,317][00318] Fps is (10 sec: 4094.7, 60 sec: 3413.2, 300 sec: 3387.8). Total num frames: 1613824. Throughput: 0: 859.7. Samples: 406878. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:46:58,318][00318] Avg episode reward: [(0, '3.678')] | |
| [2025-02-23 13:47:03,314][00318] Fps is (10 sec: 2459.5, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 1622016. Throughput: 0: 854.9. Samples: 410868. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:47:03,315][00318] Avg episode reward: [(0, '3.720')] | |
| [2025-02-23 13:47:06,293][02476] Updated weights for policy 0, policy_version 200 (0.0018) | |
| [2025-02-23 13:47:08,314][00318] Fps is (10 sec: 2458.4, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 1638400. Throughput: 0: 844.3. Samples: 412896. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:47:08,315][00318] Avg episode reward: [(0, '3.538')] | |
| [2025-02-23 13:47:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.4, 300 sec: 3415.6). Total num frames: 1662976. Throughput: 0: 844.8. Samples: 418986. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:47:13,316][00318] Avg episode reward: [(0, '3.638')] | |
| [2025-02-23 13:47:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 1679360. Throughput: 0: 870.6. Samples: 425298. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:47:18,317][00318] Avg episode reward: [(0, '3.771')] | |
| [2025-02-23 13:47:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 1695744. Throughput: 0: 871.8. Samples: 427446. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:47:23,317][00318] Avg episode reward: [(0, '3.919')] | |
| [2025-02-23 13:47:28,316][00318] Fps is (10 sec: 3276.1, 60 sec: 3276.7, 300 sec: 3387.9). Total num frames: 1712128. Throughput: 0: 872.6. Samples: 431658. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:47:28,319][00318] Avg episode reward: [(0, '3.892')] | |
| [2025-02-23 13:47:30,967][02476] Updated weights for policy 0, policy_version 210 (0.0013) | |
| [2025-02-23 13:47:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.9, 300 sec: 3387.9). Total num frames: 1728512. Throughput: 0: 831.3. Samples: 436086. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:47:33,316][00318] Avg episode reward: [(0, '3.840')] | |
| [2025-02-23 13:47:38,314][00318] Fps is (10 sec: 3277.5, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 1744896. Throughput: 0: 829.7. Samples: 439308. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:47:38,315][00318] Avg episode reward: [(0, '3.732')] | |
| [2025-02-23 13:47:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 1769472. Throughput: 0: 868.3. Samples: 445950. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:47:43,317][00318] Avg episode reward: [(0, '4.081')] | |
| [2025-02-23 13:47:43,319][02456] Saving new best policy, reward=4.081! | |
| [2025-02-23 13:47:48,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 1777664. Throughput: 0: 871.6. Samples: 450090. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:47:48,322][00318] Avg episode reward: [(0, '3.723')] | |
| [2025-02-23 13:47:53,315][00318] Fps is (10 sec: 2457.4, 60 sec: 3277.2, 300 sec: 3360.1). Total num frames: 1794048. Throughput: 0: 872.4. Samples: 452154. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:47:53,316][00318] Avg episode reward: [(0, '4.007')] | |
| [2025-02-23 13:47:53,563][02476] Updated weights for policy 0, policy_version 220 (0.0016) | |
| [2025-02-23 13:47:58,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3413.5, 300 sec: 3415.7). Total num frames: 1818624. Throughput: 0: 845.3. Samples: 457026. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:47:58,316][00318] Avg episode reward: [(0, '3.670')] | |
| [2025-02-23 13:48:03,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 1835008. Throughput: 0: 837.6. Samples: 462990. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:48:03,317][00318] Avg episode reward: [(0, '3.777')] | |
| [2025-02-23 13:48:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 1851392. Throughput: 0: 858.3. Samples: 466068. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:48:08,315][00318] Avg episode reward: [(0, '3.812')] | |
| [2025-02-23 13:48:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 1867776. Throughput: 0: 859.5. Samples: 470334. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:48:13,316][00318] Avg episode reward: [(0, '3.624')] | |
| [2025-02-23 13:48:18,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 1875968. Throughput: 0: 854.1. Samples: 474522. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:48:18,315][00318] Avg episode reward: [(0, '3.651')] | |
| [2025-02-23 13:48:18,324][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000229_1875968.pth... | |
| [2025-02-23 13:48:18,683][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000130_1064960.pth | |
| [2025-02-23 13:48:18,901][02476] Updated weights for policy 0, policy_version 230 (0.0015) | |
| [2025-02-23 13:48:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 1900544. Throughput: 0: 848.1. Samples: 477474. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:48:23,319][00318] Avg episode reward: [(0, '3.878')] | |
| [2025-02-23 13:48:28,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3550.0, 300 sec: 3443.4). Total num frames: 1925120. Throughput: 0: 847.5. Samples: 484086. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:48:28,315][00318] Avg episode reward: [(0, '3.610')] | |
| [2025-02-23 13:48:33,315][00318] Fps is (10 sec: 3276.4, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 1933312. Throughput: 0: 860.0. Samples: 488790. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 13:48:33,316][00318] Avg episode reward: [(0, '3.719')] | |
| [2025-02-23 13:48:38,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3360.1). Total num frames: 1949696. Throughput: 0: 860.5. Samples: 490878. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:48:38,315][00318] Avg episode reward: [(0, '3.947')] | |
| [2025-02-23 13:48:42,397][02476] Updated weights for policy 0, policy_version 240 (0.0013) | |
| [2025-02-23 13:48:43,314][00318] Fps is (10 sec: 3277.2, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 1966080. Throughput: 0: 845.6. Samples: 495078. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:48:43,315][00318] Avg episode reward: [(0, '3.515')] | |
| [2025-02-23 13:48:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.7). Total num frames: 1990656. Throughput: 0: 847.3. Samples: 501120. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 13:48:48,315][00318] Avg episode reward: [(0, '4.047')] | |
| [2025-02-23 13:48:53,342][00318] Fps is (10 sec: 4084.7, 60 sec: 3548.3, 300 sec: 3443.1). Total num frames: 2007040. Throughput: 0: 852.0. Samples: 504432. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:48:53,343][00318] Avg episode reward: [(0, '4.159')] | |
| [2025-02-23 13:48:53,350][02456] Saving new best policy, reward=4.159! | |
| [2025-02-23 13:48:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 2023424. Throughput: 0: 864.4. Samples: 509232. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:48:58,318][00318] Avg episode reward: [(0, '3.915')] | |
| [2025-02-23 13:49:03,314][00318] Fps is (10 sec: 2464.4, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 2031616. Throughput: 0: 856.5. Samples: 513066. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:49:03,316][00318] Avg episode reward: [(0, '4.090')] | |
| [2025-02-23 13:49:06,722][02476] Updated weights for policy 0, policy_version 250 (0.0014) | |
| [2025-02-23 13:49:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 2048000. Throughput: 0: 835.1. Samples: 515052. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:49:08,315][00318] Avg episode reward: [(0, '3.589')] | |
| [2025-02-23 13:49:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 2072576. Throughput: 0: 811.7. Samples: 520614. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:49:13,315][00318] Avg episode reward: [(0, '4.015')] | |
| [2025-02-23 13:49:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 2088960. Throughput: 0: 847.1. Samples: 526908. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:49:18,318][00318] Avg episode reward: [(0, '4.137')] | |
| [2025-02-23 13:49:23,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2105344. Throughput: 0: 845.7. Samples: 528936. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:49:23,318][00318] Avg episode reward: [(0, '4.048')] | |
| [2025-02-23 13:49:28,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3140.2, 300 sec: 3360.1). Total num frames: 2113536. Throughput: 0: 841.1. Samples: 532926. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:49:28,316][00318] Avg episode reward: [(0, '3.773')] | |
| [2025-02-23 13:49:31,815][02476] Updated weights for policy 0, policy_version 260 (0.0021) | |
| [2025-02-23 13:49:33,314][00318] Fps is (10 sec: 2457.7, 60 sec: 3276.9, 300 sec: 3360.1). Total num frames: 2129920. Throughput: 0: 813.9. Samples: 537744. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 13:49:33,318][00318] Avg episode reward: [(0, '3.804')] | |
| [2025-02-23 13:49:38,314][00318] Fps is (10 sec: 4096.2, 60 sec: 3413.3, 300 sec: 3415.7). Total num frames: 2154496. Throughput: 0: 801.8. Samples: 540492. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:49:38,320][00318] Avg episode reward: [(0, '3.936')] | |
| [2025-02-23 13:49:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2170880. Throughput: 0: 832.7. Samples: 546702. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:49:43,316][00318] Avg episode reward: [(0, '3.950')] | |
| [2025-02-23 13:49:48,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.3, 300 sec: 3360.1). Total num frames: 2179072. Throughput: 0: 838.8. Samples: 550812. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 13:49:48,315][00318] Avg episode reward: [(0, '3.795')] | |
| [2025-02-23 13:49:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3278.3, 300 sec: 3387.9). Total num frames: 2203648. Throughput: 0: 840.3. Samples: 552864. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:49:53,323][00318] Avg episode reward: [(0, '3.965')] | |
| [2025-02-23 13:49:55,044][02476] Updated weights for policy 0, policy_version 270 (0.0018) | |
| [2025-02-23 13:49:58,314][00318] Fps is (10 sec: 4095.8, 60 sec: 3276.8, 300 sec: 3415.7). Total num frames: 2220032. Throughput: 0: 832.8. Samples: 558090. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:49:58,316][00318] Avg episode reward: [(0, '3.903')] | |
| [2025-02-23 13:50:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 2244608. Throughput: 0: 837.1. Samples: 564576. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:50:03,315][00318] Avg episode reward: [(0, '4.047')] | |
| [2025-02-23 13:50:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2252800. Throughput: 0: 845.9. Samples: 567000. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 13:50:08,319][00318] Avg episode reward: [(0, '3.799')] | |
| [2025-02-23 13:50:13,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 2269184. Throughput: 0: 848.7. Samples: 571116. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:50:13,317][00318] Avg episode reward: [(0, '4.070')] | |
| [2025-02-23 13:50:18,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 2285568. Throughput: 0: 833.9. Samples: 575268. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 13:50:18,317][00318] Avg episode reward: [(0, '3.922')] | |
| [2025-02-23 13:50:18,324][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000279_2285568.pth... | |
| [2025-02-23 13:50:18,669][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000180_1474560.pth | |
| [2025-02-23 13:50:19,526][02476] Updated weights for policy 0, policy_version 280 (0.0021) | |
| [2025-02-23 13:50:23,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 2301952. Throughput: 0: 832.5. Samples: 577956. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:50:23,315][00318] Avg episode reward: [(0, '3.837')] | |
| [2025-02-23 13:50:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 2326528. Throughput: 0: 840.0. Samples: 584502. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:50:28,316][00318] Avg episode reward: [(0, '3.784')] | |
| [2025-02-23 13:50:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2334720. Throughput: 0: 858.9. Samples: 589464. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:50:33,319][00318] Avg episode reward: [(0, '4.026')] | |
| [2025-02-23 13:50:38,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 2351104. Throughput: 0: 857.1. Samples: 591432. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:50:38,316][00318] Avg episode reward: [(0, '3.977')] | |
| [2025-02-23 13:50:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 2367488. Throughput: 0: 831.7. Samples: 595518. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 13:50:43,317][00318] Avg episode reward: [(0, '4.080')] | |
| [2025-02-23 13:50:44,895][02476] Updated weights for policy 0, policy_version 290 (0.0023) | |
| [2025-02-23 13:50:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 2392064. Throughput: 0: 822.9. Samples: 601608. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:50:48,315][00318] Avg episode reward: [(0, '3.768')] | |
| [2025-02-23 13:50:53,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 2416640. Throughput: 0: 840.9. Samples: 604842. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 13:50:53,315][00318] Avg episode reward: [(0, '3.832')] | |
| [2025-02-23 13:50:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 2424832. Throughput: 0: 863.2. Samples: 609960. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:50:58,318][00318] Avg episode reward: [(0, '3.872')] | |
| [2025-02-23 13:51:03,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 2441216. Throughput: 0: 863.2. Samples: 614112. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 13:51:03,321][00318] Avg episode reward: [(0, '3.959')] | |
| [2025-02-23 13:51:07,838][02476] Updated weights for policy 0, policy_version 300 (0.0013) | |
| [2025-02-23 13:51:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 2457600. Throughput: 0: 849.1. Samples: 616164. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:51:08,319][00318] Avg episode reward: [(0, '4.184')] | |
| [2025-02-23 13:51:08,326][02456] Saving new best policy, reward=4.184! | |
| [2025-02-23 13:51:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3415.7). Total num frames: 2473984. Throughput: 0: 823.9. Samples: 621576. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:51:13,315][00318] Avg episode reward: [(0, '4.259')] | |
| [2025-02-23 13:51:13,317][02456] Saving new best policy, reward=4.259! | |
| [2025-02-23 13:51:18,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3413.2, 300 sec: 3387.9). Total num frames: 2490368. Throughput: 0: 844.6. Samples: 627474. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:51:18,317][00318] Avg episode reward: [(0, '3.974')] | |
| [2025-02-23 13:51:23,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3413.2, 300 sec: 3360.1). Total num frames: 2506752. Throughput: 0: 846.0. Samples: 629502. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:51:23,319][00318] Avg episode reward: [(0, '4.246')] | |
| [2025-02-23 13:51:28,315][00318] Fps is (10 sec: 3277.0, 60 sec: 3276.7, 300 sec: 3360.1). Total num frames: 2523136. Throughput: 0: 847.6. Samples: 633660. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:51:28,318][00318] Avg episode reward: [(0, '4.013')] | |
| [2025-02-23 13:51:32,690][02476] Updated weights for policy 0, policy_version 310 (0.0018) | |
| [2025-02-23 13:51:33,320][00318] Fps is (10 sec: 3275.5, 60 sec: 3413.0, 300 sec: 3387.8). Total num frames: 2539520. Throughput: 0: 830.4. Samples: 638982. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:51:33,321][00318] Avg episode reward: [(0, '4.088')] | |
| [2025-02-23 13:51:38,314][00318] Fps is (10 sec: 3277.1, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2555904. Throughput: 0: 832.9. Samples: 642324. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:51:38,323][00318] Avg episode reward: [(0, '3.920')] | |
| [2025-02-23 13:51:43,314][00318] Fps is (10 sec: 4098.3, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 2580480. Throughput: 0: 835.2. Samples: 647544. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:51:43,317][00318] Avg episode reward: [(0, '4.214')] | |
| [2025-02-23 13:51:48,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3276.7, 300 sec: 3360.2). Total num frames: 2588672. Throughput: 0: 836.8. Samples: 651768. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:51:48,317][00318] Avg episode reward: [(0, '4.222')] | |
| [2025-02-23 13:51:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.3, 300 sec: 3360.1). Total num frames: 2605056. Throughput: 0: 838.3. Samples: 653886. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:51:53,322][00318] Avg episode reward: [(0, '3.983')] | |
| [2025-02-23 13:51:56,519][02476] Updated weights for policy 0, policy_version 320 (0.0013) | |
| [2025-02-23 13:51:58,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 2621440. Throughput: 0: 842.4. Samples: 659484. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:51:58,316][00318] Avg episode reward: [(0, '4.001')] | |
| [2025-02-23 13:52:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 2646016. Throughput: 0: 857.0. Samples: 666036. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:52:03,320][00318] Avg episode reward: [(0, '4.182')] | |
| [2025-02-23 13:52:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2662400. Throughput: 0: 864.7. Samples: 668412. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 13:52:08,320][00318] Avg episode reward: [(0, '3.993')] | |
| [2025-02-23 13:52:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2678784. Throughput: 0: 861.4. Samples: 672420. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:52:13,316][00318] Avg episode reward: [(0, '4.165')] | |
| [2025-02-23 13:52:18,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.9, 300 sec: 3360.1). Total num frames: 2686976. Throughput: 0: 836.8. Samples: 676632. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:52:18,315][00318] Avg episode reward: [(0, '3.998')] | |
| [2025-02-23 13:52:18,323][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000328_2686976.pth... | |
| [2025-02-23 13:52:18,623][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000229_1875968.pth | |
| [2025-02-23 13:52:20,137][02476] Updated weights for policy 0, policy_version 330 (0.0013) | |
| [2025-02-23 13:52:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 2711552. Throughput: 0: 830.3. Samples: 679686. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:52:23,320][00318] Avg episode reward: [(0, '4.232')] | |
| [2025-02-23 13:52:28,321][00318] Fps is (10 sec: 4911.7, 60 sec: 3549.5, 300 sec: 3415.6). Total num frames: 2736128. Throughput: 0: 860.4. Samples: 686268. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:52:28,323][00318] Avg episode reward: [(0, '3.947')] | |
| [2025-02-23 13:52:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.7, 300 sec: 3387.9). Total num frames: 2744320. Throughput: 0: 870.3. Samples: 690930. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:52:33,315][00318] Avg episode reward: [(0, '3.890')] | |
| [2025-02-23 13:52:38,314][00318] Fps is (10 sec: 2459.3, 60 sec: 3413.3, 300 sec: 3360.1). Total num frames: 2760704. Throughput: 0: 868.9. Samples: 692988. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 13:52:38,318][00318] Avg episode reward: [(0, '3.917')] | |
| [2025-02-23 13:52:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 2777088. Throughput: 0: 834.4. Samples: 697032. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:52:43,317][00318] Avg episode reward: [(0, '4.352')] | |
| [2025-02-23 13:52:43,320][02456] Saving new best policy, reward=4.352! | |
| [2025-02-23 13:52:44,960][02476] Updated weights for policy 0, policy_version 340 (0.0013) | |
| [2025-02-23 13:52:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3415.7). Total num frames: 2801664. Throughput: 0: 828.0. Samples: 703296. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:52:48,317][00318] Avg episode reward: [(0, '3.836')] | |
| [2025-02-23 13:52:53,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 2818048. Throughput: 0: 849.9. Samples: 706656. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:52:53,319][00318] Avg episode reward: [(0, '4.040')] | |
| [2025-02-23 13:52:58,315][00318] Fps is (10 sec: 3276.5, 60 sec: 3549.8, 300 sec: 3387.9). Total num frames: 2834432. Throughput: 0: 866.8. Samples: 711426. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:52:58,318][00318] Avg episode reward: [(0, '4.036')] | |
| [2025-02-23 13:53:03,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 2842624. Throughput: 0: 864.5. Samples: 715536. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:53:03,315][00318] Avg episode reward: [(0, '4.013')] | |
| [2025-02-23 13:53:07,969][02476] Updated weights for policy 0, policy_version 350 (0.0013) | |
| [2025-02-23 13:53:08,314][00318] Fps is (10 sec: 3277.1, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2867200. Throughput: 0: 849.5. Samples: 717912. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:53:08,315][00318] Avg episode reward: [(0, '4.085')] | |
| [2025-02-23 13:53:13,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 2883584. Throughput: 0: 843.9. Samples: 724236. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:53:13,316][00318] Avg episode reward: [(0, '3.949')] | |
| [2025-02-23 13:53:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 2899968. Throughput: 0: 857.2. Samples: 729504. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:53:18,315][00318] Avg episode reward: [(0, '4.083')] | |
| [2025-02-23 13:53:23,315][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3332.3). Total num frames: 2908160. Throughput: 0: 857.6. Samples: 731580. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 13:53:23,324][00318] Avg episode reward: [(0, '3.975')] | |
| [2025-02-23 13:53:28,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.6, 300 sec: 3360.1). Total num frames: 2924544. Throughput: 0: 859.1. Samples: 735690. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 13:53:28,319][00318] Avg episode reward: [(0, '4.008')] | |
| [2025-02-23 13:53:32,353][02476] Updated weights for policy 0, policy_version 360 (0.0021) | |
| [2025-02-23 13:53:33,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 2949120. Throughput: 0: 847.9. Samples: 741450. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:53:33,319][00318] Avg episode reward: [(0, '4.021')] | |
| [2025-02-23 13:53:38,314][00318] Fps is (10 sec: 4915.3, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 2973696. Throughput: 0: 847.5. Samples: 744792. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:53:38,316][00318] Avg episode reward: [(0, '3.907')] | |
| [2025-02-23 13:53:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 2990080. Throughput: 0: 862.3. Samples: 750228. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:53:43,319][00318] Avg episode reward: [(0, '4.218')] | |
| [2025-02-23 13:53:48,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.4). Total num frames: 2998272. Throughput: 0: 860.1. Samples: 754242. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:53:48,321][00318] Avg episode reward: [(0, '4.293')] | |
| [2025-02-23 13:53:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 3014656. Throughput: 0: 852.5. Samples: 756276. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:53:53,318][00318] Avg episode reward: [(0, '4.355')] | |
| [2025-02-23 13:53:53,322][02456] Saving new best policy, reward=4.355! | |
| [2025-02-23 13:53:56,795][02476] Updated weights for policy 0, policy_version 370 (0.0023) | |
| [2025-02-23 13:53:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3031040. Throughput: 0: 836.5. Samples: 761880. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 13:53:58,325][00318] Avg episode reward: [(0, '3.866')] | |
| [2025-02-23 13:54:03,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 3055616. Throughput: 0: 864.5. Samples: 768408. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:54:03,321][00318] Avg episode reward: [(0, '3.947')] | |
| [2025-02-23 13:54:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3072000. Throughput: 0: 869.5. Samples: 770706. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 13:54:08,318][00318] Avg episode reward: [(0, '4.007')] | |
| [2025-02-23 13:54:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 3088384. Throughput: 0: 870.8. Samples: 774876. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:54:13,315][00318] Avg episode reward: [(0, '4.013')] | |
| [2025-02-23 13:54:18,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 3096576. Throughput: 0: 834.4. Samples: 778998. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 13:54:18,315][00318] Avg episode reward: [(0, '4.111')] | |
| [2025-02-23 13:54:18,325][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000378_3096576.pth... | |
| [2025-02-23 13:54:18,631][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000279_2285568.pth | |
| [2025-02-23 13:54:20,131][02476] Updated weights for policy 0, policy_version 380 (0.0022) | |
| [2025-02-23 13:54:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3415.7). Total num frames: 3121152. Throughput: 0: 828.0. Samples: 782052. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:54:23,319][00318] Avg episode reward: [(0, '3.941')] | |
| [2025-02-23 13:54:28,314][00318] Fps is (10 sec: 4915.1, 60 sec: 3686.4, 300 sec: 3443.4). Total num frames: 3145728. Throughput: 0: 855.3. Samples: 788718. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:54:28,318][00318] Avg episode reward: [(0, '4.443')] | |
| [2025-02-23 13:54:28,332][02456] Saving new best policy, reward=4.443! | |
| [2025-02-23 13:54:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3153920. Throughput: 0: 862.5. Samples: 793056. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:54:33,315][00318] Avg episode reward: [(0, '4.219')] | |
| [2025-02-23 13:54:38,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3170304. Throughput: 0: 862.9. Samples: 795108. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:54:38,319][00318] Avg episode reward: [(0, '4.114')] | |
| [2025-02-23 13:54:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 3186688. Throughput: 0: 839.1. Samples: 799638. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 13:54:43,315][00318] Avg episode reward: [(0, '3.835')] | |
| [2025-02-23 13:54:44,324][02476] Updated weights for policy 0, policy_version 390 (0.0013) | |
| [2025-02-23 13:54:48,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 3211264. Throughput: 0: 828.0. Samples: 805668. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:54:48,320][00318] Avg episode reward: [(0, '4.087')] | |
| [2025-02-23 13:54:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3219456. Throughput: 0: 849.6. Samples: 808938. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:54:53,322][00318] Avg episode reward: [(0, '4.024')] | |
| [2025-02-23 13:54:58,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3360.1). Total num frames: 3235840. Throughput: 0: 851.9. Samples: 813210. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 13:54:58,318][00318] Avg episode reward: [(0, '3.834')] | |
| [2025-02-23 13:55:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3252224. Throughput: 0: 854.7. Samples: 817458. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:55:03,318][00318] Avg episode reward: [(0, '4.054')] | |
| [2025-02-23 13:55:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3268608. Throughput: 0: 846.8. Samples: 820158. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 13:55:08,319][00318] Avg episode reward: [(0, '4.129')] | |
| [2025-02-23 13:55:08,609][02476] Updated weights for policy 0, policy_version 400 (0.0014) | |
| [2025-02-23 13:55:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 3293184. Throughput: 0: 845.9. Samples: 826782. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:55:13,315][00318] Avg episode reward: [(0, '4.011')] | |
| [2025-02-23 13:55:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 3309568. Throughput: 0: 862.5. Samples: 831870. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:55:18,319][00318] Avg episode reward: [(0, '4.074')] | |
| [2025-02-23 13:55:23,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3325952. Throughput: 0: 861.6. Samples: 833880. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:55:23,316][00318] Avg episode reward: [(0, '3.998')] | |
| [2025-02-23 13:55:28,315][00318] Fps is (10 sec: 2457.4, 60 sec: 3140.2, 300 sec: 3387.9). Total num frames: 3334144. Throughput: 0: 852.8. Samples: 838014. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:55:28,320][00318] Avg episode reward: [(0, '4.309')] | |
| [2025-02-23 13:55:32,183][02476] Updated weights for policy 0, policy_version 410 (0.0037) | |
| [2025-02-23 13:55:33,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 3358720. Throughput: 0: 838.8. Samples: 843414. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:55:33,315][00318] Avg episode reward: [(0, '4.114')] | |
| [2025-02-23 13:55:38,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3413.4, 300 sec: 3415.6). Total num frames: 3375104. Throughput: 0: 835.7. Samples: 846546. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:55:38,316][00318] Avg episode reward: [(0, '4.083')] | |
| [2025-02-23 13:55:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3391488. Throughput: 0: 857.3. Samples: 851790. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:55:43,317][00318] Avg episode reward: [(0, '4.231')] | |
| [2025-02-23 13:55:48,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3140.3, 300 sec: 3332.3). Total num frames: 3399680. Throughput: 0: 852.8. Samples: 855834. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:55:48,315][00318] Avg episode reward: [(0, '4.103')] | |
| [2025-02-23 13:55:53,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3424256. Throughput: 0: 836.4. Samples: 857796. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 13:55:53,316][00318] Avg episode reward: [(0, '4.214')] | |
| [2025-02-23 13:55:57,843][02476] Updated weights for policy 0, policy_version 420 (0.0015) | |
| [2025-02-23 13:55:58,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3440640. Throughput: 0: 816.1. Samples: 863508. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:55:58,315][00318] Avg episode reward: [(0, '4.170')] | |
| [2025-02-23 13:56:03,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3549.8, 300 sec: 3415.6). Total num frames: 3465216. Throughput: 0: 849.7. Samples: 870108. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:56:03,318][00318] Avg episode reward: [(0, '4.316')] | |
| [2025-02-23 13:56:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3473408. Throughput: 0: 854.7. Samples: 872340. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:56:08,319][00318] Avg episode reward: [(0, '3.892')] | |
| [2025-02-23 13:56:13,314][00318] Fps is (10 sec: 2457.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3489792. Throughput: 0: 853.7. Samples: 876432. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 13:56:13,317][00318] Avg episode reward: [(0, '4.117')] | |
| [2025-02-23 13:56:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3506176. Throughput: 0: 838.9. Samples: 881166. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:56:18,318][00318] Avg episode reward: [(0, '4.127')] | |
| [2025-02-23 13:56:18,391][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000429_3514368.pth... | |
| [2025-02-23 13:56:19,010][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000328_2686976.pth | |
| [2025-02-23 13:56:21,854][02476] Updated weights for policy 0, policy_version 430 (0.0014) | |
| [2025-02-23 13:56:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3522560. Throughput: 0: 827.2. Samples: 883770. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 13:56:23,318][00318] Avg episode reward: [(0, '4.036')] | |
| [2025-02-23 13:56:28,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3549.9, 300 sec: 3415.7). Total num frames: 3547136. Throughput: 0: 851.2. Samples: 890094. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:56:28,316][00318] Avg episode reward: [(0, '3.931')] | |
| [2025-02-23 13:56:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3555328. Throughput: 0: 857.7. Samples: 894432. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:56:33,315][00318] Avg episode reward: [(0, '4.222')] | |
| [2025-02-23 13:56:38,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3579904. Throughput: 0: 859.7. Samples: 896484. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 13:56:38,317][00318] Avg episode reward: [(0, '4.090')] | |
| [2025-02-23 13:56:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3588096. Throughput: 0: 843.7. Samples: 901476. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:56:43,315][00318] Avg episode reward: [(0, '4.069')] | |
| [2025-02-23 13:56:44,403][02476] Updated weights for policy 0, policy_version 440 (0.0017) | |
| [2025-02-23 13:56:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 3612672. Throughput: 0: 843.2. Samples: 908052. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 13:56:48,319][00318] Avg episode reward: [(0, '3.976')] | |
| [2025-02-23 13:56:53,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 3637248. Throughput: 0: 853.1. Samples: 910728. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:56:53,318][00318] Avg episode reward: [(0, '4.033')] | |
| [2025-02-23 13:56:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3645440. Throughput: 0: 854.4. Samples: 914880. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 13:56:58,315][00318] Avg episode reward: [(0, '3.918')] | |
| [2025-02-23 13:57:03,314][00318] Fps is (10 sec: 1638.4, 60 sec: 3140.3, 300 sec: 3360.1). Total num frames: 3653632. Throughput: 0: 843.1. Samples: 919104. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:57:03,320][00318] Avg episode reward: [(0, '4.163')] | |
| [2025-02-23 13:57:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3678208. Throughput: 0: 845.7. Samples: 921828. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:57:08,315][00318] Avg episode reward: [(0, '4.300')] | |
| [2025-02-23 13:57:09,667][02476] Updated weights for policy 0, policy_version 450 (0.0035) | |
| [2025-02-23 13:57:13,315][00318] Fps is (10 sec: 4914.9, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 3702784. Throughput: 0: 853.7. Samples: 928512. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:57:13,316][00318] Avg episode reward: [(0, '4.145')] | |
| [2025-02-23 13:57:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3710976. Throughput: 0: 872.0. Samples: 933672. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:57:18,321][00318] Avg episode reward: [(0, '4.131')] | |
| [2025-02-23 13:57:23,316][00318] Fps is (10 sec: 3276.5, 60 sec: 3549.8, 300 sec: 3387.9). Total num frames: 3735552. Throughput: 0: 870.6. Samples: 935664. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 13:57:23,318][00318] Avg episode reward: [(0, '3.844')] | |
| [2025-02-23 13:57:28,320][00318] Fps is (10 sec: 2456.2, 60 sec: 3140.0, 300 sec: 3360.0). Total num frames: 3735552. Throughput: 0: 846.8. Samples: 939588. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 13:57:28,321][00318] Avg episode reward: [(0, '4.403')] | |
| [2025-02-23 13:57:33,267][02476] Updated weights for policy 0, policy_version 460 (0.0014) | |
| [2025-02-23 13:57:33,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 3768320. Throughput: 0: 832.8. Samples: 945528. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 13:57:33,319][00318] Avg episode reward: [(0, '4.364')] | |
| [2025-02-23 13:57:38,314][00318] Fps is (10 sec: 5737.7, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 3792896. Throughput: 0: 846.8. Samples: 948834. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:57:38,316][00318] Avg episode reward: [(0, '4.149')] | |
| [2025-02-23 13:57:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 3801088. Throughput: 0: 874.0. Samples: 954210. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 13:57:43,316][00318] Avg episode reward: [(0, '4.308')] | |
| [2025-02-23 13:57:48,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3817472. Throughput: 0: 871.2. Samples: 958308. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:57:48,315][00318] Avg episode reward: [(0, '4.078')] | |
| [2025-02-23 13:57:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3833856. Throughput: 0: 857.2. Samples: 960402. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:57:53,319][00318] Avg episode reward: [(0, '4.165')] | |
| [2025-02-23 13:57:56,882][02476] Updated weights for policy 0, policy_version 470 (0.0015) | |
| [2025-02-23 13:57:58,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 3858432. Throughput: 0: 832.7. Samples: 965982. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 13:57:58,317][00318] Avg episode reward: [(0, '3.946')] | |
| [2025-02-23 13:58:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3387.9). Total num frames: 3866624. Throughput: 0: 856.5. Samples: 972216. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:58:03,319][00318] Avg episode reward: [(0, '4.142')] | |
| [2025-02-23 13:58:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 3883008. Throughput: 0: 858.2. Samples: 974280. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 13:58:08,318][00318] Avg episode reward: [(0, '4.345')] | |
| [2025-02-23 13:58:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 3899392. Throughput: 0: 864.4. Samples: 978480. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:58:13,319][00318] Avg episode reward: [(0, '3.848')] | |
| [2025-02-23 13:58:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.7). Total num frames: 3915776. Throughput: 0: 848.9. Samples: 983730. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 13:58:18,316][00318] Avg episode reward: [(0, '4.141')] | |
| [2025-02-23 13:58:18,322][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000478_3915776.pth... | |
| [2025-02-23 13:58:18,653][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000378_3096576.pth | |
| [2025-02-23 13:58:21,026][02476] Updated weights for policy 0, policy_version 480 (0.0013) | |
| [2025-02-23 13:58:23,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3413.4, 300 sec: 3443.4). Total num frames: 3940352. Throughput: 0: 847.6. Samples: 986976. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:58:23,316][00318] Avg episode reward: [(0, '4.290')] | |
| [2025-02-23 13:58:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.8, 300 sec: 3415.6). Total num frames: 3956736. Throughput: 0: 847.2. Samples: 992334. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 13:58:28,316][00318] Avg episode reward: [(0, '4.014')] | |
| [2025-02-23 13:58:33,314][00318] Fps is (10 sec: 2457.7, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 3964928. Throughput: 0: 849.2. Samples: 996522. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 13:58:33,315][00318] Avg episode reward: [(0, '4.082')] | |
| [2025-02-23 13:58:38,314][00318] Fps is (10 sec: 2457.5, 60 sec: 3140.2, 300 sec: 3360.1). Total num frames: 3981312. Throughput: 0: 849.6. Samples: 998634. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:58:38,316][00318] Avg episode reward: [(0, '4.001')] | |
| [2025-02-23 13:58:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4005888. Throughput: 0: 850.1. Samples: 1004238. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:58:43,318][00318] Avg episode reward: [(0, '4.221')] | |
| [2025-02-23 13:58:44,117][02476] Updated weights for policy 0, policy_version 490 (0.0028) | |
| [2025-02-23 13:58:48,316][00318] Fps is (10 sec: 4914.6, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 4030464. Throughput: 0: 860.1. Samples: 1010922. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:58:48,320][00318] Avg episode reward: [(0, '4.196')] | |
| [2025-02-23 13:58:53,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4046848. Throughput: 0: 871.3. Samples: 1013490. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:58:53,318][00318] Avg episode reward: [(0, '4.273')] | |
| [2025-02-23 13:58:58,314][00318] Fps is (10 sec: 2458.0, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4055040. Throughput: 0: 867.5. Samples: 1017516. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:58:58,320][00318] Avg episode reward: [(0, '4.044')] | |
| [2025-02-23 13:59:03,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 4071424. Throughput: 0: 840.9. Samples: 1021572. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:59:03,315][00318] Avg episode reward: [(0, '3.910')] | |
| [2025-02-23 13:59:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 4087808. Throughput: 0: 840.4. Samples: 1024794. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:59:08,316][00318] Avg episode reward: [(0, '4.311')] | |
| [2025-02-23 13:59:09,469][02476] Updated weights for policy 0, policy_version 500 (0.0015) | |
| [2025-02-23 13:59:13,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 4112384. Throughput: 0: 863.5. Samples: 1031190. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 13:59:13,316][00318] Avg episode reward: [(0, '4.048')] | |
| [2025-02-23 13:59:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 4120576. Throughput: 0: 875.2. Samples: 1035906. Policy #0 lag: (min: 0.0, avg: 2.1, max: 6.0) | |
| [2025-02-23 13:59:18,315][00318] Avg episode reward: [(0, '3.935')] | |
| [2025-02-23 13:59:23,315][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3360.1). Total num frames: 4136960. Throughput: 0: 874.0. Samples: 1037964. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:59:23,316][00318] Avg episode reward: [(0, '4.124')] | |
| [2025-02-23 13:59:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4153344. Throughput: 0: 839.9. Samples: 1042032. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:59:28,316][00318] Avg episode reward: [(0, '4.299')] | |
| [2025-02-23 13:59:32,798][02476] Updated weights for policy 0, policy_version 510 (0.0013) | |
| [2025-02-23 13:59:33,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3549.9, 300 sec: 3415.7). Total num frames: 4177920. Throughput: 0: 831.2. Samples: 1048326. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 13:59:33,318][00318] Avg episode reward: [(0, '4.143')] | |
| [2025-02-23 13:59:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 4194304. Throughput: 0: 845.9. Samples: 1051554. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 13:59:38,315][00318] Avg episode reward: [(0, '4.241')] | |
| [2025-02-23 13:59:43,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3413.2, 300 sec: 3387.9). Total num frames: 4210688. Throughput: 0: 869.7. Samples: 1056654. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 13:59:43,323][00318] Avg episode reward: [(0, '4.465')] | |
| [2025-02-23 13:59:43,325][02456] Saving new best policy, reward=4.465! | |
| [2025-02-23 13:59:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.9, 300 sec: 3415.6). Total num frames: 4227072. Throughput: 0: 868.0. Samples: 1060632. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 13:59:48,317][00318] Avg episode reward: [(0, '4.070')] | |
| [2025-02-23 13:59:53,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 4243456. Throughput: 0: 844.3. Samples: 1062786. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 13:59:53,317][00318] Avg episode reward: [(0, '4.215')] | |
| [2025-02-23 13:59:55,610][02476] Updated weights for policy 0, policy_version 520 (0.0022) | |
| [2025-02-23 13:59:58,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4259840. Throughput: 0: 842.7. Samples: 1069110. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 13:59:58,315][00318] Avg episode reward: [(0, '3.990')] | |
| [2025-02-23 14:00:03,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4276224. Throughput: 0: 860.8. Samples: 1074642. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 14:00:03,316][00318] Avg episode reward: [(0, '4.349')] | |
| [2025-02-23 14:00:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 4292608. Throughput: 0: 860.3. Samples: 1076676. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 14:00:08,316][00318] Avg episode reward: [(0, '4.471')] | |
| [2025-02-23 14:00:08,347][02456] Saving new best policy, reward=4.471! | |
| [2025-02-23 14:00:13,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3276.7, 300 sec: 3387.9). Total num frames: 4308992. Throughput: 0: 860.8. Samples: 1080768. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:00:13,322][00318] Avg episode reward: [(0, '4.310')] | |
| [2025-02-23 14:00:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 4325376. Throughput: 0: 845.6. Samples: 1086378. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:00:18,318][00318] Avg episode reward: [(0, '4.126')] | |
| [2025-02-23 14:00:18,326][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000529_4333568.pth... | |
| [2025-02-23 14:00:18,658][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000429_3514368.pth | |
| [2025-02-23 14:00:21,363][02476] Updated weights for policy 0, policy_version 530 (0.0015) | |
| [2025-02-23 14:00:23,314][00318] Fps is (10 sec: 4096.7, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4349952. Throughput: 0: 844.7. Samples: 1089564. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:00:23,315][00318] Avg episode reward: [(0, '4.328')] | |
| [2025-02-23 14:00:28,316][00318] Fps is (10 sec: 4095.3, 60 sec: 3549.8, 300 sec: 3415.6). Total num frames: 4366336. Throughput: 0: 858.3. Samples: 1095276. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:00:28,317][00318] Avg episode reward: [(0, '4.076')] | |
| [2025-02-23 14:00:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4382720. Throughput: 0: 858.7. Samples: 1099272. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:00:33,318][00318] Avg episode reward: [(0, '4.236')] | |
| [2025-02-23 14:00:38,314][00318] Fps is (10 sec: 2458.0, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4390912. Throughput: 0: 857.1. Samples: 1101354. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:00:38,318][00318] Avg episode reward: [(0, '4.181')] | |
| [2025-02-23 14:00:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3443.4). Total num frames: 4415488. Throughput: 0: 837.5. Samples: 1106796. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:00:43,320][00318] Avg episode reward: [(0, '4.040')] | |
| [2025-02-23 14:00:43,570][02476] Updated weights for policy 0, policy_version 540 (0.0013) | |
| [2025-02-23 14:00:48,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4440064. Throughput: 0: 862.7. Samples: 1113462. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:00:48,315][00318] Avg episode reward: [(0, '4.002')] | |
| [2025-02-23 14:00:53,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4456448. Throughput: 0: 874.7. Samples: 1116036. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:00:53,321][00318] Avg episode reward: [(0, '4.176')] | |
| [2025-02-23 14:00:58,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 4464640. Throughput: 0: 877.2. Samples: 1120242. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:00:58,315][00318] Avg episode reward: [(0, '4.337')] | |
| [2025-02-23 14:01:03,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4481024. Throughput: 0: 840.3. Samples: 1124190. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:01:03,315][00318] Avg episode reward: [(0, '4.090')] | |
| [2025-02-23 14:01:08,071][02476] Updated weights for policy 0, policy_version 550 (0.0016) | |
| [2025-02-23 14:01:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4505600. Throughput: 0: 840.0. Samples: 1127364. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:01:08,315][00318] Avg episode reward: [(0, '4.234')] | |
| [2025-02-23 14:01:13,316][00318] Fps is (10 sec: 4095.2, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 4521984. Throughput: 0: 860.0. Samples: 1133976. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:01:13,319][00318] Avg episode reward: [(0, '4.329')] | |
| [2025-02-23 14:01:18,315][00318] Fps is (10 sec: 3276.4, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 4538368. Throughput: 0: 878.0. Samples: 1138782. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:01:18,320][00318] Avg episode reward: [(0, '4.306')] | |
| [2025-02-23 14:01:23,314][00318] Fps is (10 sec: 2458.1, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4546560. Throughput: 0: 877.3. Samples: 1140834. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:01:23,315][00318] Avg episode reward: [(0, '4.046')] | |
| [2025-02-23 14:01:28,314][00318] Fps is (10 sec: 3277.2, 60 sec: 3413.4, 300 sec: 3443.4). Total num frames: 4571136. Throughput: 0: 855.2. Samples: 1145280. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:01:28,320][00318] Avg episode reward: [(0, '4.330')] | |
| [2025-02-23 14:01:32,910][02476] Updated weights for policy 0, policy_version 560 (0.0017) | |
| [2025-02-23 14:01:33,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4587520. Throughput: 0: 840.0. Samples: 1151262. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:01:33,316][00318] Avg episode reward: [(0, '4.467')] | |
| [2025-02-23 14:01:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4603904. Throughput: 0: 856.1. Samples: 1154562. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:01:38,316][00318] Avg episode reward: [(0, '4.054')] | |
| [2025-02-23 14:01:43,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4620288. Throughput: 0: 862.9. Samples: 1159074. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:01:43,316][00318] Avg episode reward: [(0, '4.177')] | |
| [2025-02-23 14:01:48,320][00318] Fps is (10 sec: 3274.9, 60 sec: 3276.5, 300 sec: 3387.8). Total num frames: 4636672. Throughput: 0: 867.8. Samples: 1163244. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:01:48,321][00318] Avg episode reward: [(0, '4.336')] | |
| [2025-02-23 14:01:53,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 4661248. Throughput: 0: 854.1. Samples: 1165800. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:01:53,315][00318] Avg episode reward: [(0, '3.979')] | |
| [2025-02-23 14:01:55,342][02476] Updated weights for policy 0, policy_version 570 (0.0017) | |
| [2025-02-23 14:01:58,314][00318] Fps is (10 sec: 4098.3, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 4677632. Throughput: 0: 855.6. Samples: 1172478. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:01:58,315][00318] Avg episode reward: [(0, '4.288')] | |
| [2025-02-23 14:02:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4694016. Throughput: 0: 864.3. Samples: 1177674. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:02:03,318][00318] Avg episode reward: [(0, '4.032')] | |
| [2025-02-23 14:02:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4702208. Throughput: 0: 865.3. Samples: 1179774. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:02:08,317][00318] Avg episode reward: [(0, '4.127')] | |
| [2025-02-23 14:02:13,317][00318] Fps is (10 sec: 2457.0, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 4718592. Throughput: 0: 859.7. Samples: 1183968. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:02:13,319][00318] Avg episode reward: [(0, '4.180')] | |
| [2025-02-23 14:02:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.4, 300 sec: 3415.7). Total num frames: 4743168. Throughput: 0: 858.3. Samples: 1189884. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 14:02:18,315][00318] Avg episode reward: [(0, '4.269')] | |
| [2025-02-23 14:02:18,326][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000579_4743168.pth... | |
| [2025-02-23 14:02:18,635][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000478_3915776.pth | |
| [2025-02-23 14:02:19,103][02476] Updated weights for policy 0, policy_version 580 (0.0013) | |
| [2025-02-23 14:02:23,314][00318] Fps is (10 sec: 4096.9, 60 sec: 3549.8, 300 sec: 3471.2). Total num frames: 4759552. Throughput: 0: 854.0. Samples: 1192992. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:02:23,316][00318] Avg episode reward: [(0, '3.880')] | |
| [2025-02-23 14:02:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4784128. Throughput: 0: 874.8. Samples: 1198440. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:02:28,318][00318] Avg episode reward: [(0, '4.135')] | |
| [2025-02-23 14:02:33,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3413.4, 300 sec: 3387.9). Total num frames: 4792320. Throughput: 0: 870.1. Samples: 1202394. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:02:33,319][00318] Avg episode reward: [(0, '4.391')] | |
| [2025-02-23 14:02:38,314][00318] Fps is (10 sec: 2457.5, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4808704. Throughput: 0: 857.6. Samples: 1204392. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:02:38,320][00318] Avg episode reward: [(0, '4.128')] | |
| [2025-02-23 14:02:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 4825088. Throughput: 0: 832.4. Samples: 1209936. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:02:43,320][00318] Avg episode reward: [(0, '4.480')] | |
| [2025-02-23 14:02:43,322][02456] Saving new best policy, reward=4.480! | |
| [2025-02-23 14:02:44,343][02476] Updated weights for policy 0, policy_version 590 (0.0026) | |
| [2025-02-23 14:02:48,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3550.2, 300 sec: 3443.4). Total num frames: 4849664. Throughput: 0: 861.9. Samples: 1216458. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 14:02:48,317][00318] Avg episode reward: [(0, '4.494')] | |
| [2025-02-23 14:02:48,330][02456] Saving new best policy, reward=4.494! | |
| [2025-02-23 14:02:53,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4857856. Throughput: 0: 864.5. Samples: 1218678. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:02:53,320][00318] Avg episode reward: [(0, '4.273')] | |
| [2025-02-23 14:02:58,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 4874240. Throughput: 0: 865.1. Samples: 1222896. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:02:58,317][00318] Avg episode reward: [(0, '4.282')] | |
| [2025-02-23 14:03:03,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 4890624. Throughput: 0: 836.8. Samples: 1227540. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:03:03,315][00318] Avg episode reward: [(0, '4.300')] | |
| [2025-02-23 14:03:07,564][02476] Updated weights for policy 0, policy_version 600 (0.0013) | |
| [2025-02-23 14:03:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 4915200. Throughput: 0: 828.9. Samples: 1230294. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:03:08,315][00318] Avg episode reward: [(0, '4.166')] | |
| [2025-02-23 14:03:13,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3686.6, 300 sec: 3471.2). Total num frames: 4939776. Throughput: 0: 855.3. Samples: 1236930. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:03:13,318][00318] Avg episode reward: [(0, '4.301')] | |
| [2025-02-23 14:03:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.7). Total num frames: 4947968. Throughput: 0: 860.1. Samples: 1241100. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:03:18,318][00318] Avg episode reward: [(0, '4.419')] | |
| [2025-02-23 14:03:23,314][00318] Fps is (10 sec: 1638.4, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 4956160. Throughput: 0: 862.8. Samples: 1243218. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:03:23,317][00318] Avg episode reward: [(0, '4.278')] | |
| [2025-02-23 14:03:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3443.4). Total num frames: 4980736. Throughput: 0: 843.2. Samples: 1247880. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:03:28,320][00318] Avg episode reward: [(0, '4.442')] | |
| [2025-02-23 14:03:31,366][02476] Updated weights for policy 0, policy_version 610 (0.0014) | |
| [2025-02-23 14:03:33,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 5005312. Throughput: 0: 843.9. Samples: 1254432. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:03:33,315][00318] Avg episode reward: [(0, '4.426')] | |
| [2025-02-23 14:03:38,317][00318] Fps is (10 sec: 3275.8, 60 sec: 3413.2, 300 sec: 3415.6). Total num frames: 5013504. Throughput: 0: 853.0. Samples: 1257066. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:03:38,318][00318] Avg episode reward: [(0, '4.130')] | |
| [2025-02-23 14:03:43,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 5029888. Throughput: 0: 850.4. Samples: 1261164. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:03:43,316][00318] Avg episode reward: [(0, '4.351')] | |
| [2025-02-23 14:03:48,314][00318] Fps is (10 sec: 2458.3, 60 sec: 3140.3, 300 sec: 3360.1). Total num frames: 5038080. Throughput: 0: 842.4. Samples: 1265448. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 14:03:48,315][00318] Avg episode reward: [(0, '4.366')] | |
| [2025-02-23 14:03:53,319][00318] Fps is (10 sec: 3275.5, 60 sec: 3413.1, 300 sec: 3415.6). Total num frames: 5062656. Throughput: 0: 844.4. Samples: 1268298. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:03:53,320][00318] Avg episode reward: [(0, '4.255')] | |
| [2025-02-23 14:03:56,197][02476] Updated weights for policy 0, policy_version 620 (0.0026) | |
| [2025-02-23 14:03:58,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5087232. Throughput: 0: 846.3. Samples: 1275012. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:03:58,320][00318] Avg episode reward: [(0, '4.260')] | |
| [2025-02-23 14:04:03,316][00318] Fps is (10 sec: 4097.3, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 5103616. Throughput: 0: 865.3. Samples: 1280040. Policy #0 lag: (min: 0.0, avg: 2.4, max: 5.0) | |
| [2025-02-23 14:04:03,317][00318] Avg episode reward: [(0, '4.228')] | |
| [2025-02-23 14:04:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 5111808. Throughput: 0: 863.2. Samples: 1282062. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 14:04:08,315][00318] Avg episode reward: [(0, '4.082')] | |
| [2025-02-23 14:04:13,314][00318] Fps is (10 sec: 2458.0, 60 sec: 3140.3, 300 sec: 3415.6). Total num frames: 5128192. Throughput: 0: 848.1. Samples: 1286046. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:04:13,320][00318] Avg episode reward: [(0, '4.232')] | |
| [2025-02-23 14:04:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 5152768. Throughput: 0: 839.6. Samples: 1292214. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:04:18,320][00318] Avg episode reward: [(0, '4.124')] | |
| [2025-02-23 14:04:18,482][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000630_5160960.pth... | |
| [2025-02-23 14:04:18,495][02476] Updated weights for policy 0, policy_version 630 (0.0014) | |
| [2025-02-23 14:04:18,770][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000529_4333568.pth | |
| [2025-02-23 14:04:23,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 5169152. Throughput: 0: 851.4. Samples: 1295376. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:04:23,322][00318] Avg episode reward: [(0, '4.290')] | |
| [2025-02-23 14:04:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5185536. Throughput: 0: 875.5. Samples: 1300560. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:04:28,315][00318] Avg episode reward: [(0, '4.130')] | |
| [2025-02-23 14:04:33,317][00318] Fps is (10 sec: 3276.1, 60 sec: 3276.7, 300 sec: 3415.6). Total num frames: 5201920. Throughput: 0: 873.0. Samples: 1304736. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:04:33,318][00318] Avg episode reward: [(0, '4.259')] | |
| [2025-02-23 14:04:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.5, 300 sec: 3415.7). Total num frames: 5218304. Throughput: 0: 856.4. Samples: 1306830. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:04:38,316][00318] Avg episode reward: [(0, '4.313')] | |
| [2025-02-23 14:04:43,314][00318] Fps is (10 sec: 3277.6, 60 sec: 3413.4, 300 sec: 3415.6). Total num frames: 5234688. Throughput: 0: 836.7. Samples: 1312662. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 14:04:43,319][00318] Avg episode reward: [(0, '4.049')] | |
| [2025-02-23 14:04:43,879][02476] Updated weights for policy 0, policy_version 640 (0.0014) | |
| [2025-02-23 14:04:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3415.6). Total num frames: 5251072. Throughput: 0: 863.6. Samples: 1318902. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 14:04:48,318][00318] Avg episode reward: [(0, '4.164')] | |
| [2025-02-23 14:04:53,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3413.6, 300 sec: 3415.6). Total num frames: 5267456. Throughput: 0: 867.1. Samples: 1321080. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 14:04:53,316][00318] Avg episode reward: [(0, '4.059')] | |
| [2025-02-23 14:04:58,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 5283840. Throughput: 0: 870.0. Samples: 1325196. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:04:58,319][00318] Avg episode reward: [(0, '4.155')] | |
| [2025-02-23 14:05:03,314][00318] Fps is (10 sec: 4096.2, 60 sec: 3413.4, 300 sec: 3443.4). Total num frames: 5308416. Throughput: 0: 852.8. Samples: 1330590. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:05:03,319][00318] Avg episode reward: [(0, '4.123')] | |
| [2025-02-23 14:05:08,094][02476] Updated weights for policy 0, policy_version 650 (0.0018) | |
| [2025-02-23 14:05:08,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 5324800. Throughput: 0: 854.5. Samples: 1333830. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:05:08,320][00318] Avg episode reward: [(0, '4.260')] | |
| [2025-02-23 14:05:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3471.2). Total num frames: 5349376. Throughput: 0: 859.1. Samples: 1339218. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:05:13,315][00318] Avg episode reward: [(0, '4.354')] | |
| [2025-02-23 14:05:18,314][00318] Fps is (10 sec: 2457.8, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 5349376. Throughput: 0: 861.4. Samples: 1343496. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:05:18,318][00318] Avg episode reward: [(0, '4.334')] | |
| [2025-02-23 14:05:23,316][00318] Fps is (10 sec: 2457.2, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5373952. Throughput: 0: 861.7. Samples: 1345608. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:05:23,319][00318] Avg episode reward: [(0, '4.635')] | |
| [2025-02-23 14:05:23,323][02456] Saving new best policy, reward=4.635! | |
| [2025-02-23 14:05:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5390336. Throughput: 0: 854.9. Samples: 1351134. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:05:28,320][00318] Avg episode reward: [(0, '4.092')] | |
| [2025-02-23 14:05:29,666][02476] Updated weights for policy 0, policy_version 660 (0.0018) | |
| [2025-02-23 14:05:33,314][00318] Fps is (10 sec: 4096.7, 60 sec: 3550.0, 300 sec: 3471.2). Total num frames: 5414912. Throughput: 0: 864.0. Samples: 1357782. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:05:33,315][00318] Avg episode reward: [(0, '4.099')] | |
| [2025-02-23 14:05:38,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 5431296. Throughput: 0: 870.0. Samples: 1360230. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:05:38,316][00318] Avg episode reward: [(0, '4.254')] | |
| [2025-02-23 14:05:43,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 5439488. Throughput: 0: 868.7. Samples: 1364286. Policy #0 lag: (min: 0.0, avg: 2.4, max: 4.0) | |
| [2025-02-23 14:05:43,326][00318] Avg episode reward: [(0, '4.056')] | |
| [2025-02-23 14:05:48,314][00318] Fps is (10 sec: 2457.8, 60 sec: 3413.3, 300 sec: 3387.9). Total num frames: 5455872. Throughput: 0: 843.9. Samples: 1368564. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:05:48,316][00318] Avg episode reward: [(0, '4.188')] | |
| [2025-02-23 14:05:53,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5480448. Throughput: 0: 841.3. Samples: 1371690. Policy #0 lag: (min: 0.0, avg: 1.6, max: 5.0) | |
| [2025-02-23 14:05:53,315][00318] Avg episode reward: [(0, '4.374')] | |
| [2025-02-23 14:05:54,770][02476] Updated weights for policy 0, policy_version 670 (0.0018) | |
| [2025-02-23 14:05:58,314][00318] Fps is (10 sec: 4095.9, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5496832. Throughput: 0: 871.6. Samples: 1378440. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:05:58,316][00318] Avg episode reward: [(0, '4.078')] | |
| [2025-02-23 14:06:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5513216. Throughput: 0: 882.9. Samples: 1383228. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 14:06:03,318][00318] Avg episode reward: [(0, '4.426')] | |
| [2025-02-23 14:06:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3415.7). Total num frames: 5529600. Throughput: 0: 882.4. Samples: 1385316. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:06:08,316][00318] Avg episode reward: [(0, '4.501')] | |
| [2025-02-23 14:06:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3415.7). Total num frames: 5545984. Throughput: 0: 848.4. Samples: 1389312. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:06:13,319][00318] Avg episode reward: [(0, '4.569')] | |
| [2025-02-23 14:06:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5562368. Throughput: 0: 839.1. Samples: 1395540. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:06:18,319][00318] Avg episode reward: [(0, '4.305')] | |
| [2025-02-23 14:06:18,327][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000679_5562368.pth... | |
| [2025-02-23 14:06:18,607][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000579_4743168.pth | |
| [2025-02-23 14:06:19,690][02476] Updated weights for policy 0, policy_version 680 (0.0017) | |
| [2025-02-23 14:06:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3443.4). Total num frames: 5586944. Throughput: 0: 855.7. Samples: 1398738. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:06:23,320][00318] Avg episode reward: [(0, '4.321')] | |
| [2025-02-23 14:06:28,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5595136. Throughput: 0: 879.1. Samples: 1403844. Policy #0 lag: (min: 0.0, avg: 2.4, max: 4.0) | |
| [2025-02-23 14:06:28,316][00318] Avg episode reward: [(0, '4.249')] | |
| [2025-02-23 14:06:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 5619712. Throughput: 0: 876.7. Samples: 1408014. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:06:33,320][00318] Avg episode reward: [(0, '4.161')] | |
| [2025-02-23 14:06:38,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 5627904. Throughput: 0: 855.3. Samples: 1410180. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 14:06:38,315][00318] Avg episode reward: [(0, '4.223')] | |
| [2025-02-23 14:06:41,714][02476] Updated weights for policy 0, policy_version 690 (0.0014) | |
| [2025-02-23 14:06:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.5). Total num frames: 5652480. Throughput: 0: 839.3. Samples: 1416210. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:06:43,319][00318] Avg episode reward: [(0, '4.226')] | |
| [2025-02-23 14:06:48,318][00318] Fps is (10 sec: 4094.6, 60 sec: 3549.7, 300 sec: 3415.6). Total num frames: 5668864. Throughput: 0: 865.3. Samples: 1422168. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:06:48,322][00318] Avg episode reward: [(0, '4.486')] | |
| [2025-02-23 14:06:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5685248. Throughput: 0: 864.3. Samples: 1424208. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:06:53,316][00318] Avg episode reward: [(0, '4.062')] | |
| [2025-02-23 14:06:58,314][00318] Fps is (10 sec: 3277.9, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5701632. Throughput: 0: 870.8. Samples: 1428498. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:06:58,315][00318] Avg episode reward: [(0, '4.403')] | |
| [2025-02-23 14:07:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 5718016. Throughput: 0: 860.8. Samples: 1434276. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:07:03,315][00318] Avg episode reward: [(0, '4.124')] | |
| [2025-02-23 14:07:05,619][02476] Updated weights for policy 0, policy_version 700 (0.0018) | |
| [2025-02-23 14:07:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 5742592. Throughput: 0: 864.7. Samples: 1437648. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:07:08,316][00318] Avg episode reward: [(0, '4.498')] | |
| [2025-02-23 14:07:13,320][00318] Fps is (10 sec: 3274.9, 60 sec: 3413.0, 300 sec: 3415.6). Total num frames: 5750784. Throughput: 0: 873.9. Samples: 1443174. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 14:07:13,321][00318] Avg episode reward: [(0, '4.559')] | |
| [2025-02-23 14:07:18,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3549.8, 300 sec: 3443.4). Total num frames: 5775360. Throughput: 0: 871.9. Samples: 1447248. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 14:07:18,317][00318] Avg episode reward: [(0, '4.247')] | |
| [2025-02-23 14:07:23,314][00318] Fps is (10 sec: 3278.7, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 5783552. Throughput: 0: 870.5. Samples: 1449354. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:07:23,323][00318] Avg episode reward: [(0, '4.592')] | |
| [2025-02-23 14:07:28,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5808128. Throughput: 0: 860.9. Samples: 1454952. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:07:28,319][00318] Avg episode reward: [(0, '4.372')] | |
| [2025-02-23 14:07:30,134][02476] Updated weights for policy 0, policy_version 710 (0.0014) | |
| [2025-02-23 14:07:33,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 5832704. Throughput: 0: 876.7. Samples: 1461618. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:07:33,315][00318] Avg episode reward: [(0, '4.804')] | |
| [2025-02-23 14:07:33,317][02456] Saving new best policy, reward=4.804! | |
| [2025-02-23 14:07:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5840896. Throughput: 0: 882.7. Samples: 1463928. Policy #0 lag: (min: 0.0, avg: 2.5, max: 5.0) | |
| [2025-02-23 14:07:38,315][00318] Avg episode reward: [(0, '4.403')] | |
| [2025-02-23 14:07:43,315][00318] Fps is (10 sec: 2457.4, 60 sec: 3413.3, 300 sec: 3415.6). Total num frames: 5857280. Throughput: 0: 882.0. Samples: 1468188. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:07:43,316][00318] Avg episode reward: [(0, '4.493')] | |
| [2025-02-23 14:07:48,325][00318] Fps is (10 sec: 3273.4, 60 sec: 3412.9, 300 sec: 3443.3). Total num frames: 5873664. Throughput: 0: 841.0. Samples: 1472130. Policy #0 lag: (min: 0.0, avg: 2.3, max: 5.0) | |
| [2025-02-23 14:07:48,326][00318] Avg episode reward: [(0, '4.550')] | |
| [2025-02-23 14:07:52,370][02476] Updated weights for policy 0, policy_version 720 (0.0017) | |
| [2025-02-23 14:07:53,314][00318] Fps is (10 sec: 4096.3, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 5898240. Throughput: 0: 838.1. Samples: 1475364. Policy #0 lag: (min: 0.0, avg: 1.5, max: 4.0) | |
| [2025-02-23 14:07:53,315][00318] Avg episode reward: [(0, '4.508')] | |
| [2025-02-23 14:07:58,314][00318] Fps is (10 sec: 4100.3, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 5914624. Throughput: 0: 863.2. Samples: 1482012. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:07:58,319][00318] Avg episode reward: [(0, '4.864')] | |
| [2025-02-23 14:07:58,327][02456] Saving new best policy, reward=4.864! | |
| [2025-02-23 14:08:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5931008. Throughput: 0: 876.8. Samples: 1486704. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:08:03,320][00318] Avg episode reward: [(0, '4.354')] | |
| [2025-02-23 14:08:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3387.9). Total num frames: 5939200. Throughput: 0: 879.2. Samples: 1488918. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:08:08,315][00318] Avg episode reward: [(0, '5.042')] | |
| [2025-02-23 14:08:08,438][02456] Saving new best policy, reward=5.042! | |
| [2025-02-23 14:08:13,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3550.2, 300 sec: 3443.4). Total num frames: 5963776. Throughput: 0: 850.5. Samples: 1493226. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:08:13,316][00318] Avg episode reward: [(0, '4.574')] | |
| [2025-02-23 14:08:17,497][02476] Updated weights for policy 0, policy_version 730 (0.0014) | |
| [2025-02-23 14:08:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 5980160. Throughput: 0: 839.5. Samples: 1499394. Policy #0 lag: (min: 0.0, avg: 1.6, max: 4.0) | |
| [2025-02-23 14:08:18,315][00318] Avg episode reward: [(0, '4.577')] | |
| [2025-02-23 14:08:18,325][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000730_5980160.pth... | |
| [2025-02-23 14:08:18,712][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000630_5160960.pth | |
| [2025-02-23 14:08:23,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 5996544. Throughput: 0: 858.7. Samples: 1502568. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:08:23,316][00318] Avg episode reward: [(0, '4.645')] | |
| [2025-02-23 14:08:28,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3413.2, 300 sec: 3415.6). Total num frames: 6012928. Throughput: 0: 863.3. Samples: 1507038. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:08:28,317][00318] Avg episode reward: [(0, '4.724')] | |
| [2025-02-23 14:08:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3443.5). Total num frames: 6029312. Throughput: 0: 869.8. Samples: 1511262. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:08:33,315][00318] Avg episode reward: [(0, '4.268')] | |
| [2025-02-23 14:08:38,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6045696. Throughput: 0: 857.7. Samples: 1513962. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:08:38,316][00318] Avg episode reward: [(0, '4.776')] | |
| [2025-02-23 14:08:40,182][02476] Updated weights for policy 0, policy_version 740 (0.0012) | |
| [2025-02-23 14:08:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6070272. Throughput: 0: 860.8. Samples: 1520748. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:08:43,315][00318] Avg episode reward: [(0, '4.552')] | |
| [2025-02-23 14:08:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.5, 300 sec: 3471.2). Total num frames: 6086656. Throughput: 0: 871.1. Samples: 1525902. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:08:48,315][00318] Avg episode reward: [(0, '4.660')] | |
| [2025-02-23 14:08:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6103040. Throughput: 0: 869.5. Samples: 1528044. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:08:53,316][00318] Avg episode reward: [(0, '5.214')] | |
| [2025-02-23 14:08:53,325][02456] Saving new best policy, reward=5.214! | |
| [2025-02-23 14:08:58,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3415.7). Total num frames: 6111232. Throughput: 0: 865.9. Samples: 1532190. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:08:58,319][00318] Avg episode reward: [(0, '4.828')] | |
| [2025-02-23 14:09:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6135808. Throughput: 0: 861.7. Samples: 1538172. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:09:03,318][00318] Avg episode reward: [(0, '4.533')] | |
| [2025-02-23 14:09:04,133][02476] Updated weights for policy 0, policy_version 750 (0.0015) | |
| [2025-02-23 14:09:08,314][00318] Fps is (10 sec: 4915.0, 60 sec: 3686.4, 300 sec: 3499.0). Total num frames: 6160384. Throughput: 0: 866.8. Samples: 1541574. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:09:08,318][00318] Avg episode reward: [(0, '4.993')] | |
| [2025-02-23 14:09:13,315][00318] Fps is (10 sec: 4095.6, 60 sec: 3549.8, 300 sec: 3471.2). Total num frames: 6176768. Throughput: 0: 891.3. Samples: 1547148. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:09:13,316][00318] Avg episode reward: [(0, '4.653')] | |
| [2025-02-23 14:09:18,314][00318] Fps is (10 sec: 2457.7, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6184960. Throughput: 0: 888.8. Samples: 1551258. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:09:18,316][00318] Avg episode reward: [(0, '4.844')] | |
| [2025-02-23 14:09:23,314][00318] Fps is (10 sec: 2457.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6201344. Throughput: 0: 877.2. Samples: 1553436. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:09:23,321][00318] Avg episode reward: [(0, '4.312')] | |
| [2025-02-23 14:09:27,710][02476] Updated weights for policy 0, policy_version 760 (0.0015) | |
| [2025-02-23 14:09:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3471.2). Total num frames: 6225920. Throughput: 0: 854.1. Samples: 1559184. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:09:28,320][00318] Avg episode reward: [(0, '4.490')] | |
| [2025-02-23 14:09:33,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6242304. Throughput: 0: 884.8. Samples: 1565718. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:09:33,315][00318] Avg episode reward: [(0, '5.343')] | |
| [2025-02-23 14:09:33,317][02456] Saving new best policy, reward=5.343! | |
| [2025-02-23 14:09:38,316][00318] Fps is (10 sec: 3276.2, 60 sec: 3549.8, 300 sec: 3471.2). Total num frames: 6258688. Throughput: 0: 884.6. Samples: 1567854. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:09:38,320][00318] Avg episode reward: [(0, '4.539')] | |
| [2025-02-23 14:09:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6275072. Throughput: 0: 886.9. Samples: 1572102. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:09:43,319][00318] Avg episode reward: [(0, '5.044')] | |
| [2025-02-23 14:09:48,314][00318] Fps is (10 sec: 3277.4, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6291456. Throughput: 0: 856.5. Samples: 1576716. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:09:48,319][00318] Avg episode reward: [(0, '4.263')] | |
| [2025-02-23 14:09:51,597][02476] Updated weights for policy 0, policy_version 770 (0.0018) | |
| [2025-02-23 14:09:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6307840. Throughput: 0: 846.3. Samples: 1579656. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:09:53,320][00318] Avg episode reward: [(0, '4.360')] | |
| [2025-02-23 14:09:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3443.4). Total num frames: 6324224. Throughput: 0: 866.4. Samples: 1586136. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:09:58,315][00318] Avg episode reward: [(0, '4.600')] | |
| [2025-02-23 14:10:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6340608. Throughput: 0: 870.9. Samples: 1590450. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:10:03,315][00318] Avg episode reward: [(0, '4.983')] | |
| [2025-02-23 14:10:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3276.8, 300 sec: 3415.6). Total num frames: 6356992. Throughput: 0: 871.2. Samples: 1592640. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:10:08,321][00318] Avg episode reward: [(0, '4.941')] | |
| [2025-02-23 14:10:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.4, 300 sec: 3499.0). Total num frames: 6381568. Throughput: 0: 859.5. Samples: 1597860. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:10:13,320][00318] Avg episode reward: [(0, '4.604')] | |
| [2025-02-23 14:10:15,176][02476] Updated weights for policy 0, policy_version 780 (0.0022) | |
| [2025-02-23 14:10:18,317][00318] Fps is (10 sec: 4094.9, 60 sec: 3549.7, 300 sec: 3471.2). Total num frames: 6397952. Throughput: 0: 861.9. Samples: 1604508. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:10:18,318][00318] Avg episode reward: [(0, '4.965')] | |
| [2025-02-23 14:10:18,324][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000781_6397952.pth... | |
| [2025-02-23 14:10:18,822][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000679_5562368.pth | |
| [2025-02-23 14:10:23,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3549.8, 300 sec: 3471.2). Total num frames: 6414336. Throughput: 0: 866.8. Samples: 1606860. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:10:23,316][00318] Avg episode reward: [(0, '5.181')] | |
| [2025-02-23 14:10:28,314][00318] Fps is (10 sec: 3277.7, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6430720. Throughput: 0: 868.7. Samples: 1611192. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:10:28,317][00318] Avg episode reward: [(0, '4.656')] | |
| [2025-02-23 14:10:33,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6447104. Throughput: 0: 860.9. Samples: 1615458. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:10:33,319][00318] Avg episode reward: [(0, '4.902')] | |
| [2025-02-23 14:10:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 6463488. Throughput: 0: 864.7. Samples: 1618566. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:10:38,318][00318] Avg episode reward: [(0, '5.130')] | |
| [2025-02-23 14:10:38,501][02476] Updated weights for policy 0, policy_version 790 (0.0013) | |
| [2025-02-23 14:10:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6488064. Throughput: 0: 872.7. Samples: 1625406. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:10:43,318][00318] Avg episode reward: [(0, '4.728')] | |
| [2025-02-23 14:10:48,317][00318] Fps is (10 sec: 4094.9, 60 sec: 3549.7, 300 sec: 3471.2). Total num frames: 6504448. Throughput: 0: 884.1. Samples: 1630236. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:10:48,318][00318] Avg episode reward: [(0, '4.805')] | |
| [2025-02-23 14:10:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6512640. Throughput: 0: 876.8. Samples: 1632096. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:10:53,320][00318] Avg episode reward: [(0, '4.649')] | |
| [2025-02-23 14:10:58,314][00318] Fps is (10 sec: 3277.6, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6537216. Throughput: 0: 855.9. Samples: 1636374. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:10:58,315][00318] Avg episode reward: [(0, '4.540')] | |
| [2025-02-23 14:11:02,212][02476] Updated weights for policy 0, policy_version 800 (0.0013) | |
| [2025-02-23 14:11:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6553600. Throughput: 0: 852.1. Samples: 1642848. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:11:03,315][00318] Avg episode reward: [(0, '4.770')] | |
| [2025-02-23 14:11:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6569984. Throughput: 0: 875.6. Samples: 1646262. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:11:08,317][00318] Avg episode reward: [(0, '5.312')] | |
| [2025-02-23 14:11:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6586368. Throughput: 0: 889.3. Samples: 1651212. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:11:13,319][00318] Avg episode reward: [(0, '5.055')] | |
| [2025-02-23 14:11:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.5, 300 sec: 3443.4). Total num frames: 6602752. Throughput: 0: 888.7. Samples: 1655448. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:11:18,318][00318] Avg episode reward: [(0, '4.956')] | |
| [2025-02-23 14:11:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 6619136. Throughput: 0: 866.7. Samples: 1657566. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:11:23,315][00318] Avg episode reward: [(0, '5.207')] | |
| [2025-02-23 14:11:25,601][02476] Updated weights for policy 0, policy_version 810 (0.0027) | |
| [2025-02-23 14:11:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6643712. Throughput: 0: 853.2. Samples: 1663800. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:11:28,320][00318] Avg episode reward: [(0, '5.205')] | |
| [2025-02-23 14:11:33,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6660096. Throughput: 0: 874.3. Samples: 1669578. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:11:33,318][00318] Avg episode reward: [(0, '5.267')] | |
| [2025-02-23 14:11:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6676480. Throughput: 0: 880.4. Samples: 1671714. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:11:38,319][00318] Avg episode reward: [(0, '5.094')] | |
| [2025-02-23 14:11:43,324][00318] Fps is (10 sec: 3273.4, 60 sec: 3412.7, 300 sec: 3471.1). Total num frames: 6692864. Throughput: 0: 880.2. Samples: 1675992. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:11:43,331][00318] Avg episode reward: [(0, '5.415')] | |
| [2025-02-23 14:11:43,337][02456] Saving new best policy, reward=5.415! | |
| [2025-02-23 14:11:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.5, 300 sec: 3471.2). Total num frames: 6709248. Throughput: 0: 866.0. Samples: 1681818. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:11:48,324][00318] Avg episode reward: [(0, '4.656')] | |
| [2025-02-23 14:11:48,767][02476] Updated weights for policy 0, policy_version 820 (0.0016) | |
| [2025-02-23 14:11:53,314][00318] Fps is (10 sec: 3280.2, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6725632. Throughput: 0: 866.3. Samples: 1685244. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:11:53,318][00318] Avg episode reward: [(0, '4.938')] | |
| [2025-02-23 14:11:58,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6750208. Throughput: 0: 865.5. Samples: 1690158. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:11:58,315][00318] Avg episode reward: [(0, '4.948')] | |
| [2025-02-23 14:12:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6758400. Throughput: 0: 864.8. Samples: 1694364. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:12:03,317][00318] Avg episode reward: [(0, '4.822')] | |
| [2025-02-23 14:12:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6782976. Throughput: 0: 864.9. Samples: 1696488. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:12:08,315][00318] Avg episode reward: [(0, '4.959')] | |
| [2025-02-23 14:12:12,681][02476] Updated weights for policy 0, policy_version 830 (0.0015) | |
| [2025-02-23 14:12:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6799360. Throughput: 0: 868.3. Samples: 1702872. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:12:13,315][00318] Avg episode reward: [(0, '4.575')] | |
| [2025-02-23 14:12:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6815744. Throughput: 0: 880.5. Samples: 1709202. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:12:18,321][00318] Avg episode reward: [(0, '5.190')] | |
| [2025-02-23 14:12:18,329][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000832_6815744.pth... | |
| [2025-02-23 14:12:18,739][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000730_5980160.pth | |
| [2025-02-23 14:12:23,315][00318] Fps is (10 sec: 3276.5, 60 sec: 3549.8, 300 sec: 3471.2). Total num frames: 6832128. Throughput: 0: 879.9. Samples: 1711308. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:12:23,323][00318] Avg episode reward: [(0, '4.284')] | |
| [2025-02-23 14:12:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6848512. Throughput: 0: 875.5. Samples: 1715382. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:12:28,315][00318] Avg episode reward: [(0, '5.490')] | |
| [2025-02-23 14:12:28,321][02456] Saving new best policy, reward=5.490! | |
| [2025-02-23 14:12:33,314][00318] Fps is (10 sec: 3277.1, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6864896. Throughput: 0: 853.2. Samples: 1720212. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:12:33,315][00318] Avg episode reward: [(0, '5.373')] | |
| [2025-02-23 14:12:36,568][02476] Updated weights for policy 0, policy_version 840 (0.0022) | |
| [2025-02-23 14:12:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6889472. Throughput: 0: 851.1. Samples: 1723542. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:12:38,315][00318] Avg episode reward: [(0, '4.801')] | |
| [2025-02-23 14:12:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.5, 300 sec: 3499.1). Total num frames: 6905856. Throughput: 0: 887.9. Samples: 1730112. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:12:43,315][00318] Avg episode reward: [(0, '4.997')] | |
| [2025-02-23 14:12:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 6922240. Throughput: 0: 890.3. Samples: 1734426. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:12:48,316][00318] Avg episode reward: [(0, '4.969')] | |
| [2025-02-23 14:12:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3443.4). Total num frames: 6930432. Throughput: 0: 890.0. Samples: 1736538. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:12:53,320][00318] Avg episode reward: [(0, '4.612')] | |
| [2025-02-23 14:12:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 6955008. Throughput: 0: 846.1. Samples: 1740948. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:12:58,316][00318] Avg episode reward: [(0, '5.150')] | |
| [2025-02-23 14:12:59,733][02476] Updated weights for policy 0, policy_version 850 (0.0015) | |
| [2025-02-23 14:13:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6971392. Throughput: 0: 849.3. Samples: 1747422. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:13:03,315][00318] Avg episode reward: [(0, '5.228')] | |
| [2025-02-23 14:13:08,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 6995968. Throughput: 0: 874.0. Samples: 1750638. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:13:08,315][00318] Avg episode reward: [(0, '5.335')] | |
| [2025-02-23 14:13:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7004160. Throughput: 0: 878.5. Samples: 1754916. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:13:13,320][00318] Avg episode reward: [(0, '5.277')] | |
| [2025-02-23 14:13:18,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7020544. Throughput: 0: 866.3. Samples: 1759194. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:13:18,315][00318] Avg episode reward: [(0, '5.042')] | |
| [2025-02-23 14:13:23,031][02476] Updated weights for policy 0, policy_version 860 (0.0013) | |
| [2025-02-23 14:13:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7045120. Throughput: 0: 861.9. Samples: 1762326. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:13:23,321][00318] Avg episode reward: [(0, '4.828')] | |
| [2025-02-23 14:13:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7061504. Throughput: 0: 852.4. Samples: 1768470. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:13:28,321][00318] Avg episode reward: [(0, '5.958')] | |
| [2025-02-23 14:13:28,328][02456] Saving new best policy, reward=5.958! | |
| [2025-02-23 14:13:33,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3549.8, 300 sec: 3498.9). Total num frames: 7077888. Throughput: 0: 866.4. Samples: 1773414. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:13:33,317][00318] Avg episode reward: [(0, '5.477')] | |
| [2025-02-23 14:13:38,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3443.4). Total num frames: 7086080. Throughput: 0: 865.9. Samples: 1775502. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:13:38,315][00318] Avg episode reward: [(0, '5.207')] | |
| [2025-02-23 14:13:43,314][00318] Fps is (10 sec: 2458.0, 60 sec: 3276.8, 300 sec: 3443.4). Total num frames: 7102464. Throughput: 0: 864.7. Samples: 1779858. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:13:43,322][00318] Avg episode reward: [(0, '5.353')] | |
| [2025-02-23 14:13:47,196][02476] Updated weights for policy 0, policy_version 870 (0.0019) | |
| [2025-02-23 14:13:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7127040. Throughput: 0: 871.2. Samples: 1786626. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:13:48,315][00318] Avg episode reward: [(0, '5.687')] | |
| [2025-02-23 14:13:53,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 7151616. Throughput: 0: 874.5. Samples: 1789992. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:13:53,319][00318] Avg episode reward: [(0, '5.833')] | |
| [2025-02-23 14:13:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7159808. Throughput: 0: 886.8. Samples: 1794822. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:13:58,316][00318] Avg episode reward: [(0, '5.317')] | |
| [2025-02-23 14:14:03,317][00318] Fps is (10 sec: 2456.8, 60 sec: 3413.2, 300 sec: 3443.4). Total num frames: 7176192. Throughput: 0: 880.6. Samples: 1798824. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:14:03,319][00318] Avg episode reward: [(0, '5.581')] | |
| [2025-02-23 14:14:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7200768. Throughput: 0: 858.5. Samples: 1800960. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:14:08,319][00318] Avg episode reward: [(0, '5.740')] | |
| [2025-02-23 14:14:10,103][02476] Updated weights for policy 0, policy_version 880 (0.0018) | |
| [2025-02-23 14:14:13,314][00318] Fps is (10 sec: 4097.3, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7217152. Throughput: 0: 867.1. Samples: 1807488. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:14:13,321][00318] Avg episode reward: [(0, '5.617')] | |
| [2025-02-23 14:14:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 7241728. Throughput: 0: 895.6. Samples: 1813716. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:14:18,317][00318] Avg episode reward: [(0, '4.882')] | |
| [2025-02-23 14:14:18,325][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000884_7241728.pth... | |
| [2025-02-23 14:14:18,705][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000781_6397952.pth | |
| [2025-02-23 14:14:23,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7249920. Throughput: 0: 894.4. Samples: 1815750. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:14:23,316][00318] Avg episode reward: [(0, '4.897')] | |
| [2025-02-23 14:14:28,315][00318] Fps is (10 sec: 2457.3, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7266304. Throughput: 0: 892.2. Samples: 1820010. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:14:28,317][00318] Avg episode reward: [(0, '5.211')] | |
| [2025-02-23 14:14:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 7282688. Throughput: 0: 852.4. Samples: 1824984. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:14:33,315][00318] Avg episode reward: [(0, '6.140')] | |
| [2025-02-23 14:14:33,317][02456] Saving new best policy, reward=6.140! | |
| [2025-02-23 14:14:33,869][02476] Updated weights for policy 0, policy_version 890 (0.0015) | |
| [2025-02-23 14:14:38,314][00318] Fps is (10 sec: 3277.1, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 7299072. Throughput: 0: 846.1. Samples: 1828068. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:14:38,325][00318] Avg episode reward: [(0, '5.526')] | |
| [2025-02-23 14:14:43,314][00318] Fps is (10 sec: 4095.9, 60 sec: 3686.4, 300 sec: 3499.0). Total num frames: 7323648. Throughput: 0: 882.4. Samples: 1834530. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:14:43,317][00318] Avg episode reward: [(0, '5.934')] | |
| [2025-02-23 14:14:48,319][00318] Fps is (10 sec: 3275.3, 60 sec: 3413.1, 300 sec: 3471.1). Total num frames: 7331840. Throughput: 0: 890.2. Samples: 1838886. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:14:48,320][00318] Avg episode reward: [(0, '5.707')] | |
| [2025-02-23 14:14:53,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 7348224. Throughput: 0: 888.0. Samples: 1840920. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:14:53,318][00318] Avg episode reward: [(0, '5.230')] | |
| [2025-02-23 14:14:57,647][02476] Updated weights for policy 0, policy_version 900 (0.0018) | |
| [2025-02-23 14:14:58,314][00318] Fps is (10 sec: 4097.9, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7372800. Throughput: 0: 866.7. Samples: 1846488. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:14:58,315][00318] Avg episode reward: [(0, '5.830')] | |
| [2025-02-23 14:15:03,314][00318] Fps is (10 sec: 4915.3, 60 sec: 3686.6, 300 sec: 3526.7). Total num frames: 7397376. Throughput: 0: 861.9. Samples: 1852500. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:15:03,315][00318] Avg episode reward: [(0, '5.092')] | |
| [2025-02-23 14:15:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7413760. Throughput: 0: 876.7. Samples: 1855200. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:15:08,321][00318] Avg episode reward: [(0, '5.556')] | |
| [2025-02-23 14:15:13,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7421952. Throughput: 0: 875.9. Samples: 1859424. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:15:13,315][00318] Avg episode reward: [(0, '5.831')] | |
| [2025-02-23 14:15:18,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 7438336. Throughput: 0: 866.7. Samples: 1863984. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:15:18,315][00318] Avg episode reward: [(0, '5.503')] | |
| [2025-02-23 14:15:21,059][02476] Updated weights for policy 0, policy_version 910 (0.0013) | |
| [2025-02-23 14:15:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7462912. Throughput: 0: 874.3. Samples: 1867410. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:15:23,316][00318] Avg episode reward: [(0, '5.347')] | |
| [2025-02-23 14:15:28,319][00318] Fps is (10 sec: 4094.1, 60 sec: 3549.7, 300 sec: 3498.9). Total num frames: 7479296. Throughput: 0: 880.0. Samples: 1874136. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:15:28,325][00318] Avg episode reward: [(0, '5.726')] | |
| [2025-02-23 14:15:33,318][00318] Fps is (10 sec: 3275.6, 60 sec: 3549.7, 300 sec: 3498.9). Total num frames: 7495680. Throughput: 0: 880.3. Samples: 1878498. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:15:33,319][00318] Avg episode reward: [(0, '5.415')] | |
| [2025-02-23 14:15:38,314][00318] Fps is (10 sec: 3278.3, 60 sec: 3549.9, 300 sec: 3471.2). Total num frames: 7512064. Throughput: 0: 880.7. Samples: 1880550. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:15:38,315][00318] Avg episode reward: [(0, '5.306')] | |
| [2025-02-23 14:15:43,314][00318] Fps is (10 sec: 3278.0, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7528448. Throughput: 0: 855.9. Samples: 1885002. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:15:43,315][00318] Avg episode reward: [(0, '5.149')] | |
| [2025-02-23 14:15:43,986][02476] Updated weights for policy 0, policy_version 920 (0.0014) | |
| [2025-02-23 14:15:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.7, 300 sec: 3526.7). Total num frames: 7553024. Throughput: 0: 875.2. Samples: 1891884. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:15:48,322][00318] Avg episode reward: [(0, '5.488')] | |
| [2025-02-23 14:15:53,315][00318] Fps is (10 sec: 4095.8, 60 sec: 3686.4, 300 sec: 3499.0). Total num frames: 7569408. Throughput: 0: 889.2. Samples: 1895214. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:15:53,316][00318] Avg episode reward: [(0, '5.327')] | |
| [2025-02-23 14:15:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 7585792. Throughput: 0: 899.1. Samples: 1899882. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:15:58,317][00318] Avg episode reward: [(0, '5.259')] | |
| [2025-02-23 14:16:03,316][00318] Fps is (10 sec: 2457.3, 60 sec: 3276.7, 300 sec: 3471.2). Total num frames: 7593984. Throughput: 0: 889.2. Samples: 1903998. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:16:03,317][00318] Avg episode reward: [(0, '5.369')] | |
| [2025-02-23 14:16:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 7610368. Throughput: 0: 858.1. Samples: 1906026. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:16:08,315][00318] Avg episode reward: [(0, '5.456')] | |
| [2025-02-23 14:16:08,549][02476] Updated weights for policy 0, policy_version 930 (0.0017) | |
| [2025-02-23 14:16:13,314][00318] Fps is (10 sec: 4916.0, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 7643136. Throughput: 0: 854.4. Samples: 1912578. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:16:13,315][00318] Avg episode reward: [(0, '5.502')] | |
| [2025-02-23 14:16:18,315][00318] Fps is (10 sec: 4914.9, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 7659520. Throughput: 0: 890.7. Samples: 1918578. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:16:18,316][00318] Avg episode reward: [(0, '5.805')] | |
| [2025-02-23 14:16:18,324][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000935_7659520.pth... | |
| [2025-02-23 14:16:18,694][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000832_6815744.pth | |
| [2025-02-23 14:16:23,317][00318] Fps is (10 sec: 2457.0, 60 sec: 3413.2, 300 sec: 3471.2). Total num frames: 7667712. Throughput: 0: 890.7. Samples: 1920636. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:16:23,322][00318] Avg episode reward: [(0, '5.692')] | |
| [2025-02-23 14:16:28,314][00318] Fps is (10 sec: 2457.7, 60 sec: 3413.6, 300 sec: 3471.2). Total num frames: 7684096. Throughput: 0: 887.9. Samples: 1924956. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 14:16:28,325][00318] Avg episode reward: [(0, '5.548')] | |
| [2025-02-23 14:16:31,242][02476] Updated weights for policy 0, policy_version 940 (0.0015) | |
| [2025-02-23 14:16:33,315][00318] Fps is (10 sec: 3277.4, 60 sec: 3413.5, 300 sec: 3471.2). Total num frames: 7700480. Throughput: 0: 857.6. Samples: 1930476. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:16:33,323][00318] Avg episode reward: [(0, '5.486')] | |
| [2025-02-23 14:16:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3499.1). Total num frames: 7725056. Throughput: 0: 849.7. Samples: 1933452. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:16:38,320][00318] Avg episode reward: [(0, '5.992')] | |
| [2025-02-23 14:16:43,316][00318] Fps is (10 sec: 4095.5, 60 sec: 3549.7, 300 sec: 3498.9). Total num frames: 7741440. Throughput: 0: 875.7. Samples: 1939290. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:16:43,317][00318] Avg episode reward: [(0, '5.831')] | |
| [2025-02-23 14:16:48,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 7757824. Throughput: 0: 877.9. Samples: 1943502. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:16:48,320][00318] Avg episode reward: [(0, '5.998')] | |
| [2025-02-23 14:16:53,314][00318] Fps is (10 sec: 3277.4, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 7774208. Throughput: 0: 880.3. Samples: 1945638. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:16:53,315][00318] Avg episode reward: [(0, '5.285')] | |
| [2025-02-23 14:16:54,475][02476] Updated weights for policy 0, policy_version 950 (0.0013) | |
| [2025-02-23 14:16:58,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 7798784. Throughput: 0: 872.3. Samples: 1951830. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:16:58,316][00318] Avg episode reward: [(0, '6.079')] | |
| [2025-02-23 14:17:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.5, 300 sec: 3499.0). Total num frames: 7815168. Throughput: 0: 887.1. Samples: 1958496. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:17:03,315][00318] Avg episode reward: [(0, '5.735')] | |
| [2025-02-23 14:17:08,315][00318] Fps is (10 sec: 2457.4, 60 sec: 3549.8, 300 sec: 3471.2). Total num frames: 7823360. Throughput: 0: 884.4. Samples: 1960434. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:17:08,316][00318] Avg episode reward: [(0, '5.723')] | |
| [2025-02-23 14:17:13,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 7839744. Throughput: 0: 883.2. Samples: 1964700. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:17:13,320][00318] Avg episode reward: [(0, '6.391')] | |
| [2025-02-23 14:17:13,327][02456] Saving new best policy, reward=6.391! | |
| [2025-02-23 14:17:18,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 7856128. Throughput: 0: 863.9. Samples: 1969350. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:17:18,316][00318] Avg episode reward: [(0, '6.123')] | |
| [2025-02-23 14:17:18,881][02476] Updated weights for policy 0, policy_version 960 (0.0013) | |
| [2025-02-23 14:17:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3499.0). Total num frames: 7880704. Throughput: 0: 871.7. Samples: 1972680. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:17:23,323][00318] Avg episode reward: [(0, '5.120')] | |
| [2025-02-23 14:17:28,314][00318] Fps is (10 sec: 4915.4, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 7905280. Throughput: 0: 893.5. Samples: 1979496. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:17:28,319][00318] Avg episode reward: [(0, '5.727')] | |
| [2025-02-23 14:17:33,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3499.0). Total num frames: 7921664. Throughput: 0: 895.3. Samples: 1983792. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:17:33,317][00318] Avg episode reward: [(0, '5.531')] | |
| [2025-02-23 14:17:38,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 7929856. Throughput: 0: 893.6. Samples: 1985850. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:17:38,320][00318] Avg episode reward: [(0, '5.929')] | |
| [2025-02-23 14:17:41,853][02476] Updated weights for policy 0, policy_version 970 (0.0015) | |
| [2025-02-23 14:17:43,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 7946240. Throughput: 0: 856.3. Samples: 1990362. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:17:43,320][00318] Avg episode reward: [(0, '6.228')] | |
| [2025-02-23 14:17:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 7970816. Throughput: 0: 857.5. Samples: 1997082. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:17:48,315][00318] Avg episode reward: [(0, '6.487')] | |
| [2025-02-23 14:17:48,328][02456] Saving new best policy, reward=6.487! | |
| [2025-02-23 14:17:53,318][00318] Fps is (10 sec: 4094.5, 60 sec: 3549.6, 300 sec: 3498.9). Total num frames: 7987200. Throughput: 0: 888.2. Samples: 2000406. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:17:53,319][00318] Avg episode reward: [(0, '5.990')] | |
| [2025-02-23 14:17:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 8003584. Throughput: 0: 895.3. Samples: 2004990. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:17:58,325][00318] Avg episode reward: [(0, '5.594')] | |
| [2025-02-23 14:18:03,321][00318] Fps is (10 sec: 3275.8, 60 sec: 3413.0, 300 sec: 3471.1). Total num frames: 8019968. Throughput: 0: 886.8. Samples: 2009262. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:18:03,322][00318] Avg episode reward: [(0, '6.194')] | |
| [2025-02-23 14:18:04,668][02476] Updated weights for policy 0, policy_version 980 (0.0013) | |
| [2025-02-23 14:18:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 8036352. Throughput: 0: 872.3. Samples: 2011932. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:18:08,315][00318] Avg episode reward: [(0, '6.163')] | |
| [2025-02-23 14:18:13,314][00318] Fps is (10 sec: 4098.7, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8060928. Throughput: 0: 855.5. Samples: 2017992. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:18:13,315][00318] Avg episode reward: [(0, '6.439')] | |
| [2025-02-23 14:18:18,318][00318] Fps is (10 sec: 3275.5, 60 sec: 3549.7, 300 sec: 3471.1). Total num frames: 8069120. Throughput: 0: 883.5. Samples: 2023554. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:18:18,319][00318] Avg episode reward: [(0, '5.722')] | |
| [2025-02-23 14:18:18,359][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000986_8077312.pth... | |
| [2025-02-23 14:18:18,741][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000884_7241728.pth | |
| [2025-02-23 14:18:23,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 8085504. Throughput: 0: 880.0. Samples: 2025450. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:18:23,321][00318] Avg episode reward: [(0, '6.423')] | |
| [2025-02-23 14:18:28,314][00318] Fps is (10 sec: 3278.1, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 8101888. Throughput: 0: 874.0. Samples: 2029692. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:18:28,315][00318] Avg episode reward: [(0, '6.232')] | |
| [2025-02-23 14:18:28,925][02476] Updated weights for policy 0, policy_version 990 (0.0013) | |
| [2025-02-23 14:18:33,314][00318] Fps is (10 sec: 4096.2, 60 sec: 3413.3, 300 sec: 3526.7). Total num frames: 8126464. Throughput: 0: 867.2. Samples: 2036106. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:18:33,315][00318] Avg episode reward: [(0, '5.694')] | |
| [2025-02-23 14:18:38,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8151040. Throughput: 0: 869.8. Samples: 2039544. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:18:38,315][00318] Avg episode reward: [(0, '5.953')] | |
| [2025-02-23 14:18:43,314][00318] Fps is (10 sec: 4095.9, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8167424. Throughput: 0: 872.4. Samples: 2044248. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:18:43,321][00318] Avg episode reward: [(0, '6.198')] | |
| [2025-02-23 14:18:48,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 8175616. Throughput: 0: 873.9. Samples: 2048580. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:18:48,316][00318] Avg episode reward: [(0, '5.913')] | |
| [2025-02-23 14:18:51,925][02476] Updated weights for policy 0, policy_version 1000 (0.0013) | |
| [2025-02-23 14:18:53,314][00318] Fps is (10 sec: 2457.7, 60 sec: 3413.5, 300 sec: 3499.0). Total num frames: 8192000. Throughput: 0: 864.7. Samples: 2050842. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:18:53,315][00318] Avg episode reward: [(0, '5.811')] | |
| [2025-02-23 14:18:58,314][00318] Fps is (10 sec: 4096.2, 60 sec: 3549.9, 300 sec: 3526.8). Total num frames: 8216576. Throughput: 0: 875.2. Samples: 2057376. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:18:58,315][00318] Avg episode reward: [(0, '6.019')] | |
| [2025-02-23 14:19:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.3, 300 sec: 3499.0). Total num frames: 8232960. Throughput: 0: 891.7. Samples: 2063676. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:03,316][00318] Avg episode reward: [(0, '7.079')] | |
| [2025-02-23 14:19:03,459][02456] Saving new best policy, reward=7.079! | |
| [2025-02-23 14:19:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 8249344. Throughput: 0: 896.8. Samples: 2065806. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:08,319][00318] Avg episode reward: [(0, '5.706')] | |
| [2025-02-23 14:19:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 8265728. Throughput: 0: 891.7. Samples: 2069820. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:13,315][00318] Avg episode reward: [(0, '5.695')] | |
| [2025-02-23 14:19:15,126][02476] Updated weights for policy 0, policy_version 1010 (0.0013) | |
| [2025-02-23 14:19:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3550.1, 300 sec: 3499.0). Total num frames: 8282112. Throughput: 0: 863.6. Samples: 2074968. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:19:18,317][00318] Avg episode reward: [(0, '6.167')] | |
| [2025-02-23 14:19:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8306688. Throughput: 0: 859.2. Samples: 2078208. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:19:23,316][00318] Avg episode reward: [(0, '6.579')] | |
| [2025-02-23 14:19:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8323072. Throughput: 0: 897.3. Samples: 2084628. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:19:28,315][00318] Avg episode reward: [(0, '6.626')] | |
| [2025-02-23 14:19:33,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3549.8, 300 sec: 3526.7). Total num frames: 8339456. Throughput: 0: 897.6. Samples: 2088972. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:33,316][00318] Avg episode reward: [(0, '6.364')] | |
| [2025-02-23 14:19:38,314][00318] Fps is (10 sec: 2457.5, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 8347648. Throughput: 0: 893.6. Samples: 2091054. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:38,316][00318] Avg episode reward: [(0, '5.309')] | |
| [2025-02-23 14:19:38,482][02476] Updated weights for policy 0, policy_version 1020 (0.0013) | |
| [2025-02-23 14:19:43,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3413.3, 300 sec: 3526.8). Total num frames: 8372224. Throughput: 0: 854.9. Samples: 2095848. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:43,316][00318] Avg episode reward: [(0, '6.046')] | |
| [2025-02-23 14:19:48,314][00318] Fps is (10 sec: 4915.4, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8396800. Throughput: 0: 868.1. Samples: 2102742. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:19:48,316][00318] Avg episode reward: [(0, '7.096')] | |
| [2025-02-23 14:19:48,323][02456] Saving new best policy, reward=7.096! | |
| [2025-02-23 14:19:53,315][00318] Fps is (10 sec: 4095.7, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8413184. Throughput: 0: 878.4. Samples: 2105334. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:19:53,326][00318] Avg episode reward: [(0, '5.664')] | |
| [2025-02-23 14:19:58,317][00318] Fps is (10 sec: 2456.9, 60 sec: 3413.2, 300 sec: 3471.2). Total num frames: 8421376. Throughput: 0: 885.5. Samples: 2109672. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:19:58,319][00318] Avg episode reward: [(0, '5.642')] | |
| [2025-02-23 14:20:01,783][02476] Updated weights for policy 0, policy_version 1030 (0.0015) | |
| [2025-02-23 14:20:03,314][00318] Fps is (10 sec: 2457.7, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 8437760. Throughput: 0: 870.7. Samples: 2114148. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:20:03,316][00318] Avg episode reward: [(0, '6.205')] | |
| [2025-02-23 14:20:08,314][00318] Fps is (10 sec: 4096.9, 60 sec: 3549.8, 300 sec: 3526.7). Total num frames: 8462336. Throughput: 0: 872.7. Samples: 2117478. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:20:08,317][00318] Avg episode reward: [(0, '6.307')] | |
| [2025-02-23 14:20:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 8478720. Throughput: 0: 870.8. Samples: 2123814. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:20:13,327][00318] Avg episode reward: [(0, '5.909')] | |
| [2025-02-23 14:20:18,315][00318] Fps is (10 sec: 3276.7, 60 sec: 3549.8, 300 sec: 3498.9). Total num frames: 8495104. Throughput: 0: 874.7. Samples: 2128332. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:20:18,317][00318] Avg episode reward: [(0, '5.770')] | |
| [2025-02-23 14:20:18,329][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001037_8495104.pth... | |
| [2025-02-23 14:20:18,684][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000935_7659520.pth | |
| [2025-02-23 14:20:23,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 8511488. Throughput: 0: 873.7. Samples: 2130372. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:20:23,318][00318] Avg episode reward: [(0, '6.063')] | |
| [2025-02-23 14:20:25,072][02476] Updated weights for policy 0, policy_version 1040 (0.0015) | |
| [2025-02-23 14:20:28,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 8527872. Throughput: 0: 872.7. Samples: 2135118. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:20:28,315][00318] Avg episode reward: [(0, '6.807')] | |
| [2025-02-23 14:20:33,314][00318] Fps is (10 sec: 4096.1, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 8552448. Throughput: 0: 868.8. Samples: 2141838. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:20:33,319][00318] Avg episode reward: [(0, '6.328')] | |
| [2025-02-23 14:20:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8568832. Throughput: 0: 887.9. Samples: 2145288. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:20:38,320][00318] Avg episode reward: [(0, '6.604')] | |
| [2025-02-23 14:20:43,315][00318] Fps is (10 sec: 2457.3, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 8577024. Throughput: 0: 890.6. Samples: 2149746. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:20:43,317][00318] Avg episode reward: [(0, '5.856')] | |
| [2025-02-23 14:20:48,315][00318] Fps is (10 sec: 2457.5, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 8593408. Throughput: 0: 881.9. Samples: 2153832. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:20:48,322][00318] Avg episode reward: [(0, '6.147')] | |
| [2025-02-23 14:20:48,413][02476] Updated weights for policy 0, policy_version 1050 (0.0016) | |
| [2025-02-23 14:20:53,314][00318] Fps is (10 sec: 4096.6, 60 sec: 3413.4, 300 sec: 3499.0). Total num frames: 8617984. Throughput: 0: 859.7. Samples: 2156166. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:20:53,315][00318] Avg episode reward: [(0, '6.041')] | |
| [2025-02-23 14:20:58,314][00318] Fps is (10 sec: 4915.5, 60 sec: 3686.6, 300 sec: 3554.5). Total num frames: 8642560. Throughput: 0: 868.5. Samples: 2162898. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:20:58,315][00318] Avg episode reward: [(0, '5.870')] | |
| [2025-02-23 14:21:03,314][00318] Fps is (10 sec: 4095.9, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8658944. Throughput: 0: 898.7. Samples: 2168772. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:21:03,319][00318] Avg episode reward: [(0, '5.567')] | |
| [2025-02-23 14:21:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.4, 300 sec: 3471.2). Total num frames: 8667136. Throughput: 0: 900.0. Samples: 2170872. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:21:08,317][00318] Avg episode reward: [(0, '6.504')] | |
| [2025-02-23 14:21:10,752][02476] Updated weights for policy 0, policy_version 1060 (0.0017) | |
| [2025-02-23 14:21:13,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3471.2). Total num frames: 8683520. Throughput: 0: 890.4. Samples: 2175186. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:21:13,315][00318] Avg episode reward: [(0, '6.855')] | |
| [2025-02-23 14:21:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.8). Total num frames: 8708096. Throughput: 0: 862.8. Samples: 2180664. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:21:18,316][00318] Avg episode reward: [(0, '5.718')] | |
| [2025-02-23 14:21:23,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8732672. Throughput: 0: 858.3. Samples: 2183910. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:21:23,316][00318] Avg episode reward: [(0, '6.643')] | |
| [2025-02-23 14:21:28,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8749056. Throughput: 0: 894.7. Samples: 2190006. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:21:28,315][00318] Avg episode reward: [(0, '6.323')] | |
| [2025-02-23 14:21:33,119][02476] Updated weights for policy 0, policy_version 1070 (0.0013) | |
| [2025-02-23 14:21:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 8765440. Throughput: 0: 900.0. Samples: 2194332. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:21:33,317][00318] Avg episode reward: [(0, '5.926')] | |
| [2025-02-23 14:21:38,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 8773632. Throughput: 0: 894.8. Samples: 2196432. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:21:38,321][00318] Avg episode reward: [(0, '6.751')] | |
| [2025-02-23 14:21:43,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3686.5, 300 sec: 3526.7). Total num frames: 8798208. Throughput: 0: 874.7. Samples: 2202258. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:21:43,321][00318] Avg episode reward: [(0, '6.139')] | |
| [2025-02-23 14:21:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3526.7). Total num frames: 8814592. Throughput: 0: 883.1. Samples: 2208510. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:21:48,316][00318] Avg episode reward: [(0, '7.044')] | |
| [2025-02-23 14:21:53,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3499.0). Total num frames: 8830976. Throughput: 0: 890.1. Samples: 2210928. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:21:53,315][00318] Avg episode reward: [(0, '6.614')] | |
| [2025-02-23 14:21:57,944][02476] Updated weights for policy 0, policy_version 1080 (0.0017) | |
| [2025-02-23 14:21:58,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 8847360. Throughput: 0: 891.1. Samples: 2215284. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:21:58,315][00318] Avg episode reward: [(0, '7.359')] | |
| [2025-02-23 14:21:58,324][02456] Saving new best policy, reward=7.359! | |
| [2025-02-23 14:22:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3526.7). Total num frames: 8863744. Throughput: 0: 875.3. Samples: 2220054. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:22:03,317][00318] Avg episode reward: [(0, '6.971')] | |
| [2025-02-23 14:22:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8888320. Throughput: 0: 879.1. Samples: 2223468. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:22:08,317][00318] Avg episode reward: [(0, '6.240')] | |
| [2025-02-23 14:22:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8904704. Throughput: 0: 894.7. Samples: 2230266. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:22:13,319][00318] Avg episode reward: [(0, '6.259')] | |
| [2025-02-23 14:22:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 8921088. Throughput: 0: 892.5. Samples: 2234496. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:22:18,315][00318] Avg episode reward: [(0, '5.972')] | |
| [2025-02-23 14:22:18,324][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001089_8921088.pth... | |
| [2025-02-23 14:22:18,679][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000000986_8077312.pth | |
| [2025-02-23 14:22:20,283][02476] Updated weights for policy 0, policy_version 1090 (0.0013) | |
| [2025-02-23 14:22:23,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3471.2). Total num frames: 8929280. Throughput: 0: 889.3. Samples: 2236452. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:22:23,317][00318] Avg episode reward: [(0, '5.790')] | |
| [2025-02-23 14:22:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 8953856. Throughput: 0: 863.2. Samples: 2241102. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:22:28,316][00318] Avg episode reward: [(0, '7.090')] | |
| [2025-02-23 14:22:33,315][00318] Fps is (10 sec: 4914.9, 60 sec: 3549.8, 300 sec: 3554.5). Total num frames: 8978432. Throughput: 0: 872.0. Samples: 2247750. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:22:33,320][00318] Avg episode reward: [(0, '6.921')] | |
| [2025-02-23 14:22:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 8994816. Throughput: 0: 894.5. Samples: 2251182. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:22:38,319][00318] Avg episode reward: [(0, '5.798')] | |
| [2025-02-23 14:22:43,019][02476] Updated weights for policy 0, policy_version 1100 (0.0013) | |
| [2025-02-23 14:22:43,316][00318] Fps is (10 sec: 3276.4, 60 sec: 3549.8, 300 sec: 3526.7). Total num frames: 9011200. Throughput: 0: 892.4. Samples: 2255442. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:22:43,319][00318] Avg episode reward: [(0, '6.713')] | |
| [2025-02-23 14:22:48,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9019392. Throughput: 0: 880.3. Samples: 2259666. Policy #0 lag: (min: 0.0, avg: 1.7, max: 4.0) | |
| [2025-02-23 14:22:48,318][00318] Avg episode reward: [(0, '6.415')] | |
| [2025-02-23 14:22:53,314][00318] Fps is (10 sec: 2458.1, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9035776. Throughput: 0: 854.1. Samples: 2261904. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:22:53,320][00318] Avg episode reward: [(0, '6.470')] | |
| [2025-02-23 14:22:58,317][00318] Fps is (10 sec: 4913.9, 60 sec: 3686.2, 300 sec: 3554.5). Total num frames: 9068544. Throughput: 0: 855.3. Samples: 2268756. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:22:58,318][00318] Avg episode reward: [(0, '7.106')] | |
| [2025-02-23 14:23:03,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9084928. Throughput: 0: 891.5. Samples: 2274612. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:23:03,315][00318] Avg episode reward: [(0, '6.216')] | |
| [2025-02-23 14:23:06,606][02476] Updated weights for policy 0, policy_version 1110 (0.0017) | |
| [2025-02-23 14:23:08,314][00318] Fps is (10 sec: 2458.2, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9093120. Throughput: 0: 896.8. Samples: 2276808. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:23:08,315][00318] Avg episode reward: [(0, '6.400')] | |
| [2025-02-23 14:23:13,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.3, 300 sec: 3526.8). Total num frames: 9109504. Throughput: 0: 888.7. Samples: 2281092. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:23:13,319][00318] Avg episode reward: [(0, '6.128')] | |
| [2025-02-23 14:23:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3554.5). Total num frames: 9134080. Throughput: 0: 872.7. Samples: 2287020. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:23:18,317][00318] Avg episode reward: [(0, '7.149')] | |
| [2025-02-23 14:23:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9150464. Throughput: 0: 858.1. Samples: 2289798. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:23:23,315][00318] Avg episode reward: [(0, '6.904')] | |
| [2025-02-23 14:23:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9166848. Throughput: 0: 887.6. Samples: 2295384. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:23:28,323][00318] Avg episode reward: [(0, '6.598')] | |
| [2025-02-23 14:23:30,026][02476] Updated weights for policy 0, policy_version 1120 (0.0017) | |
| [2025-02-23 14:23:33,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.4, 300 sec: 3499.0). Total num frames: 9183232. Throughput: 0: 889.3. Samples: 2299686. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:23:33,317][00318] Avg episode reward: [(0, '6.644')] | |
| [2025-02-23 14:23:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9199616. Throughput: 0: 886.1. Samples: 2301780. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:23:38,315][00318] Avg episode reward: [(0, '6.938')] | |
| [2025-02-23 14:23:43,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3554.5). Total num frames: 9224192. Throughput: 0: 878.3. Samples: 2308278. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:23:43,319][00318] Avg episode reward: [(0, '6.121')] | |
| [2025-02-23 14:23:48,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9240576. Throughput: 0: 890.1. Samples: 2314668. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:23:48,319][00318] Avg episode reward: [(0, '6.344')] | |
| [2025-02-23 14:23:51,623][02476] Updated weights for policy 0, policy_version 1130 (0.0039) | |
| [2025-02-23 14:23:53,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3686.3, 300 sec: 3526.7). Total num frames: 9256960. Throughput: 0: 886.0. Samples: 2316678. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:23:53,317][00318] Avg episode reward: [(0, '6.365')] | |
| [2025-02-23 14:23:58,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.9, 300 sec: 3499.0). Total num frames: 9265152. Throughput: 0: 885.6. Samples: 2320944. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:23:58,315][00318] Avg episode reward: [(0, '5.920')] | |
| [2025-02-23 14:24:03,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3413.3, 300 sec: 3526.7). Total num frames: 9289728. Throughput: 0: 866.3. Samples: 2326002. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:24:03,317][00318] Avg episode reward: [(0, '6.481')] | |
| [2025-02-23 14:24:08,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9314304. Throughput: 0: 881.2. Samples: 2329452. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:24:08,316][00318] Avg episode reward: [(0, '7.175')] | |
| [2025-02-23 14:24:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9330688. Throughput: 0: 903.9. Samples: 2336058. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:24:13,315][00318] Avg episode reward: [(0, '6.248')] | |
| [2025-02-23 14:24:16,141][02476] Updated weights for policy 0, policy_version 1140 (0.0013) | |
| [2025-02-23 14:24:18,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9347072. Throughput: 0: 906.4. Samples: 2340474. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:24:18,316][00318] Avg episode reward: [(0, '6.464')] | |
| [2025-02-23 14:24:18,330][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001141_9347072.pth... | |
| [2025-02-23 14:24:18,812][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001037_8495104.pth | |
| [2025-02-23 14:24:23,321][00318] Fps is (10 sec: 2455.9, 60 sec: 3412.9, 300 sec: 3498.9). Total num frames: 9355264. Throughput: 0: 903.2. Samples: 2342430. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:24:23,323][00318] Avg episode reward: [(0, '6.427')] | |
| [2025-02-23 14:24:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9379840. Throughput: 0: 863.6. Samples: 2347140. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:24:28,319][00318] Avg episode reward: [(0, '6.254')] | |
| [2025-02-23 14:24:33,314][00318] Fps is (10 sec: 4098.9, 60 sec: 3549.9, 300 sec: 3554.5). Total num frames: 9396224. Throughput: 0: 869.9. Samples: 2353812. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:24:33,315][00318] Avg episode reward: [(0, '6.817')] | |
| [2025-02-23 14:24:38,315][00318] Fps is (10 sec: 3276.6, 60 sec: 3549.8, 300 sec: 3526.7). Total num frames: 9412608. Throughput: 0: 900.4. Samples: 2357196. Policy #0 lag: (min: 0.0, avg: 2.2, max: 5.0) | |
| [2025-02-23 14:24:38,316][00318] Avg episode reward: [(0, '5.864')] | |
| [2025-02-23 14:24:38,420][02476] Updated weights for policy 0, policy_version 1150 (0.0015) | |
| [2025-02-23 14:24:43,316][00318] Fps is (10 sec: 4095.4, 60 sec: 3549.8, 300 sec: 3526.7). Total num frames: 9437184. Throughput: 0: 905.2. Samples: 2361678. Policy #0 lag: (min: 0.0, avg: 1.8, max: 5.0) | |
| [2025-02-23 14:24:43,317][00318] Avg episode reward: [(0, '6.385')] | |
| [2025-02-23 14:24:48,314][00318] Fps is (10 sec: 3277.0, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9445376. Throughput: 0: 889.3. Samples: 2366022. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:24:48,315][00318] Avg episode reward: [(0, '7.493')] | |
| [2025-02-23 14:24:48,324][02456] Saving new best policy, reward=7.493! | |
| [2025-02-23 14:24:53,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3550.0, 300 sec: 3554.5). Total num frames: 9469952. Throughput: 0: 874.4. Samples: 2368800. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:24:53,316][00318] Avg episode reward: [(0, '6.689')] | |
| [2025-02-23 14:24:58,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9486336. Throughput: 0: 864.1. Samples: 2374944. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:24:58,320][00318] Avg episode reward: [(0, '6.464')] | |
| [2025-02-23 14:25:00,651][02476] Updated weights for policy 0, policy_version 1160 (0.0048) | |
| [2025-02-23 14:25:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9502720. Throughput: 0: 885.5. Samples: 2380320. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:25:03,315][00318] Avg episode reward: [(0, '6.100')] | |
| [2025-02-23 14:25:08,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3276.8, 300 sec: 3499.0). Total num frames: 9510912. Throughput: 0: 889.7. Samples: 2382462. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:25:08,315][00318] Avg episode reward: [(0, '6.891')] | |
| [2025-02-23 14:25:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3526.7). Total num frames: 9535488. Throughput: 0: 881.3. Samples: 2386800. Policy #0 lag: (min: 0.0, avg: 2.3, max: 4.0) | |
| [2025-02-23 14:25:13,321][00318] Avg episode reward: [(0, '6.369')] | |
| [2025-02-23 14:25:18,314][00318] Fps is (10 sec: 4915.2, 60 sec: 3549.9, 300 sec: 3554.5). Total num frames: 9560064. Throughput: 0: 878.8. Samples: 2393358. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:25:18,316][00318] Avg episode reward: [(0, '7.382')] | |
| [2025-02-23 14:25:23,319][00318] Fps is (10 sec: 4094.0, 60 sec: 3686.5, 300 sec: 3554.4). Total num frames: 9576448. Throughput: 0: 878.8. Samples: 2396748. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:25:23,326][00318] Avg episode reward: [(0, '5.712')] | |
| [2025-02-23 14:25:25,097][02476] Updated weights for policy 0, policy_version 1170 (0.0012) | |
| [2025-02-23 14:25:28,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9592832. Throughput: 0: 885.0. Samples: 2401500. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:25:28,320][00318] Avg episode reward: [(0, '6.847')] | |
| [2025-02-23 14:25:33,317][00318] Fps is (10 sec: 3277.3, 60 sec: 3549.7, 300 sec: 3526.7). Total num frames: 9609216. Throughput: 0: 885.1. Samples: 2405856. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:25:33,320][00318] Avg episode reward: [(0, '6.592')] | |
| [2025-02-23 14:25:38,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3554.5). Total num frames: 9625600. Throughput: 0: 871.3. Samples: 2408010. Policy #0 lag: (min: 0.0, avg: 1.8, max: 4.0) | |
| [2025-02-23 14:25:38,317][00318] Avg episode reward: [(0, '6.598')] | |
| [2025-02-23 14:25:43,314][00318] Fps is (10 sec: 3277.9, 60 sec: 3413.4, 300 sec: 3554.5). Total num frames: 9641984. Throughput: 0: 879.1. Samples: 2414502. Policy #0 lag: (min: 0.0, avg: 1.7, max: 5.0) | |
| [2025-02-23 14:25:43,317][00318] Avg episode reward: [(0, '6.061')] | |
| [2025-02-23 14:25:47,059][02476] Updated weights for policy 0, policy_version 1180 (0.0013) | |
| [2025-02-23 14:25:48,322][00318] Fps is (10 sec: 4092.9, 60 sec: 3685.9, 300 sec: 3554.4). Total num frames: 9666560. Throughput: 0: 901.0. Samples: 2420874. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:25:48,323][00318] Avg episode reward: [(0, '7.078')] | |
| [2025-02-23 14:25:53,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9682944. Throughput: 0: 902.4. Samples: 2423070. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:25:53,315][00318] Avg episode reward: [(0, '6.662')] | |
| [2025-02-23 14:25:58,314][00318] Fps is (10 sec: 2459.5, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9691136. Throughput: 0: 898.4. Samples: 2427228. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:25:58,318][00318] Avg episode reward: [(0, '6.310')] | |
| [2025-02-23 14:26:03,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3549.9, 300 sec: 3554.5). Total num frames: 9715712. Throughput: 0: 858.7. Samples: 2431998. Policy #0 lag: (min: 0.0, avg: 2.1, max: 4.0) | |
| [2025-02-23 14:26:03,315][00318] Avg episode reward: [(0, '7.571')] | |
| [2025-02-23 14:26:03,325][02456] Saving new best policy, reward=7.571! | |
| [2025-02-23 14:26:08,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9732096. Throughput: 0: 855.2. Samples: 2435226. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:26:08,316][00318] Avg episode reward: [(0, '7.724')] | |
| [2025-02-23 14:26:08,325][02456] Saving new best policy, reward=7.724! | |
| [2025-02-23 14:26:10,337][02476] Updated weights for policy 0, policy_version 1190 (0.0014) | |
| [2025-02-23 14:26:13,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 9756672. Throughput: 0: 895.6. Samples: 2441802. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:26:13,315][00318] Avg episode reward: [(0, '6.926')] | |
| [2025-02-23 14:26:18,316][00318] Fps is (10 sec: 3276.3, 60 sec: 3413.2, 300 sec: 3498.9). Total num frames: 9764864. Throughput: 0: 895.5. Samples: 2446152. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:26:18,317][00318] Avg episode reward: [(0, '6.715')] | |
| [2025-02-23 14:26:18,327][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001192_9764864.pth... | |
| [2025-02-23 14:26:18,733][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001089_8921088.pth | |
| [2025-02-23 14:26:23,314][00318] Fps is (10 sec: 2457.6, 60 sec: 3413.6, 300 sec: 3499.0). Total num frames: 9781248. Throughput: 0: 894.3. Samples: 2448252. Policy #0 lag: (min: 0.0, avg: 2.1, max: 5.0) | |
| [2025-02-23 14:26:23,318][00318] Avg episode reward: [(0, '6.394')] | |
| [2025-02-23 14:26:28,314][00318] Fps is (10 sec: 3277.3, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9797632. Throughput: 0: 853.3. Samples: 2452902. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:26:28,315][00318] Avg episode reward: [(0, '6.489')] | |
| [2025-02-23 14:26:33,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.1, 300 sec: 3554.5). Total num frames: 9822208. Throughput: 0: 862.1. Samples: 2459664. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:26:33,315][00318] Avg episode reward: [(0, '7.523')] | |
| [2025-02-23 14:26:34,520][02476] Updated weights for policy 0, policy_version 1200 (0.0017) | |
| [2025-02-23 14:26:38,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9838592. Throughput: 0: 878.4. Samples: 2462598. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:26:38,315][00318] Avg episode reward: [(0, '7.178')] | |
| [2025-02-23 14:26:43,315][00318] Fps is (10 sec: 3276.5, 60 sec: 3549.8, 300 sec: 3526.7). Total num frames: 9854976. Throughput: 0: 881.9. Samples: 2466912. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:26:43,320][00318] Avg episode reward: [(0, '7.306')] | |
| [2025-02-23 14:26:48,314][00318] Fps is (10 sec: 3276.7, 60 sec: 3413.8, 300 sec: 3526.7). Total num frames: 9871360. Throughput: 0: 872.1. Samples: 2471244. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:26:48,317][00318] Avg episode reward: [(0, '6.162')] | |
| [2025-02-23 14:26:53,314][00318] Fps is (10 sec: 3277.1, 60 sec: 3413.3, 300 sec: 3526.7). Total num frames: 9887744. Throughput: 0: 875.7. Samples: 2474634. Policy #0 lag: (min: 0.0, avg: 2.2, max: 4.0) | |
| [2025-02-23 14:26:53,315][00318] Avg episode reward: [(0, '7.172')] | |
| [2025-02-23 14:26:58,314][00318] Fps is (10 sec: 3276.9, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9904128. Throughput: 0: 873.3. Samples: 2481102. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:26:58,317][00318] Avg episode reward: [(0, '7.318')] | |
| [2025-02-23 14:26:58,771][02476] Updated weights for policy 0, policy_version 1210 (0.0013) | |
| [2025-02-23 14:27:03,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3549.9, 300 sec: 3526.7). Total num frames: 9928704. Throughput: 0: 879.8. Samples: 2485740. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:27:03,321][00318] Avg episode reward: [(0, '6.602')] | |
| [2025-02-23 14:27:08,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3499.0). Total num frames: 9936896. Throughput: 0: 879.2. Samples: 2487816. Policy #0 lag: (min: 0.0, avg: 2.0, max: 4.0) | |
| [2025-02-23 14:27:08,316][00318] Avg episode reward: [(0, '6.208')] | |
| [2025-02-23 14:27:13,314][00318] Fps is (10 sec: 3276.8, 60 sec: 3413.3, 300 sec: 3526.7). Total num frames: 9961472. Throughput: 0: 877.9. Samples: 2492406. Policy #0 lag: (min: 0.0, avg: 1.9, max: 5.0) | |
| [2025-02-23 14:27:13,316][00318] Avg episode reward: [(0, '6.712')] | |
| [2025-02-23 14:27:18,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3550.0, 300 sec: 3554.5). Total num frames: 9977856. Throughput: 0: 878.8. Samples: 2499210. Policy #0 lag: (min: 0.0, avg: 2.0, max: 5.0) | |
| [2025-02-23 14:27:18,320][00318] Avg episode reward: [(0, '6.191')] | |
| [2025-02-23 14:27:20,469][02476] Updated weights for policy 0, policy_version 1220 (0.0015) | |
| [2025-02-23 14:27:23,314][00318] Fps is (10 sec: 4096.0, 60 sec: 3686.4, 300 sec: 3554.5). Total num frames: 10002432. Throughput: 0: 890.1. Samples: 2502654. Policy #0 lag: (min: 0.0, avg: 1.9, max: 4.0) | |
| [2025-02-23 14:27:23,315][00318] Avg episode reward: [(0, '6.803')] | |
| [2025-02-23 14:27:26,493][02456] Stopping Batcher_0... | |
| [2025-02-23 14:27:26,496][02456] Loop batcher_evt_loop terminating... | |
| [2025-02-23 14:27:26,496][00318] Component Batcher_0 stopped! | |
| [2025-02-23 14:27:26,530][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001222_10010624.pth... | |
| [2025-02-23 14:27:26,899][02476] Weights refcount: 2 0 | |
| [2025-02-23 14:27:26,960][00318] Component InferenceWorker_p0-w0 stopped! | |
| [2025-02-23 14:27:26,965][02476] Stopping InferenceWorker_p0-w0... | |
| [2025-02-23 14:27:26,978][02456] Removing /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001141_9347072.pth | |
| [2025-02-23 14:27:26,993][02476] Loop inference_proc0-0_evt_loop terminating... | |
| [2025-02-23 14:27:27,010][02456] Saving /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001222_10010624.pth... | |
| [2025-02-23 14:27:29,935][02494] Stopping RolloutWorker_w17... | |
| [2025-02-23 14:27:29,935][00318] Component RolloutWorker_w17 stopped! | |
| [2025-02-23 14:27:29,943][02494] Loop rollout_proc17_evt_loop terminating... | |
| [2025-02-23 14:27:29,990][00318] Component RolloutWorker_w19 stopped! | |
| [2025-02-23 14:27:29,990][02498] Stopping RolloutWorker_w19... | |
| [2025-02-23 14:27:29,995][02498] Loop rollout_proc19_evt_loop terminating... | |
| [2025-02-23 14:27:30,040][02486] Stopping RolloutWorker_w9... | |
| [2025-02-23 14:27:30,040][00318] Component RolloutWorker_w9 stopped! | |
| [2025-02-23 14:27:30,043][02486] Loop rollout_proc9_evt_loop terminating... | |
| [2025-02-23 14:27:30,060][02492] Stopping RolloutWorker_w15... | |
| [2025-02-23 14:27:30,059][00318] Component RolloutWorker_w15 stopped! | |
| [2025-02-23 14:27:30,065][02492] Loop rollout_proc15_evt_loop terminating... | |
| [2025-02-23 14:27:30,066][00318] Component LearnerWorker_p0 stopped! | |
| [2025-02-23 14:27:30,067][02456] Stopping LearnerWorker_p0... | |
| [2025-02-23 14:27:30,070][02456] Loop learner_proc0_evt_loop terminating... | |
| [2025-02-23 14:27:30,113][02487] Stopping RolloutWorker_w11... | |
| [2025-02-23 14:27:30,113][00318] Component RolloutWorker_w11 stopped! | |
| [2025-02-23 14:27:30,113][02487] Loop rollout_proc11_evt_loop terminating... | |
| [2025-02-23 14:27:30,144][00318] Component RolloutWorker_w13 stopped! | |
| [2025-02-23 14:27:30,144][02490] Stopping RolloutWorker_w13... | |
| [2025-02-23 14:27:30,147][02490] Loop rollout_proc13_evt_loop terminating... | |
| [2025-02-23 14:27:30,171][00318] Component RolloutWorker_w1 stopped! | |
| [2025-02-23 14:27:30,172][02478] Stopping RolloutWorker_w1... | |
| [2025-02-23 14:27:30,179][02478] Loop rollout_proc1_evt_loop terminating... | |
| [2025-02-23 14:27:30,249][00318] Component RolloutWorker_w3 stopped! | |
| [2025-02-23 14:27:30,251][02480] Stopping RolloutWorker_w3... | |
| [2025-02-23 14:27:30,258][00318] Component RolloutWorker_w7 stopped! | |
| [2025-02-23 14:27:30,259][02484] Stopping RolloutWorker_w7... | |
| [2025-02-23 14:27:30,263][02484] Loop rollout_proc7_evt_loop terminating... | |
| [2025-02-23 14:27:30,265][02480] Loop rollout_proc3_evt_loop terminating... | |
| [2025-02-23 14:27:30,289][00318] Component RolloutWorker_w5 stopped! | |
| [2025-02-23 14:27:30,290][02482] Stopping RolloutWorker_w5... | |
| [2025-02-23 14:27:30,291][02482] Loop rollout_proc5_evt_loop terminating... | |
| [2025-02-23 14:27:30,642][02485] Stopping RolloutWorker_w8... | |
| [2025-02-23 14:27:30,641][00318] Component RolloutWorker_w8 stopped! | |
| [2025-02-23 14:27:30,654][02495] Stopping RolloutWorker_w18... | |
| [2025-02-23 14:27:30,653][00318] Component RolloutWorker_w18 stopped! | |
| [2025-02-23 14:27:30,655][02485] Loop rollout_proc8_evt_loop terminating... | |
| [2025-02-23 14:27:30,654][02495] Loop rollout_proc18_evt_loop terminating... | |
| [2025-02-23 14:27:30,748][02481] Stopping RolloutWorker_w4... | |
| [2025-02-23 14:27:30,747][00318] Component RolloutWorker_w4 stopped! | |
| [2025-02-23 14:27:30,757][02481] Loop rollout_proc4_evt_loop terminating... | |
| [2025-02-23 14:27:30,793][02489] Stopping RolloutWorker_w12... | |
| [2025-02-23 14:27:30,792][00318] Component RolloutWorker_w12 stopped! | |
| [2025-02-23 14:27:30,816][02489] Loop rollout_proc12_evt_loop terminating... | |
| [2025-02-23 14:27:30,855][02493] Stopping RolloutWorker_w16... | |
| [2025-02-23 14:27:30,856][02493] Loop rollout_proc16_evt_loop terminating... | |
| [2025-02-23 14:27:30,855][00318] Component RolloutWorker_w16 stopped! | |
| [2025-02-23 14:27:30,872][02491] Stopping RolloutWorker_w14... | |
| [2025-02-23 14:27:30,872][00318] Component RolloutWorker_w14 stopped! | |
| [2025-02-23 14:27:30,893][02491] Loop rollout_proc14_evt_loop terminating... | |
| [2025-02-23 14:27:30,923][02477] Stopping RolloutWorker_w0... | |
| [2025-02-23 14:27:30,924][02477] Loop rollout_proc0_evt_loop terminating... | |
| [2025-02-23 14:27:30,923][00318] Component RolloutWorker_w0 stopped! | |
| [2025-02-23 14:27:30,966][02479] Stopping RolloutWorker_w2... | |
| [2025-02-23 14:27:30,974][02479] Loop rollout_proc2_evt_loop terminating... | |
| [2025-02-23 14:27:30,966][00318] Component RolloutWorker_w2 stopped! | |
| [2025-02-23 14:27:31,064][02483] Stopping RolloutWorker_w6... | |
| [2025-02-23 14:27:31,064][00318] Component RolloutWorker_w6 stopped! | |
| [2025-02-23 14:27:31,065][02483] Loop rollout_proc6_evt_loop terminating... | |
| [2025-02-23 14:27:31,133][02488] Stopping RolloutWorker_w10... | |
| [2025-02-23 14:27:31,133][00318] Component RolloutWorker_w10 stopped! | |
| [2025-02-23 14:27:31,135][02488] Loop rollout_proc10_evt_loop terminating... | |
| [2025-02-23 14:27:31,134][00318] Waiting for process learner_proc0 to stop... | |
| [2025-02-23 14:27:33,785][00318] Waiting for process inference_proc0-0 to join... | |
| [2025-02-23 14:27:33,786][00318] Waiting for process rollout_proc0 to join... | |
| [2025-02-23 14:27:40,012][00318] Waiting for process rollout_proc1 to join... | |
| [2025-02-23 14:27:40,015][00318] Waiting for process rollout_proc2 to join... | |
| [2025-02-23 14:27:40,017][00318] Waiting for process rollout_proc3 to join... | |
| [2025-02-23 14:27:40,026][00318] Waiting for process rollout_proc4 to join... | |
| [2025-02-23 14:27:40,027][00318] Waiting for process rollout_proc5 to join... | |
| [2025-02-23 14:27:40,028][00318] Waiting for process rollout_proc6 to join... | |
| [2025-02-23 14:27:40,047][00318] Waiting for process rollout_proc7 to join... | |
| [2025-02-23 14:27:40,050][00318] Waiting for process rollout_proc8 to join... | |
| [2025-02-23 14:27:40,051][00318] Waiting for process rollout_proc9 to join... | |
| [2025-02-23 14:27:40,052][00318] Waiting for process rollout_proc10 to join... | |
| [2025-02-23 14:27:40,055][00318] Waiting for process rollout_proc11 to join... | |
| [2025-02-23 14:27:40,056][00318] Waiting for process rollout_proc12 to join... | |
| [2025-02-23 14:27:40,058][00318] Waiting for process rollout_proc13 to join... | |
| [2025-02-23 14:27:40,059][00318] Waiting for process rollout_proc14 to join... | |
| [2025-02-23 14:27:40,060][00318] Waiting for process rollout_proc15 to join... | |
| [2025-02-23 14:27:40,061][00318] Waiting for process rollout_proc16 to join... | |
| [2025-02-23 14:27:40,062][00318] Waiting for process rollout_proc17 to join... | |
| [2025-02-23 14:27:40,063][00318] Waiting for process rollout_proc18 to join... | |
| [2025-02-23 14:27:40,064][00318] Waiting for process rollout_proc19 to join... | |
| [2025-02-23 14:27:40,067][00318] Batcher 0 profile tree view: | |
| batching: 108.7955, releasing_batches: 0.0405 | |
| [2025-02-23 14:27:40,068][00318] InferenceWorker_p0-w0 profile tree view: | |
| wait_policy: 0.0091 | |
| wait_policy_total: 2120.3083 | |
| update_model: 7.0036 | |
| weight_update: 0.0013 | |
| one_step: 0.0504 | |
| handle_policy_step: 828.0029 | |
| deserialize: 23.2753, stack: 2.7238, obs_to_device_normalize: 103.4719, forward: 501.2885, send_messages: 45.1772 | |
| prepare_outputs: 125.9323 | |
| to_cpu: 67.1381 | |
| [2025-02-23 14:27:40,069][00318] Learner 0 profile tree view: | |
| misc: 0.0069, prepare_batch: 32.5341 | |
| train: 190.8577 | |
| epoch_init: 0.0057, minibatch_init: 0.0079, losses_postprocess: 0.8576, kl_divergence: 2.8878, update: 23.0906, after_optimizer: 80.9018 | |
| calculate_losses: 81.8405 | |
| losses_init: 0.0040, forward_head: 2.7634, bptt_initial: 51.3426, tail: 5.4934, advantages_returns: 0.7619, losses: 15.6203 | |
| bptt: 5.0716 | |
| bptt_forward_core: 4.8707 | |
| [2025-02-23 14:27:40,069][00318] RolloutWorker_w0 profile tree view: | |
| wait_for_trajectories: 0.8228, enqueue_policy_requests: 68.1073, env_step: 2658.9875, overhead: 39.4144, complete_rollouts: 1.5150 | |
| save_policy_outputs: 71.6175 | |
| split_output_tensors: 25.9914 | |
| [2025-02-23 14:27:40,070][00318] RolloutWorker_w19 profile tree view: | |
| wait_for_trajectories: 0.8946, enqueue_policy_requests: 71.0988, env_step: 2684.3196, overhead: 44.7331, complete_rollouts: 2.7244 | |
| save_policy_outputs: 71.4273 | |
| split_output_tensors: 25.6390 | |
| [2025-02-23 14:27:40,072][00318] Loop Runner_EvtLoop terminating... | |
| [2025-02-23 14:27:40,074][00318] Runner profile tree view: | |
| main_loop: 3071.5029 | |
| [2025-02-23 14:27:40,077][00318] Collected {0: 10010624}, FPS: 3259.2 | |
| [2025-02-23 14:37:30,586][00318] Loading existing experiment configuration from /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/config.json | |
| [2025-02-23 14:37:30,587][00318] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-02-23 14:37:30,588][00318] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-02-23 14:37:30,589][00318] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-02-23 14:37:30,590][00318] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-02-23 14:37:30,590][00318] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-02-23 14:37:30,591][00318] Adding new argument 'max_num_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-02-23 14:37:30,592][00318] Adding new argument 'max_num_episodes'=1 that is not in the saved config file! | |
| [2025-02-23 14:37:30,593][00318] Adding new argument 'push_to_hub'=False that is not in the saved config file! | |
| [2025-02-23 14:37:30,594][00318] Adding new argument 'hf_repository'=None that is not in the saved config file! | |
| [2025-02-23 14:37:30,595][00318] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-02-23 14:37:30,596][00318] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-02-23 14:37:30,597][00318] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-02-23 14:37:30,597][00318] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-02-23 14:37:30,598][00318] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-02-23 14:37:30,654][00318] Doom resolution: 160x120, resize resolution: (128, 72) | |
| [2025-02-23 14:37:30,660][00318] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-02-23 14:37:30,671][00318] RunningMeanStd input shape: (1,) | |
| [2025-02-23 14:37:30,717][00318] ConvEncoder: input_channels=3 | |
| [2025-02-23 14:37:31,049][00318] Conv encoder output size: 512 | |
| [2025-02-23 14:37:31,051][00318] Policy head output size: 512 | |
| [2025-02-23 14:37:31,635][00318] Loading state from checkpoint /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001222_10010624.pth... | |
| [2025-02-23 14:37:33,476][00318] Num frames 100... | |
| [2025-02-23 14:37:33,572][00318] Avg episode rewards: #0: 7.060, true rewards: #0: 7.060 | |
| [2025-02-23 14:37:33,573][00318] Avg episode reward: 7.060, avg true_objective: 7.060 | |
| [2025-02-23 14:37:35,329][00318] Replay video saved to /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/replay.mp4! | |
| [2025-02-23 14:39:54,971][00318] Loading existing experiment configuration from /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/config.json | |
| [2025-02-23 14:39:54,972][00318] Overriding arg 'num_workers' with value 1 passed from command line | |
| [2025-02-23 14:39:54,974][00318] Adding new argument 'no_render'=True that is not in the saved config file! | |
| [2025-02-23 14:39:54,976][00318] Adding new argument 'save_video'=True that is not in the saved config file! | |
| [2025-02-23 14:39:54,977][00318] Adding new argument 'video_frames'=1000000000.0 that is not in the saved config file! | |
| [2025-02-23 14:39:54,977][00318] Adding new argument 'video_name'=None that is not in the saved config file! | |
| [2025-02-23 14:39:54,978][00318] Adding new argument 'max_num_frames'=100000 that is not in the saved config file! | |
| [2025-02-23 14:39:54,979][00318] Adding new argument 'max_num_episodes'=10 that is not in the saved config file! | |
| [2025-02-23 14:39:54,980][00318] Adding new argument 'push_to_hub'=True that is not in the saved config file! | |
| [2025-02-23 14:39:54,981][00318] Adding new argument 'hf_repository'='mjkim0928/rl_course_vizdoom_deadly_corridor' that is not in the saved config file! | |
| [2025-02-23 14:39:54,982][00318] Adding new argument 'policy_index'=0 that is not in the saved config file! | |
| [2025-02-23 14:39:54,983][00318] Adding new argument 'eval_deterministic'=False that is not in the saved config file! | |
| [2025-02-23 14:39:54,983][00318] Adding new argument 'train_script'=None that is not in the saved config file! | |
| [2025-02-23 14:39:54,984][00318] Adding new argument 'enjoy_script'=None that is not in the saved config file! | |
| [2025-02-23 14:39:54,985][00318] Using frameskip 1 and render_action_repeat=4 for evaluation | |
| [2025-02-23 14:39:55,013][00318] RunningMeanStd input shape: (3, 72, 128) | |
| [2025-02-23 14:39:55,015][00318] RunningMeanStd input shape: (1,) | |
| [2025-02-23 14:39:55,026][00318] ConvEncoder: input_channels=3 | |
| [2025-02-23 14:39:55,058][00318] Conv encoder output size: 512 | |
| [2025-02-23 14:39:55,059][00318] Policy head output size: 512 | |
| [2025-02-23 14:39:55,081][00318] Loading state from checkpoint /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/checkpoint_p0/checkpoint_000001222_10010624.pth... | |
| [2025-02-23 14:39:55,563][00318] Avg episode rewards: #0: 3.544, true rewards: #0: 3.544 | |
| [2025-02-23 14:39:55,565][00318] Avg episode reward: 3.544, avg true_objective: 3.544 | |
| [2025-02-23 14:39:55,591][00318] Num frames 100... | |
| [2025-02-23 14:39:55,741][00318] Num frames 200... | |
| [2025-02-23 14:39:55,876][00318] Avg episode rewards: #0: 6.954, true rewards: #0: 6.954 | |
| [2025-02-23 14:39:55,877][00318] Avg episode reward: 6.954, avg true_objective: 6.954 | |
| [2025-02-23 14:39:55,953][00318] Num frames 300... | |
| [2025-02-23 14:39:56,068][00318] Avg episode rewards: #0: 5.991, true rewards: #0: 5.991 | |
| [2025-02-23 14:39:56,068][00318] Avg episode reward: 5.991, avg true_objective: 5.991 | |
| [2025-02-23 14:39:56,165][00318] Num frames 400... | |
| [2025-02-23 14:39:56,252][00318] Avg episode rewards: #0: 5.333, true rewards: #0: 5.333 | |
| [2025-02-23 14:39:56,253][00318] Avg episode reward: 5.333, avg true_objective: 5.333 | |
| [2025-02-23 14:39:56,374][00318] Num frames 500... | |
| [2025-02-23 14:39:56,480][00318] Avg episode rewards: #0: 5.443, true rewards: #0: 5.443 | |
| [2025-02-23 14:39:56,481][00318] Avg episode reward: 5.443, avg true_objective: 5.443 | |
| [2025-02-23 14:39:56,583][00318] Num frames 600... | |
| [2025-02-23 14:39:56,738][00318] Num frames 700... | |
| [2025-02-23 14:39:56,815][00318] Avg episode rewards: #0: 6.261, true rewards: #0: 6.261 | |
| [2025-02-23 14:39:56,815][00318] Avg episode reward: 6.261, avg true_objective: 6.261 | |
| [2025-02-23 14:39:56,948][00318] Num frames 800... | |
| [2025-02-23 14:39:57,056][00318] Avg episode rewards: #0: 6.280, true rewards: #0: 6.280 | |
| [2025-02-23 14:39:57,057][00318] Avg episode reward: 6.280, avg true_objective: 6.280 | |
| [2025-02-23 14:39:57,168][00318] Num frames 900... | |
| [2025-02-23 14:39:57,297][00318] Avg episode rewards: #0: 6.127, true rewards: #0: 6.127 | |
| [2025-02-23 14:39:57,298][00318] Avg episode reward: 6.127, avg true_objective: 6.127 | |
| [2025-02-23 14:39:57,422][00318] Avg episode rewards: #0: 5.580, true rewards: #0: 5.580 | |
| [2025-02-23 14:39:57,423][00318] Avg episode reward: 5.580, avg true_objective: 5.580 | |
| [2025-02-23 14:39:57,434][00318] Num frames 1000... | |
| [2025-02-23 14:39:57,596][00318] Num frames 1100... | |
| [2025-02-23 14:39:57,704][00318] Avg episode rewards: #0: 5.736, true rewards: #0: 5.736 | |
| [2025-02-23 14:39:57,705][00318] Avg episode reward: 5.736, avg true_objective: 5.736 | |
| [2025-02-23 14:40:03,792][00318] Replay video saved to /scratch/sample_factory/train_dir/vizdoom/vizdoom_doom_deadly_corridor_3333/replay.mp4! | |